Has Research Quality Found its’ Right Size?!

An Interview with Merrill Dubrow, President and CEO of M/A/R/C® Research

In a recent conversation with Merrill Dubrow, M/A/R/C Research President and CEO, he discusses the issues surrounding online research and how the trends in technology are continually changing the industry. “The future of online research faces a second generation,” suggests Merrill, “this means an influx of technology that focuses on online data as the leading source of data collection.  This technology includes data mining and spidering of blogs, social sites, and the like.”

When asked about the state of online research, Merrill suggested that our industry’s progress in the next 24 months would need to demonstrate an understanding of the impact of quality on panels.  This also implies that there needs to be an increasing focus on policies and procedures that set a standard for quality in online research. “Online research is in a state where you need to ask yourself about supply and demand,” says Merrill, “and the fact is that demand is outrunning research supply.  Yet, while demand is high, supply is low, and the price of services is decreasing.”  Dubrow states that this is counter to everything all of us were taught in school.

Mr. Dubrow explains that one of the major issues facing online research is the industry’s inability to collaborate and work together on key issues, such as quality, standards, and acceptable pricing models.  Merrill recommends research companies and associations  “come together to discuss the quality issues of online research and agree on new standards.”  He also adds that “in reality, clients want a balance between quality and price and right now some companies aren’t delivering that.   We use companies that we feel comfortable with, which are not always the least expensive panel companies but rather can deliver what we and our clients want and need to ensure a successful project.”

And returning to the main question, Merrill believes M/A/R/C is the right size for their clients. “Industry needs and client needs are really the same, and we find that what really separates us in the marketplace is our belief that strong brands start from good research.  We are the right size to communicate effectively, and we are a group of truly intelligent and caring researchers.  We are able to dedicate senior management on every project, bringing best in class practices to every data quality element of a study.”  In addition, Merrill welcomes the idea of increasing efficiencies through technology; including cell phones and text messaging, while also understanding the impacts of social media, including Merrill’s own ever popular blog.

All said, it is clear that size does matter, and that the ability to work together in a collaborative environment is going to make or break many players in the online research industry in the years to come.

Technology Review

The year’s largest technology conference in Market Research is happening this week in New York. Among other things, the hot topics should be SaaS (”Software as a Service”) and the Research Quality panel. SaaS is a trend that has been happening in the wider software industry where consumers are increasingly using hosted services and software. The preeminent example of a SaaS company is Benioff’s Salesforce which has used its exclusively SaaS CRM platform to compete effectively against older, more entrenched competitors. Will this trend carry over into market research? I believe so, given a few years. Companies such as ClearView, Decipher, Insight Express, Zoomerang and others are offering fully automated, self-service SaaS research software that is web-based and operated through any browser. This is also the future that Google’s new Google Docs is betting on – a world where people will use Word / Excel / Powerpoint through the browser. Enterprise software should be worried. Very worried.

The other interesting to come out of the conference will be Data Quality, which has been the most important issue in the online research industry over the last couple of years and which was the subject of a recent Forrester study. As traditional market research faces increasing pressure from new web-based technologies, what sets researchers apart from everyone else is the validity of their predictions. The science and rigor of creating appropriate sample frames and adjusting for bias will never go away. This is the competitive edge of traditional market research – both online and offline. It can “predict the future” based on a consistent and replicable scientific method. Other techniques such as web polls are not representative of the general population or the target market. Poor data quality from cheaters and spammers therefore hits right at the center of what should be market research’s forte – valid, reliable data. It scares researchers.

Peanut Labs, OTX, Burke and Kantar will be participating on the panel at CASRO Tech discussing data quality and various approaches being developed in the industry to address this problem. We welcome you to drop in and share your thoughts.

Team work – the ‘crack’ of startups

I’ve come to realize (after 8 years of doing this) that the best part of being in a startup are the people you work with. Sure there are ups and down and some people don’t work out and we let them go. But looking back if I had to generalize, i would say the reason i do what i do, is because i get to work with an incredible group of highly talented people everyday. And we get shit done.

Thank you EVERYONE in Peanut Labs for making this such an incredible journey. I can’t imagine doing anything else, but getting up everyday and working with you all to kick some butt!

Forrester:”game-changing year for online panels”

Panel quality tools set to ‘change the online research game’

http://www.research-live.com/news_story.aspx?pageid=30&r=y&newsid=4728

Advisory Board Rocks the Press!

“Peanut Labs’ A-Team to Tackle Bad Respondents”

http://www.mrweb.com/drno/news8405.htm

 

The Coverage Continues: Peanut Labs Announces Internal Advisory Board

Peanut Labs Announces Internal Advisory Board to Resolve Market Research Data Quality Issues Through Technology

Renowned Market Research Leaders Join Peanut Labs Board to Discuss and Enact Technological Solutions Like OptimusID^TM to Fight Suspect Online Survey Responses

http://www.marketwire.com/mw/release.do?id=861630

 

 

Advisory Board: Industry leaders discuss technology solutions to data-quality

Tomorrow Peanut Labs will be announcing a new internal advisory board around its Optimus digital fingerprinting technology. You got the info a day before it happened, right here on this blog.

The Advisory Board currently consists of 17 market research leaders from companies such as WaMu, Warners Bros, Knowledge Networks, Decipher and Millward Brown. The group is coming together to help shape how digital fingerprinting and other technologies can best be used for data-quality control in online research. This involves sharing internal research-on-research, creating a dialogue and evaluating tools and technologies that make the process of data-cleansing cheaper and less labor-intensive. Simon Chadwick, the former Global CEO of NOP World, will be chairing this group.

This is not an industry-wide initiative to improve data quality, such as the ARF Online Quality Council (which has been bringing the industry together on this in a collaborative manner). Peanut Labs is an active participant in the ARF ORQC and fully supports that initiative. This advisory board is a group that looks at specific ways to use Optimus and other technologies in online research. We highly commend the work that industry organizations are doing in this regard and continue to be involved in those initiatives.

We’d like to open the forum and invite your views on the subject. How have you used technology to automate and improve quality-control and data management?

Re-Rez on Research: Insight from Industry Innovators

Joined on the conference lines by the leaders of Re-Rez (Malcolm Williamson, Debbie Peternana, and Eric Bell), we are offered a brilliant opportunity to hear more about how this Texas-based company is earning its street cred.  ”Traditional web survey is here to stay,” says Eric, “but we must gravitate to access points such as web 2.0 sites, mobile platforms, and social networks.”  Malcolm points out that as the research industry shifts its attention to available resources, we are being flooding with an overabundance of junk pouring in from small panel and online companies trying to get a piece of the action.  ”We don’t care if the sample we pull from is good or bad, we’ll make it good, and this happens through effective questioning and implementing identification technologies, such as OptimusID.”  

Although many researchers voice that good survey design will turn out good data, Debbie takes this a step-further, “with increasing variance in the level of quality amongst identification technologies, Re-Rez adds strong screening and machine/digital fingerprinting technologies to better ensure we weed out the good from the bad.”  Debbie adheres to her beliefs that there is honesty online, and that good data does come from online respondents.  ”Good data.” says Debbie, “does comes from writing the survey and effectively analyzing the results.”  

When asked about how Re-Rez is answering the challenges of online data quality, Eric shared that “we are not just an aggregator, rather, we are able to get the ‘best-of-breed’ to what’s available for a particular study.”  Eric reminds us that Re-Rez even guarantees their work, while taking on the burden of managing multiple vendors so the end-clients don’t have to.  ”We monitor best practices from sources such as CASRO and MRA, and we continue to keep our eye on the ball.  We monitor and implement.”  Debbie adds that Re-Rez makes sure that a client is checking a study within 24-48 hours of launch, ensuring that they are showing appropriate survey behavior, data integrity, and addressing the removal of speeders and cheaters.

“Re-Rez has built its reputation on trust and innovation,” says Malcolm.  ”We are focused on the research experience, we monitor and align with upcoming trends, we focus on the best current practices, and we continue to be progressive in our service to clients.”  ”And we serve our core clients by finding better ways to serve them,” says Debbie jumping in with a jolt of enthusiasm, “we help our clients go out and earn business.”

With that said, I offer many thanks to the folks at Re-Rez for offering their insight and energy around the state of online research.  It’s great to see a group of able professionals so devoted to offering real data quality and client-focused results!

What is the true cost of BAD Market Research?

I have often heard this question asked, but rarely have I seen a real answer – so I am shouting out to the readers of this blog in search of one!  We all know that MR quality stems from making sure that we are asking the right people the right questions, and applying valid statistics to the data to draw conclusions.  Taking it a step further, we also need the smart, informed people to interpret the results and formulate meaningful actions based on their interpretation.  Hopefully this happens flawlessly every time, but what happens if a mistake is made?  Does it cost

• nothing?
• $20,000?
• $1,000,000?
• $30,000,000?
• An FDA approval?
• A hard earned promotion?
• The loss of a valued client?
• The loss of a job?
• An entire brand?

I think that all of the above situations are not only possible, but probably have occurred over the course of MR history.  That said, can someone send me an answer or add to the list?  Even better, if you have a horror story or real statistic that can aid in answering this question, don’t hold back – post it!

If we can better understand the costs of BAD research, then we can start to address the potential problems from the right perspectives.  In doing so, we can find meaningful solutions and best practices to PREVENT poor decisions from ever being made.

Where’s the Love? Media-Screen’s CEO Shares his Vision of Market Research

Housed in a beautifully restored historic building in San Francisco’s mission district, Media-Screen’s CEO Josh Crandall offers his unique perspective into the future of market research. “Well it’s evident that research as a whole is headed online,” said Josh, “we (the industry) have seen the change from random-digit dial as becoming less representative of research, to online surveys becoming a more acceptable methodology.”  

 

Josh Crandall, CEO (Media-Screen)

 

As I continue to point out, albeit ad nauseam, there continue to be challenges with integrating effective online survey strategy, and at the peak of these issues we find data quality.  ”With effective survey design, respondent recruiting, and the use of behavioral-based analytics, we should essentially be able to weed out many of the quality issues plaguing online research.”  Mr. Crandall continues by adding, “The real issues facing the online research world are two part: one, is technical, the other is interpersonal.”  Josh makes the point that technical aspects of online research means that we face multiple analytic systems measuring data with their own set of standards.  And as we know, end-clients and market research firms draft their own standards too.  So in trying to cope with a lack of quality standards, Josh reiterated the need for two primary elements of effective online research: consumer engagement and effective survey design.

“It seems the industry doesn’t look favorably on professional survey takers, and I think that’s crazy, we should cherish these consumers, even reward them.”  Josh is speaking at the heart of consumer engagement.  It appears that there’s a bit of love lost amongst the survey takers and market researchers, and not enough to be done to bridge the gap between the huge online population willing to boast their opinions anywhere they can, to honest respondents taking the time to thoroughly complete an online survey.  ”We need to respect respondents’ time,” said Josh,” while continuing to conduct good research.”

Though I have conducted many an interview, there are times when someone can really strike a cord, priming your heart strings for a refreshing new tonality.  This was the case as Josh spoke so passionately about consumer engagement. “Online research is a reflection of where the market lives today.  We simply need to ask the right questions to really engage consumers in the place where they live.”