Defining the Data Quality Problem: First Dates with Respondents

 /  News

Data quality presents a peculiar problem in our industry.

On the one hand, sample companies and panel providers pretend it’s not a real issue that should concern clients, that they’ve rid their panels of fraud through a complex cocktail of validation techniques and data scrubbing. On the other hand, clients continue to reject results and demand more completes because their definition of quality just isn’t there. What’s more, some fear that more aggressive quality checks will constrain capacity in an industry where supply is already a concern.

The reality is, supply is not a problem in our industry, but quality is, and understanding the root causes of quality is a greater problem still. There’s a disconnect among suppliers and clients that goes well beyond differences of opinion and approaches to data cleaning. If we’re going to make a sincere attempt to tackle this problem as an industry, we need to first agree upon the problem.

What is contributing to this confusion? Part of the issue is that we haven’t clearly defined the data quality problem. What characterizes bad data? How do we spot it? What is driving data quality issues? Depending on who you ask, you’ll get wildly different answers to these questions. And that’s because depending on who you ask—panel recruiter, sample supplier, technology provider or research firm—you are actually talking about a different aspect of quality.

Our industry often lumps data quality into one messy problem, when in fact, there are two separate conditions contributing to data quality issues. In basic terms, there is respondent-level quality and response-level quality. Respondent-level quality is like a first date—when the panelist and panel company get to know each other and decide if they want to begin a relationship. Response-level quality is the long-term relationship itself—with all the ups and downs and fun and growth that relationships bring about. Respondent-level quality is the first line of defense in the war on fraud. It’s a top-of-the funnel approach: the more good you put in, the better chance you’ll see good coming out.

The burden of respondent-level quality and identifying fraudulency sits squarely with the panel providers and their technology partners. While great strides have been made in combating fraud and validating respondents, there’s no silver bullet. The key to respondent-level quality is layering validation in a particular order that effectively reduces risk. Think about the airport experience. The first layer of security that an airport uses—checking boarding passes—is of minimal inconvenience to travelers. Once in the appropriate queue, travelers must show an ID and boarding pass. The third layer of security is the bane of travelers everywhere: metal detectors and body scanners and the always frustrating removal of shoes.

Respondent validation is a bit like airport security and is best employed with the same layered approach of least to greatest inconvenience for the panelist. Technology like geo-IP and IP validation are great starting points and of little nuisance to panelists. Email address and name/address validation require more participation from respondents, but are essential to ensuring quality. New technologies like device fingerprinting provide even more security while asking even more of panelists.

Again, these techniques are the building blocks of data quality. Or course, quality goes beyond the recruitment and registration process. A survey taker’s overall quality can vary based on their experience in the panel. The other obvious, yet important truth to keep in mind is that people change. The person who registers to your panel one day is not the same person who completes a survey three, six or even 15 months later. So while respondent-quality checks are essential, staying abreast of what is happening in the life of a panelist and delivering appropriate surveys and content presents a whole new set of complex problems that can dramatically impact data quality. Research suppliers play a critical role here, too, but the first date is all about making the right impression.

– By Joe Jordan, Vice President of Panel Operations, Instantly & Mark Menig, CEO, TrueSample

  Back to All Articles
Like this article? Share it!