Are online surveys reliable?
Online media has been increasingly used as a platform for implementing surveys. Since the pandemic accelerated the shift to remote forms of data gathering to minimize physical contact, at least in the short-term, online survey has become a ubiquitous tool for offsite data collection.
As such, you have probably encountered reports generated from an online survey or perhaps you came across an invitation to participate in a social media survey. Given the increasing prevalence of online surveys, what do we need to know about it as an audience of online survey reports and as prospective online survey respondents?
Could online surveys be representative of the population?
Online surveys should practice due diligence when claiming that it is representative of the general Filipino population.
First and foremost, even if internet penetration in our country has been increasing and is estimated to be at 67% (from DataReportal, as of January 2021), there are still segments of the Filipino population that are not internet-connected.
Thus, the online population does not entirely cover the Filipino population and, for this reason, online surveys should practice due diligence when claiming that it is representative of the general Filipino population.
But if an online survey claims that it is indeed representative of the general population, then we should check whether it followed the protocols for generating a representative sample.
An adequately sized sample should be complemented by the random manner of drawing the members of the population who will comprise the pool of survey respondents. But while quickly achieving a substantially sized sample is relatively easy with online surveys, the random manner of respondent selection can be quite challenging or it can even be missing in the conduct of some online surveys.
Why is random selection challenging or even missing in some online surveys?
Conventionally, random sampling utilizes a sampling frame, which is an accurate and exhaustive listing of all the members of the population, and employs probability sampling techniques to randomly pick respondents from the sampling frame.
With data privacy law in place, a sampling frame that contains online contact details is not always accessible nor easy to generate. However, there are private research agencies that offer online survey services and that have managed to systematically generate an online survey panel that would be representative of, or at least closely represent, the general population or segments of the population. While these online survey services do not come cheap, the quality of data generated substantially reflects the value of its price tag.
Alternatively, there are online survey platforms that can be used to implement online surveys. Some online survey platforms are free while some are subscription-based, though the price of subscription is not as hefty as the online survey services of private research agencies.
This introduces bias in sampling as members of the population are not equally predisposed to volunteer as survey participants.
However, without a sampling frame, surveys rolled-out through these online survey platforms would typically reach out to the members of the population through an open call for participation (i.e., through social media invitations with survey links disseminated through social media; through website-linked online surveys).
Hence, instead of the survey team randomly selecting respondents from a sampling frame, the open call for participation essentially depends on the response of the members of the population who will heed the invitation to participate. This introduces bias in sampling as members of the population are not equally predisposed to volunteer as survey participants. In effect, the sample will be skewed toward a segment of the population who are predisposed to participate as survey respondents.
What is an "open-call" approach?
Then again, what if the population of interest is the online population? Can an open-call approach generate a sample representative of the online population? The same issue would arise with respect to the sample being skewed towards a segment of the online population who are predisposed to participate as survey respondents.
You must also check if the survey project exercised due diligence in verifying the identity of its pool of online survey respondents and in weeding out dubious characters and submissions from inauthentic coordinated acts.
Can this concern be addressed by adjustments to sample size? Statistical techniques may be introduced to account for the extent of heterogeneity in the population in order to arrive at a sample size that captures population heterogeneity. However, strictly-speaking, even if this open-call approach can generate a substantial sample size and even if statistical adjustments can be made to account for heterogeneity, its non-adherence to rigorous probability sampling procedures makes the sample non-representative of the population.
Moreover, apart from checking if the online survey’s sampling design adhered to the protocols of representative sampling, you must also check if the survey project exercised due diligence in verifying the identity of its pool of online survey respondents and in weeding out dubious characters and submissions from inauthentic coordinated acts.
As a prospective online survey respondent. Before volunteering to participate in an online survey, you should check the legitimacy of the source of the online survey, confirm that it is from a reputable institution or organization, and assess whether you can entrust your personal information and data with them.
Clear and sufficient information
The survey should provide clear and sufficient information about the purpose of the project; the terms, benefits, and possible risks of participation; data privacy and security protocols; and survey proponent contact details. Thus, you should carefully and thoroughly read the information provided before giving your consent to participate. Use the provided contact information to get in-touch with the proponents for any survey-related question or concern. Verify the terms of your participation and make sure that you will not be harmed in any manner during and even after the conduct of the online survey.
Keep in mind that participating in the survey is voluntary and that you can withdraw from the survey at any point. Check the clause that states that the data you shared will be properly discarded in case you opt to withdraw your participation from the study. Make sure that the survey has proper data security and data management protocols in place to safeguard the data that you will be sharing.
Unless you are giving your consent to be identified in any report that would be generated from the survey, look for a clause that guarantees that the survey project will keep your identity confidential. Remember that surveys should clearly honor and respect your rights as a participant. If you feel any form of discomfort about the survey and its terms, then it is your right not to participate or to terminate your participation.
By taking note of these basic points on the importance of adequately sized, randomly drawn, and verified sample, we can be a more judicious audience of online survey reports. Moreover, by exercising due diligence before consenting to participate and in knowing our rights as respondents, we can be better outfitted as prospective online survey respondents.