Generalizability and Representativeness: Considerations for Internet Based Research on Substance Use Behaviors
To offer an empirically driven viewpoint on the distinctions between the concepts of (1) scientific generalizability vs. (2) statistical representativeness and then use these two concepts to explore the strengths and weaknesses of internet based sampling for substance use research.
Part 1 of this paper presents a broad narrative review of the scientific literature and simulation results to help readers understand the difference between the concepts of scientific generalizability vs. statistical
representativeness obtained via random sampling. Simulations are also used to clarify how hypothetical internet sample selection mechanisms could potentially bias regression model results. Part 2 presents another,
more focused narrative review that examines the current state of internet sampling for substance use research. Additionally, empirical data on cannabis use collected via two Facebook advertising strategies and Qualtrics surveys are presented to help readers solidify the abstract scientific concepts presented in Part 1.
Probability based sampling is not always a prerequisite for valid scientific generalization.
Despite their many limitations, internet based methods and samples can contribute valuable scientific insights about substance use.
Internet based samples should not be used to make statements about prevalence (e.g., The prevalence of past month cannabis use in the U.S. population is X% X%”).
There are many different types of internet platforms and sampling methods each of which has unique strengths and weaknesses.
Substance use researchers are encouraged to think carefully about platform specific dynamics (e.g., motives for visiting a particular webpage, mechanisms influencing the probability of study participation, etc.) and the sociological “characteristics” of the substance to be studied (e.g., stigma, norms, laws, prevalence) when using an internet platform to collect data and interpret results.
Internet based sampling methods are an increasingly popular tool among substance use researchers, but the extrapolative specificity of resulting data is often unclear.
This paper summarizes several core concepts in the philosophy of science and provides a set of guidepost considerations to help researchers collect, analyze, and interpret internet based samples of substance use behaviors.
Borodovsky JT. Generalizability and representativeness: Considerations for internet based research on substance use behaviors. Exp Clin Psychopharmacol . 2022;30(4):466 477. doi:10.1037/pha0000581