Never fails! By recklessly holding forth on a topic that is obviously more complicated than I am making it out to be, I have again provoked a reflective, informed response from someone who really knows something!
Recently I dashed off a maddeningly abstract post on the “operational validity” of empirical science communication studies. A study has “high” operational validity, I suggested, if it furnishes empirical support for a science-communication practice that real-world actors can themselves apply and expect to work more or less “as is”; such a study has “low operational validity” if additional empirical studies must still be performed (likely in field rather than lab settings) before the study’s insights, as important as they might be, can be reliably brought to bear on one or another real-world science communication problem.
I wanted to distinguish the contribution that this concept, adapted from managerial studies (Schellenberger 1974), makes to assessment of a study’s practical value from those made by assessments of the study’s “internal” and “external” validity.
For a study to be of practical value, we must be confident from the nature of its design that its results can be attributed to the mechanisms the researcher purports to be examining and not some other ones (“internal validity”). In addition, we must be confident that the mechanisms being investigated are ones of consequence to the real-world communication dynamics that we want to understand and influence—that the study is modeling that and not something unrelated to it (“external validity”).
But even then, the study might not tell real-world communicators exactly what to do in any particular real-world setting.
Indeed, to be confident that she had in fact isolated the relevant mechanisms, and was genuinely observing their responsiveness to influences of interest, the researcher might well have resorted (justifiably!) to devices intended to disconnect the study from the cacophony of real-world conditions that account for our uncertainty about these things in everyday life.
In this sense, low operational validity is often built into strategies for assuring internal and external validity (particularly the former).
That’s not bad, necessarily.
It just means that even after we have gained the insight that can be attained form a study that has availed itself of the observational and inferential advantages furnished by use of a simplified “lab” model, there is still work to be done—work to determine how the dynamics observed in the lab can reliably be reproduced in any particular setting. We need at that point to do studies of higher “operational validity” that build on what we have learned from lab studies.
How should we go about doing studies that add high operational validity to studies of insights gained “in the lab”?
Science communication scholar Neil Stenhouse has something to say about that!
How to achieve operational validity: Translation Science
It is very unlikely that any real organization would want to use the stimuli from a messaging study, for example, without at least a few substantial changes. They would certainly want their organization to be identified as the source of the message. These changes would influence the effect the messages had on their audience. What kind of changes would the organization want to make? How much would that change the effects of the message? How could the message be made acceptable and usable by these organizations, yet still retain the effectiveness it had in previous experiments?
Communication practitioners wanting to put social science insights to use could very well ask questions like: how do you use the insights of cultural cognition experiments to design an effective large-scale messaging campaign for the Environmental Defense Fund? Alternatively, how do you use these insights to design a town hall meeting on climate change in Winchester, VA? How could you take a short passage about geoengineering, for example, that had a depolarizing effect on hierarchs and egalitarians (Kahan et al., 2012), and design a meeting that had a similar depolarizing effect? And if you did so, how well would it work?
I recently wrote a paper about research designed to answer questions like these (Stenhouse, 2014). It turns out that at least in one discipline, people are already doing a substantial amount of research that tests not only which kinds of interventions are effective, but figures out the nitty gritty points of what’s needed to effectively transplant the core of the lab-tested intervention into actual operational use in the real world. It addresses an important part of Dan’s concern with making communication research “evidence-based all the way down” (Kahan, 2013).
In public health, there is a whole subdiscipline – and multiple journals – on what is known as translation science, or implementation science (Glasgow et al., 2012). Researchers in public policy and international development are beginning to address this also (Cartwright & Hardie, 2012; Woolcock, 2013).
Translation science can be summarized with an example of testing an exercise program. With traditional public health research, a research team, often from a university, would design an exercise program, implement it, and measure and carefully document the results. Who lost weight? How much? Do they intend to keep exercising? And so on.
With translation research, as well as these kinds of outcomes, there is an additional focus on recording and describing the things involved in implementing these programs in the field, at scale (Glasgow et al., 1999).
For example, the research team might take their exercise program to a sample of the kinds of organizations that would be delivering the intervention if its use actually became widespread – e.g. hospital staff, community health organizations, church recreation group organizers (Bopp et al., 2007). The researchers would aim to answer questions like: how many of the organizations we approached actually wanted to implement the intervention?
Some organizations might be against it, for cost reasons, or political reasons (e.g. perhaps a hospital’s doctors have pre-existing arrangements with the providers of another intervention).
When an organization agrees to use an intervention, do they implement it correctly? Perhaps the intervention has multiple complex steps, and busy hospital staff may occasionally make errors that cause the intervention to be ineffective.
In short, traditional tests measure whether something works in the lab, under ideal, controlled conditions. Translation science measures whether something works in the real world, under typical real-world conditions (Flay, 1986; Glasgow et al., 2003). And in addition, by measuring the things that can be expected to affect whether it works in the real world – such as whether organizations like it, or how easy it is to implement – translation science can help figure out how to make interventions more likely to work in the real world.
For example, if researchers find out that an intervention is difficult for hospital staff for implement, and find out precisely which part is most difficult to understand, then they might be able to find a way of making it simpler without compromising the efficacy of the intervention.
Translation science provides the “operational validity” Dan was talking about. It answers questions like: What does it even look like when you try to put the results of experiments into real-world practice? How do you do that? What goes wrong? How can you fix it so it works anyway?
These kinds of questions are important for anyone who wants their insights to be applied in the real world – and especially important if you want them to be applied at scale. I think many researchers on climate communication would be in the latter category. While good traditional research can help us understand a lot about human psychology and behavior, it only does part of the job in putting that knowledge to use.
One question likely to come up is: Why should social scientists do this work, as opposed to the practitioners themselves?
I argue that they should do this work for the same reasons they should do any work – their skill in recording, conceptualizing and describing social processes (Stenhouse, 2014).
If we want rigorous, generalizable, cumulative knowledge about human behavior, we need social scientists. If we want rigorous, generalizable, cumulative knowledge about how to apply social interventions, we need social scientists there too. We need people who understand both the inner workings of the intervention and the context in which it is deployed, so that they can effectively negotiate between the two in creating the optimal solution.
Questions about division of labor here are certainly open to debate. Should all social scientists doing work with an applied purpose do some translation research? Should some specialize in lab work, and others in translation science, and occasionally collaborate?
These questions, as well as questions about how to shift academic incentives to reward translation science adequately, remain to be decided.
However, I would argue that especially in areas with urgent applied purposes, people are currently not doing nearly enough of this kind of work. We want our findings to be applied in the real world. Currently there are gaps in our knowledge of how to translate our findings to the real world, and other disciplines provide practical ideas for how to fill those gaps in our knowledge. We are not doing our jobs properly if all of us refuse to try taking those steps.
Neil Stenhouse (firstname.lastname@example.org) is a PhD candidate from the George Mason University Center for Climate Change Communication.
Bopp, M., Wilcox, S., Hooker, S. P., Butler, K., McClorin, L., Laken, M., ... & Parra-Medina, D. (2007). Using the RE-AIM Framework to Evaluate a Physical Activity Intervention in Churches. Preventing chronic disease, 4(4).
Cartwright, N., & Hardie, J. (2012). Evidence-based policy: A practical guide to doing it better. Oxford University Press.
Flay, B. R. (1986). Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Preventive medicine, 15(5), 451-474.
Glasgow, R. E., Lichtenstein, E., & Marcus, A. C. (2003). Why don't we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. American Journal of Public Health, 93(8), 1261-1267.
Glasgow, R. E., Vinson, C., Chambers, D., Khoury, M. J., Kaplan, R. M., & Hunter, C. (2012). National Institutes of Health approaches to dissemination and implementation science: current and future directions. American Journal of Public Health, 102(7), 1274-1281.
Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: the RE-AIM framework.American Journal of Public Health, 89(9), 1322-1327.
Kahan, D. M. (2013). Making climate-science communication evidence-based—all the way down. Culture, Politics and Climate Change. London: Routledge. Available at: http://papers. ssrn. com/sol3/papers. cfm.
Kahan, D. M., Jenkins-Smith, H., Tarantola, T., Silva, C. L., & Braman, D. (2012). Geoengineering and climate change polarization: Testing a two-channel model of science communication. Annals of the American Academy of Political and Social Science.
Schellenberger, R. E. (1974). Criteria for Asssessing Model Validity for Mangerial Purposes. Decision Sciences, 5(4), 644-653. doi: 10.1111/j.1540-5915.1974.tb00643.x
Stenhouse, N. (2014). Spreading success beyond the laboratory: Applying the re-aim framework for effective environmental communication interventions at scale. Paper to be presented at the 2014 National Communication Association Annual Convention.
Woolcock, M. (2013). Using case studies to explore the external validity of ‘complex’ development interventions. Evaluation, 19(3), 229-248.