Why we must work toward a recognised international standard in evaluation of upstream interventions
Author:Matthew Johnson, Elliott Johnson, Laura Webber and Kate Pickett
This post was originally published by the Evidence & Policy blog on 6 January 2021.
We have re-issued the article that has already been published by the Evidence & Policy blog. We would like to express gratitude to the kind offer of the editorial board of the Evidence & Policy blog.
The COVID-19 pandemic has increased interest in Universal Basic Income (UBI) as a means of addressing a range of socio-economic insecurities. While previous trials of cash transfer schemes have often focused on low-level transfers inadequate to satisfy the needs for which the policy was originally developed, emerging pilots are moving toward a position of increasing generosity. Our multidisciplinary project, Examining the Health Case for UBI, has brought together colleagues in behavioural science, public health, epidemiology and economics to establish pathways to health impact outlined in Figure 1 below. Our work suggests the potential for significant health impact and attendant economic benefit via reduced healthcare costs and increased economic activity. The model suggests that elements of impact may only be felt if payment is set at a more generous level. This could create greater return on investment and, ironically, a more cost-effective system.
Figure 1: UBI Model of impact (Johnson, Johnson, Nettle & Pickett, 2020)
The UK Government is committed to a ‘prevention agenda’ aimed at efficient, upstream investments in the nation’s health. UBI helps deliver that commitment, and support for UBI may be increasing. It has the potential to address ‘social determinants of health’, and reduce health inequalities, in ways that benefit those in the workforce as much as outside it. However, we have only a patchy evidence base upon which to evaluate that model, and much of that research examines labour market participation without health impact in mind.
Our Evidence & Policy article, ‘Measuring the health impact of Universal Basic Income as an upstream intervention: holistic trial design that captures stress reduction is essential’, highlights the need for serious interdisciplinary collaboration to design research protocols capable of capturing health impacts, and not just over the duration of the trial. We need to establish a gold standard in evaluation by developing research protocols that capture health impact comprehensively during different types of upstream intervention trials.
Achieving this is complicated. Given the nature of the pathways to impact, and the difficultywith which stress is measured, it is essential that the effects of the intervention on stress levels, nutrition and behaviour are captured comprehensively to enable assessment via statistical modelling of short, medium and long-term impacts beyond the duration of a trial.
As a starting point, we wish to develop two research protocols: a) for study of a pilot trial of a relatively small targeted number of people, such as a prospective scheme in Bradford and b) a larger scale trial involving a small town, such as Dunfermline in Fife, Scotland (population 50,000), where a UBI trial has been proposed. The two research protocols will necessarily differ according to scale and measures. The former will establish feasibility in terms of ethics, payment and proof of research concept. The latter will record data relating to a larger number of variables insofar as the intervention will also be concerned with evaluating collective level efficacy and broader socio-economic outcomes.
Protocol development needs to proceed according to the following principles:
routine collection ought to be the foundation for baseline comparison
measurement ought to capture wellbeing in its broadest form
only measures validated against an established standard will be deployed
self-reporting requires simplicity to ensure accuracy
cost ought to be minimised where similar outcomes can be produced via cheaper procedures.
Research to develop protocols needs to establish the value of existing data from: the Census, which includes a self-rated health score; Sport Active Lives Survey, which includes some wellbeing measures, disability by impairment and physical activity; primary care activity; secondary care use; inpatient and accident and emergency activity data in England; accidents via Public Health England’s Public Health Profiles and Deaths, and the Office for National Statistics’ cause of deaths data.
The next stage is to examine the possibility of integrating survey frameworks, such as perceived stress scale for chronic stress, GAD-7 Anxiety score sheet, PHQ-9 Depression score sheet, self-efficacy scale for agency, collective level efficacy, societal level trust, EuroQol (EQ)-5 quality of life survey, WEMWBS scale for Wellbeing and Behavioral Risk Factor Surveillance System (BRFSS) life satisfaction scale as a proxy for subjective wellbeing.
This will enable consideration of the need for collection of biomarkers, such as the Whitehall II method of stress measurement, and hair for study of chronic stress as an indicator of longer-term morbidity. Assessing viability of biomarker collection is important insofar as individuals may either not perceive their biological stress response accurately – since some studies have found no significant association between biological stress and mood – or self-report it differently for a range of social reasons. Formal sample size calculations can then be conducted for each of the measures to establish minimum size of trials and minimum number of participants.
Finally, it is necessary to establish the cost of conducting the research protocols. This is essential to determine the viability of trials, since the cost of conducting the research is likely to be significant. For example, a scoping exercise on a potential Scottish Longitudinal Study of Ageing used figures from the English and European equivalents to establish a cost of approximately £2-3 million per wave covering 10,000 participants in the first wave, with a series of four waves over a period of 10 years costing £8-12 million.
This work is a significant undertaking. However, we cannot continue to have trials of UBI, and upstream interventions more broadly, conducted without a protocol that enables a comprehensive understanding of their impact on health in the long term and at population level. A significant effort now will pave the way for the effective evidence-based policy in future.
----------------------------------------------------------------------------------------------------------------------
Matthew Thomas Johnson, PhD, Senior Lecturer in Politics, Lancaster University. Politics, Philosophy and Religion, County South, Lancaster University Matthew Johnson is a Senior Lecturer in Politics, Lancaster University. He has conducted participatory research into the effect of modern economic circumstances on people’s health, including through collaboration with GPs.
Elliott Aidan Johnson, MA, Disability Research Consultant Elliott Johnson is a Research Consultant in the disability sector. He has published, with Matthew Johnson, on the psychobiological mechanisms by which hierarchies can harm physical health and, separately, on the impact of the UK welfare system on disabled people’s physical activity.
Laura Webber, PhD, Chief Operating Officer and Co-founder, HealthLumen; Honorary Assistant Professor, London School of Hygiene and Tropical Medicine Laura Webber is COO and Co-founder of HealthLumen and Honorary Assistant Professor, London School of Hygiene and Tropical Medicine. She has written widely on the need for effective modelling and upstream interventions capable of advancing a ‘health in all policies’ approach.
Kate Pickett, Professor of Epidemiology, University of York Kate Pickett trained in biological anthropology at Cambridge, nutritional sciences at Cornell and epidemiology at UC-Berkeley. She is currently Professor of Epidemiology in the Department of Health Sciences, and the University’s Research Champion for Justice and Equality. Kate was an UK NIHR Career Scientist from 2007-12, is a Fellow of the RSA and a Fellow of the UK Faculty of Public Health. She is co-author, with Richard Wilkinson, of The Spirit Level chosen as one of the Top Ten Books of the Decade by the New Statesman, winner of Publication of the Year by the Political Studies Association and translated into 23 languages. She is a co-founder and trustee of The Equality Trust.
----------------------------------------------------------------------------------------------------------------------
Image credit:Photo by Javardh on Unsplash
----------------------------------------------------------------------------------------------------------------------
This post was originally published by Transforming Society on January 2021.
You can read the original research in Evidence & Policy:
Johnson, E.A. Johnson, M.T and Webber, L. (2020). Measuring the health impact of Universal Basic Income as an upstream intervention: holistic trial design that captures stress reduction is essential. Evidence & Policy, DOI: 10.1332/174426420X15820274674068.
----------------------------------------------------------------------------------------------------------------------
If you enjoyed this blog post, you may also be interested to read:
---------------------------------------------------------------------------------------------------------------------------------------