Disability policy and research evidence

Progressive disability policy has long been recognized as a key strategy to systematically address the structural disadvantages faced by persons with disabilities 1 (World Health Organisation [WHO] and World Bank 2011; United Nations Department of Economic and Social Affairs [UN-DESA] 2019). Disability policy addresses these disadvantages by supporting programs that promote opportunities for persons with disabilities and establishing and enforcing legislation to prohibit discrimination and exclusion (United Nations [UN] 2006). Policy interventions require planning and the allocation of resources to ensure effective and sustained implementation.

It is generally understood that research evidence can be useful to inform policy development and implementation. The importance of linking evidence with policy is reflected in the provisions of the United Nations Convention on the Rights of Persons with Disabilities (UN 2006). According to the UNCRPD's article 31, States Parties are obliged to, 'undertake to collect appropriate information, including statistical and research data, to enable them to formulate and implement policies.' The UNCRPD stimulated disability policy development in many countries that previously had little to no disability policy. Subsequent to the arrival of the UNCRPD and expansion of disability policy, more research is being conducted on disability, particularly in low- and middle-income countries (LMICs). Disability prevalence has been a prominent focus of this research (Skempes and Bickenbach 2015).

Measuring disability prevalence

Prevalence, 'the proportion of individuals in a population who have the condition at a specified point in time' (Porta 2014) is a foundational descriptive measure in the field of epidemiology. Although the notion of prevalence was developed to measure disease, disability researchers and advocates have found utility in this measurement. Scholars such as Loeb, Eide, and Mont (2008) propose that disability prevalence can be used to monitor functioning in a population and to design services. Meanwhile, disability advocates propose that there is a 'demographic argument,' supported by a high prevalence of disability, that can be deployed in the political and social spheres to draw more attention to issues of concern for people with disabilities (Amundson and Tresky 2008; Handicap International 2012).

The calculation of prevalence is considered to be straightforward, as long as there is a clear, dichotomous case definition that can be applied reliably and consistently. However, with respect to disability, the classification of 'cases' and 'controls' is not straightforward (Pfeiffer 1999), especially since disability results from a complex interaction between person and environment (World Health Organization [WHO], 2001). Given this complexity, some scholars argue that disability should not be conceptualized in dichotomous terms (Zola 1993), while others argue that classification should be abandoned altogether (Hammell 2004). Meanwhile, there is a movement to refine classification and quantification led by the 'Washington Group for Disability Statistics,' which has developed tools for the identification of disability among adults (Madans, Loeb, and Eide 2017) and children (Loeb et al. 2018).

The Washington Group's tools measure disability based on self-reported activity performance ('How much difficulty do you experience when you try to do [a given task]?') rather than self-identification ('Do you consider yourself to be disabled?'). These tools allow for the quantification of disability according to various types (e.g., seeing, hearing, mobility) and severities (e.g., 'some difficulty,' 'unable to do'; difficulty in one activity versus difficulties in multiple activities). The use of these tools offers many possible measures of prevalence, most of which classify more individuals as disabled than tools based on self-identification. Alongside development of the Washington Group tools, the general estimates of the number of persons with disabilities in the world has gradually been revised upward over time. An estimate of '10% of the world's population' was popular from 1981 (World Health Organization [WHO], 1981; Mitra and Sambamoorthi 2014) and in 2011 this was replaced by an estimate of 15% in the World Report on Disability (WHO and World Bank 2011).

Zambia: a site of disability policy expansion and divergent research results

In the past decade, Zambia has revised and expanded its disability-related policies. Zambia signed the UNCRPD in 2008. Subsequent domestication of the Convention led to a significant reformulation of the Persons with Disabilities Act (Zambia 2012) and the development of a first National Policy on Disability (Ministry of Community Development Mother and Child Health [MCDMCH], 2015).

Like all signatories to the UNCRPD, Zambia is obliged to engage in research activities to inform its disability policies. Although there is not currently a disability research repository in Zambia (e.g., Centre for Applied Disability Research 2019), the country has a history of robust disability prevalence measurement when compared to many LMICs. Initiatives to quantify disability prevalence in Zambia include the national census (Central Statistical Office [CSO] 2012), a large-scale, national survey on disability and living conditions conducted by the Norwegian organization SINTEF (Eide and Loeb 2006; Eide et al. 2011), and most recently, a government led National Disability Survey (Central Statistical Office [CSO] and Ministry of Community Development & Social Services [MCDSS] 2018).

The multiple initiatives to quantify disability in Zambia have produced different results (see Table 1). For example, the disability prevalence measured through the census (0.9-2.7%) was markedly lower than that measured through the surveys (7.7-13.7%). A comparison of methods between these initiatives reveals important differences in the questions used for data collection and the calculation of disability 'cases.' In the three census waves, citizens were asked a form of the question, 'Do you have a disability?' By contrast, the survey results were based upon the more complex questions and calculations of the tools developed by the Washington Group. Given that questions about disability identity have historically produced lower calculations of disability prevalence than those derived from a series of questions about function (Me and Mbogoni 2006), the lower prevalence results from the census are consistent to what is expected due to their design.

Table 1. Quantifications of disability prevalence in Zambia (Loeb, Eide, and Mont 2008; Central Statistical Office [CSO] 2012; Central Statistical Office [CSO] and Ministry of Community Development & Social Services [MCDSS] 2018).

Year	Initiative	Result
1990	National census	0.9%
2000	National census	2.7%
2006	SINTEF survey	13.4%
2010	National census	2.0%
2015	National Disability Survey	7.7%

Whereas the differences in data collection and analysis clearly account for the lower disability prevalence calculations in the census as compared to the surveys, the marked difference in the survey results cannot be easily explained: both surveys had large representative samples and used the same six Washington Group questions (Loeb, Eide, and Mont 2008; CSO and MCDSS 2018). The Washington Group questions allow researchers to calculate prevalence using a case definition that is more inclusive, thereby increasing the disability prevalence, or more restrictive, thereby reducing disability prevalence. Contrary to what one would anticipate from the respective survey results, the primary prevalence calculations from the 2006 survey were based upon a more restrictive case definition: the disability prevalence calculated in the 2006 survey was 13.4% (Loeb, Eide, and Mont 2008), 2 nearly double the prevalence of 7.7% that was calculated from data collected in 2015 (CSO and MCDSS 2018). 3 When directly comparing the results from these two surveys – as is possible, since both used the same survey tool 4 – it is necessary to take into consideration the differing definitions. Fortunately, the researchers leading the 2006 survey published results according to multiple definitions of disability that can be derived from the Washington Group questions, including the one used for the National Disability Survey 2015. When the definition used for the National Disability Survey is applied to the 2006 survey data, it generates a disability prevalence of 17.8% (Loeb, Eide, and Mont 2008), revealing that there is a much larger difference between the results of the two surveys than that which is initially apparent from an overview. With the notable differences in disability prevalence results in Zambia, it is understandable that there could be widespread confusion or disagreement with respect to 'the actual prevalence of disability' in the country.

The need for contemporary critiques of disability prevalence

Given the value of robust research evidence to support new disability policy initiatives and the visibility, prominence, and variability of disability prevalence research, it is surprising that there has been very little critical attention devoted to disability prevalence. Although there are published critiques of disability prevalence (e.g., Kirchner 1990; Abberley 1992; Hahn 1993; Zola 1993), these pre-date important developments like the World Report on Disability (WHO and World Bank 2011) and the establishment of the Washington Group (United Nations Economic and Social Council [ECOSOC] 2003). Moreover, there has been no investigation of the perspectives of disability prevalence research expressed by stakeholders likely to bring disability prevalence evidence into the policymaking process (e.g., policymakers and civil society leaders advocating for the improvement of the situation of persons with disabilities).

The purpose of this study is to explore the perspectives of disability policy stakeholders, specifically disability advocates and policymakers, on disability prevalence research within the context of a single national jurisdiction. With a significant history of disability prevalence research and a recent expansion of disability policy, Zambia is an ideal country in which to conduct this research.

Study design

We used a qualitative descriptive design (Sandelowski 2000), guided by a social constructivist interpretive framework (Creswell and Poth 2018), to explore one element of a larger knowledge translation project. Consistent with social constructivism's interest in the ways individuals 'develop subjective meanings…directed towards objects or things' (Creswell and Poth 2018), the project aimed to better understand the ways that Zambian disability policy stakeholders use and perceive information, including research evidence. Here we report findings specific to the issue of disability prevalence, a topic that emerged as important through the inductive analysis of the collected data.

Sampling and recruitment

We aimed to recruit participants from two categories of disability policy stakeholders: advocates and policymakers. Disability advocates included leaders of disabled persons' organizations and non-governmental organizations. Policymakers were employed with the Government of Zambia or international organizations that are part of the United Nations' operations in Zambia. We recruited participants from these two categories in order to access civil society and policymaker perspectives.

We began recruitment by developing a list of potential participants according to investigator contacts and discussions with disability advocates known to the principal investigator from his previous research on disability in Zambia (Cleaver et al. 2018). From the list, we purposively selected individuals to approach with the intent of having a diverse sample with respect to gender, disability type, and substantive area of expertise. Participants were approached directly by telephone or email when contact information was available, or through formal government channels (i.e., written contact with the Permanent Secretaries of government ministries).

Participants

Eight advocates and four policymakers participated in this study (refer to Table 2 for participant details). All participants were based in Lusaka, the capital city of Zambia.

Table 2. Participants.

Participant number	Stakeholder category	Gender	Self-identifies as disabled	Self-declared disability type
01	Advocate	Man	Yes	Deaf
02	Advocate	Woman	Yes	Physical
03	Advocate	Man	Yes	Hard of hearing
04	Advocate	Man	Yes	Psychosocial
05	Policymaker	Woman	No	NA
06	Advocate	Man	Yes	Blind
07	Advocate	Woman	Yes	Physical
08	Advocate	Man	No	NA
09	Policymaker	Man	No	NA
10	Policymaker	Woman	No	NA
11	Advocate	Woman	No	NA
12	Policymaker	Woman	No	NA

Data collection

Data were collected through individual, semi-structured interviews lasting 30-90 minutes. Participants were asked to identify the disability-related policies that they thought were important and then describe their involvement with the development of these policies according to the policy cycle (Jack 2016). Participants were also asked about the structures, relationships, and processes (Lencucha, Kothari, and Hamel 2010) that guided the development of those specific policies. Particular attention was devoted to the ways that research evidence was used to inform policy development. All interviews were audio-recorded and transcribed. Data were collected in March 2018, prior to the release of the National Disability Survey 2015 final report (CSO and MCDSS 2018).

Data analysis

Data were first analyzed using content analysis (Hsieh and Shannon 2005) to organize the data and better understand its scope. We initially located all instances when participants referred to research. We then further analyzed these data, identifying that participants frequently discussed disability prevalence research. Through a thorough reading and re-reading of this data set, we organized the participants' accounts of the value of disability prevalence research into a typology (Patton 2015) that we later identified to be stakeholder priorities for prevalence studies.

Ethical considerations

This study was approved by the McGill University Institutional Research Board (Protocol reference #: A12-B68-17B), the University of Zambia Biomedical Research Ethics Committee (Protocol reference #: 011-01-18), and the Zambia National Health Research Authority. Informed consent was provided by all participants prior to the beginning of the interview. All data were anonymized during the transcription process and securely stored on encrypted media.

Inclusion: 'Involve us [for] good results.'

According to the participants who emphasized this priority, the inclusion of persons with disabilities is manifested in two ways: through the process of studying prevalence and through the data that is generated. This priority was expressed exclusively by disability advocates (participants 03, 06, 07, and 11). Two participants (03 and 07) expressed this priority emphatically. Participants 06 and 11 generally supported the issues related to this priority but presented these as more of a concern for their disability advocacy colleagues than for themselves (refer to Table 3 for the details of their presentation).

The disability advocates who presented this priority were critical of the Zambian National Disability Survey and concerned with its impending release. When these participants spoke of the national survey, they perceived it as being doubly deficient. They expressed disappointment that persons with disabilities were denied opportunities to participate in the conduct of the study and that the prevalence results under-represented the Zambian disabled population. Whereas these participants were primarily concerned with the results of the studies, they proposed that the exclusion of persons with disabilities from the process led to the low prevalence calculations.

Pragmatism: 'We have to use that [number]'

According to this priority, the greatest value of prevalence research is its potential to be supported by government stakeholders and applied through efficient policy levers like quotas. This priority was expressed by participant 12, a program officer within the government of Zambia. In contrast to participants 03 and 07 above, participant 12 viewed the situation in reverse. She presented the recent national survey as Zambia's first accurate quantification of disability: 'Like recently, you are aware, we conducted the National Disability Survey. In the past we'd used 10 percent but now that we have some preliminary results… the studies are showing it's now 7.7 percent.'

For participant 12, the single prevalence result of the national survey was a pragmatic tool that could inform the design and monitoring of programs from her Ministry: 'We have to use that in putting it as a quota for persons to benefit from the program. So basically, it's 7.7 percent of the population that is supposed to benefit from the program.'

Like 03 and 07, participant 12 was concerned about who was involved in conducting the research. However, from her perspective, the key player in the research needed to be the government:

When you want to conduct research, bring government on board. Let there be buy-in from government. I don't know if government were already part of [the SINTEF 2006 study] and this is where some private institution goes ahead and does… I think it was in collaboration with ZAFOD [the Zambian Federation of Disability Organisations].

Participant 12 was aware of disability advocates' attempts to promote a higher disability prevalence, considering these to be inappropriate applications of the worldwide disability prevalence estimate of 15%. Participant 12's response to this discrepancy was to also draw upon international data, but she instead chose to focus on disability prevalence estimates from neighboring countries, in support of the results of the national survey:

If you look at Zimbabwe, if you look at South Africa, those are the statistics. For the two studies that have… that have been done also a bit recently. Zimbabwe is about 7 some… 7-point- something percent, equally South Africa is. So the problem that we have is that people want the results to… to be in line with the World… is it the World Report on Disability? But then, you know, circumstances are different.

Granularity: 'This will help us get in-depth information'

Whereas the previously described priorities were focused upon a single result (the overall prevalence of disability), some participants were more interested in multiple results with greater detail. This priority was identified most clearly by participants 05, 08, and 09, all of whom provide support for the implementation of disability policies across multiple ministries. For example, participant 09 considered the national survey's initially reported prevalence of 7.2% to be 'proxy data,' stating:

Participant 09: It's national disability…it is coming from UNICEF [funder of the national survey], so that's the one that's actually being used, yes. But it does not disaggregate…

Interviewer: Between types of disabilities?

09: …yeah, among the disability categories, yeah. It's just, like a crude figure.

Participant 05 presented a need for greater detail with respect to the measurement of disability prevalence, since her office was frequently approached by,

…various institutions, saying 'Okay, we are asking for this data. How many persons with disabilities, probably in our area have this, that?' But we don't have that information and it will surely help institutions when planning for persons with disabilities.

For participant 05, the Washington Group questions were a tool that should be used more widely, 'because this will help us get in-depth information about persons with disabilities.' The Washington Group questions are premised upon a greater specificity of disability types and severity. In seeking to confirm that these types of details were of the greatest interest, the interviewer asked participant 05:

Interviewer: If we are talking about statistics, I think part of what you are talking about has to do with disability prevalence. Isn't it? Just knowing the number of people who are there, and then more details about their disabilities?

05: Their economic statuses, probably their educational statuses.

Unlike participants 05 and 09, both of whom work in government, participant 08 was an employee of a non-governmental organization. Nonetheless, his perspective was similar to his government colleagues with regards to the need for more details:

First of all, we need to have well documented data or statistics about how many persons with disabilities are there, in the country. How many have qualifications? How many are in primary schools? How many are in schools? For example, and so on.

Participants 05, 08, and 09 all expressed interest in the quantification of the details of the disabled population, specifically with respect to disability types, demographics, and socioeconomics. According to these participants, the details would be more effective at informing programming than a single disability prevalence result.

Priorities and policy stakeholder positionalities

For participants who expressed perspectives according to the priority of inclusion, prevalence research could substantiate (or undermine) the extent to which disability was a public policy concern. All stakeholders expressing this perspective were disability advocates and most considered themselves to be disabled. These participants felt that it was important that prevalence studies demonstrate that there is a high percentage of persons with disabilities in the population to justify governmental budget allocations and disability programming.

For the participants who expressed priorities of pragmatism and granularity, prevalence research was also important but its purpose was different: these participants valued the research for its capacity to refine, rather than justify, disability policy. According to perspectives that emphasized pragmatism in relation to prevalence research, the refinement was aimed at aligning the distribution of social programs to the distribution of disability in the population. For perspectives aligned with granularity, the development of policies and programs requires more than knowing the number of persons with disabilities, it requires the quantification of more specific categories of impairment and socio-economic status. It should be noted that the participants expressing these priorities did not self-identify as disabled; three of four were policymakers employed by the government while the remaining participant was an advocate working for a non-governmental organization (rather than an advocate engaged with a disabled persons' organization).

The association of the positionalities of the participants with the priorities that they emphasize for prevalence research could be indicative of more fundamental relationships to policy and research in the different stakeholder categories. Whereas the impact of 'good' prevalence research is interpreted in similar ways – all would agree that it is useful even if there are varied perspectives about the nature of good research and its application – the effects of 'bad' research are likely to be experienced very differently. For policymakers without disabilities, bad research might be useless or lead to suboptimal policies. Meanwhile for disability advocates, especially those who are disabled themselves, bad research might be dangerous if it justifies a lack of attention to disability or a diversion of resources away from disability issues.

In addition to the different perspectives on the policy implications of research, perspectives on who should be involved in the research process also appear to be associated with positionality. For the advocates with disabilities who emphasized the priority of inclusion, prevalence research must involve persons with disabilities as part of the process. Conversely, for a policymaker who emphasized pragmatic policy applications of disability prevalence research, government involvement was necessary for the research to be trustworthy.

Research to estimate the prevalence of disability attracts outsized attention

This study was drawn from a larger knowledge translation project that explored the connection between disability and research more broadly. There have been multiple research projects about disability in Zambia, using a variety of methods and designs (e.g., Trani and Loeb 2012; Banda-Chalwe, Nitz, and De Jonge 2014; Annie, Ndhlovu, and Kasonde-Ng'andu 2015; Parsons, Bond, and Nixon 2015; Cleaver et al. 2018), therefore it was unforeseen that prevalence research would figure so prominently in the participants' accounts. All participants who spoke about prevalence research emphasized the importance of this type of research, even though they disagreed about the ways in which it was important.

Equally unanticipated was the possibility of organizing the participants according to their priorities for prevalence research. The priorities emphasized by participants could effectively be considered 'camps,' each of which was aligned with a different vision for disability prevalence research and its relevance for policy. In the case of the priorities of inclusion and pragmatism, the respective visions were oppositional; indeed, those who emphasized these priorities spoke not only of the merits of their preferred disability prevalence study, they also explicitly discounted the alternative. Between the stakeholders who express these respective priorities, disability prevalence is contested, both in terms of the way that this research should be conducted and in terms of the single number that the process produces (i.e., the percentage of the population that is disabled).

There is reason to believe that these two findings, the unexpected interest in disability prevalence research and the contested perspectives on this research, are related to one another. In effect this study might have identified an example of a 'science-related public controversy,' a public controversy that 'result[s] from uncertainty—such as that which arises from either an inconclusive set of scientific findings or disagreements within the scientific community about how to interpret the results of science' (National Academies of Sciences 2017, 51). To be clear, this example of contested priorities regarding disability policy evidence in Zambia is a much smaller controversy than the 'wicked and messy' (Yamamoto 2012) science-related public debates on (for example) climate change or genetically modified organisms in food (National Academies of Sciences 2017). The controversy identified in this study is more contained, involving stakeholders who agree about the value of prevalence research and the need for more robust disability policies in Zambia. Nonetheless, it appears as though each of two diverging perspectives on how research should be done and the result that it should generate has aligned with a prevalence survey (Eide and Loeb [2006] versus CSO and MCDSS [2018]) to create a scenario in which, 'Differing bodies of evidence provide ammunition for competing views' (Sarewitz 2015, 414). In turn, this dynamic enables a polarization of perspectives with respect to the research evidence (Ingwersen et al. 2021) that can draw attention to a specific research program as a site of promise and threat and potentially hamper collaboration.

All eyes on the numbers?

As qualitative researchers, we were well positioned to notice that participants spoke frequently and emphatically about quantitative research while rarely acknowledging the qualitative research about disability that has been conducted in Zambia. While the differential interest might be partially explained by quantitative researchers mounting more successful knowledge mobilization campaigns than their qualitative researcher counterparts, we expect that another phenomenon should be considered: the tendency for quantitative evidence to carry more weight in the policy process due to the 'certainty' of numbers (Jerrim and de Vries 2017). Furthermore, quantitative evidence 'often simplifies complex social problems into a single set of numbers,' (Jerrim and de Vries 2017, 118).

Among the perspectives expressed by participants in this study, there are those that align well with the proposition that single sets of numbers – in fact one number, the population prevalence of disability – is a valuable tool for the policy process. According to the participant who emphasized the priority of 'pragmatism,' an accurate prevalence estimate of disability offered an important, straightforward mechanism to improve the capacity of a social program to reach Zambians with disabilities, by using this prevalence estimate to establish a beneficiary quota. For the participants who expressed the priority of 'inclusion,' a large population prevalence of disability justified the need for focused policies and programs for Zambians with disabilities, simultaneously supporting the need for larger budget allocations for disability programs and services. The use of large population numbers to validate the importance of disability issues has been articulated as a 'demographic argument' (Amundson and Tresky 2008; Handicap International 2012).

Simplicity and persuasive power are compelling reasons to emphasize a single disability prevalence number to encourage policy change. There are also downsides to this approach. One risk is the possibility that a small prevalence estimate undermines the importance of the issue. Concern for this risk could belie the participants' unanimous denunciation of disability data from the census, even though these data could be useful for multiple purposes when users account for the fact that the census uses a more restrictive case definition. The promotion of a single large number also incurs the risk that opponents will scrutinize the data to undermine advocates' claims of significance (see Zola 1993; Mechanic 2003). With respect to the use of disability prevalence to drive quota systems, there is a precedent of a disability prevalence estimate having been created first through a research and diplomacy process that was later applied to an employment quota system. When applied to the employment quota system, the categorization created for disability prevalence measurement classified workers as disabled or non-disabled according to characteristics that were confusing and seemingly arbitrary (Kohrman 2003).

The biggest limitation of a single number could be the loss of complexity for a complex phenomenon. This simplicity could lead end-users to overlook the uncertainty of the original research and pursue policies that do not match the data (Jerrim and de Vries 2017). With respect to disability, a phenomenon that is inherently diverse with respect to type, severity, and experience (WHO, 2001), the action of trying to identify a single percentage or number of persons with disabilities could be a misguided 'attempt to make fixed and dichotomous something which is better conceptualized as fluid and continuous' (Zola 1993, 18, italics in original). While the number of persons with disabilities might place the general issue of disability on the agenda, it would not inform the content of policy to adequately respond to different needs within this category.

Implications and recommendations

Consistent with the wave of disability prevalence critique from the early 1990s (e.g., Kirchner 1990; Abberley 1992; Hahn 1993; Zola 1993) we hope to illuminate phenomena that are generally ignored by drawing attention to structures and practices that are often taken for granted. Without this type of critique, we suspect that the most intuitive 'solution' to the 'problem' of stakeholders contesting disability prevalence research will be another prevalence study, with the goal of the next study to provide 'the correct' measurement of disability prevalence in Zambia. Tremendous effort and resources are necessary to conduct a national disability prevalence survey, frustrating the possibilities of this occurring soon. From our perspective, the lack of additional prevalence research could indeed be positive since we are sympathetic to Abberley's (1992) assertion that the project of 'providing the "true" figure [of disability prevalence] is an impossible one' (p. 153). By contrast, we think that the identification of the varied priorities for prevalence research provides insight with respect to the values of Zambian disability policy stakeholders and their understandings of disability research. From this insight we see immediate implications and recommendations that can be applied with far fewer resources than those required for a national survey.