Module 2: Empirical Law and Policy Methods

The variety of researchable issues and disciplines that may be relevant means that there are many possible methods. For example: economic modelling can inform competition law and policy reform; psychology and sociology can help to understand criminal and social justice issues; systems thinking and dynamic modelling can provide insights into many law or policy issues; ethnography can inform indigenous policy or help understand the needs of disadvantaged people; mathematics and statistics are used in analysing law and policy issues; and strategic planning can be used to design interventions. In this section I introduce some approaches to illustrate what is possible, to encourage researchers to explore suitable and feasible methods. However this is far from exhaustive, and new methods are developing all the time.

Handbooks or methods encyclopedias from the Sage website indicate many possible research methods, and the concepts that underpin them. These include social experimentation, case studies, the use of art imagery, grounded theory, observational methods, document analysis, modelling, focus groups, ethnography among hundreds of possible methods.

It may be informative to browse for examples in:

SAGE Handbook of Mixed Methods in Social & Behavioral Research
The SAGE Handbook of Qualitative Data Collection
The SAGE Handbook of Qualitative Data Analysis
The SAGE Encyclopedia of Qualitative Research Methods

Readers should remember that articles about research concepts and methods reflect the sometimes-differing views of experts from different disciplines.

Empirical studies

What methods are appropriate can depend on whether the research has theoretical or applied aims. A further consideration is what resources and capabilities are available to the researcher. Where the intention is purely scholarly, limiting what is investigated to fit available methods and resources may be reasonable, even if all relevant issues are not researched. However, if the intention is to propose practical solutions to complex challenges in society, excluding issues will increase the risk that outcomes will be unreliable. Real world investigations often face the conundrum that important questions are beyond the capacity of the researchers to investigate with an appropriate empirical method.

Because law and policy interventions can affect the welfare of people (or other creatures), a pragmatic choice not to investigate all aspects of an issue can have ethical implications. For example, research seeking to protect vulnerable migrant women employees is likely to require understanding of what happens in different workplaces, what the law is and how it is implemented, the dynamics of everyday oppression, the attitudes of workers and employers, and so forth. A narrow focus only on what the law is and how it is implemented means ignoring (for example) the cultural dynamic of oppression, or the economic pressures on migrant families or small business owners. The resulting ‘solutions’ would address only the aspects that have been investigated, which increases the risk that conclusions will fail to solve the problem, or perhaps make it worse.

Deciding the research strategy for complex law and policy issues involves difficult choices, with implications for the value and reliability of the investigations. Researchers should be able to justify the tradeoffs that they have made, the reasons for their choices and what aspects of the complete problem have been excluded from investigation.

The methodology of law and policy research is changing. Langbroek, P., van den Bos, K., Thomas, M. S., Milo, M., & van Rossum, W. (2017). Methodology of legal research: Challenges and opportunities. Utrecht Law Review, 13(3), 1–8. https://doi.org/10.18352/ulr.411 provides an overview of two of the shifts in legal scholarship, towards making methodology explicit, and towards non-doctrinal approaches. Other chapters in the Methodology of legal research book illustrate many qualitative and quantitative methods relevant to law and policy research.

McConville, M. (2017). Development of Empirical Techniques and Theory. In M. McConville & W. H. Chui (Eds.), Research Methods for Law (pp. 280–302). Edinburgh University Press. Retrieved from https://www.jstor.org/stable/10.3366/j.ctt1g0b16n.17%0AJSTOR tells the story of a legal scholar who gradually embraced social science methods. It is itself an example of “auto-ethnography”. It illustrates how political and institutional problems can arise for researchers. It also illustrates how empirical evidence can trigger insights about the law, that would not otherwise occur. The author used multiple empirical methods, to form a theory to explain legal phenomena. Though the analysis does not provide the explicit link between theory, research design, evidence gathering and structured analysis that a scientific approach would involve, it demonstrates an evidence-based approach to theory formation (note that the the original intention was not theory formation. A “grounded theory” method would arguably been more appropriate, if this had been the original purpose).

Majchrzak, A. (1984). The Nature of Policy Research. In Methods for Policy Research (pp. 12–22). https://doi.org/10.4135/9781412985024.n1 (republished in 2011 on the Sage Research Methods ) examines some variables in policy research. She refers to the need for an “empirico-inductive research orientation”, when it is necessary to rely on judgement because it is not feasible to obtain sufficient evidence about all of the research issues. Though in such situations pragmatic compromises may be unavoidable, science principles suggests that the researcher should explain their compromises and the limits to the generalisability of their findings.

Mixed methods and multi-disciplinarity

The examples we have discussed above show that a law and policy question can involve interwoven factors such as: the technicalities of a legal instrument or doctrine, the effectiveness and efficiency of implementation processes, the capacity of the implementing bodies and the characteristics of the parties involved, the dynamics of the transactions, and the influence of context variables (e.g. social, economic, cultural, political or other factors). Each sub-factor has many components, which can be investigated using various methods. A researcher wanting a complete understanding of the issues will use various methods, and each will yield different (and sometimes contradictory) intelligence. Having different forms of intelligence enables “triangulation” – judging the likely truth by considering an issue from different perspectives. When different types of evidence all point towards a finding, a researcher can have confidence that the finding is reasonable, though this is short of any finding being “proven”.

The term ‘mixed method’ refers to research that uses diverse methods. In the methods literature, it is often assumed the combination is of qualitative and quantitative approaches. However, mixed method research in law and policy may combine methods within either a qualitative or a quantitative paradigm. For example, investigating reforms to protect vulnerable women could involve discussions with ‘victims’ and ‘bullies (or meta-analysis of data from previous surveys), examining police or hospital records, focus groups or workshops to ‘brainstorm’ solutions (noting that the qualitative mix might also be supplemented with quantitative surveys, statistics or economic analysis).

The papers that follow explore the dynamics and the politics involved in mixing research methods from disciplines with different world-views (paradigms).

Morgan L., D. (2007). Paradigms Lost and Pragmatism Regained: Methodological Implications of Combining Qualitative and Quantitative Methods. Journal of Mixed Methods Research, 1(1), 48–76 is a complicated paper (available from Sage), most useful to those interested in the underpinnings of mixed methods research. It explains research ‘paradigms’, and the evolution of pragmatic approaches to messy social issues. Of particular interest is the discussion of axiology. In pure science the values and beliefs of the researcher should be irrelevant, but for important law and policy issues beliefs and values are significant. For example, in indigenous, feminist, or environmental law and policy studies researchers may adopt a liberal worldview without questioning their own perspective; and economic analysis typically accepts that economic growth and consumption are fundamental and perhaps the paramount concerns. These are hidden biases that often affect the research. Hidden biases are not unique to the social sciences – they exist in the empirical sciences as well – but they undermine objectivity, contrary to scientific principles.
The report by Bammer, G. (2012). Strengthening Interdisciplinary Research: What it is, what it does, how it does it and how it is supported (download from https://acola.org/wp-content/uploads/2018/08/strengthening-interdisciplinary-research.pdf , see particularly pp7-16) shows how interdisciplinarity is evolving, and providing opportunities for innovation. The idea that multi-methods or interdisciplinary research always involves both quantitative and qualitative methods is anachronistic –many combinations of methods for the law and policy researcher are possible within either tradition. What is important is the integrity of the research design and implementation, the quality and transparency of the analysis and reporting, and a disciplined approach to improving methods and the knowledge that they yield.

Why are “qualitative” methods widely used?

Quantitative methods use numbers to represent things which can be measured or counted that are studied. Because pure science concerns objective facts (which can be measured) it can use mathematical and statistical analysis, or quantitative models. However, because the law and many public policies also concern intangible values (e.g. justice, fairness, freedom, etc), non-quantifiable variables are important matters that often require investigation.

Implementation of the law also involves dynamics such as bargaining, power relationships, and politic, and judgement is needed to understand these. Law or policy transactions involve both objective and subjective variables, and respond to volatile ideas and situations, making the variables dynamic. Beliefs and emotions are not quantities, and opinions are often unstable. Though it is possible to use quantifiable approximators of subjective and volatile phenomena (and analyse the data mathematically) what is measured is not the actual variable. An example is using numerical measures in attitude surveys; another is attaching quantified economic value to intangible things such as amenity or aesthetics. Quantification reduces some of the complexity and ambiguity of the artefact or phenomenon being analysed, which makes numerical analysis possible. However there is an obvious risk that the analysis will be misleading because of what has been excluded .

Because aspects of socio-cultural issues cannot be reliably quantified, qualitative methods are widely used in law and policy research, illustrated by the following papers.

Dobinson, I., John, F., & Johns, F. (2017). Legal research as qualitative research. In M. McConville & W. H. Chui (Eds.), Research Methods for Law (pp. 18–47). Edinburgh University Press, UK suggest that doctrinal research is a form of qualitative research, and that the process for designing research is the same regardless of whether the subject is doctrinal, philosophical or empirical. The chapter outlines some of many qualitative methods.
Webley, L. (2012). Qualitative Approaches to Empirical Legal Research. In P. Cane & H. Kritzer (Eds.), The Oxford Handbook of Empirical Legal Research (pp. 1–21). Oxford University Press provides detail of some methods (limited to survey or observational methods, excluding experimental methods or innovations such as‘photo-voice’ or auto-ethnography). It highlights an important distinction between science positivist approaches (that seek a definite ‘truth) and interpretative approaches that embrace subjectivity and ambiguity. The paper provides guidance for researchers about possible methods.