Module 1: Introduction

Please click on the video link to view “Non-doctrinal research methods in law and policy: Introduction”. That video provides information about these resources, about the author, and introduces key aspects of non-doctrinal law and policy research.

The videos that form part of these resources were prepared during the Covid epidemic in mid-2020, using video calls and amateur video and editing. This has unavoidably limited their production quality, for which we apologise.

Non-doctrinal research methods in law and policy: Introduction

Methods - terms and concepts

Discussions about research methods involve specialized concepts and language. Use Wikipedia or a dictionary to explore the following terms/concepts (writing them down will help you to remember them).  Authors from different disciplines do use the same terms in different ways!

  • Axiology
  • Data
  • Discourse
  • Doctrine (legal)
  • Empirical
  • Epistemology
  • Hypothesis
  • Methodology
  • Methods
  • Paradigm
  • Philosophy
  • Philosophy of Science
  • Positivist
  • Scientific method
  • Theory
  • Triangulation

Experts may also have strongly held contradictory beliefs about the ‘right way’ of doing research, and some of what is written about methods is hard to understand because the authors are focused on technical or philosophical details (and because academic writers often do not use simple English).

My approach to dealing with ambiguity is to not worry too much! Ambiguity does give some freedom, and understanding will improve with exposure and experience. You should not be shy about asking those with more experience for their views, but remember that experts may have very different ideas.

Adapting the­­­­­ scientific method to law and policy

In traditional sciences (e.g. chemistry, physics, mathematics etc.) credible researchers must follow an empirical scientific method. Frequently their theory, hypotheses, methodology, methods, data and findings are stated in publications (particularly in research theses). To be fully scientific, the data should be able to be observed and measured with reasonable precision. Such scientific standards of objectivity have gradually infused areas that traditionally used less prescriptive methods, such as psychology, political science, and economics, and scientific norms increasingly applied in law and policy research.

This adoption of scientific conventions into law and policy research has been slow, for many reasons. Law and policy questions often involve non-measurable matter as well as concrete facts. For example, the concept of justice involves subjective views and philosophical questions that cannot be empirically tested. This type of subject matter defies purely empirical investigation. Many phenomena that are important are caused by the interaction of very many variables, and to limit investigation to those that can be empirically tested can lead to unreliable conclusions. In addition, law and policy issues often concern complex tradeoffs between competing interests, and politics and self-interest create complex dynamics. Mainstream legal scholarship and public policy are, for these reasons, often legitimately discursive, political, and multifaceted.

Even though law and policy research may not fit neatly within the pure scientific method, researchers still should respect scientific principles to the maximum possible degree. They should do all they can to minimize subjectivity, be explicit about methods and their data, and be transparent about the limits to the reliability and “generalizability” of their research. Open disclosure will contribute to disciplined improvement in the research field, and limit the risks of self-interest, bias or ignorance.

There is a growing tradition of law and public policy researchers using empirical evidence and scientific methods to improve understanding of the law, particularly when the research concerns making law and policy instruments more effective, efficient, and fair. The non-doctrinal methods used to inform legal scholarship include observational research, surveys of attitude and behavior, economic or dynamic systems modelling, political science, and evidence and analysis from many other disciples.

Traditional doctrinal legal research focuses on legal instruments, and to a lesser degree the processes, that constitute the law. Central to these investigations are dialectic approaches; paying particular attention to the language of instruments (such as judgements and administrative instruments). Jurisprudential research often examines the values and philosophies embedded in the law and generally also relies on discourse. Empirical “non-doctrinal” investigations often concern the organisations, dynamics and contexts that create, interpret and apply laws, and with the dynamics and effects of the law. Computers and statistical methods have opened up new research avenues, even within doctrinal research, and as machine intelligence increases new legal methods will emerge. Other disciplines such as sociology or economics or criminology also study law as a mechanism to achieve social ends, or examine the forces that drive the dynamics of the law in society, and are less concerned with doctrinal technicalities.

Law and policy scholars who use methods from other disciplines should pay attention to the norms of those disciplines, to ensure that the methods they use are sound and properly implemented. This helps to ensure that their work is credible. A concern for methodological credibility has increased as the responsibility for legal education and research has shifted from narrow legal craft guilds (e.g. the Inns of Court or articles within the legal profession) to academic institutions with scientific research cultures. This has fostered an increasing professionalization of legal academia, bringing an expectation that academic research will be scientifically credible.

Though legal scholarship is becoming more scientific, axiological and discursive elements are inevitably embedded in the law, and this limits the extent to which legal scholarship can become a science (in the narrow sense of that word). Postgraduate researchers are particularly being expected to follow science norms: to be explicit about the research questions, methodology, data and analysis that they use. These expectations are most salient when law and policy research examines empirical matters, and uses methods from explicitly scientific disciplines. The requirement for a strict scientific approach can be challenging for researchers who are not imbued with the norms and culture of mainstream science; or the norms and culture of the disciplines whose methods they choose to use. However credibility requires that they overcome these challenges.

A package of resources

This package of resources is designed assist law and policy researchers to bridge the gap between traditional law and policy research practices and culture, and the methods which are used in empirical sciences. The resources are designed for self-directed use; particularly for research students to use them with their supervisors. The materials have should help students to write up their methods.

The package contains four types of resources:

  • Readings selected for their content and suitability for students not familiar with the jargon of academic disciplines. For most, there are links to be found by searching on Google Scholar, and for a few I provide the links. For a very few, you need to purchase or borrow the book in which they are found.
  • SAGE Research Methods resources (http://ezproxy.une.edu.au/login?url=http://methods.sagepub.com) which is a concentrated source for research students and supervisors for information about research theory, methods, and practice.
  • Short videos (including examples of researchers using different methods) to help the researcher navigate the materials, to identify what is useful and practical for them.
  • Pointers to research “tools” and supports to facilitate research and writing.

Every discipline has preferred methods, and often un-questioned “biases”, about method. Academics discussing methods are generally aiming to communicate with their scholarly peers, so they tend to focus on less obvious aspects of method and philosophy and they use complex terms. Sometimes their personal views are presented as universal truths, rather than positions that reflect assumptions that vary across disciplines, and with situations. This can lead other researchers into trying to use methods that are not suitable or feasible for them or their research, only to learn the difficulties through painful experience.

Students who have completed a methods course designed for one discipline (e.g. business or sociology) can have problems when their research is in a field where different considerations and research norms apply (e.g. with law and policy research). For example, some statistical research norms may not be relevant if the sample size is so small or the issues so complex that statistical analysis is not feasible.  Slavishly following a method without thinking critically about its efficiency for what you are doing can lead to errors. Law and policy researchers (like other researchers) may need to innovate, and tailor methods to the specific investigations they are conducting. However, regardless of what method you choose, high standards of integrity and transparency remain fundamental.

Applying a scientific discipline to law and policy research

The readings and the video that follow consider whether modern law and policy research is, or can be, truly ‘scientific’ (viz. following the epistemology and methodology of science). They suggest that the science paradigm has limits when investigating some issues in law or public policy, and highlight the implications of these limitations.

  • Ulen, T. S. (2002). A Nobel prize in legal science: Theory, empirical work, and the scientific method in the study of law. University of Illinois Law Review, 4.(4), 875–920. https://doi.org/10.2139/ssrn.419823
    This paper indicates that legal research is evolving towards a more ‘scientific’ approach. This involves a shift from document focused discourse methodologies, to the use of explicit theories and objective evidence.
  • Martin, P., & Craig, D. (2015). Accelerating the evolution of environmental law through continuous learning from applied experience. In P. Martin & A. Kennedy (Eds.), Environmental jurisprudence – improving governance effectiveness (pp. 1–23). London UK: Edward Elgar. https://doi.org/10.4337/9781783479313.00006

This paper suggests that a strategic, rather than a scientific, paradigm may provide a sound basis for much applied law and policy research.

There are different definitions of “science” or the “scientific method”. The philosophy of science is complex and there are heated debates about abstract issues, but the core concepts of science recur in that literature. There are plenty of web resources that give simple explanations – but these are not always consistent, partly because they reflect different theories of science, or different research subject matter.

An example of a simple explanation of the scientific method can be seen at: https://youtu.be/zST6xoW-GUw.

Please watch the video, then think about what issues might be involved in researching the following law and policy topics (while reflecting on the questions involved in the topic that you intend to research):

  1. Investigating public policy or law reform to protect female migrant workers from exploitation.
  2. Researching legal or other strategies to protect endangered species on public and private land.
  3. Exploring the potential to implement full cost recovery from the users of government services.

Consider the following questions:

  1. What aspects of these questions involve subjective matters such as beliefs or values, and what methods might be useful to explore these intangible issues?
  2. What aspects of these questions should/can be addressed using objective evidence? What forms of objective analysis, and what types of data, would assist in answering these questions?
  3. What aspects of these issues involve social issues such as politics, political or social power, or bargaining? How might these issues be investigated?
  4. What scientific norms should apply to all research regardless of method or issues, and which norms might need to be modified or abandoned for some questions?

Empirical research is increasingly important in law and policy scholarship, and there are many possible approaches. Some examples are provided in a later module. The next paper highlights the contribution of empirical work to law and public policy (noting that the authors do consider only a narrow form of empirical methods).

The saying that “to a man with a hammer, every problem is a nail” reflects one challenge of research methods. Researchers can have strong preconceptions about methods, including believing that their preferred method is the only ‘correct’ way to do research. This bias can be counterproductive.  The specifics of the question being investigated will affect which scientific, philosophic, doctrinal or other issues should be researched, and what methods are appropriate. For example: investigating whether a law actually works will require considering objective evidence about policy goals and outcomes (noting however that “effectiveness” of laws and policies itself has a philosophical and political dimension); and determining how a law is implemented by the courts can involve doctrinal considerations alongside other types of evidence (e.g. opinions, statistics, observations). Deciding whether the legal system is implementing particular social values (e.g. indigenous or feminist values) will involve factual and philosophic or doctrinal matters.

What research approach is best will vary depending on whether the investigation is theoretical, or meant to develop practical solutions to problems.  A scholarly study is likely to be more abstract (philosophic or doctrinal) than a study intended find practical solutions. Designing applied instruments or systems may require consideration of implementation issues such as the capabilities of those responsible for implementation, political and other interest dynamics, and other social, economic or cultural contexts. In particular the diverse issues within an applied legal issue often creates the need for a variety of methods – which is why ‘mixed methods’ approaches are common in empirical law and policy research.

Pragmatic approaches to cope with complexity

It is logical to use a purely scientific approach to investigate a physical phenomenon where there are few variables (e.g. the response of a plant to changed carbon dioxide in the atmosphere); or for a descriptive study (e.g. describing the physiology or behaviour of a species). Public policy research however often involves interwoven complex and dynamic issues. This is methodologically challenging because many diverse variables are at work at the same time, and different contexts create different outcomes. The constraints of time, skills and money can make it infeasible to use a ‘pure’ scientific approach when the characteristics of the relevant phenemona vary, where many variables operate at the same time, and where objective facts are mixed with subjective variables (e.g. about justice, fairness, power and politics). The three examples of law and policy topics mentioned earlier (female worker exploitation, threatened species, and the economics of legal administration) demonstrate these characteristics, which are found with many law and policy studies:

  1. situations where the issues arise vary greatly, and each instance has unique characteristics and dynamics;
  2. when the characteristics of the actors and organisations involved vary between instances, and this affects the dynamics of the legal/administrative system, and its outcomes; and
  3. when many social, economic, institutional, and cultural variables are involved in the key transactions, and these are constantly changing and interacting.

To research such situations requires ‘pragmatic’ choices about what can be realistically investigated (given available resources and time); about what methods will work; and about how subjective judgement can be used to cut through ambiguities or gaps in evidence.

Many issues involving the law are so complex that they can be characterised as ‘wicked’, which is discussed in the following paper.

  • Head, B. W. (2008). Wicked Problems in Public Policy. Public Policy, 3(2), 101–118. https://doi.org/10.1128/MCB.00567-07. Researching wicked issues requires considering many factors together. Proposing solutions will often require conclusions using subjective judgement (supported by evidence). Effective responses may require interwoven actions that go beyond law or policy, such as social, institutional, economic and cultural interventions.

Pure science rejects subjective interpretation, and aims for transparent deduction based on facts alone. However, conclusions about complex social issues may require inference to complement what it is possible to know by deduction alone.  This has fostered a research approach that accepts partly subjective judgement supported by evidence, as a pragmatic alternative to the ideal of pure deduction. Evidence-based judgement about complex situations is a part of how the law works in practice (e.g. in court judgements), and this spills over into law and policy research.

The issues (and thus research questions) relevant to law and public policy have led to diverse methods. The next 2 readings consider some of these.

  • Langbroek, P., van den Bos, K., Thomas, M. S., Milo, M., & van Rossum, W. (2017). Methodology of legal research: Challenges and opportunities. Utrecht Law Review, 13(3), 1–8. https://doi.org/10.18352/ulr.411.
    This chapter surveys contemporary legal research that is documented in 10 chapters of a book about modern legal research, particularly multi-disciplinary approaches. It directs readers to the International Network to Promote the Rule of Law (INPROL) research memoranda to provide examples particularly of law reform research (https://www.inprol.org/inprol-publications?f%5B0%5D=field_publication_type%3A1449 ).
  • Tyler, T. R. (2017). Methodology in Legal Research. Utrecht Law Review, 13(3), 130–141. https://doi.org/10.18352/ulr.410.
    This chapter from the same book details evidence-based empirical methods, pointing to the value of theories from the social sciences, notably institutional analyses (e.g. reflecting institutional science and economics paradigms) and behavioural theories (reflecting psychology, sociology and other behavioural sciences).

For most law and policy non-doctrinal topics there is no “natural” limit to the interwoven issues might be investigated, particularly with wicked problems. Many questions of context, social dynamics, culture, economics and other matters are often found to be involved when the researcher begins to ‘unpack’ the research question. No law or policy researcher will have the full platform of resources to carry out exhaustive studies of all possible dimensions. Researchers cope with this in many ways: they might focus on only some of the issues; limit the methods that they use; or extrapolate from the limited data; or use personal judgement/opinion when the evidence is incomplete or inconclusive. These can be sound pragmatic choices to limit the investigation to what is feasible, but each compromise introduces risk to the reliability of the findings. Pragmatic compromises should be implemented with integrity, and transparently justified. Unfortunately is not usual for law and policy researchers to explain their pragmatic choices of method, or state the limits to their research that result.