Evaluating sources
Learning objectives
At the end of this module, you will be able to:
- distinguish between quantitative and qualitative research
- recognise the implications of different research approaches
- locate material which will assist you in critically assessing published research.
Types of research
Most research can be classified as either as quantitative or qualitative. Quantitative research deals with what can be measured in numerical form ("quantified"). Qualitative relates to the unquantifiable, those personal, dynamic aspects of the world that cannot be reduced to numbers. Neither form of research is inherently superior to the other. Both have their advantages and drawbacks. It is esssential, however, that you become familiar with the differences between quantitative and qualitative approaches, as these are evaluated in different ways.
Be aware that not all research falls squarely into one or the other categories. Mixed-mode research, which incorprates both quantitative and qualitative methods, is becoming incereasingly common.
What is quantitative research?
Quantitative research is research which seeks to explain events through the collection and analysis of information in numerical form.
There are many different types of quantitative research. Quantitative research methods include:
- laboratory experiments
- field studies
- questionnaires and surveys
- observations of natural phenomena
- data-mining.
Definitions of what constitutes quantitative research differ between disciplines. As a postgraduate student, you should become familiar with the theoretical perspectives that underpin your descipline.
The scientific method
At the heart of most quantitative research is the use of what is termed the scientific method. Although there are different accounts of the scientific method, most physical scientists at least would argue that this method involves:
- the intial observation of an event or phenomenon
- the formulation of an hypothesis to explain the initial observation
- use of the hypothesis to make predictions regarding the outcome of additional observations or experiments
- confirmation or falsification of the hypothesis through observation and experiment
- replication of these results by independent, objective observers.
The advantages of quantitative research approaches in terms of the scientific method are obvious. Quantitative approaches facilitate
- objective measurement of phenomena
- correction of error through statistical tests of validity
- the use of controls to exclude extraneous factor
- replication of the results by other investigators.
Although most quantitative researchers would affirm their adherence to the scientific method, there are differences between disciplines. Researchers working in the pure sciences often understand the scientific method in a way that differs from the understandings common among, say, sociologists or psychologists.
Experimenter effects
Scientific fraud is not unknown — it may in fact be more common than most of us assume. However, most researchers are more likely to attribute anomalous results to accidential error or unconscious bias (expectancy effects). Experimenter effects occur when the researcher's attitude effects the outcome of the experiment.
 |
Mendel and his peas Monwhea Jeng, A selected history of expectation bias in physics, arXiv:physics/0508199v1, 27 August 2005. Awareness of experimenter effects is particularly high among researchers in the social sciences, psychology and medicine. However, these effects have also been at work in the hard sciences. They have been detected in such important experiments, as Gregor Mendel's pioneering work on genetics. Read this unpublished article on experimenter effects. Do you agree with the author's conclusions? |
Researchers usually attempt to control for expectancy effects (which can influence the response of participants as well as the observations of experimenters) using procedures such as double-blind trials. In such experiments, neither the experimenters nor the subjects of the experiment are aware of the research aims. The idea of a double-blind trial comes from medical research. In such a trial, patients are divided into two identical or near-identical groups. One group is given a placebo, the other the real treatment. Neither the subjects nor the researchers directly involved in the experiment know which group has received the placebo.
 |
The Economist and the BMJ are alarmed Sloppy stats shame science: Scientific publishing, The Economist, 5 June 2004, 71. Signs of the times: Medical statistics, The Economist, 24 February 2007, 82. Travelling but never arriving: reflections of a retiring editor, BMJ, 329, 31 July 2004, 242-244. Read this these articles. Do you think that they are unnecessarily alarmist? Scepticism regarding the quality of quantitative research has occasionally reached alarming levels. Professor Richard Smith has gone so far as to assert that only 5% of scientific papers in medical journals "came up to scratch". According to Professor Smith, "In most journals less than 1%". Do you think that the situation is as serious as he believes? |
Evaluating quantitative research
Unfortunately, the use of quantitative methods in research do not guarantee objectivity. Quite apart from the risk of experimental error, the overuse and misuse of statistics in articles published in scholarly journals has long been a subject of controvesy.
Many of the articles published in scientific and medical journals contain errors in terms of statistical techniques or in the reporting of the statistical methods employed. Although such errors do not automatically invalidate the findings of these articles, they make it difficult to accept their results in isolation.
|
Nature Medicine agrees Editorial: Statistically significant, Nature Medicine, 11 (1), January 2005, 1. Read this recent editorial from Nature Medicine. Compare the editor's comments with the statement in The Economist that: Far too many scientists have only a shaky grasp of the statistical techniques they are using. They employ them as an amateur chef employs a cook book, believing the recipes will work without understanding why (The Economist, 5 June 2004, 71) Does this editorial tend to confirm the worst fears of journalists from The Economist? |
|
Common pitfalls in statistics Alexander M Strasak et al (2007) "Statistical errors in medical research — a review of common pitfalls", Swiss Medical Weekly, 137, 44-49. The authors of this review article provide a thorough survey of many common pitfalls in quantitative research. Don't be put off by the fact that the article title refers to medical research. The points made in the article apply to published research in many disciplines. |
The Null Hypothesis Statistical Test
If you are still not convinced of the extent of the problem, consider the case of the null hypothesis statistical test (NHST). This test is the primary method of data analysis in the social sciences. It is widely employed by researchers in economics, education, information technology, politcal science, psychology, sociology and many other related disciplines.
The manner in which the NHST is used by many social scientists has been subject to scathing criticism at least since the 1950s. Whole books have been written casting doubt on the way in which researchers typically misapply the NHST. The problems with misunderstandings of NHST are so common that some statisticians have suggested that this technique should be abandoned completely in social science research.
You don't need to agree with the critics of the NHST, but you need to be aware of the issues. Ignorance of the points raised by critics of the NHST will limit your ability to effectively evaluate quantitative research.
If your research involves quantitative analysis, ensure that you are thoroughly familiar with the statistical principles involved. If you have forgotten exactly what Student's t-test involves or the differences between single-tailed or two-tailed distributions, you might like to review the field.
|
The null hypothesis: don't ask and don't tell Raymond S Nickerson (2000) Null hypothesis significance testing: A review of an old and continuing controversy, Psychological Methods, 5(2), 241-301. |
There are a number of excellent books in the Library on quantitative research methods. Many of these will provide you with useful check-lists that you can apply in the evaluation of quantitative research.
| Further reading | |
 |
Geoffrey R Marczyk and David DeMatteo (2005) Essentials of Research Design and Methodology, Wiley & Sons. Click here to view this book. |
 |
Seymour Sudman (2004) Asking Questions: The Definitive Guide to Questionnaire Design, Wiley & Sons, Newark. Click here to view this book. |
What is qualitative research?
Whereas quantitative research deals with numbers, qualitative research is concerned with words; with rich description and in-depth enquiry. In many disciplines, qualitative research approaches predominate. These methods allow researchers to capture the meaning of events in a way that quantitative approaches cannot.
There are many definitions of qualitative research. Some investigators see the essence of qualitative research as being the systematic, first-hand observation of phenomena in the real world. Scholars in other disciplines, such as history, would argue that qualitative research is not limited to first-hand observations.
The methods employed in qualitative research include:
- archival research
- literary analysis
- interviews
- use of questionnaires
- participant observation
- case studies.
Different methodologies are tied to different philosophies and theoretical perspectives. Although most quantitative researchers are positivists (ie they assume that research is a search for "truth"), there are qualitative investigators who take a non-positivist approach, defending phenomenological or interpretative ("hermeneutic") approaches. Understanding these different reseach philosophies is essential if you are to engage effectively with qualitative research.
Evaluating qualitative research
The sort of questions you might like to ask include:
- What was the purpose of the study?
- What, if any, theoretical framework has been employed?
- Would another theoretical perspective have been more effective or appropriate?
- How has the researcher's theoretical perspective informed their study?
- What methodology did the researcher employ?
- What, if any, reasons were given for this choice?
- Were there any frequently used methodologies which might have been used and were not?
- Has the researcher justified this decision?
- Has the researcher thoroughly evaluated the relevant literature?
- Has he or she referred to research pointing to different conclusions?
- Were the conclusions reached in the study justified?
- Are there alternative explanations for the data?
- Are there logical weaknesses in the argumentation
- Does the study make a significant contribution in terms of knowledge or theoretical insight?
- How does this study relate to my topic?
|
Issues in qualitative research If you are interested in methodological issues relating to qualitative research, there is a Web page on the topic at the University of Alberta in Canada. |
The University Library also has a number of books on conducting or criticising qualitative research.
|
Qualitative Report Another source of information on qualitative research methodologies is the Qualitative Report. This is an open-access, peer-reviewed journal. The journal site also has links to Web resources on qualitative research. |
Summary
This module dealt with the following
- the distinction between quantitative and qualitative research
- the implications of different research approaches
- finding information that will assist you in assessing published research.