Shortcomings in a Study Design
You are asked to identify three shortcomings in the design of a study and to propose
Begin by reading the following account of a fictitious study.
A researcher was interested in whether the time of day affected how people reacted to
unexpected noise. To explore this the researcher set up an MP3 player behind a tree in
a local park, so that it was hidden from view. Each time a person walked past the tree,
the MP3 player automatically played the sound of a man clearing his throat. Over a
period of 24 hours the researcher, or an assistant, sat on a nearby bench where they
could observe and record the behaviour of people passing by the tree.
The researcher found that the time of the day did affect people’s reaction to the noise:
the greatest reactions occurred in the early hours of the morning. Also, women showed
greater reactions than men.
From the list below, identify three shortcomings in the design of this study (these concepts
are explained in Online Activity 11.3: Preparing for TMA 02 (Part 2):
1 the participants used
2 acquiescence response bias
3 whether the method is appropriate
4 ethical issues
5 confounding variables
6 ability to generalise results from laboratory to everyday situations
7 closed questions
8 subjectivity of researcher/confirmatory bias
9 reactivity – influencing what happens
10 genuineness of response.
Write a paragraph saying what the three shortcomings are, why they are an issue for the
above study, and propose a solution for each one. Write no more than 250 words in total.
Shortcomings in a Study Design
The choice of study designs is usually the key to conducting a reliable and accurate research. The design encompasses the entire aspect of, participants’ selection, methodology, data analysis and presentation, and interpretation. In the hypothetical study presented, there are three significant shortcomings that are easily identifiable including the participants used, subjectivity, and ethical values.
From the short description of the study, the participants used do not give a reliable data and validity of response received. Arguably, there are several variables that were not considered while choosing on the participants; in fact, the study did not choose any participants at all. It is imperative that while designing a study, the participants used must meet certain criteria, since the reactions of the participants could be subject to various personal factors such as health (B.A. T.S.C., 1966; Wunderlich, & Kalsbeek, 1999). In order to solve this, it is important that participants are screened, and a defined method of selection be used, such as random procedure (YIN, 2003).
The study seems to have a confirmatory bias or some level of subjectivity, since the researcher arguably primarily wanted to confirm his/her hypothesis or belief that the time of day actually does affect people’s reaction to unexpected noise. In order to eliminate the effects of confirmatory bias in a study, one has to adopt an objective methodological approach, with proper controls and significantly defined variables (Stanovich, West, Toplak, & August 2013).
Lastly, there is an obvious ethical breach in the study, as the participants have been engaged without their consent. The researcher ought to have sought an official consent, signed by each participant, before engaging them, since such act can lead to legal tussle (Parsons et al, 2015).
Bay Area Transportation Study Commission, 1966. Study design.
Parsons, K, McCormac, A, Pattinson, M, Butavicius, M, & Jerram, C 2015, ‘The design of phishing studies: Challenges for researchers’, Computers & Security
Stanovich, K. E.; West, R. F.; Toplak, M. E., 2013. “Myside Bias, Rational Thinking, and Intelligence”. Current Directions in Psychological Science 22 (4): 259–264.
Wunderlich, G. S., & Kalsbeek, W. D., 1999. Review of disability evaluation study design: third interim report. http://site.ebrary.com/id/10055553.
YIN, R. K., 2003. Case study research: design and methods. Thousand Oaks, Calif, Sage Publications.