Understanding Test Response Process: A Study Of Test-Taking Strategies In A Reading Comprehension Test

UNDERSTANDING TEST RESPONSE PROCESS: A STUDY OF TEST-TAKING STRATEGIES IN A READING COMPREHENSION TEST

— Michelle Chen, January 30, 2018

When test developers design and assemble a test, they purposefully develop multiple tasks to measure the same general skill, such as English reading comprehension. They also purposefully develop tasks that test different specific skills (such as ‘making an inference’) and which are at different levels of difficulty. For example, reading an email about a daily scenario is often easier than reading an academic article on abstract and complex ideas. Similarly, identifying the main point of an email is often easier than inferring a writer’s stance in an academic article. Tasks are purposefully selected to ensure that the test covers the skills needed in reading and test developers’ selections are based on expectations that they have about the way in which test takers will engage with the tasks.

‘Response processes’ is a term often used when discussing how test takers engage with test tasks. Defined broadly, response processes can include many cognitive and non-cognitive reactions and mechanisms which happen during the process of test taking. For instance, when answering a reading comprehension item, a test taker first needs to recognize the words and then construct their meaning. Test-taking strategies can be seen as part of a test taker’s response process. In a reading comprehension test, these strategies can generally be grouped into three different types: the comprehending-meaning strategy, the score-maximizing strategy, and the test-wiseness strategy. Table 1 offers a definition and an example of each category.

Type Definition Example
Comprehending-meaning strategy Strategies that require test takers to use their language knowledge and skills to understand the reading material. To answer the test questions, I needed to understand the main ideas of the whole passage.
Score-maximizing strategy Strategies that rely on test takers’ partial understanding or their use of other clues in the test. To answer the test questions, I used clues in the other questions to guess the answer.
Test-wiseness strategy Strategies that test takers use when they focus on getting an answer without engaging in the expected processes of reading comprehension. To answer the test questions, I selected an option that had an important word from the passage.
Table 1. Three types of test-taking strategies in a reading comprehension test

We may hypothesize that an individual test taker has some preferred test-taking strategies when responding to reading comprehension tasks, and these strategies do not drastically change across different tasks. We may also hypothesize that test takers use different test taking strategies when responding to different tasks. One reason for them to use different strategies might be the different difficulty levels of the tasks.

There has been some research into test-taking strategies, but no-one has looked specifically at whether and how test takers will change their strategies in reaction to tasks that are at different levels of difficulty. This was the focus of a paper that Dr. Amery Wu, Dr. Jake Stone and I published recently.

As response processes (such as test-taking strategies) cannot be directly observed, researchers often rely on test takers’ self-report to study these processes. A commonly used method is the think-aloud protocol in which participants talk aloud their thought processes as they complete a task. Using this method, researchers can obtain detailed information regarding test takers’ cognitive processes, strategy use, metacognition, and other related factors involved in responding to test questions. It has been shown to be an effective way to assess higher-level thought processes.

Another commonly used method is the questionnaire. Test takers can reflect on the strategies they use by filling out a strategy questionnaire immediately after they finish each reading task. The test takers can be given the same strategy questionnaire each time, making it feasible to collect individual test takers’ responses on the same set of strategy questions repeatedly. This type of repeated measure facilitates studies of changes in test-taking strategies between tasks of different difficulty levels. Questionnaires also have a number of other advantages over think-aloud protocols. They are very time and resource efficient, enabling the collection of responses from a relatively large group of participants in a relatively short period (often as little as two or three hours). Larger sample sizes enable the application of sophisticated statistical methods to further explore patterns in the data. Most importantly, questionnaires can be easily incorporated into the testing process with minimal interruptions to the standard flow of the examination. The more an experimental setting mimics the real testing context, the more valid the conclusions are when generalizing the findings from the experimental setting to what might happen in a real test setting (so-called external validity).

The research data were obtained during a pilot study of the CELPIP-General test. Test takers responded to a test-taking strategy questionnaire immediately after they finished each reading task. The test-taking strategy survey was developed to capture the three major test-taking strategy types described in Table 1: the comprehending-meaning strategy, the score-maximizing strategy, and the test-wiseness strategy. The findings showed that test takers exhibited three different patterns in using the three types of strategies. Figure 1 shows the average reported usage of each strategy type by each of these three groups. The three types of strategies are color coded: Blue for the comprehending-meaning strategy, orange for the score-maximizing strategy, and grey for the test-wiseness strategy. The first group, represented by the first three bars on the left, was labelled the “Using-All” class. Test takers in this group used a large number of all three types of test-taking strategies. The second group, represented by the three bars in the middle, comprised test takers who made substantial use of the comprehending-meaning strategy and moderate use of the score-maximizing strategy. They were named the “Moderate-Score-Maximizing” group. The final group of test takers, represented by the three bars on the right, relied almost entirely on the comprehending-meaning strategy, and they were named the “Only-Comprehending-Meaning” group.

UNDERSTANDING TEST RESPONSE PROCESS: A STUDY OF TEST-TAKING STRATEGIES IN A READING COMPREHENSION TESTFigure 1. Average probability of each strategy type per group

We were also interested in whether test takers changed their strategies when they encountered more difficult tasks. To explore this question, we analysed the test takers’ self-reported strategy use on two reading comprehension tasks, an easy one and a more difficult one. Our results showed that many test takers changed their strategies when the task was more difficult, and the changes were dependent on both their reading ability and the task difficulty. The general trend of the results showed that the test takers tended to use only comprehending-meaning strategies when the task was easy in relation to their ability level. When task difficulty increased, higher ability test takers were able to continue using comprehending-meaning strategies but less able test takers showed a tendency to use additional types of test-taking strategies such as score-maximizing and test-wiseness strategies. Test takers who persisted in using only comprehending-meaning strategies in both tasks turned out to have the highest scores on the more difficult task.

The findings of this study confirm that test takers use different types of test-taking strategies and to different degrees. More interestingly, the study also shows that as task difficulty increases, test takers may start using more types of strategies. Test takers’ reading proficiency influences the changes in their strategy use when the task gets more difficult. In turn, the different transition patterns of strategy use are associated with test takers’ performance on the task. We can conclude from these results that strategies such as test-wiseness may seem useful but they are often a last resort. Test takers may turn to such strategies when the test questions are too difficult. Crucially, test takers who use strategies which are unrelated to comprehension do not perform better on the test than those who rely mostly on comprehending-meaning strategies. A reasonable inference we can draw from the study is that rather than resorting to test-wiseness strategies, a test taker who studies to improve their language proficiency has a better chance of improved test performance.

References

Wu, A. D., Chen, M. Y., & Stone, J. E. (2018). Investigating how test-takers change their strategies to handle difficulty in taking a reading comprehension test: Implications for score validation. International Journal of Testing18(3), 253-275.

Michelle Chen is a Psychometrician at Paragon Testing Enterprises.

Michelle Chen is a Psychometrician at Paragon Testing Enterprises.