CONTACT US

Assessment Insights

Online and Paper Testing: Are We Looking at the Differences Closely Enough?

[fa icon="calendar"] April, 2016 / by Chad Barrett

Chad Barrett

Recently Parcc, Inc. reported student performance differences on the Spring 2015 PARCC® assessments based on the mode of testing: online or paper. Benjamin Herold summarized the findings in two recent Education Week articles. He reported that the pattern of lower student performance for those taking the computer-based tests was most pronounced in English Language Arts and middle- and high-school mathematics (Herold, 2016a). Parcc, Inc. has not yet released research evidence or announced a plan to conduct research that may explain the mode differences. Herold reported that Parcc, Inc. is asking participating states to examine the differences and draw conclusions appropriate for their context.

Assessment researchers have struggled for over 20 years to identify and explain performance differences between students who took a test on paper and those that took the test online. In the PARCC case, as in other cases, the source of the differences is being attributed to students’ exposure to technology during instruction and to their familiarity with the testing platform. If students are unfamiliar with the technology or testing platform, then they may struggle to use that technology to express their answers and reasoning.

Although I agree that technology access may partially explain student performance differences, I am not convinced that it is a sufficient explanation. Research suggests that people interact with technology differently than they interact with paper. In particular, people read differently on screen than on paper. In educational assessment, if students interact with screens and paper differently, there may be implications for test developers and policy makers. In the remainder of this article, I describe the research findings related to the differences in reading on screen and on paper and point out potential implications for assessment.

Is Reading on a Screen Different?

In 2014, Maria Konnikova wrote an article for The New Yorker that identified three major differences in how people read on a screen when compared to paper.
  • Readers skim more often than read closely when reading on screen.
  • Readers become exhausted more quickly when reading on screen.
  • Readers have trouble putting events in a story in order because there are fewer visual cues on screen to help readers locate their place within a story.

Her third point indicates that readers may have poorer comprehension when reading on screen when compared to paper. If this is the case, test developers and policy makers need to design assessments with this factor in mind.

Naomi Barton, in her recent book Words Onscreen: The Fate of Reading in a Digital World, spent a chapter discussing two typical behaviors of people who read on screen that could impact student assessment results. First, she corroborates Konnikova’s conclusion that readers tend to skim web pages. According to Barton’s research, the median amount of time someone spends on a web page may be as little as 12 seconds. Given that duration, most readers can read, word-for-word, only about 20 to 30 percent of the words on that page.

Barton’s second conclusion is that someone reading on screen tends to stop before reading all the text on a lengthy web page. She used the phrase “too long; didn’t read” to describe this behavior. Readers seem predisposed to scan the first screen and try to understand the main idea of the text. They will not read, word-for-word, a web page with a lot of text. 

In the last chapter of Words Onscreen, Barton identified another behavior exhibited by those that read on a screen. Now that many reading devices are connected to the Internet, readers are often distracted from their reading by notifications, email, and unrelated content available on the device. Such distractions are costly to comprehension.

online_vs_paper_blog.png

Inconsistent Evidence

Despite these assertions, research into the differences between reading on screen and on paper is inconclusive. Konnikova noted that there is no longitudinal data to help us understand how reading habits are changing. Ferris Jabr, in a 2013 Scientific American article, identified the problem. “Before 1992 most studies concluded that people read slower, less accurately and less comprehensively on screens than on paper. Studies published since the early 1990s, however, have produced more inconsistent results: a slight majority has confirmed earlier conclusions, but almost as many have found few significant differences in reading speed or comprehension between paper and screens.”

Where Do We Go from Here?

I believe that a detailed exploration of student performance differences between online and paper tests is required. Are performance differences explained solely by how much access a student has to technology? Or, are the differences also explained by a student’s behavior when taking an online test when compared to a paper test? We need to conduct further research to better explore how students interact with online and paper tests. Few studies have examined these interactions closely.

One way to investigate these questions is to use a think-aloud study to explore how examinees take tests on screen and on paper. Think-aloud studies are frequently used to investigate how students engage in many educational tasks. During a think-aloud study, examinees talk about how they read and responded to the test questions. This strategy can help us examine the testing experiences of the same examinees in two different testing modes. 

If differences in test scores are explained by differences in testing mode, then we must be careful about the inferences we make from the test results. Perhaps, as these performance differences become better understood, researchers can provide clear guidance about how to accurately design tests to measure intended constructs. Then test developers and policy makers can evaluate their test design and administration decisions in order to align the construct being measured to the testing mode being used.

References

Baron, N. S. (2015). Words onscreen: The fate of reading in a digital world. United Kingdom: Oxford University Press 

Herold, B. (2016a). PARCC scores lower for students who took exams on computers. Education Week35(20), 1, 11. Retrieved 8 March 2016, from http://www.edweek.org/ew/articles/2016/02/03/parcc-scores-lower-on-computer.html 

Herold, B. (2016b). Comparing paper and computer testing: 7 key research studies. Education Week35(22), 8. Retrieved 8 March 2016, from http://www.edweek.org/ew/articles/2016/02/23/comparing-paper-and-computer-testing-7-key.html 

Jabr, F. (2013, April 11). The reading brain in the Digital Age: the science of paper versus screens. Retrieved 5 December 2014, from http://www.scientificamerican.com/article/reading-paper-screens/  

Konnikova, M. (2014, July 16). Being a better online reader. Retrieved 5 December 2014, from http://www.newyorker.com/science/maria-konnikova/being-a-better-online-reader 


 PARCC® is a registered mark of Parcc, Inc. Measured Progress is not owned by or affiliated in any fashion with Parcc, Inc., the Partnership for the Assessment of Readiness for College and Careers, nor any state of the Union. Neither Parcc, Inc., nor the Partnership for the Assessment of Readiness for College and Careers, nor any member state has endorsed Measured Progress products and services. 

Topics: Classroom Assessment, Formative Assessment, Accountability, Interim Assessments

Chad Barrett

Written by Chad Barrett

Chad Barrett is a Measurement Services Senior Advisor at Measured Progress, working with content and psychometrics teams to develop test designs and contribute to other large-scale assessment solutions. Chad has more than 15 years of educational assessment experience, particularly in content development and program design.