Assessment Insights

Mutual Admiration Society: Measured Progress and our Psychometrics Interns

[fa icon="calendar"] August, 2018 / by Press Room

Press Room

Annually since 2005, Measured Progress has provided summer internships for graduate students pursuing doctorates in psychometrics or related fields. This year we welcomed three outstanding students, each already immersed in research to advance the practice of psychometrics and our understanding of testing.

Meet the interns

Jane Hong, from South Korea, is a doctoral student in Research, Educational Measurement, and Psychometrics at UMass Amherst. At the UMass Center for Educational Assessment, she began working with Measured Progress on equating studies for the Massachusetts Comprehensive Assessment System. Her research centers on “item parameter drift”—the phenomenon in which anchor items (those items used repeatedly over years to statistically “anchor” the results from different tests and different cohorts of students) lose statistical stability over time. The reliability and accuracy of the score scale over multiple years depends on the anchor items. Jane is working on a new way to identify items that might be susceptible to drift, and the causes of that drift.

Scarlett Bian, originally from Hebei, China, is working toward her doctorate in Educational Statistics and Measurement at Rutgers University. She expects to receive her degree in December 2019. Her research focuses on diagnostic assessments. She’s working on creating models that give more information to teachers than the typical binary results that indicate whether a student has or doesn’t have a particular skill. Other models give more nuanced information—about both the items and the students’ understanding. Scarlett is working on a method that gives very specific information about how well an item “discriminates” students’ abilities.

Luping Niu, also from China, is now a resident of Austin Texas, where he attends the University of Texas. In his doctoral program on Quantitative Methods, his field of study is Item Response Theory, Response Time Modeling, and Multilevel Modeling. His expected degree date is May 2020. In Luping’s study of response-time models, he’s examining both the average response times of items and the speed with which individuals complete a response. This data can help answer questions about cheating or low motivation. For example, if a complex question is answered quickly and correctly, that might indicate cheating or other unusual behaviors. If it’s answered quickly and incorrectly, that might point to low motivation. Luping is also investigating the effect of item position on response time.

Psychometrics Interns Interns 2018

Learning the business

All three interns remarked that they came to Measured Progress based on its reputation with their professors. They were told that they’d spend time working directly with our psychometricians—which wouldn’t be the case in other internship programs. According to the students, the predictions turned out to be correct. They appreciated the amount of time staff members spent with them, discussing the interns’ research as well as the research taking place at Measured Progress. It also seems that psychometricians at all their universities know Louis Roussos, senior psychometrician and research scientist at Measured Progress, and encouraged their students to work with him. Louis also put Jane in touch with Ben Shear, a researcher in her field. Now a professor at University of Colorado, Ben was a 2014 summer intern himself.

While the program came highly recommended to Jane, Scarlett, and Luping, they were each pleasantly surprised by the extent to which they were exposed to operational psychometrics and the business of assessments. A full schedule of “chats” with directors and managers plus lectures from outside experts gave them a broad overview of our work. They also met with the interns from the neighboring Center for Assessment. Jane noted that she had never heard of Dover, NH, and was surprised to arrive and find that two major testing companies are located here. “Dover should be famous!” said Luping—at least for assessment.

The “berry best” of New England

This was the first time Jane, Scarlett, and Luping had been in this part of the country—and they’re big fans. A berry-picking expedition (to Berry Best Farm) with Louis was a highlight. “This was the first time I ever picked fruit!” said Luping. He also especially enjoyed the Independence Day fireworks in Rochester. Jane enjoyed hiking beside the ocean at Acadia National Park in Maine, and Portland’s restaurant scene. “I was surprised that we were so close to Maine,” she said. Scarlett named berry picking and exploring Portland among her favorite New England experiences.

Professional impact

On the last day of their program here, each of the interns gave a presentation on their research. The presentations will form the basis of proposals to present sessions at the annual conference of the National Council on Measurement in Education. Watch the News and Events page in spring 2019 for announcements of all Measured Progress presentations at the conference.

The internship program is a boost for our own psychometricians as well as for the interns. Psychometricians/research scientists Xiao Luo and Xi Wang joined Louis as mentors to Jane, Scarlett, and Luping. The three professionals found it exciting to engage with the students and hear their ideas and questions. “The research projects helped answer questions that I’ve been curious about for a long time,” commented Xiao. Xi noted that she learned from the projects, and got new research ideas as a result. Louis called the internship program “the ideal outlet” for conducting research and disseminating the results. “I get to work with the best students in the country,” he added. Further, he noted that because the projects are directly related to psychometric issues in our work, “the research results are always of value to Measured Progress.”

Best of luck to Scarlett, Luping, and Jane! We can’t wait to see what they accomplish.

Topics: Assessment Literacy, Formative Assessment, Accountability, Connecting Teaching and Learning

Press Room

Written by Press Room

Our Press Room chooses topics of interest from activities around the company, to provide a closer look at ways to improve learning and instruction, and to help demystify assessment issues.