Research Article: Effects of test item disclosure on medical licensing examination

Date Published: July 31, 2017

Publisher: Springer Netherlands

Author(s): Eunbae B. Yang, Myung Ae Lee, Yoon Soo Park.

http://doi.org/10.1007/s10459-017-9788-8

Abstract

In 2012, the National Health Personnel Licensing Examination Board of Korea decided to publicly disclose all test items and answers to satisfy the test takers’ right to know and enhance the transparency of tests administered by the government. This study investigated the effects of item disclosure on the medical licensing examination (MLE), examining test taker performance, psychometric characteristics, and factors affecting pass rates. This paper analyzed examinee performance data (n = 20,455) from 41 medical schools who took the MLE before (2009–2011) and after (2012–2014) the item disclosure policy (5548 total items). Changes in passing rates, performance of examinee, difficulty and reliability of the test, and factors affecting pass rate of the medical licensing examination before and after item disclosure were analyzed. In order to identify changes caused by item disclosure in the effects of student and school variables on the passing rate of MLE, Binary Logistic Hierarchical Linear Model was used. There was no significant change in pass rates before and after item disclosure. There was a modest increase in the proportion of test takers in the high-scoring group, following item disclosure. Degree completion status, gender, age of applicants and school mean were significant factors affecting pass rates, regardless of item disclosure. There was no difference between passing rates before and after item disclosure with respect to student- and school-level variables. Despite potential concerns for changes in test and examinee characteristics, empirical findings indicate that there was no significant difference caused by implementing item disclosure.

Partial Text

Medical licensing examination is a selection process that identifies qualified candidates and grants them license to practice medicine. In this paper, we refer to the general term medical licensing examination (MLE) as a high-stakes examination, administered at a national level for medical licensing purposes. The agency administering MLEs typically maintains item banking systems, enforces test security measures, and has policies against item disclosure—publicly releasing all exam questions, answer keys, and examinee performance data following exam administration. Maintaining the security of test items and scores is paramount if tests are to provide useful inferences on examinee performance (Cohen and Wollack 2006). However, publicly disclosing item-level information has historically faced challenges on grounds of fairness and validity (Bersoff 1981; Camilli 2006; Park and Yang 2015). Proponents of item disclosure emphasize the need for test takers to know the actual test items and answer keys in addition to other relevant performance data, based on the principle of fairness and public accountability. They assert that testing is a type of social policy, which deserves thorough public scrutiny and discussion. On the other hand, opponents state that the security of testing already support fairness, and thus, disclosure of test items may undermine the validity of the test; for example, exposure of reused items may inflate the equated test scores (Gilmer 1989). Opponents also emphasize the burden in costs that may be accompanied by each new cycle of test development if full item disclosure were to be implemented.

Following the implementation of the item disclosure policy, the MLE in South Korea began releasing all test items, answers, and performance data to the public. This study investigated whether the disclosure of test items impacted passing rate, average scores before and after the item disclosure, and factors that influence pass rates. Overall, the results did not show meaningful changes in the passing rate before and after item disclosure. While, passing rates in the high-scoring group and a specific score section rose slightly, the overall change in pass rates was modest at 1.6%. As for the effects of variables affecting the passing rate of the MLE, regardless of the item disclosure policy, degree completion status, gender, and age of students were significant predictors, and the school mean had statistical significance in terms of school-level variable. As such, this research was unable to identify any meaningful change brought about by item disclosure in MLE. These findings are consistent with previous research by Stricker (1984), which concluded that following test item disclosure of Scholastic Aptitude Test of the United States, there were no significant changes or effects on the applicants’ performance. Likewise, Wood (2009) also found no change in test characteristics and that repeat examinees do not appear to be advantaged by encountering reused question. It is worth noting that prior computer simulation studies by Gilmer (1989) in the measurement literature revealed that item disclosure could possibly lead to the passing unqualified examinees and bring about an approximately 10% increase of passing rate as compared with the time of non-disclosure. However, this study provides empirical findings based on actual data motivated by a national policy change, which has allowed a natural quasi experiment to examine the impact of item disclosure in large-scale high-stakes testing conditions.

 

Source:

http://doi.org/10.1007/s10459-017-9788-8

 

Leave a Reply

Your email address will not be published.