A Study of Interrater Reliability in an In-house Oral Proficiency Test

Abstract

Many oral interview tests such as the Foreign Service Institute (FSI), the Speaking Proficiency English Assessment Kit (SPEAK), the Oral Interview (OI) and the Oral Proficiency Interview (OPI) are used in the U.S.A. However very few standardized tests of oral skills are available in Japan. Most oral proficiency tests used are in-house tests designed by individual institutes and the validity and reliability of those tests are somewhat questionable. This study, attempts to measure the validity and reliability of an in-house oral proficiency test by focusing on the interrater reliability. It was found that there were multidimensional causes for the judges' subjectivity in their evaluation process. The findings in this study are closely linked to the variables vital to solving the problems with the interrater reliability of the test and also enabling us to find out which attributes are important to measure in our students' oral performance.