Intended and unintended test constructs in a multiple-mini admission interview. A validity study

Lotte Dyhrberg O'Neill, Eva Lykkegaard, Kulamakan Kulasageram

Research output: Contribution to journalJournal articleResearchpeer-review

Abstract

Construction errors in multiple-choice items are quite prevalent and constitute threats to test validity of multiple-choice tests. Currently very little research on the usefulness of systematic item screening by local review committees before test administrationseem to exist. The aim of this study was therefore to examine validity and feasibility aspects of review committee screening for item flaws. We examined the reliability of item reviewers’ independent judgments of the presence/absence of item flaws with a generalizability study design and found only moderate reliability using five reviewers. Statistical analyses of actual exam scores could be a more efficient way of identifying flaws and improving average item discrimination of tests in local contexts. The question of validity of human judgments of item flaws is important - not just for sufficiently sound quality assurance procedures of tests in local test contexts - but also for the global re-search on item flaws.
Original languageEnglish
JournalDansk Universitetspaedagogisk Tidsskrift
Volume14
Issue number26
Pages (from-to)66-81
ISSN1901-5089
Publication statusPublished - 1. Apr 2019

Fingerprint

interview
quality assurance
discrimination
threat

Keywords

  • Multiple Mini Interview
  • Admission interviews
  • Validity
  • Internal structure
  • Factor analysis

Cite this

@article{f10d36659db64a579f4b04fae7cfed22,
title = "Intended and unintended test constructs in a multiple-mini admission interview. A validity study",
abstract = "Construction errors in multiple-choice items are quite prevalent and constitute threats to test validity of multiple-choice tests. Currently very little research on the usefulness of systematic item screening by local review committees before test administrationseem to exist. The aim of this study was therefore to examine validity and feasibility aspects of review committee screening for item flaws. We examined the reliability of item reviewers’ independent judgments of the presence/absence of item flaws with a generalizability study design and found only moderate reliability using five reviewers. Statistical analyses of actual exam scores could be a more efficient way of identifying flaws and improving average item discrimination of tests in local contexts. The question of validity of human judgments of item flaws is important - not just for sufficiently sound quality assurance procedures of tests in local test contexts - but also for the global re-search on item flaws.",
keywords = "Admission Tests, Multiple-Mini Interviews, Validity, Medical Education, Multiple Mini Interview, Admission interviews, Validity, Internal structure, Factor analysis",
author = "{Dyhrberg O'Neill}, Lotte and Eva Lykkegaard and Kulamakan Kulasageram",
year = "2019",
month = "4",
day = "1",
language = "English",
volume = "14",
pages = "66--81",
journal = "Dansk Universitetspaedagogisk Tidsskrift",
issn = "1901-5089",
publisher = "Syddansk Universitetsforlag",
number = "26",

}

Intended and unintended test constructs in a multiple-mini admission interview. A validity study. / Dyhrberg O'Neill, Lotte; Lykkegaard, Eva ; Kulasageram, Kulamakan .

In: Dansk Universitetspaedagogisk Tidsskrift, Vol. 14, No. 26, 01.04.2019, p. 66-81.

Research output: Contribution to journalJournal articleResearchpeer-review

TY - JOUR

T1 - Intended and unintended test constructs in a multiple-mini admission interview. A validity study

AU - Dyhrberg O'Neill, Lotte

AU - Lykkegaard, Eva

AU - Kulasageram, Kulamakan

PY - 2019/4/1

Y1 - 2019/4/1

N2 - Construction errors in multiple-choice items are quite prevalent and constitute threats to test validity of multiple-choice tests. Currently very little research on the usefulness of systematic item screening by local review committees before test administrationseem to exist. The aim of this study was therefore to examine validity and feasibility aspects of review committee screening for item flaws. We examined the reliability of item reviewers’ independent judgments of the presence/absence of item flaws with a generalizability study design and found only moderate reliability using five reviewers. Statistical analyses of actual exam scores could be a more efficient way of identifying flaws and improving average item discrimination of tests in local contexts. The question of validity of human judgments of item flaws is important - not just for sufficiently sound quality assurance procedures of tests in local test contexts - but also for the global re-search on item flaws.

AB - Construction errors in multiple-choice items are quite prevalent and constitute threats to test validity of multiple-choice tests. Currently very little research on the usefulness of systematic item screening by local review committees before test administrationseem to exist. The aim of this study was therefore to examine validity and feasibility aspects of review committee screening for item flaws. We examined the reliability of item reviewers’ independent judgments of the presence/absence of item flaws with a generalizability study design and found only moderate reliability using five reviewers. Statistical analyses of actual exam scores could be a more efficient way of identifying flaws and improving average item discrimination of tests in local contexts. The question of validity of human judgments of item flaws is important - not just for sufficiently sound quality assurance procedures of tests in local test contexts - but also for the global re-search on item flaws.

KW - Admission Tests

KW - Multiple-Mini Interviews

KW - Validity

KW - Medical Education

KW - Multiple Mini Interview

KW - Admission interviews

KW - Validity

KW - Internal structure

KW - Factor analysis

M3 - Journal article

VL - 14

SP - 66

EP - 81

JO - Dansk Universitetspaedagogisk Tidsskrift

JF - Dansk Universitetspaedagogisk Tidsskrift

SN - 1901-5089

IS - 26

ER -