TY - JOUR
T1 - Do performance indicators predict regulator ratings of healthcare providers?
T2 - Cross-sectional study of acute hospitals in England.
AU - Allen, T
AU - Walshe, K
AU - Proudlove, N
AU - Sutton, M
PY - 2020/4
Y1 - 2020/4
N2 - Objective: To determine whether a large set of care performance indicators ('Intelligent Monitoring (IM)') can be used to predict the Care Quality Commission's (CQC) acute hospital trust provider ratings. Design: The IM dataset and first-inspection ratings were used to build linear and ordered logistic regression models for the whole dataset (all trusts). This was repeated for subsets of the trusts, with these models then applied to predict the inspection ratings of the remaining trusts. Setting: The United Kingdom Department of Health and Social Care's Care Quality Commission is the regulator for all health and social care services in England. We consider their first-inspection cycle of acute hospital trusts (2013-2016). Participants: All 156 English NHS acute hospital trusts. Intervention(s): None. Main Outcome Measure(s): Percentage of correct predictions and weighted kappa. Results: Only 24% of the predicted overall ratings for the test sample were correct and the weighted kappa of 0.01 indicates very poor agreement between predicted and actual ratings. This lack of predictive power is also found for each of the rating domains. Conclusion: While hospital inspections draw on a much wider set of information, the poor power of performance indicators to predict subsequent inspection ratings may call into question the validity of indicators, ratings or both. We conclude that a number of changes to the way performance indicators are collected and used could improve their predictive value, and suggest that assessing predictive power should be undertaken prospectively when the sets of indicators are being designed and selected by regulators.
AB - Objective: To determine whether a large set of care performance indicators ('Intelligent Monitoring (IM)') can be used to predict the Care Quality Commission's (CQC) acute hospital trust provider ratings. Design: The IM dataset and first-inspection ratings were used to build linear and ordered logistic regression models for the whole dataset (all trusts). This was repeated for subsets of the trusts, with these models then applied to predict the inspection ratings of the remaining trusts. Setting: The United Kingdom Department of Health and Social Care's Care Quality Commission is the regulator for all health and social care services in England. We consider their first-inspection cycle of acute hospital trusts (2013-2016). Participants: All 156 English NHS acute hospital trusts. Intervention(s): None. Main Outcome Measure(s): Percentage of correct predictions and weighted kappa. Results: Only 24% of the predicted overall ratings for the test sample were correct and the weighted kappa of 0.01 indicates very poor agreement between predicted and actual ratings. This lack of predictive power is also found for each of the rating domains. Conclusion: While hospital inspections draw on a much wider set of information, the poor power of performance indicators to predict subsequent inspection ratings may call into question the validity of indicators, ratings or both. We conclude that a number of changes to the way performance indicators are collected and used could improve their predictive value, and suggest that assessing predictive power should be undertaken prospectively when the sets of indicators are being designed and selected by regulators.
KW - British
KW - Data analysis
KW - Government regulation
KW - Hospital administration
KW - National health service
KW - Patient safety
KW - Standards
KW - Statistical
KW - Cross-Sectional Studies
KW - Humans
KW - Quality of Health Care/organization & administration
KW - England
KW - Hospitals, State/organization & administration
KW - Quality Indicators, Health Care/statistics & numerical data
KW - State Medicine/standards
U2 - 10.1093/intqhc/mzz101
DO - 10.1093/intqhc/mzz101
M3 - Journal article
C2 - 31725874
SN - 1353-4505
VL - 32
SP - 113
EP - 119
JO - International Journal for Quality in Health Care
JF - International Journal for Quality in Health Care
IS - 2
ER -