Nonparametric Limits of Agreement for Small to Moderate Sample Sizes: A Simulation Study

Maria Elisabeth Frey*, Hans Christian Petersen*, Oke Gerke*

*Corresponding author for this work

Research output: Contribution to journalJournal articleResearchpeer-review

92 Downloads (Pure)


The assessment of agreement in method comparison and observer variability analysis of quantitative measurements is usually done by the Bland--Altman Limits of Agreement, where~the paired differences are implicitly assumed to follow a normal distribution. Whenever this assumption does not hold, the 2.5% and 97.5% percentiles are obtained by quantile estimation. In the literature, empirical quantiles have been used for this purpose. In this simulation study, we applied both sample, subsampling, and kernel quantile estimators, as well as other methods for quantile estimation to sample sizes between 30 and 150 and different distributions of the paired differences. The~performance of 15 estimators in generating prediction intervals was measured by their respective coverage probability for one newly generated observation. Our results indicated that sample quantile estimators based on one or two order statistics outperformed all of the other estimators and they can be used for deriving nonparametric Limits of Agreement. For sample sizes exceeding 80 observations, more~advanced quantile estimators, such as the Harrell--Davis and estimators of Sfakianakis--Verginis type, which use all of the observed differences, performed likewise well, but may be considered intuitively more appealing than simple sample quantile estimators that are based on only two observations per quantile.
Original languageEnglish
Issue number3
Pages (from-to)343-355
Number of pages13
Publication statusPublished - 2020

Cite this