Variability of quantitative health measurements: On method comparison, norm curves, and optimal cut-off point selection

The quality and usability of every quantitative health measurement depend on the accuracyand reliability of the respective measurement devices, algorithms, or observers. Patient data canvary widely within and between patients. For instance, when a general practitioner measurespulse and systolic and diastolic blood pressures, one way is to perform the measurement threetimes, and using only the measurement of the third round to avoid inflated measurementsďeĐause of the patieŶt’s ŶeƌǀousŶess. This thesis foĐuses oŶ the ǀaƌiaďilitLJ of ƋuaŶtitatiǀe healthmeasurements across three domains: method comparison studies, deriving nonparametric normcurves based on a representative sample, and estimating the optimal cut-off points for acontinuous marker or measurement. 

Bland-Altman Limits of Agreement (BA LoA) are widely known and excessively applied;however, the diligence of the involved analyses as well as the extent and thoroughness of theirreporting in the literature vary notably. Several research groups have proposed reportingguidelines for BA LoA. In paper I, we reviewed such methodological reviews and concluded thatthe study by Abu-Arafeh et al. (doi: 10.1093/bja/aew320) was the most comprehensive andconcise but lacked sample size considerations as a reporting item. 

Application of the BA LoA requires that paired differences follow a roughly normaldistribution, constant bias, and variance homogeneity across the measurement range. Sometimes,the application of a regression approach or data transformation to another scale (such as a naturallogarithm) can ameliorate the violation of these assumptions. Whenever this is impossible, theLoA must be estimated using alternative methods. Because the BA LoA are simply the 2.5% and 97.5% quantiles of the paired differences, any nonparametric quantile estimator represents acandidate. Paper II describes a simulation study of six nonparametric quantile estimators under sixdistributional assumptions for paired differences, with sample sizes varying between 50 and 1,000.A simple sample quantile estimator (a weighted average of the observations closest to the targetquantile) outperformed the Harrell-Davies estimator and estimators of the Sfakianakis-Verginistype for n = 50 and was slightly better for n = 100, whereas all three estimators performedidentically well for n ≥ 150.  

Doug Altman and Martin Bland proposed the BA LoA in the 1980s. Arthur Agatston proposeda score for coronary calcification based on cardiac computed tomography scans in 1990 thatmanifested itself in preventive cardiology. Paper III reviews how BA analysis has been applied toreproducibility analyses of the Agatston score over three decades. Based on the 49 identifiedstudies, the sample sizes were highly variable (ranging from 10 to 9,761) and the focus was onboth intra- and inter-rater and intra- and inter-scanner variability. Simple analysis tools, such asscatterplots and correlation coefficients, which Bland and Altman deemed inappropriate, were stillpopular. At first, Tukey difference plots supplemented these; later, BA plots followed.Unfortunately, very few publications have been capable of deriving LoA that fit the observed datavisually in a convincing way. 

The Agatston score is an important risk marker for cardiovascular disease and is part of therespective prevention guidelines. It is nonnegative, characterized by large inter-patient variability,and often follows a right-skewed distribution in screening populations, with an overexpression ofzero values for absent calcification. In paper IV, we derived the Agatston score reference curvesfor middle-aged and elderly Danish populations based on two screening cohorts, aged 50–75 years and stratified by sex. We implemented these percentile curves in a freely available calcificationcalculator (http://flscripts.dk/cacscoreͿ that plaĐes a patieŶt’s AgatstoŶ sĐoƌe, ďased oŶ thepatieŶt’s age aŶd sex, in a plot with 25%, 50%, 75%, and 90% percentile curves across the entireage range. This calculator simplifies the interpretation of whether a given Agatston score is low,moderate, or high during a patient-physician consultation.

Paper V is a methodology paper associated with paper IV. First, positive Agatston scoreswere log-transformed and nonparametrically regressed on age for each sex with lowesssmoothing. The respective residuals were then ranked, and their percentiles were derived. Addingthese percentiles to the model-based mean value for a particular age resulted in the estimatedpercentiles for log-transformed Agatston scores. Finally, these percentile curves were backtransformed and transposed according to the proportion of zero Agatston scores, resulting inpercentile curves for males and females. 

When analyzing the ability of a continuous marker to discriminate between diseased andnon-diseased subjects, the area under the receiver operating characteristics curve is a commonsummary statistic. Cut-off points that dichotomize a continuous marker are used in the clinicalroutine as an aid for treatment planning and often stem from fixed sample size studies. In thesimulation study reported in paper VI, we investigated how the optimal cut-off points derivedfrom the receiver operating characteristics curve analysis with to the closest-to-(0,1) criterionconverge with increasing sample sizes. Moreover, we explored a heuristic and path-basedalgorithm for cut-off point determination. Here, the sample sizes were iteratively increased, andthe cut-off points were evaluated. The algorithm identified a cut-off point as optimal when itdeviated by less than 1% from the cut-off point of the previous iteration. This heuristic determination required only a few iterations and meant practicable sample sizes; however, theresult was, at best, a rough estimate of an optimal cut-off point, and the estimate was unbiasedfor a prevalence of 0.5, and positively and negatively biased for a prevalence smaller than 0.5 andlarger than 0.5, respectively.