Appendix H. Calculating Sensitivity and Specificity for Validation

Run algorithm on all N records for which truth is known:
 Algorithm
+-
Truth+AB
-CD

Alg sensitivity = A/(A+B)
Alg specificity = D/(C+D)

From the Algorithm-positive charts (A and C), do human review on a sample (a from A and c from C)

The number, a*, from the sample that the human review finds to be positive is a hypergeometric random variable (A, AP, a) = (population size, subpopulation size, sample size)

From the True positive charts, A, a are sampled

 Sampled?Total
yesno
Human Review+a*AP - a*AP
-a-a*AN-a-a*AN
TotalaA-aA

a* = STP (sample true positives)
a-a* = SFN (sample false negatives)

Similarly for c*
From the True negative charts, C, c are sampled

 Sampled?Total
yesno
Human Review+c*CP - c*CP
-c-c*CN-c-c*CN
TotalcC-cC

c* = SFP (sample false positives)
c-c* = STN (sample true negatives)

Estimated TP = STPA/a
Estimated FN = SFNA/a
Estimated TN = STNC/c
Estimated FP = SFPC/c

Estimated HR Sensitivity = Estimated TP/(Estimated TP + Estimated FN) = (STPA/a)/( STPA/a+ SFNA/a) = … = a*/a

Estimated HR Specificity = Estimated TN/(Estimated TN + Estimated FP) = (STNC/c)/( STNC/c+ SFPC/c) = … = (c-c*)/c

Var(c*) = var(c-c*) = (cCP(C-c)CN)/(C2(C-1))
Var(a*) = (aAP(A-a)AN)/(A2(A-1))

So the variance of the estimated HR sensitivity and HR specificity:
Var((c-c*)/c) = (CP(C-c)CN)/(cC2(C-1))
Var(a*/a) = (AP(A-a)AN)/(aA2(A-1))

Overall:
Sensitivity = Alg sensitivity x HR sensitivity
Specificity = Alg specificity + (1 – Alg specificity) x HR specificity

Method 1: Assuming Alg sensitivity and specificity are known constants (because all charts will undergo algorithm and the truth is known).

Var(Sensitivity) = (Alg sensitivity)2 x Var(HR sensitivity)

Var(Specificity) = Var(HR specificity) + Var(HR specificity) x (Alg specificity)2

Table for sample size a = c = 50 (so 50 true + and 50 true -)

Algorithm SensitivityAlgorithm SpecificityHuman Review SensitivityHuman Review SpecificitySensitivity 95% CI Half-widthSpecificity 95% CI Half-width
0.950.500.800.850.090.11
0.950.500.800.900.090.09
0.950.500.800.950.090.07
0.950.700.800.850.090.12
0.950.700.800.900.090.10
0.950.700.800.950.090.07
0.950.500.850.850.080.11
0.950.500.850.900.080.09
0.950.500.850.950.080.07
0.950.700.850.850.080.12
0.950.700.850.900.080.10
0.950.700.850.950.080.07
0.950.500.900.850.070.11
0.950.500.900.900.070.09
0.950.500.900.950.070.07
0.950.700.900.850.070.12
0.950.700.900.900.070.10
0.950.700.900.950.070.07
0.990.500.800.850.090.11
0.990.500.800.900.090.09
0.990.500.800.950.090.07
0.990.700.800.850.090.12
0.990.700.800.900.090.10
0.990.700.800.950.090.07
0.990.500.850.850.080.11
0.990.500.850.900.080.09
0.990.500.850.950.080.07
0.990.700.850.850.080.12
0.990.700.850.900.080.10
0.990.700.850.950.080.07
0.990.500.900.850.070.11
0.990.500.900.900.070.09
0.990.500.900.950.070.07
0.990.700.900.850.070.12
0.990.700.900.900.070.10
0.990.700.900.950.070.07

Method 2: Assuming Alg sensitivity and specificity are not known constants and use Delta Method for variance. The variance for the Alg assumes the A is binomial(A+B, sensitivity) and C is binomial(C+D,1-specificity)

Var(Sensitivity) = (Alg sensitivity)2 x Var(HR sensitivity) + (HR sensitivity)2 x Var(Alg sensitivity)

Var(Specificity) = Var(HR specificity) + Var(HR specificity) x (Alg specificity)2
                + Var(Alg specificity) x (HR specificity)2 + Var(Alg specificity)

There was no perceptible change in CI width. The variance in the estimates were small due to the large sample size (C+D) and high sensitivity of the algorithm.

Page last reviewed December 2012
Internet Citation: Appendix H. Calculating Sensitivity and Specificity for Validation. December 2012. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/research/findings/final-reports/ssi/ssiaph.html