ssiaph.htm

Appendix H. Calculating Sensitivity and Specificity for Validation

Run algorithm on all N records for which truth is known:

Alg sensitivity = A/(A+B)
Alg specificity = D/(C+D)

From the Algorithm-positive charts (A and C), do human review on a sample (a from A and c from C)

The number, a*, from the sample that the human review finds to be positive is a hypergeometric random variable (A, AP, a) = (population size, subpopulation size, sample size)

From the True positive charts, A, a are sampled

a* = S_TP (sample true positives)
a-a* = S_FN (sample false negatives)

Similarly for c*
From the True negative charts, C, c are sampled

c* = S_FP (sample false positives)
c-c* = S_TN (sample true negatives)

Estimated TP = S_TPA/a
Estimated FN = S_FNA/a
Estimated TN = S_TNC/c
Estimated FP = S_FPC/c

Estimated HR Sensitivity = Estimated TP/(Estimated TP + Estimated FN) = (S_TPA/a)/( S_TPA/a+ S_FNA/a) = … = a*/a

Estimated HR Specificity = Estimated TN/(Estimated TN + Estimated FP) = (S_TNC/c)/( S_TNC/c+ S_FPC/c) = … = (c-c*)/c

Var(c*) = var(c-c*) = (cCP(C-c)CN)/(C²(C-1))
Var(a*) = (aAP(A-a)AN)/(A²(A-1))

So the variance of the estimated HR sensitivity and HR specificity:
Var((c-c*)/c) = (CP(C-c)CN)/(cC²(C-1))
Var(a*/a) = (AP(A-a)AN)/(aA²(A-1))

Overall:
Sensitivity = Alg sensitivity x HR sensitivity
Specificity = Alg specificity + (1 – Alg specificity) x HR specificity

Method 1: Assuming Alg sensitivity and specificity are known constants (because all charts will undergo algorithm and the truth is known).

Var(Sensitivity) = (Alg sensitivity)² x Var(HR sensitivity)

Var(Specificity) = Var(HR specificity) + Var(HR specificity) x (Alg specificity)²

Table for sample size a = c = 50 (so 50 true + and 50 true -)

Method 2: Assuming Alg sensitivity and specificity are not known constants and use Delta Method for variance. The variance for the Alg assumes the A is binomial(A+B, sensitivity) and C is binomial(C+D,1-specificity)

Var(Sensitivity) = (Alg sensitivity)² x Var(HR sensitivity) + (HR sensitivity)² x Var(Alg sensitivity)

Var(Specificity) = Var(HR specificity) + Var(HR specificity) x (Alg specificity)²
                + Var(Alg specificity) x (HR specificity)² + Var(Alg specificity)

There was no perceptible change in CI width. The variance in the estimates were small due to the large sample size (C+D) and high sensitivity of the algorithm.

Return to Contents
Proceed to Next Section