Decisions Encountered During Key Task Number 4: Checking Data Quality and Completeness

Methodological Considerations in Generating Provider Performance Scores

Whether a Chartered Value Exchange (CVE) receives raw data, uses a distributed data model, or reports "prescored" measures, the quality and completeness of performance data are key factors that determine whether a performance report can provide useful information to patients and providers. The delegation of "data auditing" tasks may depend on how a CVE is handling performance data:

  • If a CVE receives and processes raw performance data (e.g., health plan claims), then the CVE itself may want to perform the "data auditing" tasks described in this section.
  • If a CVE contracts with a vendor to process raw performance data, the CVE may request from its vendor a plan for data auditing and a report of what was done (once data auditing and preparation are finished). The data auditing plan may identify the processes the vendor will use to edit, clean, quality check, and amend the data. The auditing report may describe the results of these activities and provide a list of known data quality and completeness issues.
  • If a CVE uses a distributed data model (see section on Task Number 3), the CVE may consider discussing these data auditing tasks with the sources of its performance data. A CVE may want to have each data source reviewed by an independent auditor.
  • If a CVE reports "prescored" measures (see section on Task Number 3), then the CVE may want to consult the existing documentation for these measures to see what kinds of data auditing steps were performed.

A discussion of practical approaches to data auditing is available in a separate AHRQ decision guide titled Selecting Quality and Resource Use Measures: A Decision Guide for Community Quality Collaboratives.14

Return to Contents 

A. How will tests for missing data be performed?

To determine the extent to which data are missing, there are two main kinds of missing data to try to detect:

  • Missing data within a record. A "record" refers to a unit of observation, such as a patient office visit. If a database contains a notation that an office visit occurred, but the diagnosis (or reason for visit) is absent, then this data element can be considered missing. Another example is in patient experience surveys. If a survey is returned but a question has been skipped, then this skipped question constitutes a missing data element within the record (the survey). It is generally easier to detect missing data within a record than to detect entire records that are missing.
  • Missing records. Examples of missing records include entire surveys that are not returned and office visits that are not included in administrative data. For surveys, a list of patients to whom the survey was mailed will allow a CVE to know the extent of missing data. But for office visits, the situation is more difficult. How can a CVE tell that an office visit occurred when there is no record? After all, maybe the office visit never occurred in the first place.

Missing data are a difficult problem, even for experienced programmers and analysts. One way to check for missing data is to compare the performance data to an external standard, such as the documentation that accompanies the data. If performance data come with documentation that lists the number of records, a good first step is to check that the number of records in the data file is equal to the number listed in the documentation. In a related example, if the number of patients in the performance data from a given health plan is much smaller than the number of patients known to be enrolled in the health plan, then it is likely that many patient records are missing. Other examples include a complete lack of mental health data from a health plan (due to the plan's use of a "carve-out" subcontract for mental health services) and a complete lack of pharmacy data (due to use of a pharmacy benefit manager). The only way to fix these problems is to go back to the data source and figure out a way to obtain the missing data.

In general, missing records are detected in two situations. First, the number of records in a dataset may not match the number of records listed in the dataset documentation (i.e., a description of the dataset that gives the number of records). Second, the existing data may be implausible (e.g., it is extremely unlikely that an entire health plan's membership would consume no mental health services or no prescription drugs in a given year). Sometimes, however, there may not be any such red flags. It is much harder to detect missing records when the existing data still look plausible. The best techniques for detecting missing data depend on the data source in question, and obtaining consultation from analysts who are experienced with each data source may be advisable.

Examples: Missing data

The New York Quality Alliance (http://www.nyqa.org) is using adjudicated health plan claims data to calculate Healthcare Effectiveness Data and Information Set (HEDIS) performance measures. However, when patients are enrolled in capitated products, the health plans do not receive claims for every clinical service that is delivered. It is difficult, therefore, to know the extent to which HEDIS performance measure data for capitated patients are missing.

Return to Contents 

B. How will missing data be handled?

There are a number of options for handling missing data. Some options are based on statistical techniques, focusing on trying to make performance reports as complete and accurate as possible. Other options for handling missing data are intended to create incentives for data sources to report data that are more complete. These options may sacrifice some short-term accuracy in exchange for the longer term goal of getting more complete data in the future. The choice between statistical techniques and creating incentives can be guided by the reasons the data are missing. For the purpose of creating reports of provider performance, there are two main reasons performance data might be missingiii:

  • Data can be missing in a way that is not related to true provider performance. For example, a survey of patient experience could have had a printing flaw that led many patients to skip certain items. Or a computer problem could have deleted all data pertaining to clinical care over a 1-month period. Some types of clinical data (e.g., lab values) might be rarely recorded in administrative databases.
  • Data can be missing in a way that is related to true provider performance. For example, if a CVE is getting performance data directly from providers, some providers might choose not to report data that are likely to show poor performance.

Missing data are problematic because they can cause performance misclassification. When data are missing in a way that is not related to true provider performance, then having more missing data will increase the risk of performance misclassification due to chance (i.e., lower the reliability of performance measurement, which is discussed in Appendix 2). Misclassification risk will rise because of fewer performance observations.

However, if data are missing in a way that is related to true provider performance, then having more missing data will increase the risk of systematic performance misclassification (i.e., introduce statistical bias, which is discussed in Appendix 1). If, for example, low-performing providers tend to selectively withhold data that would indicate poor performance, then they will be systematically misclassified as having performance that is higher than their true performance.

To determine the reasons for missing data, a CVE can query the suppliers of the performance data and perform data audits. In some cases, a statistician may be able to help distinguish between the reasons for missing data. The following options for handling missing data are intended to give an overview of the types of strategies a CVE can use. Additional discussion of options for handling missing data is available in an earlier RAND report.22

  1. Option 1: Imputation.mputation refers to a family of statistical methods that use the available data from a given provider (i.e., the data that are not missing) to "fill in" the missing data for that provider. In addition to providing estimated values for the missing data, these imputation methods can compute the amount of uncertainty associated with these estimated values. In other words, statistical imputation gives an educated guess for each missing data element as well as a sense of how good the guess is likely to be.

    Advantages:

    • Given the available data, enables calculation of performance estimates that are as accurate as possible. However, the successfulness of imputation techniques will depend on the reasons the data are missing: Imputation will be most successful when data are missing in a way that is not related to true provider performance.
    • Maximizes the number of measures and providers that can be included in a performance report.

    Disadvantages:

    • Methodologically complex; may not be needed in many situations. Statistical imputation will require consultation with a statistician experienced in these techniques who can advise on the potential advantages of imputation.
    • May produce inadequate results. Imputed data are only as good as the data that are not missing. If too many data are missing, then imputation may not produce performance estimates with enough certainty to be useful for reporting.
    • May create a disincentive to report more complete data in the future.
    • May be difficult to explain to stakeholders. Imputation may raise the likelihood that stakeholders will mistrust performance reports.
  2. Option 2: Report the average score for measures with missing data. Suppose a CVE wants to include six performance measures in a report, but for a given provider ("Provider X"), performance data on two of these measures are missing. For these two measures, the CVE could report, for example, the average performance of all providers as the performance for Provider X.

    Advantages:

    • Methodologically simple; easy to explain to stakeholders.

    Disadvantages:

    • If data are missing in a way that is related to performance, then imputing the average score is likely to systematically misclassify performance.
    • May create a disincentive to report more complete data in the future. When performance data are obtained directly from providers, this approach may create an incentive for providers to withhold data for any measure on which performance is lower than average.
  3. Option 3: Report only the available data. Suppose a CVE wants to include six performance measures in a report, but for a given provider, the data needed to generate scores on two of these measures are missing. In this case, the CVE could report performance for this provider on the four measures for which data exist, placing a "not reported" marker in its report for the other two measures.

    Advantages:

    • Methodologically simple; easy to explain to stakeholders.

    Disadvantages:

    • May confuse patients, who might not understand what the "not reported" marker means.
    • May result in systematic performance misclassification if data are missing in a way that is related to performance and only the remaining measures are reported.
    • May create a disincentive to report more complete data in the future.
  4. Option 4: Report performance only for providers that are not missing data on any measure. Under this approach, if a provider were missing data on any measure reported by a CVE, then this provider would receive a "not reported" marker on all measures. This "not reported" marker would even apply to measures for which data are available.

    Advantages:

    • Methodologically simple; easy to explain to stakeholders.

    Disadvantages:

    • Many providers may have no reported performance data, limiting the usefulness of public reports to patients.
    • As the number of measures grows, the number of providers with reported performance may fall (since there are more chances to have missing data).
    • This approach carries unclear incentives for future reporting and may encourage nonreporting by low performers.
    • This approach may be unacceptable to providers and patients.
  5. Option 5: Report the lowest possible score when data are missing. When a provider has missing data on a given measure, a CVE can report the provider's performance as the lowest possible score.

    Advantages:

    • Methodologically simple; easy to explain to stakeholders.
    • Creates an incentive for complete data reporting in the future.
    • May result in less systematic performance misclassification than Options 1 through 4 if data are more likely to be missing when performance is low..

    Disadvantages:

    • If data are missing in a way that is not related to performance (i.e., missing due to chance alone), likely to systematically misrepresent performance as being much lower than it really is.
    • May be unacceptable to providers.
  6. Option 6: Report the lowest observed score when data are missing. When a provider has missing data on a given measure, a CVE can report the provider's performance as being equal to the lowest observed score among providers who are not missing data on the measure.

    Advantages:

    • Methodologically simple; easy to explain to stakeholders.
    • Creates an incentive for complete data reporting in the future.
    • May result in less systematic performance misclassification than Options 1 through 4 if data are more likely to be missing when performance is low..
    • May be more acceptable to providers than imputing the lowest possible score.

    Disadvantages:

    • If data are missing in a way that is not related to performance (i.e., missing due to chance alone), likely to systematically misrepresent performance as being much lower than it really is. 

Table 1 summarizes the relationship between the reasons data are missing and the strengths and weaknesses of the options for handling missing data.

Different approaches to dealing with missing data can be used for different measures within the same performance report. For example, it is common to impute missing values in composite measures without imputing missing values when the individual measures are presented (i.e., in a drilldown screen). This combination of strategies is attractive because the cumulative risk of having one or more missing data elements increases with the number of individual measures included in a composite. In addition, the impact of any one missing element decreases as the number of indicators in a composite increases, so the misclassification risk associated with erroneous imputation is less for a composite measure than for an individual measure.

Examples: Approach to "missing" performance data

Each of the nine CVEs we interviewed described working with data sources to minimize the amount of missing data. After this step, CVEs report the available data (Option 3 above). When performance data for a provider cannot be reported due to concerns about the risk of misclassification due to chance (e.g., insufficient numbers of observations), a symbol is generally used to indicate that performance cannot be reported. For examples of using such symbols, go to the reports of the Healthy Memphis Common Table (http://www.healthymemphis.org) and the Puget Sound Health Alliance (http://www.wacommunitycheckup.org).

Return to Contents 

C. How will accuracy of data interpretation be assessed?

Even when performance data are present, it is possible for these data to be misinterpreted during the computation of measure performance. Problems with data accuracy are likely to be greatest when a CVE plans to generate performance scores (on quality or cost measures) using health plan claims. For example, measure specifications may assume that patients with diabetes are identified using a particular set of diagnosis codes, when in fact such patients are identified using a different set of diagnosis codes. This is a particular problem when combining performance data from multiple data sources, because sources may not all have the same ways of coding conditions, health care services, and patient outcomes (as discussed in the section on Task Number 3.)

Generally speaking, the same kinds of approaches used to check for missing records can be used to assess the accuracy of data interpretation. Some of these approaches for detecting missing records are discussed in the section on Task Number 4. These approaches include:

  • Compute overall number of patients who qualify for a measure, and make sure this number makes sense. For example, if a measure is supposed to apply to all patients with diabetes, does the number of diabetics identified seem realistic based on other known data? Comparison data may be available from local health departments or from the data source itself (e.g., a health plan that supplies data may also conduct disease management outreach and have a roster of its diabetic enrollees).

    This step also gives a CVE the opportunity to see how many patients are excluded by continuous enrollment criteria. In their specifications, some performance measures have "continuous enrollment criteria." This usually means that in order to contribute performance data, a patient must be a member of the same health plan (or a patient of the same provider) for at least 1 or 2 years. However, a significant percentage of patients may switch health plans from year to year, becoming ineligible for inclusion in a measure. This may result in a large reduction in the percentage of patients who are contributing performance data to a report. In some cases, more than half of all patients in a CVE's area may be excluded by these criteria.

  • Recheck the number of patients who qualify for a measure on a provider-by-provider basis. The question again is: "Do the numbers make sense, relative to some kind of external standard?" The advantage of calculating provider-by-provider patient counts for a measure is that these counts can, when providers have disease registries, be verified by the providers themselves. A measure's specifications may include a patient population that is somewhat different from what a provider would report (e.g., because of continuous enrollment requirements). Thus, some degree of disagreement between the measure-identified and provider-identified patient counts can be expected. But the figures should at least be in the same ballpark. A notable caveat to this approach is that some providers probably will not participate in data verification.
  • Check the range of numeric values within each type of data, and make sure these are consistent with what the data are supposed to represent. For example, if a given variable is supposed to represent hospital length of stay, this variable should never be a negative number. The same is true of patient age, which should never be negative and will infrequently be greater than 100 years.
  • Compute population-level performance measure scores and compare to external benchmarks. For example, a CVE may compute the overall performance rate within its geographic area on a given HEDIS measure. Then this overall rate can be compared to national performance data: does it seem to be in the right ballpark? The ability to make comparisons to national benchmarks is one advantage of using nationally endorsed measure specifications (also discussed in the section on Task Number 2).

When problems are discovered in these data checking steps, there may be inaccuracies in data interpretation. In other words, the measure specifications may assume that the raw performance data mean one thing when in fact they mean another. Just as when missing record problems are detected, the best approach to correcting data interpretation problems may be to consult with analysts who are experienced with each data source. Such analysts are likely to be in the best position to understand how data misinterpretations have occurred and to find solutions.


 iii Note that the reasons data might be missing are distinct from the kinds of missing data that were discussed in the preceding section.

Page last reviewed September 2011
Internet Citation: Decisions Encountered During Key Task Number 4: Checking Data Quality and Completeness: Methodological Considerations in Generating Provider Performance Scores. September 2011. Agency for Healthcare Research and Quality, Rockville, MD. http://archive.ahrq.gov/professionals/quality-patient-safety/quality-resources/value/perfscoresmethods/perfsctask4.html