Basis Upon Which Winner Will be Selected
Applications will be evaluated based on two broad areas: (1) Reliability and (2) Validity. In general, reliability is assessed by how closely the model or method deployed predicts the actual utilization rates for 2017. Validity is assessed by how well the model performs on earlier years of data.
Applicants will be provided with an Excel spreadsheet that includes columns where the predicted values are to be inserted and will return it to AHRQ. For counties where 2017 data are available to AHRQ by June 28, 2019, an overall evaluation metric will be calculated. An overall evaluation metric for the submitted model or method is calculated in the following way:
- For each cell, the absolute percentage difference between the predicted value and the actual value is determined.
- For each cell, the calculated value is weighted by the share of the population in that county to the total population of all selected counties in the dataset.
- Each cell is weighted equally based on the assigned weights for each category. The category weights are assigned as follows:
- Item #1: 80%
- Item #2: 20%
- The sum of scores of all cells determine an applicant's overall score. The lower the score, the more reliable and valid the predictive method.
After scoring is complete, AHRQ will convene and internal team of experts to review the top scoring submissions and make final winner determinations. Winners are determined by applicants with the lowest scores, as long as materials submitted under Item #3 are verified by AHRQ. In particular, AHRQ will assess whether an applicant's predicted values can be replicated. AHRQ will be the final arbiter of winners for this Challenge.
AHRQ anticipates three (3) winners, but the number of winners may vary, depending on circumstances. In the event of a tie, the prize money will be split equally.