Appendix E: Blinded Reviewer Comments (continued)

Health Care Efficiency Measures: Identification, Categorization, and Evaluation

Contents - By Section:
Explanation of Interest in Efficiency Measures
Executive Summary
Chapter 1 - Introduction
Chapter 2 - Methods
Chapter 3 - Results
Chapter 4 - Assessing Measures
Chapter 5 - Discussion
  Editorial Comment
  Which measures are ready for use?
  Are there published measures not included?
  Are there vendor developed measures not included?

Chapter 1 - Introduction

Chapter 1 - IntroductionIn this introductory chapter efficiency (as well as value) should be defined more explicitly upfront. Efficiency is later defined as an attribute of performance in Ch 2 and we are told how it is measured (relationship of inputs to outputs) but need to be clear on the parameters of efficiency here (mainly costs).Done.
Chapter 1 - IntroductionLists Leapfrog as a measures developer — this is a stretch — No mention of the PCPIWe added PCPI
Chapter 1 - IntroductionParagraph 3 - first real mention of the concept of value - it is mentioned in the first paragraph in the executive summary - but then disappears from the paper."Value" is useful for framing the debate but we omit it from the rest of the document since our task was to develop a typology of "efficiency"- "value" was not part of the nomenclature adopted for our typology.
Chapter 1 - IntroductionChapter 1, page 11, 2nd paragraph, I suggest adding AQA to the list of groups defining health care quality measures.This was added
Chapter 1 - IntroductionPage 12 -nice definition of efficiency that should be pulled into the executive summaryThis definition has been added in the executive summary.
Chapter 1 - IntroductionBullet in perspective – refers to intermediaries who “act on behalf of providers or individuals --- no mention of their own profit motive or self interest.It is true that there is no mention of the other motives that intermediaries may have but there is no implication nor indeed expectation that intermediaries are free of other interests. We do not expect that readers will assume that intermediaries, nor providers, are free of other motives including profit or self-interest.

Return to Top

Chapter 2 - Methods

Chapter 2 - MethodsData Sources: As acknowledged in the report, academics and vendors/purchasers/plans rely on administrative data sources for measuring efficiency.We have revised this section to reflect this observation.
Chapter 2 - MethodsData Sources: Administrative data was termed problematic by the authors, without any clear evidence that for this specific use it might be adequate. I have seen no evidence in the literature that indicates that the use of administrative data for measuring efficiency is inadequate.We agree and have modified references to the problems with administrative data.
Chapter 2 - MethodsData Sources: Severity adjustment tools and groupers utilizing administrative data have been shown to have high C statistics for certain conditions using certain products-since no one has looked at administrative data for efficiency measurement it is somewhat arbitrary to indicate that the data source is problematic and that the resulting measurement would be better using clinical data sources. Is it not true that clinical data sources without administrative data would hinder the development and use of efficiency measures? Imagine searching through ambulatory clinical records for visits provided by others.We agree and have modified the text to reflect this.
Chapter 2 - MethodsData Sources: For this document to have greater value to vendors/purchasers/plans adopting a more neutral stance on methods would be useful; this document as written has an academic bias. It could be noted that academic models have the same potential problem (secondary use) that is applied to administrative data, given that academics are not designing their models to be used in the real world of pay-for-performance.We have modified the text to try to communicate a more neutral stance.
Chapter 2 - MethodsI am not that familiar with the DEA and SFA type measures. Could you provide a simple example of each early on in the report? Then the concepts would be easier to follow later on. This is a non-economist speaking, of course.

It seems like the efficiency index concept is getting applied in more and more places (I understand IHA is considering using it, for example). The report spent a good deal of time on the academic papers, and less on physician efficiency measures. Given the increasing pervasiveness of efficiency index measures, they may deserve more space and discussion.
We now reference the box in the results chapter, which describes the methods in greater detail and includes references to these methods.
Chapter 2 - MethodsA scan of Appendix F, Characteristics of Health Care Efficiency Measures Published in Peer-Reviewed Literature (1982-2006) raises some very useful questions that pertain to the implications of existing research for future, practical efforts to develop efficiency measures for large-scale implementation and use.

Three basic approaches to resource (input) enumeration seem apparent in the literature: [A] operationalization/measurement of inputs using costs; and [B] operationalization/measurement of inputs using number of units - especially manpower units (e.g. physicians, nurses); [C] a hybrid of the two.

RAND defines efficiency as "an attribute of performance that is measured by examining the relationship between a specific product of the health care system (also called an output) and the resources used to create that product (also called inputs). Others have defined efficiency as the cost of producing a given level of output, or quality.

Risk adjustment and episode groupers have attempted to increase comparability across patients. There is a question whether heterogeneity in some fundamental characteristics of producers and the inputs/technology that they use should similarly be addressed. In other words, it would seem that a measure of efficiency should hold constant variation in patient characteristics that might affect the amount of resources used, but also some physician characteristics that might affect resource use. For example, two physicians of similar training and background may differ in efficiency if one uses more tests and time, than the other. On the other hand, medical students and new doctors may use more resources (diagnostic tests, for example) in producing care initially because they have not built up their stock of human capital that comes from years of learning-by-doing and experience (whether direct or vicarious). More seasoned doctors may be able to treat the same case more quickly or with fewer inputs. Without accounting for heterogeneity in the sheer number of years as a proxy for human capital, a strict interpretation of efficiency may identify newer producers as inefficient relative to older producers. This may not be desirable.
We tried to discuss in greater depth and nuance these various themes.
Chapter 2 - MethodsBased on your review of the literature, how much variation is there in the way costs are identified and measured across studies, and what implications are there for developing some form of standardized cost reporting for use in efficiency measurement?There is a great deal of variation in way costs are identified and measures across studies and we expect it would be a substantial challenge to standardize this.
Chapter 2 - MethodsOn page 13 -- the whole section on Outputs being either health services (number of visits, drugs, admissions, etc.) misses the point. There will always be some exchange between, say, higher drug compliance and use and lower admissions or visits to the E/R. Neither one is inherently "better" than the other in terms of number of services. Similarly, while I would love to have good measures of health outcomes for every health status/disease/treatment, this is likely not possible in the near future. Thus, we are left with only dollar-cost denominated measures as the only practical ones.We have edited the text to reflect that neither approach is inherently better or worse.
Chapter 2 - MethodsTop of page 13, of course CDHPs are attempts to put individuals directly in contact with providers.We have added this observation.
Chapter 2 - MethodsPage 13, discussion of outputs: Is there a place here to discuss quality of outputs? Also, where does patient-centeredness fit in? I'm working with a fellow with a six-sigma background, and he talks about connecting the set of patient preferences (the “voice of the customer”) to the set of desired outcomes and then measuring the costs to get there.

Next paragraph, last sentence “Greater opportunity for conflict may arise…” I'm not sure why this should be. Perhaps a concrete example would clarify. Is there a particular issue you are getting at?
We have added a reference to consumer experience as being an extension of the health outcomes category.

We moved the sentence referring to conflict.
Chapter 2 - MethodsP13, paragraph 7 - Re comparability of outputs: Using the example of “by specialty for physicians” begs the question of how to define specialty. When pooling provider directories across health plans we identified many inconsistencies across plans in the listed primary and secondary specialties for the same physician. The Bd of Registration in Medicine files are not that helpful either. Should the mix of ETGs be used in some fashion (e.g. specialty “fit” statistic)?We have added this example.
Chapter 2 - Methodsp.14 par.1 In this paragraph it would be helpful to present the rationale why the AQA defines efficiency as “a measure of the relationship of cost of care associated with a specific level of performance measured in respect to the other 5 IOM aims (effectiveness, safety, equity, timeliness, patient-centeredness) of quality”. For example, the unintended consequences of only measuring costs without regards to patient outcomes. Or deeming a provider efficient (i.e. performed a CABG perfectly efficiently) without considering appropriateness of the intervention (should it have been done in the first place).We have added the rationale.
Chapter 2 - MethodsPage 14, last two sentences before “Input” section: Yes, this is why well specified definable desired outcomes are a key research question.Added to future research section.
Chapter 2 - MethodsP14, last 3 sentences - Here again, there's a seeming ambivalence re whether the failure to use health outcomes as the output of interest for efficiency measurement is something that should be remedied or if outcomes should be addressed in the quality domain instead. The authors could add something like “However, health outcome measures of quality can be used in the side-by-side comparisons referred to above (e.g. risk-adjusted cost of cancer care and 5-year survival rates).”We have edited the text to reflect that either health services or health outcomes are reasonable outputs to include in efficiency measures.
Chapter 2 - MethodsP14, paragraph 4 - The last sentence in this paragraph reads, “The way in which inputs are measured may influence the way the results are used”. One could just as easily say, “The measurement objectives may dictate which inputs are measured.” I prefer the idea of beginning with an objective. At the least, I would say, “…will dictate the way results should be used.”Done.
Chapter 2 - Methodsp14 bottom - p. 15 top You also need to set up social (allocative) efficiency here for the discussion later on p 21. It is part of the stated typology in regards to inputs.We have introduced this concept here as suggested.
Chapter 2 - MethodsP15, paragraph 2 - Just as the authors point out the need for comparability of outputs, it would seem that a key issue in the discussion of inputs is whether the inputs are comparable. One problem area we've encountered in establishing comparable inputs is aligning cost data for physicians paid on a FFS basis with that for physicians or groups paid on a capitated basis. There are payments in capitated contracts that should be allocated to services in order to get truly equivalent service level payments. This can also be true when contracting with groups on behalf of physicians who are paid FFS; the group may receive payments that should be allocated to services. This is an argument for using neutral pricing.Done.
Chapter 2 - MethodsPart I
Page 15, last paragraph: This is where judgmental systems break down. The physician with a 20 percent co-insurance has a competitive advantage over the other two LASIK providers. It is to her advantage to maintain that advantage and increase her share of the local LASIK business. Therefore she has an incentive not to share any “best practice” knowledge. The stronger the judgment, the more the pressure to keep best practice proprietary. This would include knowing about generic medication. Also, she would never tell another surgeon that her nurse was faster – the nurse would get poached. (A shortage of any non-physician helper leads to this problem, which I have actually seen in a different specialty in our community. A larger practice was hiring away a certain type of technician from smaller practices, making them less able to compete.) We were able to maintain cooperation in our PFP system I think because we had counterbalancing system forces (for example, gain-sharing on non-physician expenses, and specialty budgets that bound together the performance of all practitioners in the specialty). Also, PFP I think would be weaker deterrent to cooperation than co-insurance tiers for elective surgery.

I don't think analysis without external reference would work. For example, physicians may have asymmetric knowledge (for example that the general anesthetic is a less expensive alternative). Asymmetric knowledge is both a competitive advantage but also by definition is not (yet) available to the other practitioners. Therefore there needs to be an outside agent with a broader perspective working with the practitioners. In the manufacturing world, this would be the idea that line workers can tell you about special case variation while management has to understand common cause variation. (I'm thinking of special causes like a worker being able to say “Last Friday we were less productive because our coworker Larry was out sick” vs common cause like management finding out that Mondays are generally least productive because they are the day after the weekend.)
We have changed the example to cataract surgery and have made it focus more on the math and the concepts than the sociology of change.
Chapter 2 - MethodsPart II
The way to make LASIK surgery less expensive in this micro-environment would be to remove tiers and organize a quality improvement initiative. This would require analysis by an organization at a level higher than the competing physicians, and that organization would have to be able to align the financial incentives. The organization could be an IPA, health plan, or integrated delivery system, but the principles would be the same (they would require different methods of aligning the financial incentives).

I'm not sure if LASIK is the best example, because as non-medically necessary procedure I presume it is rarely covered by an insurance company. On the other hand, it may respond more to market forces than other procedures precisely because it is elective and paid directly by patients. To be more realistic you might make the costs be more like $400 to $1000, and the charge might be $1000 to $2000 (all figures per eye). Of course, the charges depend on many factors including the region (what each market will bear). In Buffalo, NY practitioners need to compete with Canadian clinics that advertise $299 (USD) per eye. Interestingly, that seems to be less a problem when you get to Rochester, only 60 miles to the east.

Another interesting economic aspect is that optometrists refer the patients to ophthalmologists, but also do the pre-op and post-op care for LASIK. The optometrists therefore negotiate a fee from the ophthalmologists for pre- and post-op care. Cataract surgery might be an example that works better for the intended audience, for example for Medicare. Similar issues with optometrist referral and participation apply in that case, although Medicare sets their fee through the use of CPT modifiers.
The example has been changed to cataract surgery.
Chapter 2 - Methodsp15 I don't care for this example (personal bias) particularly since it is an elective procedure. It is very “production” oriented. I think a common chronic condition such as diabetes or even low back pain would resonate better here.The example has been changed to cataract surgery.
Chapter 2 - MethodsI did not feel, however, that the Lasik example was particularly helpful. It certainly illustrates the difficulty of comparison when dealing with multiple variables, but then again, anyone not really well versed in quadratic math could say that solving any equation with more than two moving variables is a tough exercise, so I'd get rid of the example.We've left the example for those readers who might find something more concrete helpful. We changed the Lasik example to cataract surgery.
Chapter 2 - MethodsP16, paragraph 2 - I thought I understood the difference between technical and productive efficiency as explained on pages 14 and 15 until I got to the last sentence in this paragraph. If I understand the last sentence, it's stating that standard pricing does not reflect productive efficiency. Is the point that actual costs or prices would reflect productive efficiency, but that standard pricing would not? I would think that standard pricing does reflect the mix of inputs, which I thought distinguished productive efficiency from technical efficiency. A physician who routinely performs a lab test that avoids a large percentage of hospitalizations should have a lower episode cost than a physician who doesn't perform that test and has more hospitalizations, whether using actual or standard pricing. Maybe I'm just hopelessly confused.Standard pricing does reflect the input mix and hence reflects productive as well as technical efficiency.
Chapter 2 - MethodsPage 16, last sentence: this is a place where a simplified example of DEA and SFA would be helpful.We've added a simpler description of these methods to the executive summary and the example in the typology.
Chapter 2 - MethodsOn page 19, the paragraph on cost per covered life mentions one reason that cost per covered life may not be an accurate measure of efficiency. “Large national employers may have some difficulty accounting for differences in market prices.” Another limitation to comparison across geography is differences in state mandates. While these might not apply to plans covered by ERISA, many national employers with ERISA plans also purchase non-ERISA local plans such as HMOs. Illinois, for example, mandates rich infertility treatment benefits for HMO plans. These mandates make it difficult to standardize benefit packages across the country.We added this comment.
Chapter 2 - MethodsP19, paragraph 6 - Generic prescribing—there is also an assumption that the availability of generic substitutes is consistent across all conditions, or at least across large population groups. Otherwise, case-mix adjustment would be needed or generic prescribing rates would need to be measured by condition.We added this observation.
Chapter 2 - MethodsPage 20 Table 4: The title includes the word “effectiveness” but from the context it seems like the term “efficiency” is what was intended.This was a typo. We changed it to efficiency.
Chapter 2 - Methodsp 20. Table 4 The measures excluded from the typology solidify my concern over the limited scope of this purely economic approach to classification. Exclusion of more system-level types of efficiency measures such as readmission rates (or hospital admissions for ambulatory sensitive conditions) says to me we are missing some important opportunities to improve quality of care while minimizing costs. What about measures of waste such as duplicate medical tests or overuse such as imaging for acute low back pain? Or not having access to good primary care and so using the emergency room for what could have been routine lower cost care?We note that there are many ways to improve the functioning of the health care system other than the development and implementation of efficiency measures. We are not saying these measures have no uses, rather that they do not meet our criteria for an efficiency measure.
Chapter 2 - MethodsPage 21, under cost-effectiveness: The first thing that occurred to me in terms of cost-effectiveness measures was comparative cost of QALY, as in the cost per QALY saved is higher for screening mammography in women between 40 and 50 than in women over 50, so it is less cost-effective. But doesn't that also qualify as a measure of efficiency? In other words, there is an output (the QALY) at a given level of input (cost).We have revised our explanation of why we excluded cost-effectiveness measures from this report.
Chapter 2 - MethodsP21, paragraph 4 - Not sure I agree that the delivery of additional services at a price above input costs and below current charges is always a win-win. I think those additional services need to be necessary and appropriate, or at least add value. If those services could possibly have negative health and/or cost consequences down the road (e.g. an injection that has an Iatrogenic effect), that would not be a win-win.We modified this sentence.
Chapter 2 - MethodsPage 22, second paragraph: This sounds like the “moral hazard” argument. Are you thinking that society needs to prioritize the outputs? E.g. society spends $2000 on LASIK but gets a very low ROI compared with spending $2000 on pre-natal care.This isn't exactly moral hazard.
Chapter 2 - MethodsPage 22: I like your comment on “more plans than is good for society as a whole.” We are very lucky in Rochester only to have a handful of major plans. Could you be more explicit that having a higher number of plans raises administrative costs, thereby decreasing a system's efficiency? Carried to its logical conclusion this becomes the argument for a national health care system, of course.We are not taking this argument to its “rational” conclusion.
Chapter 2 - MethodsPage 22, second to last paragraph: which is “this setting?”Changes have been made to the text.
Chapter 2 - MethodsP22, paragraph 5 - In the last sentence, the word “setting” is unclear. Did the author's mean “perspective”?
Chapter 2 - MethodsP24, figure 1 - Either remove the text box outline around “Inputs” or outline the text box around “Outputs” to be consistent.We added the text box to the Inputs category.
Return to Top
Page last reviewed April 2008
Internet Citation: Appendix E: Blinded Reviewer Comments (continued): Health Care Efficiency Measures: Identification, Categorization, and Evaluation. April 2008. Agency for Healthcare Research and Quality, Rockville, MD.