Mental Health Quality Measures

Contents

• Overview
• Experience of Care and Health Outcomes Survey (ECHO™)
• National Inventory of Mental Health Quality Measures

 Overview

The lack of reliable and useful quality measures for mental and behavioral health services has been frequently cited by State policymakers. This section tries to address these concerns through a review of two national mental health initiatives that target or include children's mental health needs.

In addition to these resources, there are two products developed under the Child and Adolescent Health Measurement Initiative (CAHMI) with measures related to child behavioral health:

• Young Adult Health Care Survey (YAHCS).

  Includes behavioral screens for 14-18-year-olds including preventive screening and counseling on risky behaviors and on emotional health and relationship issues.

• Promoting Health Development Survey (PHDS).

  Includes related measures including parental guidance on child development.

Online Resources:

• Information on YAHCS.
• Information on PHDS.

Experience of Care and Health Outcomes Survey (ECHO™): The ECHO™ survey collects consumers' assessments about their behavioral health treatment, including mental health and chemical dependency services. There are both adult and child versions of the ECHO™ survey.

National Inventory of Mental Health Quality Measures: Developed by the Center for Quality Assessment and Improvement in Mental Health (CQAIMH), this is a searchable database of over 300 measures for quality assessment and improvement in mental health and substance abuse care.

Return to Contents

 Experience of Care and Health Outcomes Survey (ECHO™)

Experience of Care and Health Outcomes Survey (ECHO™) collects consumers' assessments about their behavioral health treatment, including mental health and chemical dependency services. The ECHO™ survey is designed for use by consumers, clinicians, managed behavioral health care organizations (MBHOs), health care plans, purchasers, States, and Federal agencies. The format and design of the ECHO™ survey is consistent with the widely used CAHPS. Its contents were largely derived from two pre-existing instruments for behavioral health care quality assessment:

• The Mental Health Statistics Improvement Program.
• The Consumer Assessment of Behavioral Health Services surveys.

There are both adult and child versions of the ECHO™ survey.

Online Resources:

• Information on CAHPS 3.0.
• For more information on the Mental Health Statistics Improvement Program, go to: https://depts.washington.edu/pbhjp/projects-programs/page/mental-health-statistics-improvement-program-mhsip-surveys
• For the ECHO™ surveys, go to: https://www.ahrq.gov/cahps/surveys-guidance/echo/index.html

Return to Contents

Child Measures Included

Separate versions of the child survey have been developed for MBHOs and managed care organizations (MCOs). Items assess consumer experience with specialty behavioral health, including care for mental or emotional illness, substance abuse, family problems, and developmental conditions. The MBHO version contains 58 items and the MCO version contains 69 items. The ECHO™ survey assesses several aspects of care, including:

• Getting treatment and counseling quickly.
• Communications with clinicians.
• Information provided by clinicians on medication side effects.
• Family involvement in care.
• Information about self-help groups and treatment options.
• Cultural competency of providers of care.
• Treatment effectiveness.
• Health plan administrative and office staff services.

Return to Contents

Users

The ECHO™ survey is targeted to children who currently use mental health, behavioral health, and/or chemical dependency services. MBHOs, MCOs, and State Medicaid agencies are the intended users of the ECHO™ survey. It is expected that the child version, like the adult version of ECHO™, will be used over time to assess the performance of contracted MCOs and to compare performance across plans and subsets of the child population (e.g., children enrolled in Medicaid versus children enrolled in a State Children's Health Insurance Program (SCHIP).

Comparisons and Trends

ECHO™ results provide statistically significant comparisons when produced for multiple MHBOs and MCOs, provided that comparable populations are used.

Return to Contents

Benchmarking and Databases

The National CAHPS Benchmarking Database (NCBD) is developing a national database of ECHO™ results.

Online Resource: Information on the CAHPS database.

Return to Contents

Service Delivery and Units of Analysis

ECHO™ was designed to assess the performance of MBHO and MCO service delivery systems.

Return to Contents

Length-of-Enrollment Requirements

Twelve months of continuous eligibility is recommended when selecting the target population for the child version of the ECHO™ survey. The principles of good survey techniques suggest that individuals be surveyed who have had sufficient enrollment to achieve the desired effects of what is being measured. For example, no minimum length of enrollment requirements may be required for assessing whether members understand their basic rights for due process within a plan. However, when assessing whether a member received appropriate preventive or followup care, a minimum of 12 months of continuous enrollment is necessary. Because ECHO™ measures are based in part on the premise that MBHOs and MCOs are accountable for providing defined services to enrolled members, the minimum period of enrollment is designed to give plans a reasonable opportunity to fulfill that responsibility prior to measurement.

Return to Contents

Data Issues

ECHO™ can be administered by:

• Telephone.
• Mail.
• A combination of both. To maximize response rates, administration by mail with telephone followup is recommended.

Return to Contents

Sample Sizes

The CAHPS family of surveys requires a sample large enough to yield 300 completed questionnaires. Larger sample sizes may be required to get valid information on specific subpopulations. The CAHPS Survey and Reporting Kit proposes several sampling alternatives that can be applied to the child version of ECHO™.

Online Resource: For more information on the CAHPS Survey and Reporting Kit, go to: http://www.ahrq.gov/cahps/surveys-guidance/cg/instructions/index.html.

Return to Contents

Resource and Burden Issues

Credible survey data require close attention to proper sampling methods and adherence to survey administration protocols. These take time and money. Senior-level agency responsibility and sufficient staff resources are needed to ensure useful results.

This survey requires a minimal burden and cost for those conducting the survey. There is also a minimal burden for those responding to the survey with respect to length and reading level.

Return to Contents

Development Process

The group that developed the ECHO™ survey included behavioral health consumers, clinicians, and behavioral health policy experts, including representatives from CAHPS®, the National Committee for Quality Assurance (NCQA), NCQA's Behavioral Health Measure Advisory Panel, the Mental Health Statistics Improvement Program, the Evaluation Center at the Human Services Research Institute, the Washington Circle Group, the American Managed Behavioral Health Care Association, and the National Alliance for the Mentally Ill.

Online Resource: For more information on the group that developed the ECHO™ survey, go to:
https://www.ahrq.gov/cahps/surveys-guidance/echo/about/Development-ECHO-Survey.html

In 1998, the Evaluation Center at the Human Services Research Institute and AHRQ funded the Harvard Medical School CAHPS® study team to determine whether two existing instruments for assessing behavioral health plan performance could be combined into a single instrument. The Consumer Assessment of Behavioral Healthcare Services (CABHS) and the Mental Health Statistics Improvement Program (MHSIP) surveys were compared with respect to eight factors:

• Response rates.
• Characteristics of respondents versus non-respondents.
• Respondent burden, understanding, and perceived utility.
• Structure of survey.
• Distribution of survey response (e.g., missing data, inapplicable items).
• Data validity.
• Internal consistency.
• How well survey items discriminate among plans.

A pilot phase was conducted to test each survey instrument as well as a combined instrument. Results were reviewed by the CAHPS instrument development team and were used to develop a single survey combining the best features of each instrument. A draft ECHO™ survey was field tested by MCOs. In June, 2003 the adult version of ECHO™ was approved under the CAHPS version 3.0 family of surveys. Approval of the child version is pending.

Return to Contents

Criteria Used

In designing the final child and adult versions of ECHO™, several criteria were essential. The survey needed to be:

• Meaningful to individuals who have used behavioral health services and to plans and staff for quality improvement efforts.
• Applicable to different types of plans, including commercial, Medicaid and Medicare, and health systems (managed care and fee-for-service) plans.
• Appropriate for consumers with a range of mental health service needs, including those with severe mental illness.
• Easy to understand for diverse demographic and racial/ethnic groups and different levels of education.
• Easy to administer in different modes (mail or telephone) with minimal effect on survey responses.

Return to Contents

More Information and User Support

Online Resource: To download the ECHO™ surveys, go to:
https://www.ahrq.gov/cahps/surveys-guidance/echo/instructions/index.html

Return to Contents

 National Inventory of Mental Health Quality Measures

This is a searchable database of over 300 measures for quality assessment and improvement in mental health and substance abuse care. Measures included in the database have been developed by government agencies, researchers, clinical and professional organizations, accreditors, health systems and facilities, employer purchasers, consumer coalitions, and commercial organizations. Information about the measures' clinical rationale and evidence base was developed by the Center for Quality Assessment and Improvement in Mental Health (CQAIMH), with funding from AHRQ.

The inventory can be searched by the following specifications:

• Diagnosis, including major depressive disorder, personality disorders, schizophrenia, and substance abuse dependence.
• Special populations, including child/adolescent and dual diagnosis.
• Data source, such as administrative claims, medical record, and survey.
• Evidence level, which is a rating by AHRQ as to the level of evidence.
• Treatment, including medication, psychotherapy, and case management.
• Domain of quality, such as access, prevention, and treatment.
• Clinical setting, such as inpatient, outpatient, home, and community.

Results of an indicator search of the database include a description of the measure, appropriate data sources to use for the numerator and denominator of rates, and the developer of the measure.

Child and adolescent measures include:

• Access to child specialty care for depression.
• Family involvement in attention-deficit hyperactivity disorder (ADHD).
• Stimulant medication treatment for ADHD.
• Antipsychotic treatment for childhood psychosis.
• Completion of treatment for substance abuse.
• Referral to post-detoxification services.

Online Resource: For more information on CQAIMH, including a guide to using the database, a directory of measures, and project publications, go to the CQAIMH Web site at http://www.cqaimh.org/index.html

Return to Contents