Guidelines for Ensuring the Quality of Information Disseminated to the Public
C. Agency for Healthcare Research and Quality
Note: Last revised in 2006, these Guidelines also appear on the HHS Web site. AHRQ presents them here with a 2015 Update.
Contents
- Agency Mission.
- Scope and Applicability of Guidelines.
- Types of Information Disseminated by the Agency to the Public.
- Types of Dissemination Methods.
- Quality Assurance Policies, Standards and Processes for Ensuring the Quality of Information Disseminated to the Public.
- Agency Administrative Complaint Procedures.
- Influential Scientific, Financial and Statistical Information.
I. Agency Mission [Select for Update]
The mission of the Agency for Healthcare Research and Quality (AHRQ) is to develop scientific evidence that enables health care decision makers—patients and clinicians, those who manage and purchase health care services, and policy makers—to make more informed health care choices. AHRQ accomplishes its mission by conducting, supporting, and disseminating scientific research designed to improve the outcomes, quality, and safety of health care, reduce its cost, broaden access to effective services, and improve the efficiency and effectiveness of the ways health care services are organized, delivered, and financed.
The Agency was created December 1989 as the Agency for Health Care Policy and Research (AHCPR) and reauthorized on December 6, 1999, as the Agency for Healthcare Research and Quality. AHRQ is an agency of the Public Health Service (PHS) within the U.S. Department of Health and Human Services (HHS).
At AHRQ, the Director will have overall responsibility for implementing the AHRQ Information Quality Guidelines.
II. Scope and Applicability of Guidelines for Agency
AHRQ will ensure that disseminated information meets the standards of quality set forth in the OMB, HHS and AHRQ guidelines. It is AHRQ's goal to ensure and maximize the quality, objectivity, utility, and integrity of information that it disseminates to the public. AHRQ strives to provide information that is accurate, reliable, clear, complete, unbiased, and useful. It is committed to integrating the principle of information quality into every phase of information development, including creation, collection, maintenance, and dissemination.
The pre-dissemination review described in the guidelines only applies to information disseminated on or after October 1, 2002. The administrative mechanism for correction applies to information that the agency disseminates on or after October 1, 2002, regardless of when the agency first disseminated the information. The guidelines apply to information in all media (e.g., print, electronic) and apply primarily to the dissemination of substantive information (e.g., reports, studies, and summaries relating to the findings and related products of AHRQ research) rather than on information pertaining to basic Agency operations not intended for public dissemination.
Many AHRQ activities result in publicly disseminated information that is subject to the OMB Guidelines including:
- Statistical information and databases.
- Results of research projects.
- Program evaluations.
- Software tools.
- Consumer and professional health information.
The following types of information produced by the Agency are not covered: scientific information produced by extramural researchers who conduct and disseminate their own work in the same manner as their academic colleagues and whose views do not represent the views of the Agency; information in which the results are identified as not being Agency views or positions; archival information; information intended solely for intra-or-interagency use or sharing of government information; press releases that support the announcement or give public notice of information that AHRQ has disseminated elsewhere; correspondence with individuals; responses to requests for Agency records under the Freedom of Information Act (FOIA); public filings; subpoenas; and the findings and determinations that AHRQ makes in the course of adjudications involving specific parties.
III. Types of Information Disseminated by the Agency to the Public
Data products—Includes print and electronic materials that describe or present aggregate statistical information such as data from the Medical Expenditure Panel Survey (MEPS), a survey of households regarding how they use and pay for health care. MEPS Public Use Data Files allow access to aggregate data. Another example is the Healthcare Cost and Utilization Project (HCUP), a family of health care databases and related tools with data provided by state data organizations, state hospital associations, and other sources. Web-based interactive analytic tools (i.e., MEPSnet, HCUPnet) are also available.
Results of research projects—Research reports and other information products such as fact sheets, research syntheses, and AHRQ's monthly newsletter Research Activities are produced in both print and electronic formats and describe the findings and methods of AHRQ-funded research projects. [Select for Update.]
Program evaluation reports—Reports summarizing the results of program evaluations are included in the departmental catalog of program evaluations maintained by the Office of the Assistant Secretary for Planning and Evaluation. Many of the Agency's program evaluations are driven by the goals established under the Government Performance and Results Act (GPRA).
Software tools—Tools developed to assist health care policy makers, health system leaders, purchasers, and others in carrying out their tasks and making health care choices. An example is the Quality Indicators software that ensures a standard, trusted approach to quality measurement to support improvements to patient care.
Consumer and professional health information [Select for updated paragraph]—AHRQ produces a wide range of print and electronic consumer and professional information. Examples include consumer publications on: various health conditions/diseases, on how to select a health plan, personal health guides, and how to improve the quality of health care you receive as a patient. Examples of publications for professionals include The Physicians Handbook of Preventive Services and evidence reports on a wide array of clinical topics that can be used by public and private entities to develop and implement their own practice guidelines, performance measures, review criteria, and other clinical quality improvement tools.
IV. Types of Dissemination Methods
AHRQ principally uses print and electronic methods of dissemination.
Print methods include:
- Reports, fact sheets, and research syntheses communicating the results of surveys and other research activities.
- Publication of Research Activities, AHRQ's monthly newsletter AHRQ [Select for Update].
Electronic methods include:
- The AHRQ Web site as well as AHRQ program specific sites.
- AHRQ's Electronic Newsletter.
Secondary methods of dissemination include, but are not limited to: workshops, audiovisual techniques, oral presentations, and teleconferences. These dissemination mechanisms largely draw on information developed and reviewed first through one of the primary print or electronic modes of information dissemination.
V. Agency Quality Assurance Policies, Standards and Processes for Ensuring the Quality of Information Disseminated to the Public
The quality assurance process begins at the inception of the information development process. AHRQ reviews the quality (including the objectivity, utility, and integrity) of information before it is disseminated and treats information quality as integral to every step of the development of information, including creation, collection, maintenance, and dissemination. AHRQ will demonstrate in its Paperwork Reduction Act (PRA) clearance packages that each draft information collection will result in information that will be collected, maintained, and used in a way that is consistent with OMB, HHS and AHRQ information quality guidelines.
Work performed by AHRQ is conducted in compliance with generally accepted professional standards appropriate to the field(s) of study being undertaken, e.g., clinical, statistics, economics, or qualitative health services research. Moreover, work produced by the Agency undergoes many levels of review before being finalized and disseminated to ensure quality, i.e., utility, objectivity (in substance and presentation), and integrity. Reviews are both hierarchical, i.e., performed by higher level supervisors and managers, and by peer experts both inside and outside of the Agency. Generally, it is the responsibility of AHRQ's Center Directors to ensure the utility, objectivity, and integrity of work produced within their research Centers.
The discussion that follows outlines the general quality standards and quality assurance procedures that AHRQ has in place. More detailed information is available on AHRQ's Web site and that information will be expanded to reflect new products (e.g. new databases) or revised over time to reflect advances in generally accepted professional standards.
Statistical information and database development
Quality standards: Statistical activities are carried out in accordance with generally accepted professional standards and modern statistical theory and practice, including scientific sampling, statistical inference and analytical techniques and practices. All statistical programs employ or have access to experts in statistics and research design.
Quality assurance procedures: Each of the Agency's data products undergoes a series of standardized quality assurance procedures to assure the validity, reliability, and consistency of the data. These procedures are posted on the AHRQ Web site. For example, the Medical Expenditure Panel Survey (MEPS), which involves data collection, uses quality assurance procedures such as the validation of interviewers' work, performance of quality control checks on the source variables across all analytic groups, and benchmarking the data (e.g. comparisons with prior years and comparable data sources such as the Current Population Survey of the U.S. Census Bureau and the National Health Interview Survey). The Healthcare Cost and Utilization Project (HCUP), which aggregates and standardizes existing data from states, has quality control procedures in place to verify that the databases contain information that is valid, internally consistent, and consistent with established norms. Procedures include automated edit checks, careful review of all summary statistics, and benchmarking data with other sources (e.g. comparison with the National Hospital Discharge Survey).
Before disseminating information in any format, data are reviewed internally to ensure that they are scientifically sound and meet or exceed standards for data quality, statistical integrity, and confidentiality and privacy protections. Similarly, description and documentation of databases, tools, products, and on-line query systems are carefully reviewed, edited, and re-reviewed before release to the public.
Results of research projects
Quality standards: AHRQ staff adhere to the highest professional standards in their appropriate/respective fields, e.g., economics, social science and statistics. They seek to create transparency in their activities to enable others to fully understand their research processes, methodologies, and assumptions so that the work can be evaluated and replicated. Staff must ensure that strong measures are in place to guarantee the integrity of research information and to protect it from unauthorized access or revision.
Quality assurance procedures: All intramural research projects—whether they involve qualitative or quantitative analysis—and all projects on which AHRQ staff collaborate with authors outside the Agency are subject to AHRQ's intramural research planning process. This is intended to ensure conformance with strategic planning, accountability, peer review, scientific objectivity, confidentiality, and privacy standards. Research reports that are the products of intramural research undergo many levels of review prior to release, including hierarchical and peer reviews (sometimes including external reviews by other agencies and outside experts). Manuscripts submitted to scientific journals are subject to additional peer-review by the journal, prior to acceptance and publication.
In keeping with widely-accepted scientific research practices, research reports published by AHRQ describe the methods, data sources, analytical techniques, assumptions, and limitations of the research so the study can be substantially assessed and replicated.
Program Evaluations
Quality standards: Program evaluations are developed and assessed in accordance with the highest professional standards of evaluation practice.
Quality assurance procedures: Evaluation proposals are reviewed by Agency management for suitability, utility, and methodology, in accordance with sound evaluation design and standards of evaluation practice. Many evaluation projects have technical advisory committees that oversee the design and conduct of the evaluation in accordance with standard evaluation theory and practice; they often provide an expert review of the final report. Results of evaluation activities are released to the public only after Agency management has completed a review of the quality, accuracy and completeness of the report.
Software tools
Quality standards: Software tools are developed by teams of subject area experts in compliance with generally acceptable professional standards appropriate to their respective fields. Many levels of review and testing are built into the development process to ensure that the final product is useful, objective, and of high quality.
Quality assurance procedures: Reviews of tools in development are stringent. To ensure that they are needed and useful, the Agency generally relies upon peer experts and user groups both inside and outside of the Department. The Agency conducts extensive testing prior to release. Feedback is obtained from users of the tools or software to identify enhancements that will make the tools or software more fully meet user needs.
Consumer and professional health information
Quality standards: Consumer and professional information developed by AHRQ is conducted in compliance with generally accepted professional standards appropriate to the field of study being undertaken, e.g., clinical, economics, statistics, and social science research, as well as sound health care communication principles.
Quality assurance procedures: AHRQ works closely with stakeholder groups to determine the need for consumer and professional publications. Publications are based on information emanating from research that has been reviewed for scientific and medical accuracy and completeness by experts within and outside of the Agency before it is disseminated. Consumer publications are tested with targeted audiences to ensure relevance, clarity, and comprehension. In addition, AHRQ's publications containing health care recommendations indicate the level of substantiating evidence.
VI. Agency Administrative Complaint Procedures
AHRQ has developed administrative mechanisms to allow affected persons to seek and obtain correction of disseminated information that does not comply with OMB, HHS and AHRQ guidelines. Persons seeking correction of AHRQ-produced information should follow the procedures below.
- Responsibility of the Complainant
To seek a correction of information disseminated by the Agency, complainants should follow the procedures described below.
Complaints or requests for review and correction of information shall be in written hard copy or electronic form. They shall be sent to the Agency by regular or electronic mail. Letters (written hard copy) should be sent to:
Office of the Director
5600 Fishers Lane
Mailstop 06E37A
Rockville, MD 20857
Electronic mail should be sent via the AHRQ website form. Complaints should state "Information Quality Concern" in the Subject. The body of the complaint shall state that a request for correction of information is being submitted.
Complaints shall contain a detailed description of the specific material that needs to be corrected including a detailed citation, e.g., the publication title, date, and publication number, if any, or the Web site and web page address (url). Complaints should describe the specific reasons for believing the information does not comply with OMB, HHS, or AHRQ guidelines and is in error and supporting documentation, if any. In addition, they should include specific recommendations for correcting the information; a description of how the person submitting the complaint is affected by the information error; and the name, mailing address, telephone number, email address, and organizational affiliation, if any, of the individual or organization making the complaint. Requesters should be aware that they bear the "burden of proof" with respect to the necessity for correction as well as with respect to the type of correction they seek.
In determining whether and how to address a request, the Agency will consider such factors as the Agency's budget, resources, and priorities, as well as the complexity of the correction task itself. Also, it needs to be understood that it may not be in the public interest for the Agency to devote significant resources to correcting information where the expenditure of such resources is not, in the Agency's view, cost effective in light of the significance of the asserted error, the benefits that are likely to be derived from such a correction, the costs of the correction, and the Agency's more pressing priorities and obligations. -
Responsibility of the Agency
Based on a review of the information provided, the Agency will determine whether a correction is warranted and if, so what action to take. The Agency will respond to the requestor by letter or email. The Agency's response will explain the findings of the review and the actions that the Agency will take, if any. The response will consider the nature and timeliness of the information involved and such factors as the significance of the correction on the use of the information and the magnitude of the correction. The response will describe how the complainant may request reconsideration. The Agency will respond to all requests for correction within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the Agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.
- Appeals
If the individual submitting the complaint does not agree with the Agency's decision (including the corrective action, if any), the complainant may send a written hard copy or electronic request for reconsideration within 30 days of receipt of the Agency's decision. The appeal shall state the reasons why the Agency response is insufficient or inadequate. Complainants shall attach a copy of their original request and the Agency response to it, clearly mark the appeal with the words, "Information Quality Appeal," and send the appeal to the addresses for hard copy or electronic submissions identified above.
The Agency will respond to all requests for appeals within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the Agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.
The Agency official who resolved the original complaint will not have responsibility for the appeal.
VII. Influential Scientific, Financial and Statistical Information
From time to time, AHRQ disseminates influential information, including statistical information, as defined in the OMB Guidelines. By "influential," OMB is referring to information "that the agency can reasonably determine...will have or does have a clear and substantial impact on important public policies or important private sector decisions." In accordance with standard Agency quality procedures, care is taken to ensure that information disseminated is substantially reproducible and replicable. This goal is accomplished by using reliable data sources and sound analytical techniques, and by employing a high degree of transparency about the data, methods, measures, assumptions and limitations used to develop the information in order to facilitate reproducibility by qualified third parties.
Because of ethical and feasibility constraints as well as confidentiality and legal obligations to third parties supplying information, there may be instances where not all of the original or supporting data may be available to the public. In such instances, AHRQ will promote understanding and transparency by clearly identifying, to the extent practicable, the resulting research methodologies and assumptions in reference documents.
