Chapter 4. Using Web-Only and Mixed-Mode Surveys

Pharmacy Survey User's Guide

In this chapter, we suggest ways to publicize your survey, describe survey administration steps for Web-only and mixed-mode surveys, describe materials that need to be developed, and highlight important best practices in Web survey design and pretesting.

Publicize and Promote the Survey

As with paper surveys, we strongly recommend publicizing the survey before and during data collection. Be sure to advertise that the survey is supported by pharmacy, chain, or system leaders. Publicity activities may include:

  • Posting flyers or posters in the pharmacy, promoting the survey during staff meetings and shift changes, sending staff emails, and posting information about the survey on a pharmacy Intranet.
  • Promoting the survey during any meetings.
  • Having a senior leader or executive send a supportive email during data collection, thanking staff if they have completed the survey and encouraging others to do so.

Return to Contents

Follow Survey Administration Steps

We recommend the following basic data collection steps to achieve high response rates.

  1. Prenotifiction email. For a Web survey, if all or most staff have email access in the pharmacy, we recommend sending staff a prenotification email telling them about the upcoming survey and alerting them that they will soon receive an invitation to complete the Web survey. You will need an up-to-date list of staff email addresses.
  2. Survey invitation email. Send the survey invitation email a few days after sending the prenotification email. Include the hyperlink to the Web survey (or instructions for accessing the survey on the pharmacy Intranet), along with the individual’s password, if applicable. Provide instructions about whom to contact about technical problems accessing and navigating the survey.
  3. Followup communications. Send an email reminder a week after sending the survey invitation. In the message, thank those who have already completed the survey and encourage others to do so. Distribute a second reminder a week later. Consider sending a third email reminder to boost response as needed. Be sure to make the subject lines of followup email reminder messages slightly different to capture the recipients’ attention.

    If you use individual identifiers and can determine who has completed the survey, you can send email reminder notices only to nonrespondents. Otherwise, reminders must be sent to everyone. Be sure to thank those who have already completed their surveys and ask them to disregard the reminder.

    Because individuals respond differently to various forms of communication, we recommend using a combination of printed reminders and electronic reminders.

  4. Calculate preliminary response rates. Calculate a preliminary response rate at least once a week to track your response progress. To calculate preliminary response rates during data collection, divide the number of returned surveys (numerator) by the number of eligible staff who received the survey (denominator).

    Response rate = Number of surveys returned/Number of eligible staff who received a survey

    If any staff members end their employment during data collection, they are still considered eligible and should be included in the denominator even if they did not complete and return the survey.

    Note: At the end of data collection, you will need to adjust your preliminary response rate to reflect decisions made about whether a survey is complete or incomplete, or possibly ineligible. Chapter 5 discusses how to calculate the final official response rate for your pharmacy.

  5. Close out data collection.Keep in mind that your goal is to achieve a high response rate. If your response rate is still too low after distributing the second survey, add another week to the data collection period.

Survey Administration Steps for Mixed-Mode Surveys

Administer the Web survey first, followed by a paper survey.

  • Week 1: Carry out Web survey administration steps for the first week of data collection.
  • Week 2: Email or distribute a followup reminder.
  • Week 3: Distribute survey packets to all staff (or to nonrespondents only if using identifiers to track response). In the cover letter, tell staff to disregard the paper survey if they completed and submitted the Web survey.
  • Follow paper survey administration steps but continue the Web survey option. For followup reminders (if needed), you can use a mix of email and printed (or in-person) reminders.

Return to Contents

Develop Survey-Related Materials

The following materials will need to be developed in preparation for Web survey data collection.

Pharmacy/Chain/System Point-of-Contact Letter and Instructions

Send a letter to POCs describing the purposes of the survey and explaining their role in the survey effort. The letter should be on letterhead, signed by a senior executive. We also recommend that you provide the POCs with a simple data collection protocol that describes their tasks, along with a proposed timeline. (Appendix B presents a sample data collection protocol.)

Publicity Materials

Your publicity materials can help legitimize the survey effort and increase your response rate by including some or all of the following types of information:

  • Endorsements of the survey from your pharmacy leaders.
  • Clear statements about the purpose of the survey, which is to assess staff attitudes and opinions about patient safety in their pharmacy.
  • Description of how the collected data will be used to identify ways to improve patient and medication safety.
  • Assurances that only summary (aggregated) data will be reported, thus keeping individual responses confidential.
  • Introductions to the survey vendor, if you have chosen to use a vendor.
  • Contact information for the designated POC in the pharmacy.

Prenotification Email

We recommend the following for the prenotification email to help boost survey response:

  • Have it signed by a senior pharmacy leader or executive.
  • Use a name or email address in the “From” line that will be easily recognizable to staff to prevent them from mistaking your email for spam and deleting it.
  • Include the following points in your message:
    • Statement that in a few days the person will receive an invitation from [XXX] to participate in a brief survey on patient safety in the pharmacy.
    • Statement about the purpose and intended use of the survey and the importance of responding.
    • Confidentiality or anonymity assurances.
    • Introduction to survey vendor (if applicable).

Survey Invitation

The survey invitation email should also be signed by a senior pharmacy leader or executive. We recommend providing hyperlinks to the Web survey in your invitation email and any email reminders. Respondents will be able to directly select the hyperlink. You may also provide passwords for beginning the survey. If the survey is located on the pharmacy Intranet, provide instructions for accessing the survey. The survey invitation message should include the following information:

  • Brief restatement of why the pharmacy is conducting the survey, how it will use the data, and why the staff member’s response is important.
  • How much time is needed to complete the survey.
  • Assurances that the survey is voluntary and can be completed during work time.
  • Assurance of individual anonymity (if no individual identifiers are used) or confidentiality of response (if individual identifiers are used).
  • Suggested reply timeframe (but do not put a deadline date on the survey itself in case data collection is extended).
  • Incentives for survey participation (optional).
  • Contact information for the pharmacy POC (and chain/system-level POC, if applicable).
  • Contact information if someone other than the POC will handle questions about possible technical problems with the survey.

Followup Reminder Notices

If needed to improve response, send email reminder notices a few days after data collection begins and again a week after that. If some staff do not have email access, you can prepare the reminders on a half-page of cardstock and distribute to staff. The reminder notice should ask them to please complete and return their surveys and should include a thank you to those who have done so already. If you use individual identifiers to track responses, you can email or distribute reminders to nonrespondents only.

Sample Survey Invitation Email (Web Survey)

You are invited to participate in an important survey. It is part of our pharmacy’s efforts to better address patient and medication safety. All staff in the pharmacy are being asked to complete this survey. Your participation is voluntary, but we encourage you to complete the survey to help us improve the way we do things in this pharmacy. It will take about 10 to 15 minutes to complete and you may take it during work time. Your individual responses will be kept anonymous [say confidential if you use respondent identifiers]. Only group statistics, not individual responses, will be prepared and reported.

To access the secure survey Web site, click on the following link: http://www.xxxxxxxx.

Then copy and paste (or enter) the following password to begin the survey: Xxxxxxxxx.

Please complete and submit your survey WITHIN THE NEXT 7 DAYS. [Optional incentive text: In appreciation for participation, staff will receive (describe incentive).]

Please contact [POC name and job position] if you have any questions about the survey [provide phone number and email address]. If you are having a technical problem with the survey, please respond to this email with a description of your problem or contact [Name, phone number].

Thank you in advance for participating in this important patient safety effort.

Electronic Signature of Senior Pharmacy Leader

Return to Contents

Design and Pretest Web Surveys

If you decide after weighing the pros and cons of conducting a Web survey that this is the approach you will take, you will need to consider a number of Web survey design elements. If you plan to use commercial off-the-shelf software rather than having a vendor design and develop a custom Web survey, assess the various available software applications and select the product that best handles the many features and recommendations discussed below.

Web Survey Design Features

Although research on the best ways to design Web-administered surveys continues to evolve, current knowledge suggests that a good Web survey has the following elements.

Do not force respondents to answer every question. There are several good reasons for allowing staff to choose not to answer a particular question:

  • Forcing respondents to answer each question may annoy respondents and lessen their motivation to complete the survey.
  • Some respondents may have legitimate reasons for not answering an item. Forcing a response may cause them to make a wild guess, rather than provide an informed answer.
  • The Web version should be similar to the paper version, which does not require an answer to every question.

Decide on the number of questions on each Web page. The Pharmacy Survey on Patient Safety Culture includes an opening page with definitions and instructions as well as six sections, three of which are somewhat lengthy. Several options exist for Web page layouts of the survey:

  • Option 1: A single scrollable Web page for the full survey (would require extensive vertical scrolling).

    Using a single page allows respondents to see the entire questionnaire with little effort, but respondents may miss questions due to extensive scrolling. In addition, answers are not usually sent to the server until the full Web page is completed and the survey is submitted. If respondents break off while taking the survey, they will need to start again at the beginning when they return to the survey. Special scripting can be used to capture answers before final submission, but that feature may be disabled on the computers staff will use.

  • Option 2: One Web page for each section of the survey (would include scrolling vertically in larger sections of the survey).

    Using one Web page for each section of the pharmacy survey would greatly reduce the amount of vertical scrolling and allow capturing of responses at the end of each section, rather than at the end of the survey. The survey can be programmed to allow respondents to move back and forth across the Web pages before they submit their completed survey.

  • Option 3: Multiple Web pages to avoid vertical scrolling.

    This option eliminates vertical scrolling as a source of response error. On the downside, it may take respondents slightly longer to answer the survey because they have to visit more pages. Also, if respondents want to review an earlier answer in the same section, they may have to move backward a page or two to locate that answer.

We do not recommend that you format the pharmacy survey with one item per page. That format would increase the time it takes to complete the survey, and you may introduce undesirable response effects if related items in the pharmacy survey are widely separated from each other.

Also, never program the survey so that respondents must scroll horizontally to see parts of the survey. That format can be annoying and may contribute to response error if respondents overlook parts of the survey.

Make sure the response categories (e.g., Strongly Disagree, Disagree) always appear on the screen. Response errors may occur if staff cannot see the response categories when answering survey items. To prevent this problem, when you program the Web survey and the format requires respondents to scroll down, ensure that the response categories are repeated as frequently as needed so that respondents see them when answering every question.

Use a screen resolution of at least 800 by 600 pixels when programming and testing the Web survey because this issue is more problematic the larger the screen resolution. The following example shows you the right way to program the survey with vertical scrolling.

Sample survey Web page with response options repeated

For format option 3 (multiple Web pages with no vertical scrolling), repeat the Section head at the top of each page in that section. Add “Continued” to the Section head to indicate it is not the first page in the section, as shown in this example:

Sample survey Web page showing continuation of section

Provide respondents with a way to assess their survey progress (optional). For a relatively short instrument like the Pharmacy Survey on Patient Safety Culture, a progress indicator is optional. Nevertheless, you could use a graphic progress bar that shows completion or indicates completion percentages at various points, such as “Survey is 50% complete” or “Page 2 of 4.”

Allow respondents to print a hard-copy version of the survey and complete it on paper (optional). Some respondents will prefer to complete a paper version of the survey, and providing this option may boost your response rate. It is possible to design your Web survey so it can be printed in paper form, but test this functionality thoroughly to ensure that the survey prints properly on different printers. Attention must be given to line lengths and page lengths in the design of the Web survey pages.

Alternatively, you can include a link to a portable document file (PDF) version of the survey on the Web site. With either alternative, respondents will need instructions to know where to return the completed paper surveys. Designated personnel then must enter the responses into your dataset (paper survey data can be entered via the Web site). Also, if you use individual or pharmacy-level identifiers, there should be a way to include the identifier on the printed version of the survey or otherwise identify the paper response.

Thoroughly Test the Survey

It is essential to conduct thorough tests of the survey. When testing:

  • Use the same type of computer that will be available to staff taking the survey in your pharmacy. If you have more than one type of computer, or if staff will have the option to take the survey on a nonpharmacy computer, be sure to test with a range of computer types and include the lower end type with slower Internet connections.
  • Test the survey with various Internet browsers (e.g., different iterations of Internet Explorer, Safari, Firefox, Chrome, Mozilla, Opera), different display settings (screen resolutions set at 800 x 600 pixels versus 1280 x 800 pixels), and so forth.
  • After you have completed the first two testing steps, submit test survey responses to ensure that the Web survey is working properly and is easy to use.
  • Check the Web survey data output. For example, check to make sure that Does Not Apply/Don’t Know responses show up with a value of 9, not a value of 1 through 5. Also, make sure that the other responses (e.g., Strongly Disagree through Strongly Agree) have the correct 1 to 5 values. If the Web responses are miscoded, there is no way to correct the data set after the survey has been administered.

Testing will help to ensure that the survey appears and performs as it should despite the different settings and personal preferences that staff may use. For more information on Web survey design principles and survey testing, see Couper (2008) and Dillman, Smyth, and Christian (2009).

Return to Contents
Proceed to Next Section

Current as of March 2013
Internet Citation: Chapter 4. Using Web-Only and Mixed-Mode Surveys: Pharmacy Survey User's Guide. March 2013. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/professionals/quality-patient-safety/patientsafetyculture/pharmacy/toolkit/pharmsopsuserguide4.html