Survey User's Guide
Chapter 5. Conducting a Web-Based Survey
We strongly recommend that medical offices administer the Medical Office Survey on Patient Safety Culture as a paper-only survey because medical office staff access to email and the Internet, as well as staff computer skills, may be very limited. Current research and evidence show that Web-based surveys typically have lower response rates than paper-based surveys (Groves, 2002; Shih and Fan, 2008). This general finding seems to hold true in health care settings. For example, as reported in the AHRQ Hospital Survey on Patient Safety Culture 2008 Comparative Database Report (Sorra, Famolaro, Dyer, et al., 2008), the average response rate for hospitals that administered the survey by paper was highest (60 percent); followed by mixed mode using Web and paper (52 percent); and Web only (44 percent). We believe it is likely that similar differences in response rates by survey mode will occur with the Medical Office Survey on Patient Safety Culture. Response rates are important because low rates may limit your ability to generalize your results to your entire medical office.
If your medical office has never conducted a Web-based survey, you will need to allow for the extra time and resources that are usually needed when conducting a Web-based survey for the first time. However, if you are going to use a vendor or have a history of conducting Web-based surveys and you are familiar with their advantages and disadvantages, your medical office may want to consider this type of approach. To help you decide which approach is best suited to your situation, or if a combination approach is warranted, we present some of the pros and cons of conducting a Web-based survey. We also outline special considerations that need to be taken into account and present guidelines that will help you make the most of a Web-based survey, should you decide to take that approach.
A major factor, of course, is cost. Although the costs of a Web-based survey may seem lower because there are no printing, postage, or data entry expenses, do not overlook the labor costs associated with Web survey programming and testing. At the same time, a Web-based approach generally tends to be more economical as the survey sample size becomes larger. Surveys of only a few hundred individuals are likely to be more cost-effective using a paper-based survey. Cost, however, is just one of the many factors that need to be considered in deciding which approach to take.
There are a number of pros and cons to conducting Web-based surveys. The relative weight given to each of these advantages and disadvantages, and the final decision on whether to conduct a Web-based survey, will be determined by your medical office's specific circumstances, capabilities, resources, and goals.
The primary advantages of Web-based surveys are:
- Simpler logistics. Web-based surveys can be virtually paperless, making them easier in some ways to manage. There are no surveys to print; no letters, labels, envelopes, or postage to handle; and no completed paper surveys to manage.
- No need for data entry and minimal need for data cleaning. Web-based surveys typically are programmed to prevent invalid responses. Moreover, the responses are automatically copied to a database, so the need for separate data entry is eliminated and the need for data cleaning is greatly reduced.
- Potential for faster data collection. Although not always the case, Web-based surveys can facilitate shorter data collection periods. Web-based surveys involving E-mail notification and followup correspondence are received immediately after being sent, so the time interval between survey administration steps often is reduced.
There also are several disadvantages to Web-based surveys:
- Time and resources needed for development and testing. Time and resources are needed to program a Web-based survey so that it meets acceptable standards of functionality, including usability requirements, log-in usernames and/or passwords, and the convenience of allowing respondents the option of saving their responses and returning later to finish the survey. Of equal importance are security safeguards for protecting the data. In addition, the Web-based survey must be pretested thoroughly to ensure that it works properly and that the resulting data set is established correctly.
- Limited access to the Internet or E-mail. A Web-based survey should be accessible to all the individuals in your survey population. Barriers to Internet service and E-mail accessibility issues will lead to poor response rates. Many medical offices have only a limited number of Internet-connected computers. If computers are located centrally, staff may be concerned about the privacy of their responses. In addition, all staff may not have E-mail access or may not access their E-mail regularly. In such cases, E-mail notification or E-mail messages with hyperlinks to the survey Web site may not be effective instruments for getting respondents to complete the survey.
- Individual differences in computer and Internet use. The intensity of computer and Internet usage is the most important predictor of participation in a Web-based survey (Groves, 2002). There are likely to be staff among your sample group who are not computer or Internet savvy and therefore may not respond to the survey if this is their only means of accessing the survey.
If you decide after weighing the pros and cons of conducting a Web-based survey that this is the approach you will take, there are a number of Web survey design aspects to consider. If you plan to use commercial off-the-shelf software, rather than having a vendor design and develop a custom Web-administered survey, assess the various software applications available to you and select the product that best handles the many features and recommendations we outline below.
Web-Based Survey Design Features
Although research on the best ways to design Internet-administered surveys continues to evolve, current knowledge suggests that a good Web-based survey has the following elements.
Do not force respondents to answer every question. Allow staff to continue completing the survey after choosing not to answer a particular question. Forcing respondents to answer each question before being allowed to move on to the next question can annoy respondents. It also is not advisable on the Medical Office Survey on Patient Safety because some respondents may have legitimate reasons for not answering an item. Forcing a response may cause them to make a wild guess, rather than provide an informed answer. In addition, you will want the Web version to be similar to the paper version which does not require an answer to every question.
Provide respondents with a way to assess their survey progress (optional). Because it is difficult to know the length of a Web-based survey, it is sometimes helpful for respondents to have some type of indicator showing their overall progress in the survey. However, for a relatively short instrument like the Medical Office Survey on Patient Safety, we think a progress indicator is optional. Nevertheless, if you prefer to use one, there are several ways to indicate progress in completing the survey. For example, you could use a graphical progress bar that indicates completion percentages at various points, such as "Survey is 50% complete." Other options include allowing respondents to move forward and backward through a multiple-page format at their convenience so that they can view the entire length of the survey. If a multiple-page format is used, however, avoid using an extreme one-question-per-page design.
Include username and/or password protection (optional). Unless access is restricted in some way, Web sites are accessible to the public. Your survey Web site can be restricted through the use of a password that is common to all users or groups of users or through the use of individual usernames and/or passwords. Use of individual identification requires confidential identifiers to link individuals to usernames and passwords. The use of passwords and/or usernames is also an effective way to ensure that respondents will not be able to complete the survey more than once, even when the survey is published to part of a restricted company or organization Intranet. We recommend providing usernames and/or passwords and hyperlinks to the survey Web site in all your E-mail survey notifications. Respondents will be able to click directly on the hyperlink, then copy and paste their individual username and/or password directly from the E-mail.
Allow respondents to print a hard-copy version of the survey and complete it on paper (optional). Some respondents will prefer to complete a paper version of the survey, and providing this option may boost your response rate. It is possible to design your Web-based survey so that it can be printed in paper form, but this functionality must be tested thoroughly to ensure that it prints properly on different printers. Attention must be given to line lengths and page lengths in the design of the survey page template. Alternatively, you can include a link to a portable document file (PDF) version of the survey on the Web site. With either alternative, respondents will need instructions to know where to return the completed paper surveys. Designated personnel then must enter the responses into your data set (paper survey data can be entered via the Web site). Also, if you use individual identifiers there should be a way to include the identifier on the printed version of the survey or otherwise identify the paper response.
Thoroughly pretest the survey (essential and mandatory). Conduct thorough pretests of the survey using low-end computers with slower Internet connections, various Internet browsers (e.g., with different iterations of Internet Explorer, Safari, Mozilla, Opera), different display settings (screen resolutions set at 800 x 600 pixels versus 1152 x 864 pixels), and so forth. Pretesting will help to ensure that the survey appears and performs as it should, despite the different settings and personal preferences selected on individual computers. For more information on Web survey design principles and pretesting, go to Dillman (2007).
A Web-based survey data collection plan is very similar to a paper-based data collection plan in its basic steps. Rather than reiterate all the necessary data collection steps in this section, we have chosen to highlight various steps and strategies unique to Web-based surveys, while offering advice on the best approaches.
A Combination of Web- and Paper-Based Survey Methods
If you prefer to use a combination of Web-based and paper survey approaches, it is most economical to implement the Web-based survey first. Later, you can distribute paper surveys to those members of the sample group who did not respond to the Web-based instrument.
Web-Based Task Timeline
For a Web-based survey, use the task timeline in Figure 2. Plan for at least 6 weeks from the beginning of the project to the end of data collection. You will need to add several weeks in the preparation and planning stage before beginning data collection to design and program the survey for the Web and to pretest to ensure that the Web version works properly. Depending on the sizes of your medical offices and the ability to send E-mail reminders about the survey, the data collection period for a Web-based survey may be shorter than for a paper survey.
For a Web-based survey, we recommend e-mailing staff a prenotification letter telling them about the upcoming survey and alerting them that they will soon receive an invitation to complete this Web-based survey. Prenotification messages are easy to prepare and inexpensive to deliver in most Web-based surveys. Also, because E-mail messages can be easily overlooked in a crowded inbox, the prenotification message will help to alert providers and staff to expect the upcoming survey invitation e-mail. Prenotification messages do require an up-to-date list of the e-mail addresses for those individuals in your survey population.
To boost response rates, we recommend personalizing any e-mail prenotification messages (i.e., addressed to each respondent, using the person's first and last name). If e-mail notification is used, the name or e-mail address in the "From" line should be easily recognizable to staff to prevent them from mistaking your e-mail for spam and deleting it. For example, you might use the title and name of the medical office, practice or system senior executive leader, or another recognized staff executive, to ensure that the e-mail gets opened and read (FROM: Dr. Joe Smith, with "X," or FROM: Jane Smith, CEO, with Health System "X").
As we recommended with paper-based surveys, you should also publicize the survey by posting paper flyers in the medical office and posting messages on office or health care system Intranet sites. In addition, you can promote the survey during staff meetings.
Followup steps improve response rates for Web-based surveys in the same way they help with paper surveys (Groves, 2002). It is important to follow up with nonrespondents in a timely manner to ensure that the data collection period does not drag on for too long.
If you have the means to conduct all contact steps via e-mail, time intervals between followup steps can be reduced. Consider sending the first E-mail reminder within a few days after the survey Web site link has been E-mailed (rather than using a 1-week reminder, as recommended with a paper survey). Include the hyperlink to the survey Web site in each E-mail reminder, along with the individual's username and/or password, if applicable. Then send a second e-mail reminder 1 week after the first reminder. A third E-mail reminder can be sent the following week. Use a larger, colored font to make the heading of the reminder E-mail more noticeable, and ensure that the subject lines and texts of the first and second reminder messages are slightly different, to capture the recipients' attention.
If you have used individual identifiers and can determine which providers and staff have completed the survey, you can send reminder notices only to nonrespondents. However, if you did not use identifiers, reminders must be sent to everyone. It is important in such cases to include a sentence thanking those who have already completed their surveys and asking them to disregard the reminder.
We recommend using a combination of printed reminders and electronic reminders—even for those who can conduct all contact steps through E-mail—to ensure that at least one message reaches each eligible provider and staff member, since individuals respond differently to various forms of communication. You may decide to send the first and second reminders via E-mail, then distribute a final reminder card or letter to nonrespondents. The final reminder card can be printed on brightly colored card stock, thanking those who have responded for their help and asking those who have not responded to please complete the survey in the next 7 days.
If all followup reminders are printed on paper and distributed to staff, more distribution time will be needed between data collection steps. The followup steps for a Web-based survey are the same as those associated with a paper survey (go to Chapter 4: Establishing Data Collection Procedures).