Conducting a Web-based Survey

User's Guide: Hospital Survey on Patient Safety Culture

As mentioned earlier in this guide, current research and evidence shows that Web-based surveys have typically lower response rates than paper-based surveys (Groves, 2002). It is important to reiterate that low response rates will limit your ability to generalize your results. However, because Web-based surveys do have certain advantages, your hospital may be considering this type of approach. To help you decide which approach is best suited to your situation, or if a combination approach is warranted, this chapter presents the pros and cons of conducting a Web-based survey. The chapter also outlines special considerations that need to be taken into account and presents guidelines that will help you make the most of a Web-based survey, should you decide to take that approach. 

A major factor, of course, is cost. While the costs of a Web-based survey may seem less because there are no printing, postage or data entry expenses, do not overlook the labor costs associated with Web survey programming and testing. At the same time, a Web-based approach generally tends to be more economical as the survey sample size becomes larger. Surveys sampling only a few hundred individuals are likely to be more cost-effective using a paper-based survey approach. Cost, however, is but one of the many factors that need to be considered in deciding which approach to take.

Consider the Pros and Cons of Web-based Surveys

There are a number of pros and cons to conducting Web-based surveys. The relative weight given to each of these advantages and disadvantages, and the final decision on whether to conduct a Web survey, will be determined by your hospital's specific circumstances, capabilities, resources, and goals.

The primary advantages to Web-based surveys are:

  • Simpler logistics. Web-based surveys can be virtually paperless, making them easier in some ways to manage. There are no surveys to print; no handling of letters, labels, envelopes, or postage; and there are no completed paper surveys to manage.
  • No need for data entry and minimal need for data cleaning. Web-based surveys typically are programmed to prevent invalid responses. Moreover, the responses are automatically copied to a database, so the need for separate data entry is eliminated and the need for data cleaning is greatly reduced.
  • Potential for faster data collection. While not always the case, Web-based surveys can facilitate shorter data collection periods. Web-based surveys involving E-mail notification and followup correspondence are received immediately after being sent, so the time interval between survey administration steps often is reduced.

There also are several disadvantages to Web surveys:

  • Time and resources needed for development and testing. Time and resources are needed to program a Web-based survey so that it meets acceptable standards of functionality including: usability requirements, log-in usernames and/or passwords, and the convenience of allowing respondents the option of saving their responses and returning later to finish the survey. Of equal importance are security safeguards for protecting the data. In addition, the Web-based survey must be pretested thoroughly to ensure that it works properly and that the resulting data set is established correctly.
  • Limited access to the Internet or E-mail. A Web-based survey should be accessible to all the individuals in your sample group. Barriers to Internet service and E-mail accessibility issues will lead to poor response rates. Many hospitals have only a limited number of Internet-connected computers. If computers are located centrally, staff may be concerned about the privacy of their responses. In addition, all staff may not have E-mail access or may not access their E-mail regularly. In such cases, E-mail notification or E-mail messages with hyperlinks to the survey Web site may not be effective instruments for getting respondents to complete the survey.
  • Individual differences in computer and Internet use. The intensity of computer and Internet usage is the most important predictor of cooperation in a Web-based survey (Groves, 2002). There are likely to be staff among your sample group who are not computer or Internet savvy, and, therefore, may not respond to the survey if this is their only means of accessing the survey.

Return to Contents

Design and Pretest the Web-based Survey

If you decide after weighing the pros and cons of conducting a Web-based survey that this is the approach your hospital will take, there are a number of Web survey design aspects to consider. If your hospital plans to use off-the-shelf commercial software, rather than having a vendor design and develop a custom Web-administered survey, assess the various software applications available to you and make your selection on the basis of capabilities and which product best handles the many features and recommendations we outline below.

Web-based Survey Design Features

While research on the best ways to design Internet-administered surveys continues to evolve, current knowledge suggests that the following are elements of a good Web-based survey:

  • Do not force respondents to answer every question. Permit respondents to continue completing the survey after choosing not to answer a particular question. Forcing respondents to answer each question before being allowed to move on to the next question is something that not only annoys respondents, but is not advisable on the Hospital Survey on Patient Safety Culture because some respondents may have legitimate reasons for not answering an item. Forcing a response would cause them to make a wild guess, rather than an informed answer.

    It may be desirable, however, to establish a minimum number or percentage of completed items in judging a survey "complete." You may not want respondents to start the Web-administered survey and submit their final survey answers after completing only a few items, particularly if you have promised an incentive of some type for "completing" the survey. Program a certain number of responses or a percentage of the total items as a minimum number to be completed, before allowing respondents to submit their final answers (50% complete would be a good starting point, but you could set your cutoff higher). If the number of completed items falls below your cutoff minimum when respondents try to submit their data, have a message inform them that they must complete at least "XX %" of the items to be eligible for the incentive. They can then choose to "save and exit" (if you provide the option for respondents to reenter the survey) and complete the minimum required number of items at a later time, or they can choose to go ahead and "submit" their data with the knowledge they not be eligible for the incentive. In either case, respondents should be given the option of submitting the incomplete data.

  • Provide respondents with a means to assess their survey progress. Because it is difficult to know the length of a Web-based survey, it is helpful for respondents to have some type of indicator showing their overall progress in the survey, particularly for a relatively short instrument like the Hospital Survey on Patient Safety Culture. For example, there could be a graphical progress bar that indicates completion percentages at various points, for example "Survey is 50% complete." Other options include programming the survey as one scrolling page, or allowing respondents to move forward and backward through a multiple-page format at their convenience, so they may view the entire length of the survey. If a multiple-page format is used, however, avoid using an extreme one-question-per-page design.
  • Include username and/or password protection (Optional). Unless access is restricted in some way, Web sites are accessible to the public. Your survey Web site can be restricted through the use of a password that is common to all users or groups of users, or through the use of individual usernames and/or passwords (which requires the use of confidential identifiers to link individuals to usernames/passwords). While the survey may be published to part of a restricted company or organization intranet, respondents will be able to complete the survey more than once unless individual passwords and/or usernames are established. Screening questions also can be developed to prevent individuals from participating in the survey multiple times, in the event usernames and/or passwords are not used. The use of usernames and/or passwords is best accomplished in conjunction with E-mail survey notifications using hyperlinks to the survey Web site. This enables respondents to easily copy and paste their username and/or password directly from the E-mail. Linking individuals to usernames and/or passwords will complicate the Web development and administrative aspects of the project.
  • Allow respondents to interrupt their session, save their answers, and complete the survey at a later time (Optional). Although it takes only 10 to 15 minutes to complete the Hospital Survey on Patient Safety Culture, respondents may get interrupted while in the middle the survey and they will not want to readdress parts of the survey they have already completed. If they choose to leave their Internet browser open and the survey idle until they can come back to it, the respondent may get "timed-out" of their Internet connection and their responses will be lost. To encourage the respondent to complete the survey at a later time, the stopping point in the survey must be bookmarked and the completed items have to be stored in computer memory. Provisions must be made in the programming to allow an individual to re-use the same identification username and/or password that was established at the initial login to again access the site at a later time for the purpose of completing the survey. The "save and exit" feature should be accessible at any point in the survey, but the "submit responses" option should be available only at the end of the survey.
  • Allow respondents to print a hard-copy version of the survey and complete it on paper (Optional). Some respondents will prefer to complete a paper version of the survey, and providing this option may boost your response rate. It is possible to design your Web-based survey so it can be printed in paper form, but this functionality must be tested thoroughly to ensure that it prints properly on different printers. Attention must be given to line lengths and page lengths in the design of the survey page template. Moreover, instructions must be provided so the respondents will know where to return the completed paper surveys, and designated personnel then must enter the responses into your data set (paper survey data can be entered via the Web site).
  • Thoroughly pretest the survey (Essential and Mandatory). Conduct thorough pretests of the survey using low-end computers with slower Internet connections, with various Internet browsers (different iterations of Netscape Navigator® and Internet Explorer®), and with different display settings (screen resolutions set at 800 x 600 pixels versus 1,152 x 864 pixels), etc. This must be done to ensure the survey appears and performs as it should, despite the different settings and personal preferences selected on individual computers.

Return to Contents

Develop a Web-based Data Collection Plan

A Web-based survey data collection plan is very similar to a paper-based data collection plan in its basic steps. Refer to Sections 4 and 5 to identify those elements central to your data collection methods, and for those collection procedures common to Web-based and paper-based surveys. Rather than reiterate all the necessary data collection steps in this section, we have chosen to highlight various steps and identify strategies for conducting those steps that are unique to Web-based surveys, while offering advice on the best approaches.

A Combination of Web- and Paper-based Survey Methods

If you desire to use a combination of Web-based and paper survey approaches, it is most economical to first implement the Web-based survey. Later, you can distribute paper surveys to those members of the sample group who did not respond to the Web-based instrument.

Prenotification

Prenotification is correspondence used to notify staff that they have been included in a sample and are being asked to complete a Web-based survey. Prenotification letters can be sent electronically, via E-mail, which requires an up-to-date list of the E-mail addresses for those individuals in your sample group. Alternatively, printed letters can be distributed through internal hospital mail on letterhead signed by the hospital CEO or president. The main criterion in deciding which prenotification method to use is staff E-mail use (e.g., whether staff in your hospital sample all have access to E-mail and read it regularly). If E-mail use is uneven, it is best to distribute a hard copy prenotification letter through the internal hospital mail. Overall, we recommend doing prenotification with a hard copy letter-even in conjunction with Web-based survey data collection-because it is another tool for capturing the respondents' attention. E-mail is then used to direct the sample group to the survey instrument. The message should contain a hyperlink to the Web site containing the survey form and individual usernames/passwords, if applicable.

To further boost response rates, it is advisable to personalize the prenotification letters or E-mails (i.e., addressed to each respondent, using their first and last name). If E-mail notification is used, the name or E-mail address in the "From" line should be easily recognizable to staff to prevent them from mistaking your E-mail for spam and deleting it. For example, you might use the title and name of the hospital CEO, or another recognized staff executive, to ensure the E-mail gets opened and read (FROM: "CEO Joe Smith, with Hospital X").

Followup

Followup steps improve response rates for Web-based surveys in the same way they help with paper surveys (Groves, 2002). It is important to follow up with nonrespondents in a timely manner to ensure the data collection period does not drag on for too long.

If you have the means to conduct all contact steps via E-mail, time intervals between followup steps can be reduced. Consider sending the first E-mail reminder 1 week after the survey Web site link has been E-mailed (rather than using a 2-week reminder, as is recommended with a paper survey). Include the hyperlink to the survey Web site in each E-mail reminder, along with the individual's username and/or password, if applicable. Then send a second E-mail reminder, 1 week after the first reminder. A third E-mail reminder can be sent the following week. Send E-mail reminders only to those who have not responded, or to those who chose to "save and exit" the survey, but have not returned to the Web site to complete the survey. Use a larger, colored font to make the heading of the reminder E-mail more noticeable, and ensure the text of the first and second reminder messages is slightly different, to capture the recipients' attention. If you have not used identifiers and have no way to determine which members of the sample group have completed the survey, then E-mail reminders must be sent to everyone. It is important in such cases to include a sentence thanking those who have already completed their surveys and asking them to disregard the reminder.

We are recommending a combination of printed reminders and electronic reminders-even for those with the capabilities to conduct all contact steps through E-mail—to ensure that at least one of the messages reaches each respondent, since individuals respond differently to various forms of communication. You may decide to send the first and second reminders via E-mail, followed by a final reminder postcard to be distributed to nonrespondents. The final reminder postcard could be printed on brightly colored card stock, thanking those who have responded for their help and asking those who have not responded to please complete the survey in the next 7 days.

If all followup reminders are printed on paper and sent through internal hospital mail, more distribution time will be needed between data collection steps. The followup steps for a Web-based survey are the same as those associated with a paper survey (Go to: Establishing Data Collection Procedures).

Return to Contents
Proceed to Next Section

Current as of September 2004
Internet Citation: Conducting a Web-based Survey: User's Guide: Hospital Survey on Patient Safety Culture. September 2004. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/professionals/quality-patient-safety/patientsafetyculture/hospital/userguide/hospcult6.html