Level: Intermediate Velda Bartek (bartekva@us.ibm.com), Senior Software Engineer, IBM
Deane Cheatham (dcheath@us.ibm.com), Human Factors Engineer, IBM
01 Feb 2003 In this second part of the Remote Usability article, Pervasive Computing specialists Velda Bartek and Deane Cheatham share the experience gained by conducting a number of remote usability studies using application-sharing technology. The authors describe some of the lessons learned as they planned and conducted remote usability evaluations for software products. The first article provides a context for remote usability testing and describes the benefits and pitfalls of remote usability evaluations and the application-sharing tools that they evaluated.
With new technology tools, such as Lotus Sametime, usability specialists can now access geographically distant and specialized users. In the past, costs and travel constraints made it difficult to integrate these users into studies. With these remote application-sharing tools, you can reach such users and eliminate the costs associated with travel and loss of work time when working with remote or specialized users.
In this, the second installment of a two-part article, we share our experience gained by conducting a number of remote usability studies using the application-sharing technology. We present the experiences gathered and lessons learned during our remote usability studies so you can use this knowledge when planning and conducting your own remote studies.
Our experiences include a discussion of participant acceptance of this type of testing and the impact that it has on test results. Although the case study presented here is specific to middleware and suite software, the observations are useful to those who conduct usability studies for any kind of application. To illustrate our observations, we cover:
- A general set of goals and challenges when designing evaluations
- How to plan and recruit for remote evaluations
- How to prepare for the remote test
- How to conduct the session
- A look at the reactions to, and acceptance of, remote testing
- Assessment of participant acceptance
- Our conclusions and future direction
In the first article, we provided a context for remote usability testing by detailing and describing the benefits and pitfalls of remote usability evaluations and the application-sharing tools that we evaluated.
Again, we'd like to remind you that found that conducting remote studies has been a convenient and positive experience for both the tester and user.
Goals and challenges when designing evaluations
During the last 10 months, our team conducted nine usability evaluations, all focused on the installation, administration, system planning, and Web application-development support in three different products.
The methodologies employed in the evaluations included task analysis, design explorations of prototypes, and design validations (or usability tests). During the testing period, we also conducted surveys and phone interviews. By nature, these evaluations are intrinsically remote activities, so we do not describe them in this article.
We had three major goals in mind while designing our evaluations (all mentioned in part 1 of this article):
- To obtain feedback from the hard-to-reach specialists
- To obtain feedback from a worldwide set of participants
- To enable our geographically dispersed development team to observe and participate in sessions
These are probably the main considerations for any usability study of the remote testing variety.
To achieve these goals, we needed to include remote participants in our usability evaluations as well as local participants. Remote participants were physically remote from the facilitators at the time of the session. Local participants were those who are with the facilitators in a controlled usability lab setting.
In all, 42 of our 60 participants (or 70 percent of the participants) in our evaluations were remote. Local and remote participants were involved in six of our evaluations. Three of our evaluations involved only remote participants.
Planning and recruiting for remote evaluations
As we designed our evaluations, we considered whether using remote participants was logistically appropriate. First, we decided the type of activity that we were planning. In our case, it was:
- A task analysis for new support
- A design exploration of a prototype for new support
- Design validation (usability) tests using pre-beta level software
Activities that focus on the participant interacting with one or two applications are ideal for remote testing. (Keep in mind that launching new Help windows, dialogs, or browser windows may require that you change the focus of the application-sharing tool.)
We realized that the time required to switch between several applications and sharing the applications would interfere with the participant's ability to complete tasks. Our remote design exploration sessions required only one change in applications -- from the existing application to the prototype.
The validation test of administration tasks only required sharing the administration console with the participant. Participants in the installation validation test only interacted with the install application. They did not perform any pre- or post-installation setup or configuration tasks.
When considering which activities to conduct remotely, we examined the types of things that participants would need to do. We chose not to conduct remote sessions for tasks that required users to perform some or all tasks on paper. Being remote, we would not be able to observe behavior, so we chose activities that required the user to interact with the application.
For a second consideration, we determined how long the evaluations would take. Based on experience, we decided to limit these activities to less than three hours. We felt that the types of evaluations and the duration was acceptable to, and appropriate for, remote participants.
After defining the evaluation methodology and determining whether it was appropriate for remote participants, we continued with the usual tasks associated with an evaluation: identifying participants and preparing materials such as background questionnaires, the test instrument, and feedback surveys.
We targeted companies that provided a variety of industries and sizes, and a range of product implementations.
As with all tests, we distributed background questionnaires to individual candidates to determine their skill and experience levels. In addition to these common screening activities, we also explained the remote testing procedure and verified that candidates were comfortable with participating remotely.
Before we move on to more of our direct observations, we should share some of our more general knowledge on the important role culture plays in designing, preparing, executing, and assessing remote testing sessions.
Preparing for the remote test
Once participants are identified, we schedule sessions at a mutually convenient time. Warning: This may not be a trivial task.
When scheduling sessions with non-US-based participants, arriving at a mutually convenient time that wasn't too early or too late for either the facilitators or the participant was not always easy. We wanted to schedule just the right amount of time between arrival and before the actual session started so we did not influence participant responses.
Another aspect to scheduling remote sessions is the need to schedule conference calls for the session's verbal communication. You may also need to schedule a remote online meeting. In our case, we have a set of remote meeting servers that are shared by all within our company. To conduct a remote meeting using the application-sharing tools, we schedule time on one of the servers.
In preparing materials, we considered that users would be remote. You can choose from several approaches for each aspect of an evaluation. For example, to solicit satisfaction feedback you could use:
- A Web survey that the participant completes after being told how to access the survey. The participant can take the survey either before or during the test.
- Verbal questioning of the participant during the test.
- A survey sent to the participant by e-mail that he could fill out and return by e-mail or fax.
All of these choices had associated advantages and disadvantages. We chose to send surveys to participants. Why? Participants are most familiar with giving feedback this way. Roughly 10 percent (4 out of a total of 42 participants) failed to return the survey materials. This is slightly higher than the number of local participants who chose not to complete sections of the surveys.
When conducting a remote test, we also considered whether to distribute materials for the test ahead of time or provide them during the session. The format of the design exploration was to review the existing support and then to demonstrate the new prototype (as a Flash movie). Because it was important to capture first impressions, we chose not to send the prototype ahead of time.
To ensure that we and the participant were viewing the same information, we shared the application to demonstrate the current support and the movie to demonstrate the proposed changes. With this format of the task analysis and validation test, we had the option to either send materials (scenarios) ahead of time or to progressively disclose the scenarios using a separate application-sharing window or Web page during the session.
When we sent materials ahead of time, participants had the option to print the materials and use them during the session. Experience taught us that local participants prefer using printed materials as they organize and arrange tasks and as they perform usability test scenarios. We were testing new support so sending the scenarios ahead did not affect the error or completion rates of participants. In all cases, we chose to send validation (usability) test materials ahead of time.
Another decision was whether to set up the prototype or software to be tested on our computer or send it to participants to set up on their computers. In making this decision, we considered the following:
- What are the consequences of giving up control of the software?
- Who receives the software?
- Will participants remember to delete the software or prototype at the end of the test as the confidentiality agreement requires?
- Can participants successfully set up the pre-beta level software and any pre-requisite software?
- Does the presence of this level of software cause a problem when participants later install the new product on their computers?
We required a considerable amount of time and effort from of each participant. In our experience, it takes a bit of baling wire, chewing gum, and duct tape to get prototype or early builds of software running. We asked ourselves: Is there a need to share this adventure with the participants?
To minimize the amount of time and problems that participants might face when setting up the software, we chose to install the software and prototypes on our own computers. Participants could then access this computer through the remote application-sharing tool.
Our preparation was similar to preparing for an evaluation with only local participant. We set up the application or prototype that was to be evaluated on our computer. The only additional task when testing with remote participants was to set up the application-sharing tool.
Conducting the remote session
Conducting a session with remote participants is similar to conducting a session with local participants. Participants stepped through prototypes, organized and grouped tasks, and completed task scenarios. Facilitators conducted the sessions. Development team members observed participants. And, sessions were taped on video.
The difference? We were in separate locations. To talk, we called participants on the phone. We and the participant started the application-sharing tool on our computers. To walk through prototypes or to complete scenarios, the participant accessed our computer.
Many of the problems that occur during a local participant session can also occur during a remote participant session. The software being tested repeatedly crashes. The network connection is slow. Evacuation drills occur. Participants fail to appear. (A tree may fall on a participant's house.)
As with any situation, try to foresee the problem, minimize its effect, and have a contingency plan (although if you plan for a tree crashing through a roof, you may be planning too much!). For example, if we were unable to contact the participant by phone, we tried again 10 minutes later. If the network was slowing application performance or there was a personal emergency, we rescheduled. To ensure adequate time for the evaluation, we planned for additional time at the end of the session in case the sessions ran long.
As with any software, problems may arise using the application-sharing tool. For remote tests, we scheduled time on one of our servers running the sharing tool. To ensure that the session started on time, we scheduled additional time prior to the beginning of the session to start the application being tested and the application-sharing tool.
To ensure participants can access your computer, consider scheduling a brief, 15-minute session with participants to test connectivity before the usability session. You can also verify that voice communications are acceptable.
Participants may not have set up their computer with the remote-meeting tool prior to the start of the session. On average, participants took 10 minutes to set up the application-sharing tool and join the remote-meeting session. To compensate for this lag, we budgeted extra time into the length of the session.
As an alternative, you can schedule a separate meeting with participants to verify that their computer is set up correctly. This technique is especially beneficial if you are conducting a walk-through or user-focus session with several customers. Rather than try to assist six participants at the same time and annoy those who are ready to start on time, you can work with each individually beforehand. Then your meeting can start on time and focus on the evaluation.
Another alternative is to send participants explicit instructions that step them through setting up their computer.
To minimize participant set-up problems, performance, or security concerns, you might have participants go to one of your company's offices. Your colleagues can assist them by ensuring that the remote application-sharing tool is set up correctly. Network connections and voice communications may be better there than that at the participant's location. Also, if you test particularly sensitive materials, your colleague may monitor the session to ensure that participants do not make copies of the test materials or software. Another advantage, your colleague can note the non-verbal feedback that you are unable to observe.
You might use the online chat capability of the application-sharing tool . Remote participants, who work in an open landscape office environment, might be more comfortable when they don't worry about being overheard. If you work with participants in another country, online chat can help minimize the effect of accents on comprehension, neutralize the differences between British and American pronunciations of English, and minimize any phone connection problems. And, it has the added benefit that some may be more comfortable reading and writing than speaking.
Reacting to and accepting remote sessions
Two critical considerations to keep in mind are how participants may react to the idea and the practice of evaluating products while being removed from the product, and how the facilitator reacts to the participants being elsewhere. A number of factors tied to these reactions can influence acceptance.
Familiarity and acceptance of remote meetings
The more familiar and comfortable that you and the participant feel working with others remotely and with using online meeting/application-sharing technology, the better the acceptance will be. You will need to rely on verbal communications to express ideas and to convey information without visual cues.
To become familiar and more comfortable with remote-meeting sessions, schedule brief opportunities with participants. As you walk them through a simple scenario, they (and you) become familiar with using the application-sharing tool and talking on the phone.
Familiarity and acceptance of online meeting technology
The participant and you also need to be comfortable with the environment of the test. You are giving up a controlled environment in which you can observe non-verbal feedback and keep disruptions and distractions to a minimum.
The advantage to remote testing? Participants are in their office without a test facilitator sitting next to them, offering as close to a real-world setting as possible. Participants work through scenarios and make decisions where they perform their day-to-day jobs. Phones and pagers may ring. People may interrupt. Loudspeaker announcements or construction noises may interfere.
Willingness to set up for the session
Participants may not set up their computer with the remote-meeting tool. We found that roughly 40 percent of our participants had not set up their computer prior to the start of the session.
To ensure participants are ready to go prior to the start of the session, you could set up a short meeting. We offered participants this option with only a few choosing to do this.
You can also plan for the set-up time in the session time. For those not set up, we allowed them to set up at the beginning of the sessions. Participants liked this approach. The separate meeting may cause the participant to have negative perceptions of the remote-meeting technology or of the facilitators.
Tolerance for system failures during the test
Sharing an application and being on the phone may test the participant's and your tolerance of failures of the application being tested. Keep in mind that if the application crashes, you may need to reboot and re-establish the remote-meeting session.
As a back-up, you could have a second computer with the application and remote-meeting software ready, thereby minimizing the recovery time. In our experience, remote participants were as tolerant as local participants with system failures. In other words, it's to be expected with prototypes and pre-beta level software.
Suitability of the activity for a remote test
When choosing an evaluation that involves remote participants, carefully consider the types of tasks that participants will perform.
We chose not to conduct remote sessions for tasks that required participants to perform some or all tasks on paper. Long silences on the phone could be awkward. And, we would be unable to observe participant actions.
We also chose scenarios that did not require frequent switching between the applications that were being shared. Switching can disrupt the participant's concentration.
Tolerance of the duration of the session
Another factor influencing suitability is the duration of the session. To determine the length of a session, we looked at our previous experiences and decided the following:
- On average, exploring and discussing early prototypes took about an hour.
- Completing scenarios and rating satisfaction took two or two and a half hours.
We also examined our experiences in remote meetings. We asked ourselves: "How long were we comfortable talking on the phone?" The answer: We found that keeping sessions under three hours works best.
More frequent breaks may be needed than when the participant is local. We relied on audio cues to determine when to take short breaks.
Tolerance of observers
A common aspect of usability testing is to have development team members observe participants. Remote testing may influence a participant's willingness to be observed.
Our lab test cells are designed with tinted glass separating the participants from observers. If the participants turn around, they can see any observers. Obviously, a participant cannot tell if they are being observed when removed from the facilitators.
When we recruit participants, we ask if they can agree to being observed. During a session, we always tell participants when observers are present and how many. We had one participant who was quite uncomfortable when unable to tell if someone was observing the session. He kept asking if any one was observing. When he participated in a later session, the participant was no longer concerned about observers.
The application-sharing tool provides a list of meeting attendees.
Our development team is dispersed throughout the world. Remote-meeting capabilities have the added advantage of enabling them to observe evaluation sessions.
Assessing participant satisfaction
To assess satisfaction with participating in remote usability evaluations, we surveyed 26 of our remote participants with 11 responding. All respondents had previous experience using online meeting tools and with participating in remote meetings.
Ten of the 11 indicated that it was easy to participate in remote usability tests and that they were satisfied with participating in remote usability sessions. These ten also indicated satisfaction from interacting with the product remotely as compared with in-person meetings.
In all cases, the eleventh participant rated these questions as "Neutral."
In addition to this survey, at the end of each session we ask participants if they would like to continue participating in usability activities. All participants, local and remote, expressed a willingness to continue participating. As proof of this willingness, 11 individuals in our participant pool of 29 individuals participated remotely in more than one evaluation. And, different individuals at six companies participated in different evaluations.
Evaluating the impact on usability test results
One question we had about using remote participants was the impact it would have on the feedback we receive. We used local and remote participants in six of our usability evaluations. In general, local and remote participants completed the same scenarios. The exception was the installation validation test. When dictated by installation environments, we counter-balanced scenarios between local and remote participants.
We assessed participant behavior during the session in the areas of think aloud, time on task, and completion and error rates. We also used question probes during the task scenarios.
After considering individual participant characteristics (such as quiet or extroverted), we found no difference between local or remote participants. Each took similar amounts of time to complete tasks and had similar error rates.
We also found that the amount and value of the comments provided as part of using the think aloud protocol were the same. All participants also responded to question probes in a similar manner.
Another area we assessed was satisfaction ratings. Again, based on observation and survey responses, we noted that being local or remote did not impact satisfaction ratings. As a result of these findings, we were comfortable in conducting tests with only remote participants.
The future of our remote testing efforts
We experienced the same type and number of procedural hiccups during remote participant sessions as during those with local participants. The advantages of obtaining participants who are hard to get or who are geographically dispersed far outweigh any minor inconveniences with recruiting, preparing, or conducting remote tests. Since participants experience little inconvenience during the tests, they are generally quite willing to participate.
Because work is not disrupted due to employee absence, companies are willing to allow employees to participate. And they are willing to allow employees with diverse skills to participate, so we achieve a larger pool of participants. As with local participants, remote participants like opportunity for an early look at new offerings.
Thanks to the relative ease and the reliability of remote testing, when appropriate, we will continue using remote participants in our usability evaluations.
Resources
-
IBM's Ease of Use site addresses the challenge of creating great user experiences through the discipline of User Engineering, supported by design guidelines, tools, services and other relevant materials.
- IBM uses a process called "User-Centered Design" to develop products, and they want you to participate.
- Check out the Lotus Sametime online meeting tool (includes a running demo).
- If you're a novice user, find methods for improved learning in "Usability engineering: Systematic methods to improve your user interface," by Dr. Jakob Nielsen (developerWorks, May 2001).
- Read Thomas Myer's article "Usability for component-based portals," which covers methodology on increasing usability in portal components (developerWorks, June 2002).
- Read Part 1 of this article for a context for remote usability testing by detailing and describing the benefits and pitfalls of remote usability evaluations and the application-sharing tools that the authors evaluated (developerWorks, January 2003).
About the authors  | | |
Velda Bartek is a User-Centered Design Specialist in IBM's Pervasive Computing Division. On weekends, she enjoys driving her mower around the yard. You can contact Velda at bartekva@us.ibm.com.
|
| | |
Deane Cheatham is a Human Factors Engineer in IBM's Pervasive Computing Division. She enjoys riding her white motorcycle on weekends. You can contact Deane at dcheath@us.ibm.com. |
Rate this page
| 
