Hello. The preparation of Guerilla Usability Testing as is the preparation to field
visits consist of selecting research questions to be answered in this particular study.
Deciding on the way of recording the study,
but instead of preparing notes,
the preparation towards usability testing includes developing a test plan.
There are different approaches to creating usability test plans that
vary in the effort needed to create them and in the resultant documents.
A formal approach implies development
a detailed test plan that can be freely transferred to a professional not
involved in the development of this plan so that
the professional will be able to successfully run usability tests.
Of course I'm talking about a person with
appropriate training and experience in usability evolution.
An informal approach implies creating a test plan for yourself,
not to forget anything while moderating the study sessions.
In this lecture, we will discuss an informal approach.
If you are interested in the formal one,
you can read about it in Jeff Rubin and Dana Chisnell's Handbook of Usability Testing.
And then the formal test plan consists of sections presented on the slide.
The core of each test plan is interview questions that precede and conclude test tasks,
interview questions that precede and conclude each task,
and of course test tasks themselves.
Before discussing each section from this structure in detail,
I'd like to point out one more time that you should know why
exactly you are going to ask participants these questions and give them these tasks.
In the previous lecture,
we were talking about sources of research questions.
They all apply to
usability testing as well as field visits to make the creation of test plan easier,
after you have selected research questions,
you might divide them in accordance with part of the user interface you will evaluate and
then write test tasks in questions
according to these research questions as is shown on the slide.
It will help you to make the whole study more focused.
By the way, you can find an example of a test plan for
mobile banking apps in the materials for this week.
In the first section,
contains a written description of recruitment criteria for each group of participants.
I'd like to remind you that groups of
participants do not always directly map to user groups.
Depending on research questions,
you may want to evaluate the usability of
product with a specific subpopulation of your users.
For instance, if you have reworked
huge part of an interface you will need to evaluate how
the changes affect the behavior of access to news and
as well as users who approached these part for the first time.
Once subpopulations can belong to one user group.
The question that begs for an answer within this section
of your test plan is how many participants do you need?
To test participant sample size depends on a number of
participant groups and how homogeneous the group are.
That is how similar people in age group are in respect of
those characteristics that influence interactions with a mobile app.
The general rule is that,
the more homogeneous a group is,
the fewer participants you need.
Since we're talking about formative evaluation,
a mathematical model proposed by Jakob Nielsen and Thomas Landauer can be applied.
That provides you with an approximate number of
participants for a given percentage of the possible findings.
According to this model you need only five participants
to uncover 80% of usability findings.
Eight participants can yield 90% percent of the possible findings.
Presuming of course a reasonably homogeneous user population.
If you conduct a study with two groups of participants to get the same results,
Jakob Nielsen recommends to lower down the number of participant per group.
For example, to uncover 80% of usability problems,
you need three to four participants per group.
This section should describe equipment that you are
going to use during test sessions including,
but not limited to equipment for video recording of the sessions, and note-taking.
As well as describe the use of a mobile device for testing.
We will discuss the former in the next lecture in detail.
A couple of thoughts on the use of the device for
guerrilla usability test when possible test on
a participant's device and via
a mobile internet connection if the intended usage context of the app implies the later.
As I said, "Oh well,
when we were discussing an ecological [inaudible] it
will allow you to uncover more interaction problems.
But there is one concern that you should keep in mind making this decision,
use participant's devices only if you are sure that the deployment
of the app or prototype is trustworthy from their point of view.
If the deployment of a new build via a testflight on IOS or
the installation with the help of APK file on Android is a usual thing for you,
I assure you that for many users of these operating systems it looks like voodoo magic.
People trust things they are familiar with.
In respect to deployment,
it comes down to two things.
The installation of apps from official app stores,
and the use of prototypes within a mobile browser.
The selection of test tasks should be guided by research questions.
As I already said,
you may divide the questions in accordance with parts of an interface.
Be it a single UI control,
the whole screen, or a navigation of your app,
and then select tasks that require test participants to use the used parts of the UI.
In case you are faced with a not very specific research question such as,
what are the interaction problems that interfere with
the performance of most common tasks by first time users?
Your choice of tasks should be guided by your knowledge of the app's usage context.
This is a pretty common situation when you work on a new application
or a new feature that affects a significant portion of an apps UI.
In this case, select key tasks,
the tasks uses before most frequently or
tasks that are important from a business standpoint.
You can figure out the later through the interviews with stakeholders.
There are also several additional considerations you should be aware of.
Imagine you designed a new way to set time for an alarm clock app.
It's not a good idea to ask participants to set
a particular time using these UI control as a test task.
It makes no sense from user's point of view.
Of course, you may ask them to change the time of one of
the app's alarms so they will have to use the control you designed.
Participants should understand why they are asked to do things.
But of course these requires some judgment.
For example, in general the task that consists of
signing up only also does not make sense from a user's perspective.
In real world context people do not use apps only to sign up and then do nothing.
However, the sign up task is a pretty standard test task.
The second consideration comes into play here.
You should strive not to make tasks too long.
The longer the task,
the harder it is for participant to remember what happened at the beginning.
It may present a problem for some moderation techniques.
For instance, think of a protocol where
you are not allowed to intervene during the task performance.
We'll examine various moderation techniques
in one of the following lectures of this week.
As a note, mobile app can be a part of a bigger service.
If the user journey is distributed across different channels,
one of which is a mobile app.
It's better to organize test tasks to cover
the whole user journey opposite to and relate different channels separately.
Gorilla test sessions are usually much shorter than lab ones.
What it means, that the number of tasks that
you can test with one participant is limited.
I'd like to remind you that it's extremely important to conduct wireless sessions,
to determine how much time your task is supposed to take.
If it's [inaudible] try to derive groups of tasks each of which will cover different sets of tasks,
but will be complete,
that is the organization of tasks in each group will make sense to the target audience.
It's okay if different groups have the same tasks.
Meaningful organization of tasks is more important.
You may want to evaluate this ability of your app for
context where users are already accustomed to,
but before you release the app to the production environment.
It can be crucial for apps designed for frequent use.
Unfortunately guerrilla usability testing is
not suitable for evaluate long-term usability.
To do that, you have other options.
You may employ lab usability testing where you
ask participants to perform the same tasks repeatedly.
For example, about five or so times,
but no less than three.
Another variation is to ask participants to perform tasks
for the first time to uncover problems related to the first time used,
and then provide some training and conduct a couple of follow up trials.
Another ways to do it is to conduct a series of feasibility test
with the same participants employing alpha or beta versions of your app.
You may also use diary studies to evaluate long term usability.
The introductory instructions section of your test plan is applied after
you find an appropriate person and convince her to participate in your study.
Since we're talking about formative feasibility evaluation,
it's usually a key to briefly explain the purpose of the study.
Just say that you want to understand how
convenient and useful the app is for the participant.
Say a few words about whether participant will be doing and that she may
withdraw at any time as well as decide that her results will not be used.
Assure the participant that she is not being tested.
Also never disclose the fact that you worked
on the app you are going to test even if it is so.
If the participant ask,
tell the truth, but do not disclose it.
Both instances can seriously affect the validity of the study results.
Then ask permission to use
the participant's device if you have chosen this option as we discussed earlier.
Also ask permission to video record the session.
I'd like to remind you that the agreement of a participant to video or
audio recording must be recorded in itself as it's against the law in some countries.
Explicitly state that the data will be kept confidential.
The last thing to do is to ask for any questions before we start.
To see an example of introductory instructions,
I'd like to refer you to the example of the test plan in the materials of this week.
It's introductory instructions are written for lab test,
but still they have much in common with gorilla ones.
I have to stop here.
We will continue the discussion in the next part of this lecture. Thank you.
