User Testing Round 1: Test the Premise
The purpose of this initial round of user testing was to determine whether or not the tasks in the application would be valuable to high school students and how students would react to those tasks.
- An interactive prototype was created on PowerPoint emulating both the Speaking and Listening activities.
- The prototype was put in front of three Spanish AP students with little instruction.
- Students were asked to think aloud as they interacted with the prototype.
- Students were interviewed about what they liked/disliked about the activity and why.
Screenshots from the prototype (click to enlarge)
Ashley: "Usually we’ll get an activity
where we’ll have to conjugate verbs or something we practiced in the class...
those are just kind of tedious. I like this better
because you don’t get a lot of time in class to actually just speak Spanish, cause
that’s what I’m probably worst at. I can conjugate stuff on paper, but I can’t
speak it as well.”
Dave: "It would be very convenient. You could do it on
the way home from school in the car. I mean, it’s just a minute of just doing
something on an app. And then provided that the animations are still kind of
fun and quirky, it’d be a great thing and wouldn’t be a hassle at all. ”
Ryan: “We do grammar work and reading and question-answering in class, but I like how this is more interactive. This seems like something I’d have more fun doing – more fun if I know that, by a certain time of night, I have to turn in a speaking portion because my friend needs it for their homework. I like how you have a sense of responsibility to somebody other than yourself."
User Testing Round 2: Validation as a Language Challenge
We asked people to complete the task in their first language.
If the task could be completed easily by people in their first language, but was more challenging in a second language, then we would have verified that this task is a test of language ability and not some other factor.
Results:
Indeed, people completing the task in their first language accomplished the task with a near-perfect record.
If the task could be completed easily by people in their first language, but was more challenging in a second language, then we would have verified that this task is a test of language ability and not some other factor.
Results:
Indeed, people completing the task in their first language accomplished the task with a near-perfect record.
User Testing Round 3: Test the Long-term Experience
The purpose of this second round of user testing was more exploratory and aimed to uncover issues that would arise when students were asked to use the product every day - the way it was intended to be used.
- Four high school students were enlisted in a two-week user testing period where active participation would be required every day.
- Interactive HTML prototypes were created using a combination of Flash and free online recording software.
- For the Speaking activity, participants were able to watch a video and record their audio descriptions online (see 1st image right).
- For the Listening activity, participants were able to listen to a peer's recording and dynamically rearrange images into the correct order (see 2nd image right).
- Links to these web-based prototypes were sent out daily.
- Participants also completed a short survey with every task.
Speaking Activity
Listening Activity
Results:
These quotes from user interviews say it all.
These quotes from user interviews say it all.
"If I had to this every day for a year, by the end, I would definitely be more comfortable speaking... yeah, I think the biggest thing is that I'd be more comfortable as a speaker."
"This activity is really necessary to boost communication skills because there really is no equivalent inside the Spanish classroom."