Gathering user feedback

As well as the quantitative data gathered by users activity in the Oppia app, most implementations will also want to gather qualitative data on how users manage to use the Oppia app and content.

Most approaches to this would use surveys, interviews and/or focus groups to ask users about how they manage to use the app and ask questions such as what you like/dislike about the app, or to get information about specific difficulties users may be having.

However, some users may be hesitant to admit they are having difficulties in using the app and content, so the qualitative data might not be representative.

An alternative approach to asking specific questions is to give users a set of short tasks to perform in the app and with the content, watching the users doing the task and noting any particular issues they are having.

This approach is modeled on the concept of ‘guerilla’ usability testing.

An overview of the process is:

  • Design the tasks - each task should be specific and discrete

  • Introduce the project and what you are doing and why. Specifically emphasise that this is about you testing the product, not the users ability to complete the task

  • Ask the user to think out loud as they are going through each task - this gives an idea about users thought processes

  • Reset the device back to either the Android main screen, or the app homepage, before each task

  • If possible video the user whilst they are doing the task, as this is a useful reference for later

  • The facilitator should take notes as the user is doing each task, especially noting where users have difficulties, or get confused

In terms of the number of users needed for this type of usability testing, even a small number of users (5-10), providing they are representative users, can give an idea of priority areas to update in the app and/or content.

Some suggested specific tasks for Oppia might be:

For end users:

  • Register in the app and download course X

  • Download the media files for course X

  • Find how many points you have earned for course X

  • Complete quiz X and tell us your score and the number of points you earned

  • Play media file X

  • Find out how many activities you have completed in course X

  • Find out how many points you earned on course X in the last month

For supervisors:

  • Find out how many activities has the user completed in the last week on course X

  • Transfer new courses and media files from your device to another device

  • Download the activity logs from a users device and submit them to the server

A very similar approach can be applied for testing the Oppia server dashboard.