Get information about people, their behaviour and experiences with technology.

• What information and why?
• Depends on research problem and phase of research/design process, e.g.,
  • Comparing two alternative interfaces
  • Understanding a context of use
  • Measure time taken to complete a task
  • Discover how users interact with an existing system

Who are the participants? How many are needed?

• Small group of stakeholders
• Criteria for involvement (“plays video games regularly”)
• Random sampling from large population
• Convenience/volunteer sampling from those available
• Users with specific skills/needs
• Snowball sampling (participants help find more)
• Researchers make a justified choice
• Number of participants: most common in HCI is 12 (Caine, 2016)

A data provider gives us data. What do they get back?

• Conduct research openly, ethically, responsibly
• Informed consent
• Clear communication of benefits and risks
• Respect and acknowledgement
• Particular care for some groups, e.g., Aboriginal and Torres Strait Islander communities.
• Practical: building rapport and understanding (not just taking)

• data can have strings attached
• privacy
• personal information (including name!)
• sensitive information (embarrassing or harmful)
• storage requirements
• do we just use Google drive for everything?
• research needs a data management plan

• data triangulation: data is drawn from different sources, times, places, people, etc
• investigator triangulation: different researchers (observers, interviewers, etc)
• triangulation of theories: different theoretical frameworks
• methodological triangulation: different data gathering or research techniques

A small initial study to help plan a larger study.

• Could involve limited number of participants
• Limited interface or study parameters
• Checks that expected data can be obtained
• Can be called a “formative study” (to form the goals of the main study)
• E.g., assess the game platform used in a larger study (Mohaddesi & Harteveld, 2020)

• Interviews make a lot of sense in HCI
• What should a system do? Just ask.
• How should it work? Just ask.
• Did it work well? Ask away.
• How would you change it? Ask.

• exploratory, similar to conversations
• go into depth on experiences
• questions are open: no expectation on the content of answers or subsequent questions
• probing: can you tell me more about …?
• benefit: generate rich, complex data
• limitation: time consuming to analyse

Could you tell me about your experience using the NewWidgetApp?

You mentioned that you enjoyed (feature X), why was that?

Can you explain more about what happened when you used (feature X)?

• questions are predetermined
• questions need to be short and clear
• questions can be closed (answers from specific options)
• typically, whole interview is scripted
• low skill to deliver
• fast to deliver
• useful when goals are clearly understood and questions and responses can be identified

What is your most used code editor: VSCode, nvim, emacs, notepad.exe, or something else?

How often do you push your assessments to Gitlab: every minute, every hour, every day, just once?

• features of both structured and unstructured
• typically: script with questions on main topics but discussion and probing follows each question
• intended to be somewhat replicable
• probe: neutral questions to gain more detail
• prompt: reminder of some part of the topic to gain specific information
• careful: prompting can preempt answers (bias!)

How are things going with the interaction experience?

Are there any [initial] impressions you want to share?

Can you describe the connections between your movements and the resulting sound?

(Reed, 2023)

• interview with multiple participants (could be called a group interview)
• facilitator prompts discussion
• group members can influence each other (good or bad?)
• good for talking to lots of people.
• raising diverse viewpoints
• facilitation: need to be careful
• groupthink: generally considered harmful

(I don’t find the “Focus Group” term super useful in my academic research—but they are widely used in industrial/government research.)

Let’s plan a semi-structured interview!

What questions should be asked in a semi structured interview about student’s user experience with the “catchbox” (soft microphone used during lectures)?

Use the poll everywhere link to suggest interview questions and vote on the best ones.

Ideate for 2-3 minutes, vote for 1 minute, then let’s discuss.

• usually have a script for introductions, consent, etc (read verbatim)
• listen more than talk
• respond with sympathy but without bias
• enjoy the experience
• interviewing is hard work!

• usually interviewing involves a combination of:
  • notes
  • audio
  • video
• different data have different issues and needs
  • notes: type up, handwriting (bias?)
  • audio: convert format, transcribe (suggest Aiko)
  • video: edit, store (large file sizes, anonymity)

• written method of gathering structured data
• sometimes called a “survey”, technically a survey is the whole study
• questionnaire or survey instrument is the paper form with questions
• questions can be open or closed
• often questionnaire used for demographic data
• standard survey instruments are often questionnaires

Questions might be something like:

What were the strengths of this course?

Please provide any suggestions about how this course could be improved?

• the researcher might have little control over how seriously these questions are taken.
• potential to gather rich data
• careful: use these only with really open concepts

• closed form questions have preset responses from which the respondent must select.
• unordered responses
• rating scale questions
• careful: closed form questions design should be neutral and include most likely or relevant responses

What is your favourite fruit (select one answer): plum, tomato, pineapple”

How interesting is this lecture? Select a number from 1 (interesting) to 10 (amazingly interesting).

• Likert scale questions are very common in questionnaires
• named after Rensis Likert (social scientist) (Likert, 1932)
• The question includes a statement, e.g.: “The workload was appropriate for this course”
• A number of levels of agreement are provided, e.g.: “strongly disagree, disagree, neutral, agree, strongly agree.”
• 5-points is common, 3, 7, 9 or a continuous slider is also possible.

• ratings of an object, concept, situation, etc
• the answer is a point between two opposite concepts
• e.g., describe your experience of using the (insert system here)
  • difficult to use — easy to use
  • boring — fun
  • slow to learn — fast to learn

1. I think that I would like to use this system frequently.
2. I found the system unnecessarily complex.
3. I thought the system was easy to use.
4. I think that I would need the support of a technical person to be able to use this system.
5. I found the various functions in this system were well integrated.

6. I thought there was too much inconsistency in this system.
7. I would imagine that most people would learn to use this system very quickly.
8. I found the system very cumbersome to use.
9. I felt very confident using the system.
10. I needed to learn a lot of things before I could get going with this system.

1. How mentally demanding was the task?
2. How physically demanding was the task?
3. How hurried or rushed was the pace of the task?
4. How successful were you in accomplishing what you were asked to do?
5. How hard did you have to work to accomplish your level of performance?
6. How insecure, discouraged, irritated, stressed and annoyed were you?

Worksheets provided!, (Hart & Staveland, 1988), extra questions weight the ratings.

• Idea: measure how well a system can support creativity Cherry & Latulipe (2014)
• creativity support tools: writing, visualisation, video editing, music tools etc
• creativity is a bit hard to define, but the CSI includes:
  • exploration, expressiveness, immersion, enjoyment, results worth effort, collaboration
• inspired by TLX, 2 questions per factor + 15 paired comparisons.

• It’s hard to write survey questions! It may be good for beginners to use a standard questionnaire.
• The more questions you ask, the more work it can be to analyse. It can be counter-productive to have lots of questions without a way to aggregate them.
• Rating scale data is generally not continuous and so you need to use non-parametric significance tests.
• The distribution of survey data is usually important, good to use plots such as box plots rather than a mean and standard error chart.

Let’s write a questionnaire about ANU students’ class enrolment experience.

• What questions should we ask on a questionnaire about ANU student’s enrolment experience? If closed list options, if open write “(open question)”

Talk for 2-3 minutes and add some examples on PollEverywhere, then let’s have some discussion.

https://PollEv.com​/charlesmarti205

• watching people for science
• using system or technology in the normal context of use
  • e.g., a researcher joins a tour group to observe use of a travel navigation app
  • e.g., a researcher watches a concert carefully to observe the performer’s use of music technology

Example framework:

• The person: who is using the technology at any given point?
• The place: where are they using it?
• The thing: what are they doing with it?

• Literally “study of culture”
• immersion and participation in a research context.
• e.g., to create a factory management system, a researcher might embed themself in a factory, talk to workers and perform tasks
• ethnography is often used in relation to specific workplaces, e.g.: factories, hospitals etc
• being within the context of the participants rather than bringing them to a lab/classroom for study
• ethnography as a methodology connected with anthropology (i.e., as “the study of culture”), but it’s not identical

• observation in a lab-based setting
• could be possible to identify close details
• record all aspects of technology use
• record data, audio, video as well as observations

• it can be difficult to remember details from a long sequence of tasks.
• techniques like video cued recall can help participants provide a commentary on an experience.
• the idea is: you record video of the participant completing an an interaction and then have an open-ended discussion while watching it back.

• participants write about their experiences with a system regularly in a diary
• easy for the researcher at the collection stage
• takes continuous effort from the participants (reminders? structure?)
• relies on participant’s memory and subjective account
• video and photos can reinforce written accounts

• Interaction log: a log of data captured from a system showing exactly what the participant did at any given time
• e.g.: key presses, mouse movements, interactions with GUI components, sensor data
• time spent on actions or using software (e.g., playing a game, using instagram)
• unobtrusive, automatic
• lots of data, should be visualised or analysed to develop findings
• scraping data from public sources (e.g., social media) can be observation

N.B. scraping large amounts of data raises ethical concerns!

As an HCI researcher, would you rather:

1. observe directly in the wild
2. observe directly in a lab
3. observe indirectly through a diary
4. observe indirectly through data analytics

Discuss for 2-3 minutes in groups, rank on PollEverywhere, and then we will hear some responses!

• data gathering should adapt to different participants
• children: think and react differently to adults, e.g., adapt scales to visual representations (Putnam et al., 2020)
• people with disabilities: caregivers might be involved to adapt interview questions
• animals(!): hard to interpret behaviour, need to design interfaces to work without hurting animals (Mancini et al., 2017)

• expectations for remote data gathering changed completely during pandemic!
• can access participants in different countries, age groups, abilities, specific or expert users
• observation could be recorded and then analysed later
• best practices might include (Mastrianni et al., 2021):
  • running pilot tests before conducting sessions
  • have backups in place in case of issues
  • informing participants of technical requierments
  • use respective questioning if issues with think-aloud.

Who has a question?

• I can take cathchbox question up until 2:55
• For after class questions: meet me outside the classroom at the bar (for 30 minutes)
• Feel free to ask about any aspect of the course
• Also feel free to ask about any aspect of computing at ANU! I may not be able to help, but I can listen.