Conversation Analysis (CA)

01  SUS i'd like to play beat the intro in a minute
02  LIA   [ oh no: ]
03  SUS [ alexa ][ (1.1)  ] beat the in[tro
04  CAR      [ °yeah°; ]
05  LIA                 [°no:::…°
06  CAR (0.6) it' mother's day? (0.4)
07  SUS it's (   ) yep (.) listen (.) you need to keep
08    on eating your orange stuff (.) liam
09    (0.7)
10  CAR and your green stuff
11  SUS alexa (1.3) alexa (0.5)=
12  CAR             =°and your brown stuff
13  SUS play beat the intro

Discourse Analysis

• Analysing dialogue: what is said, how words convey meaning
• Interpreting context, psychological and social aspects of words.
• Language as social reality, open to interpretation
• Constructivist approach: Language as a constructive tool: analyse the process of world construction
• Identify subtle and implicit meaning between sources

Content Analysis

• Classifies data into categories and studying frequency of occurrences
• Applicable to diverse media formats including text, video, images, etc
• can involve predefined frameworks or classification systems to systematically examine content across specified dimensions
• can be combined with other analytical techniques such as sentiment analysis

Interaction Analysis (Jordan & Henderson, 1995)

• investigate human interactions with each other, artifacts, and technologies using video recordings of naturally occurring activities
• can be teamwork: researchers watch videos together, discuss in real-time
• coding and annotation through repeated video playing
• hypotheses based on observable participant actions and behaviors
• find patterns inductively by assembling instances of salient events

Grounded Theory

Grounded Theory (GT) is an old and important qualitative analysis technique (Corbin & Strauss, 2014; Galser & Strauss, 1967)

• main idea: identify categories (a kind of theme) through iterative data collection and analysis.
• any kind of data but often ethnographic and interview data
• analysis procedure:
  • open coding: categories, properties and dimensions discovered inn data
  • axial coding: systematically establishing categories/sub-categories
  • selective coding: refine and integrating categories
• heavier and less flexible in comparison to Reflexive Thematic Analysis (not suggested for first-timers)
• Rogers et al. (2023) claim that GT “uses reflexive thematic analysis”, which I guess is accurate.

Grounded Theory Example: Idle Games

Systems-Based Frameworks

• large projects involve many sources of data and stakeholders
• e.g., hospital, corporation, local council, airport, (or university…)
• need ways to understand how the system works together
• manage complex interdependencies
• common theme of management speak: “systems thinking”

Socio-technical Systems Theory (STS)

• STS: technology and people in a work system are interdependent (Klein, 2014)
• treat the whole system as a whole, applied in complex work places.
• More of a philosophy than a methodology, a holistic perspective to address challenges.

Notable aspects:

1. Task interdependence
2. Socio-technical systems are “open systems”: influenced by environmental factors
3. Heterogeneity of system components: subsystems have different technical components
4. Practical contributions: analysing systems, evaluating changes, designing co-optimised systems
5. Fragmentation of design processes

Distributed Cognition of Teamwork (DiCoT)

• Distributed cognition unpacks how multiple people and technologies interact complete tasks and solve problems.
  • information flow model
  • physical model
  • artifact model
  • social structure model
  • system evolution model
• models have underlying principles, e.g., for physical model:
  • horizon of observation: What an individual can see or hear
  • perceptual: How spatial representations aid computation
  • arrangement of equipment: arrangement of the environment affects access to information
• useful in collaboration contexts, e.g., software development, medicine

Which Analytical Framework to Use?

adapted from Rogers et al. (2023) table 9.6

Big Research Writing Tip: Cite your methods

• Applies to research projects in this class and at Honours, master, PhD and research-focussed workplace
• Work with supervisors/mentors/managers to choose methodology and analytical frameworks
• Read and understand the framework from (recent) scholarly sources (not just nngroup.com)
• Change and evolution is allowed, but understand that there is a wealth of example and established approaches

Structured Notations

Specific interaction information can be represented in a formal/structured way when presented.

• presenting information through formal notations related to particular domains
• e.g., music applications might involve musical notation or symbolic data formats
• could be related to an analytical framework (e.g., conversation analysis)
• tradeoff between precision and flexibility; structured notation can be precise but potentially less accessible to a reader or limited in scope

Using Stories

Context of use and examples of user experience can be seen as stories or narratives.

• Participants tell stories during data gathering which can illustrate research findings.
• Observations can be framed as stories
• Stories can be written or in the form of storyboards or videos.
• Can be used to support research findings and provide authenticity.

Summarising Findings

Overall advice about findings…

• Multi-modal is often good: combine styles such as stories, plots, data excerpts, numerical tables
• Developing plots and visualisations is critical and hard work, just like crafting text. We spend hours getting it right in Python/R!
• Important to balance the weight of a claimed finding against supporting evidence; however, doesn’t mean that small studies are not useful.
• Reviewers hate over-generalisation: careful with terms like “most,” “all,” “majority,” and “none” without justification
• Statistical claims require care to avoid misleading the reader

What is knowledge?

By this stage, you could be excused for being a bit confused about qualitative research in HCI.

• Research is often defined as “knowledge creation”, but it’s not always clear what that knowledge is in HCI:
  • Is the knowledge from an interview different to a measured interaction?
  • Does our interpretation matter?
  • Do the users have to be objective?
• E.g., if you create a new app, and then evaluate it, can your evaluation ever be objective? (remember you created the app in the first place!)
• Not a new question: epistemology is a (philosophy) discipline to understand knowledge.

Epistemology

This may feel firmly off topic, but we need to surface some friction about knowledge to properly explain the different approaches in qualitative research.

• Postivism/Post-positivism knowledge is true by definition or provable via generalisable methods. The “post” bit accepts qualitative research but emphasise sample size and eliminating bias.
• Interpretivism: knowledge can be socially constructed and meaning made by people
• Critical theory: examine power structures and hidden inequalities
• Constructivism: knowledge is created and shaped by human experience and social interaction, including with the researcher
• Pragmatism: apply frameworks that work best to solve the problem (e.g., mixed methods research)
• New-Materialism: things can create meaning too with HCI defined by intra-action (Barad, 2007) of things and people

Activity: What should this mean to you

• Some aspects of HCI, e.g., “user experience” aren’t well uncovered by (post-)positivist frameworks.
• Interpretivist/Constructivist stances more popular in qualitative HCI research.
• New-Materialism/Agential-realism (Barad, 2007) is emerging in HCI as a relevant mode of inquiry
• but this has some implications in terms of how findings are described!

Does any of this make sense? What kind of knowledge would you want to deal with?

Discuss with someone near you for 2 minutes, then let’s hear some answers.

Phases of thematic analysis

1. Familiarise yourself with the data
2. Code the data
3. Initial theme generation
4. theme development and review
5. theme refining, defining and naming
6. writing up

Phase 1: Familiarise yourself with your data

• considered poor practice to jump to themeing before understanding the data.
• familiarisation starts during data collection (thinking about the content while/directly after collecting)
• angles for thought: key knowledge, simalarities/differences, surprises, adjustments to interview technique/script.

Reading

• Read data in an active way: search for meaning
• make notes while reading
• familiarise while transcribing
• read and re-read transcripts

Phase 2: Code the data

A code is: a name or label applied to a chunk of data

• reduces volume of data
• connects data items together.
• remember last week: inductive vs deductive coding (in this class, please do inductive!)
• chunks can have multiple codes
• code choice: short phrase, or a pithy label (shorter than the data it describes!)
• code small chunks: start with each sentence.
• do this thoroughly

What to look for when coding?

• initial coding: lots of new codes
• later: reuse existing codes.
• length: sometimes one word can be too general (links too much data)
• don’t overlook data: code the obvious

Example codes for “uncertainty”

• “uncertainty about what to do next”
• “uncertain about whether command was received”
• “uncertain about whether information is true”
• “uncertain whether other options would be better”

Coding your data in a text editor

Types of codes

• In vivo codes: based on the literal words of a participant (Given, 2008)
• Researcher denoted: based on the researcher’s interpretation
• Semantic: surface level, close to in vivo but may not be exact words
• Latent: based on deeper interpretation of data

Code book

• Some qualitative research involves collecting codes in a document and sharing between researchers
• Researchers argue this can help eliminate bias
• Controls on number and type of codes, aim for consistency and repeatability
• Often not recommended in reflexive thematic analysis

Activity: Do some coding

Let’s code some interview data.

David is explaining how orders groceries online.

Use the poll everywhere link to code statements and we will see them all together. We’ll code each statement for 1 minute and then discuss the results.

Phase 3: Initial theme generation

A theme:

“A theme captures something important about the data in relation to the research question, and represents some level of patterned response or meaning within the data set” (Braun and Clarke 2006, p.82)

How do we find them?

Themeing

In this class:

1. Use affinity diagrams to cluster codes / data extracts
2. Look for patterned responses/meanings (may help to write higher level codes)
3. prototype themes that capture “something important” (may help to promote a code)
4. Refine and question your themes and codes, not all themes are well-supported by data.

Themes do not emerge

Braun and Clarke insist that “themes do not emerge”, (Braun & Clarke, 2022)

• themes are created by the researcher, not exacavated from the data
• work goes into analysing data and deciding on themes that can be supported
• when folks write “themes emerged”, it’s as if the themes were either there to begin with or developed themselves.

Terry & Hayfield (2021) suggest approaching themeing by prototyping, an iterative process where “the goal isn’t to finish”.

Code Hierarchies

• Some TA methods suggest hierarchies of codes to find themes.
• I suggest: codes, sub- or prototype- themes, then themes
• In this example, are the themes thick or thin?

Phase 4: Develop and review themes

Are your themes good? Test them.

• Are themes supported by (enough) data?
• Answer your research question?
• Provide strong organising concepts?
• Conceptually rich? (Could you write 500 words about each?)
• Do the themes reflect the overall meaning of the data set?

Thin vs thick themes

Phase 5: Refine and name themes

• Does your theme name reflect its ‘essence’?
• Does the theme contain subthemes?
• Are your theme names descriptive but concise?
• Can you describe your theme in a couple of sentences?

Activity: Let’s theme

Let’s cluster some codes from the HCI grocery interview.

• Use this miro board link: https://quicklink.anu.edu.au/o095

This is fairly uncontrolled so be kind 😇

1. Yellow notes are “codes”, cluster them.
2. Make red notes to represent prototype themes.

Cluster for 2-3 minutes, discuss, theme for 2-3 minutes, discuss.

Phase 6: Write up a report

• determine the narrative for your themes
• include quotes and examples from your data
• include analysis: an argument in relation to your research question
• in your user research project: support themes with data
• in your final project: present design implications arising from the themes

Reflexive Thematic Analysis Bingo (Braun & Clarke, 2022)

A bingo card of potential researcher problems with (R)TA… which make sense so far?

What is reflexivity?

• turns back on or accounts for the person’s self
• analytic attention to the researcher’s role in research
• continual dialogue and critical self-evaluation of positionality
• honouring oneself and others through awareness
• thoughtful, conscious self-awareness
• using subjectivity to examine social and psychosocial phenomena
• attending to the context of knowledge production

What is reflexivity in HCI?

• researcher is a non-objective instrument! (to detect phenomena such as “user experience”, among others)
  • “an individual’s experiences and background make up a unique perspective on the world… influences how they interact with participants” (Liang et al., 2021)
  • “Reflexivity calls upon researchers to self-reflect and understand their own possible biases, their role in power relations, and how these factors might manifest in their work” (Liang et al., 2021)

Positionality

• “how a researcher’s perspective compares to others’ perspectives” (Liang et al., 2021)
• not necessarily about disclosing your identity
• disclosing, or examining the aspects of the researcher that is relevant to understand their situated context

What is reflexivity in HCI?

Burroway’s definition (Rode, 2011):

1. reflexivity, unlike positivism, embraces intervention as a data gathering opportunity
2. reflective texts aim to understand how data gathering impacts the quality of the data itself. This approach “commands the observer to unpack those situational experiences by moving with the participants through their time and space”
3. reflexive practitioners attempt to find structural patterns in what they have observed, and fourth, in doing so they extend theory

Types of reflexivity

Statement of Positionality – Example

Activity: Positionality statement (in a sentence)

Let’s try it:

What is your 1-sentence statement of positionality as a researcher?

Use the poll everywhere link to provide it.

Why should you care about reflexivity?

• Considering researcher’s perspective is important part of developing thick themes.
• Needed to address challenges of subjectivity in HCI research.
• Positionality statement and reflexive consideration required for postgraduates (COMP6390) in Final Project!

Questions: Who has a question?

Who has a question?

• I can take cathchbox question up until 2:55
• For after class questions: meet me outside the classroom at the bar (for 30 minutes)
• Feel free to ask about any aspect of the course
• Also feel free to ask about any aspect of computing at ANU! I may not be able to help, but I can listen.