• search engine for scholarly sources
• add more search terms to get more specific
• you can use the “time” selector on the left column to find recent work

Careful:

• Google Scholar indexes anything that looks like scholarly research (e.g., any PDFs on a website in a conference/journal format). Need to use critical thinking to decide whether sources are good quality or not.
• Google Scholar can give you a formatted citation but it may not have all information (e.g., URL, DOI)

• ACM Digital Library archives proceedings of ACM conferences and journals.
• not as good at searching, but will show you only peer-reviewed works

Careful:

• Works best on-campus or via virtual.anu.edu.au so that you can access all papers.
• CHI and ACM are centres of HCI research, but there are other non-ACM venues that could be missed.

• metadata for a reference can be used with any citation format
• academics often use special tools for storing reference metadata
• computer scientists tend to use BibTeX format
• BibTeX is part of the venerable (La)TeX ecosystem originally developed by CS luminary Donald Knuth in the late 70s
• in LaTeX you can select citation format (e.g., ACM, IEEE, Chicago, Harvard, APA)

@inproceedings{adiwangsa-charades:2025,
author = {Adiwangsa, Michelle and Bransky, Karla and Wood, Erika and Sweetser, Penny},
title = {A Game of ChARades: Using Role-Playing and Mimicry with and without Tangible Objects to Ideate Immersive Augmented Reality Experiences},
year = {2025},
isbn = {9798400714863},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3715668.3736382},
abstract = {Using tangible objects for immersive augmented reality (AR) experiences offers various benefits, such as providing a physical means of interacting with virtual objects and enhancing the functionality of everyday objects. However, designing AR experiences with tangible objects presents unique challenges, particularly due to the diverse physical properties that can influence user interactions. In this provocation, we explore effective approaches for ideating such AR experiences, by designing two exergames intended for AR head-mounted displays (HMDs). We found that role-playing and mimicry, both with and without tangible objects, provide valuable benefits in the design of such experiences. Building on this insight, we introduce ChARades, an iterative and playful gamestorming technique that incorporates role-playing and mimicry in both forms, to ideate immersive AR experiences involving tangible objects.},
booktitle = {Companion Publication of the 2025 ACM Designing Interactive Systems Conference},
pages = {440–445},
numpages = {6}
}

The metadata entry for Adiwangsa et al. (2025) which is cited using [@adiwangsa-charades:2025] in this Markdown document. In LaTeX you would cite it like this: \cite{adiwangsa-charades:2025}.

Big reference libraries are part of academic work.

• Charles’ “main” .bib file has ~1300 entries (started in 2013, first year of Charles’ PhD!)
• references.bib for this course has about 100 entries.
• Charles uses BibDesk to manage the big file, but usually just VSCode for smaller bibtex libraries.
• Other popular reference managers are Zotero, Mendeley and Endnote.
• Opinion time: Bibdesk is nice because the database is just a text file.

• Not expecting you to use BibTeX (yet)
• Expecting you to list references in “ACM format”.
• ACM format is inspired by APA referencing
  • supports both numerical [1, 2] and author-date (Martin, 2020) referencing.
  • includes full names of authors and publication details for clarity
  • includes DOI as a full URI
• ACM page plain text examples for references to different types of source
• CSL version (Citation Style Langauge)
• BST version (BibTeX Style)

• Command Line
• Graphical
• Multimedia
• Virtual reality
• Web
• Mobile
• Appliance
• Voice
• Pen
• Touch
• Touchless

• Haptic
• Multimodal
• Shareable
• Tangible
• Augmented reality
• Wearables
• Robots and drones
• Brain-computer
• Smart
• Shape-changing
• Holographic

• Type in commands (e.g., ls)
• pressing certain combinations of keys (e.g., Ctrl + V)
• fixed from the keyboard (e.g., delete, enter, esc) or user-defined
• largely superseded by graphical interfaces such as menus, icons, predictable text commands
• still useful for complex software (e.g., CAD), scripting batch operations, coding
• is chatGPT a CLI?

• Information represented within a graphical interface
• use of color, typography, and imagery (Mullet & Sano, 1996)
• interface features abbreviated as WIMP
  • Windows
  • Icons
  • Menus
  • Pointer

A single interface combines different media such as graphics, text, video, sound, and links them together with various forms of interactivity. E.g., Wikipedia.

• better information presentation.
• facilitate rapid access to multiple representations of information.

Interfaces can sit on a spectrum between fully virtual and fully real interaction (Milgram & Kishino, 1994).

• The big middle area includes “mixed reality” (MR) interfaces
• Augmented reality usually closer to “real” reality.
• eXtended reality (XR) is a more recent term.

Has often seemed like a great idea, but hasn’t cracked mainstream yet (or has it??)

• blending of digital content with the physical world to create an enhanced real-world experience
• 1960s: Ivan Sutherland’s development of the first head-mounted three-dimensional display

• AR systems have evolved significantly, particularly displays and interaction models (Billinghurst et al., 2015; Speicher et al., 2019)
• displays: see-through, screen-based, projection-based.
• “spatial Computing”, another way of thinking about it, defined by Greenwold (2003)
  • “human interaction with a machine in which the machine retains and manipulates referents to real objects and spaces.”
• emphasising not only the augmentation of reality but also the meaningful interaction between digital and physical elements.

Emerged in 1970s with computer-generated graphical simulations.

Goal: to create user experiences that feel real when interacting with an artificial environment.

• stereoscopically displayed image.
• interact with objects through input devices such a joystick within the field of vision.
• higher level of fidelity compared to other graphical interfaces, provides immersion.
• different viewpoints: first-person perspective, third-person perspective, etc.

• learning and training for specific skills.
  • driving, pilot training, surgery, medicine, complex/confronting environments.
  • build up skills with lower costs and low risk.
• navigation for accessibility, treatment (e.g, mental health)
• entertainment.
• virtual body to enhance the feeling of presence; reduce cybersickness; support natural user experience; the level of realism to target, etc.

• early websites were largely text-based, with hyperlinks to different places or pages of text.
  • “how best to structure information at the interface level to enable users to navigate and access it easily and quickly?”
• shift from information usability to aesthetics and visual design.

Much of the content on a web page is not read. Web designers are “thinking great literature” (or at least “product brochure”), while the viewer’s reality is much closer to a “billboard going by at 60 miles an hour” (Krug et al., 2014).

• web development involves multiple technologies
• CSS, HTML, JavaScript, node.js, Python, etc.
• breadcrumb navigation (key interface element): “way finding” to navigate without losing track
• design for smartphone or table interaction modality, for smaller-sized displays and for infinite scrolling.
• three core questions proposed by Keith Instone: “Where am I? What’s here? Where can I go?” (Veen, 2000)

Smartphones, fitness trackers, smartwatches, large-sized tablets on the flight, educational tablets, etc.

• embedded sensors, such as accelerometer for movement detection, thermometer for temperature measurement, bio/fitness sensors.
• new affordances led to novel and creative apps e.g.,
  • Ocarina (Wang, 2014)
  • contextual information access: scanning QR codes

• machines for everyday use in the home
• washing machines, microwaves, refrigerators, toasters, bread makers, blenders
• connections to the internet, control by remote apps
• do such designs help?
• Cooper et al. (2014) suggest that appliance interfaces require the designer to view them as transient interfaces, where the interaction is short.
• Two fundamental design principles: simplicity and visibility.

• lets users interact with apps (e.g., search engines, chatbots, or travel planners) through spoken language
• commonly used to request information (like flight times or weather) or issue commands (e.g., playing music or selecting a movie)
• Voice interfaces rely on command- or conversation-based interaction.
• early speech systems earned a reputation for mishearing all too often what a person said (still true?)

What conversational mechanisms to use to structure the voice interface and how human-like they should be?

• natural conversation
• system navigation efficiency
• synthesised voice or voice actor: male, female, neutral, dialect, and pronunciation

Pros and cons of dialogue structures, error handling, and etiquette remain key for modern voice interfaces (Cohen et al., 2004).

• light pens, styluses, or scanners for drawing and writing
• write, draw, select, and move objects on a page or tablet.
• very early use in CRT displays with lightpen!
• touchscreen versions most common now
• another option: digital paper that scans what you write (IR sensor, special dot pattern on the paper)

Pens are great! We should do more with them.

• common in kiosks, ATMs, checkouts, phones, tablets, computers.
• Now weird to find a non touchscreen.
• Single-touch respond to single taps.
• Multitouch supports multiple simultaneous touches and gestures like pinching, rotating
• Multitouch devices (e.g., smartphones, tablets, tabletops) enable intuitive interactions using one or both hands.
• Finger gestures enhance how users interact
• but there are limitations. How do you “undo” on an iPhone?

• vibration and forces (via actuators) to provide tactile feedback
• can be embedded in clothing or mobile devices such as smartphones and watches.
• gaming consoles and driving simulators use haptics to enhance realism.
• vibrotactile feedback can simulate remote physical communication, like hugs or squeezes, using actuators in clothing.
• Haptics can also be used for skill training, such as learning musical instruments.
  • “novice players responded well to vibrotactile cues, adjusting their actions accordingly.”

• Ultrahaptics uses ultrasound to create 3D shapes and textures in midair that can be felt but not seen.
• It simulates touch-based interfaces (e.g., buttons, sliders) that appear in midair.
• In automotive interfaces to replace physical controls with invisible, tactile controls: adjust volume or change radio stations.
• Haptic Exoskeletons: bedded into wearable exoskeletons, inspired by “Techno Trousers” from Wallace and Gromit (Rossiter, 2018).
  • Graphene parts are used to stiffen or relax the trousers to assist movement.
  • Application in mobility assistance, fitness.

• multiple input/output modalities (e.g., touch, sight, sound, speech) to enhance user interaction (Bouchet & Nigay, 2004);
• natural, flexible, and expressive interactions, similar the real world interaction experience (Oviatt et al., 2017).
• Common combinations: speech and gesture; eye-gaze and gesture; haptic and audio, pen input and speech (Dumas et al., 2009).
• Speech + vision processing is the most common (Deng & Huang, 2004)!

E.g., Kinect for Xbox — combined RGB camera, depth sensor, and microphones for real-time gesture and voice recognition.

must recognise multiple user aspects: handwriting, speech, gestures, eye movements, and body movements.

• more complex to design than single modality systems.
• most researched interaction modes are: speech, gesture, eye-gaze tracking.
• What are the actual benefits of combining multiple input/output modalities?
• Is natural human-like interaction (e.g., talk + gesture) effective when applied to computer interaction?

Design guidelines, see (Oviatt et al., 2017).

Designed for multi-user interaction, unlike typical single-user devices (PCs, laptops, phones).

multiple simultaneous inputs by co-located groups. E.g., large wall displays (e.g., SmartBoards); interactive tabletops for users to interact using fingertips on a shared surface.

• A large shared space for group collaboration.
• Allow simultaneous interaction, unlike working on a single PC.
• Users can point, touch, and see the same content.
• This creates a shared point of reference, enhancing group coordination and participation (Rogers et al., 2009).

• tabletop systems used in museums and galleries.
• support interactive, group-based learning (Clegg et al., 2020).
• some shareable interfaces are software platforms for remote collaboration.
• Early example: ShRedit (1990s) – supported shared document editing.
• Google Docs, Microsoft Excel, Miro, Canva, etc.

Research and Design Consideration

from single-device interactions (e.g.,handwriting) to cross-device collaboration; key design issues around display layout, participation, and balancing personal and shared spaces.

Link physical objects (bricks, cubes, clay) with digital representations through embedded sensors (Fishkin, 2004; Ishii & Ullmer, 1997)

Manipulating objects triggers digital effects like sounds, animations, or lights, either on the object or in surrounding media (e.g., Tangible Bits (Ishii & Ullmer, 1997)).

Some use physical models on digital surfaces (tabletops), where moving objects influences digital events (e.g., Urp for urban planning).

Physical artifacts can be lifted, rearranged, and manipulated directly, distinguishing from purely screen-based or mobile interfaces.

• Digital devices worn on the body – e.g., smartwatches, fitness trackers, smart glasses, and fashion tech.
• Early wearable computing enabled mobile recording and access to digital info.
• Advances in flexible displays, e-textiles, and physical computing (e.g., Arduino) have made wearables more practical and appealing.
• Items like jackets, jewelry, hats, and shoes have been designed to interact with digital info on the go.
• From convenience design focus to expressive and communicative functions.

Robots assist in manufacturing, hazardous environments, rescue, and remote exploration with remote controls using cameras and sensors.

Domestic robots help with chores and support elderly or disabled people.

Pet robots like Paro provide companionship and reduce loneliness, especially for dementia patients.

Drones, once military and hobbyist tools, now serve in delivery, entertainment, agriculture, construction, and wildlife monitoring, offering real-time data and automated operations.

Create a link between brain activity and external devices (e.g., cursor, robot, game controller).

Detect neural signals via electrodes in headsets placed on the scalp.

Assist or augment cognitive and motor functions—especially for people with disabilities (e.g., BrainGate lets paralyzed users type via thought).

…and entertainment – e.g., Brainball, where relaxation controls a ball’s movement.

Aim to transfer mental states (e.g., “focused,” “relaxed”) between people via stimulation.

Ethical concerns arise around mind reading and mental state manipulation, especially with companies like Neuralink pursuing direct brain implants.

• Use physical form changes as input/output (e.g., 3D bar charts moving rods to show data) (Alexander et al., 2018).
• Enable tactile interaction beyond screens
• Make data more relatable by embedding it in everyday contexts

Research and Design Consideration

• Do they improve data understanding and engagement?
• Are they better than 2D/3D digital charts for health monitoring?
• Size of rod grids; number of cubes for easy learning and recognition.

• Create the illusion of a 3D person being present through taking advantage of the human perceptual system.
• The projection and display technology have achieved some convincing results (e.g., ABBA Voyage in London).

Research and Design Consideration

• A lot research conducted in the tech industry exploringhow to represent people in virtual spaces in ways that feel natural, comfortable, engaging, and not creepy.
• Design considerations include hologram size and how users can interact and communicate with projections in their space.

🙋🏽‍♀️🤷💁🏻🧠🗣️

Imagine you are developing a new interface for the classic game “connect four”

1. what interface what would you choose?
2. how do you think it would evaluate in terms of usability and user experience?

Chat for 3 minutes with someone next to you then let’s hear some ideas.

A natural user interface (NUI) is designed to allow people to interact with a computer in the same way that they interact with the physical world—using their voice, hands, and bodies.

• but how natural are NUIs?

Don Norman (Norman, 2013) argues “natural” depends on a number of factors:

• how much learning is required,
• the complexity of the app or device’s interface,
• and whether accuracy and speed are needed.

A gesture may worth a thousand words; other times a word is worth a thousand gestures. It depends on how many functions the system supports.

Who has a question?

• I can take cathchbox question up until 2:55
• For after class questions: meet me outside the classroom at the bar (for 30 minutes)
• Feel free to ask about any aspect of the course
• Also feel free to ask about any aspect of computing at ANU! I may not be able to help, but I can listen.