From Thought to Prompt: Cognitive Design Challenges in Human-LLM Interactions

A recording of the talk can be found here:

Event is hybrid. Participants may attend in person (T2, Computer Science Building, Aalto University) or online on Zoom.

Speaker: Hari Subramonyam
Research Assistant Professor
Graduate School of Education and Computer Science, Stanford University

Talk Abstract: 

Large language models (LLMs) exhibit dynamic capabilities and appear to comprehend complex and ambiguous natural language prompts. However, calibrating LLM interactions is challenging for HCI designers and end-users alike. A central issue is our limited grasp of how human cognitive processes begin with a goal and form intentions for executing interface actions, a blindspot even in established interaction models such as Norman's gulfs of execution and evaluation. In this talk, I theorize how end-users 'envision' translating their goals into clear intentions and craft prompts to obtain the desired LLM response. By mapping different interaction pathways to LLM-powered interfaces, I highlight three types of gaps in envisioning LLM interactions: (1) knowing whether LLMs can accomplish the task, (2) how to instruct the LLM to do the task, and (3) how to evaluate the success of the LLM's output in meeting the goal. I then present the design and evaluation of a specific LLM interface, Spellburst, for generative art composition and highlight design affordances to prompt-based interactions. Finally, I conclude by presenting high-level recommendations for the design and use of LLM interfaces.