Multimodal systems seek to support effective human-computer interaction leveraging people's natural capabilities. While screen-based touch, keyboard, and mouse input have been the mainstream, we see the growing popularity of speech input. Inspired by speech's fast, flexible, and expressive nature, I examine how speech input complements traditional touch input on smartphones in supporting self-tracking practices.
 
In this talk, I will describe several research projects that incorporate speech input into self-tracking systems. Specifically, I examine how people adopt speech and touch to capture different types of personal data in various contexts including exercise, food, and productivity. Taking a mix of research methods involving co-designing with healthcare professionals, in-situ data collection, and technology deployment, my research reveals the benefits and limitations of speech input in capturing structured and unstructured data, and expands our knowledge of how people practice self-tracking with natural language.

Yuhan Luo is a Ph.D. Candidate in Information Studies at University of Maryland College Park. Her research focuses on Human-Computer Interaction (HCI), Health Informatics, Personal Informatics, and Ubiquitous Computing. Yuhan is passionate about bringing positivity to individuals’ everyday health and well-being through empowering them to fully leverage their personal health data. She has designed and evaluated several multimodal self-tracking systems such as mobile apps and Alexa skills. Before joining the Ph.D. program at UMD, Yuhan received her master’s degree in Information Science and Technology at Pennsylvania State University and her bachelor’s degree in Computer Science at Southeast University in China. Outside Academia, she has interned as a User Experience Researcher at Facebook and Google.
 
More information can be found on her website: https://www.terpconnect.umd.edu/~yuhanluo/