Designing AI Tools for Underserved Populations from the Ground Up

It can be difficult for anyone to keep up with the latest artificial intelligence (AI) technologies, but minority groups face additional challenges using systems that were not specifically tailored for them. Researchers say this is why we must design AI from the ground up for marginalized communities.

The Northwestern Center for Advancing Safety of Machine Intelligence (CASMI) supported a special interest group (SIG) event on Oct. 17 in Minneapolis called “Purposeful AI.” Its organizers ‒ Francisco Iacobelli, associate professor of computer science at Northeastern Illinois University and principal investigator for the CASMI project, “Intelligent Tutoring System for Non-Native, Low-Literacy Individuals”; and Tianyi Li, assistant professor in the department of computer and information technology at Purdue University ‒ coordinated the SIG at the Association for Computing Machinery (ACM) Conference On Computer-Supported Cooperative Work And Social Computing (CSCW) to discuss the challenges and best practices associated with research in AI with underserved populations.

“People in academia or people who work on community-driven research want to support and serve marginalized communities,” Li said. “However, in many cases, we as researchers are not members of those communities. We wanted to contribute a multi-disciplinary discussion on how we can purposefully develop AI solutions.”

The following panelists spoke at the SIG: Neha Kumar, Georgia Institute of Technology associate professor at the Sam Nunn School of International Affairs and the School of Interactive Computing, who has done extensive research with underserved populations in India; Zhicong Lu, City University of Hong Kong assistant professor of computer science, who worked with marginalized communities in China; Calvin Liang, Northwestern University postdoctoral fellow in the department of communication studies, who has worked extensively with the Lesbian, Gay, Bisexual, Transgender, and Queer (LGBTQ) community; Tzu-Sheng Kuo, PhD student in the Human-Computer Interaction (HCI) Institute at Carnegie Mellon University, who is exploring how to support community participation in the design and evaluation of impactful AI systems, such as AI tools that prioritize public housing; and Sheena Erete, associate professor in the College of Information at the University of Maryland, College Park, who has done extensive work with marginalized African-American communities and applications that solve their unique problems.

Researchers said five dimensions must be considered while conducting this research: 1) power structures, 2) validity, 3) sustainability, 4) trust between researchers and communities, and 5) impact. This framework says interdisciplinary teams need to include community members.

“All of these new technologies will face the same problem in that they are mainly designed by people with power,” Kuo said. “However, HCI research has a long history of engaging marginalized communities in evaluating these technologies. I do think there are lots of rich literature or methods to build upon within the HCI space. I think AI researchers can learn a lot from drawing inspiration from HCI research to think about how to address these challenges.”

The SIG included a working session which divided groups by dimension to explore the topics further. The group that focused on sustainability noted that the community they work with sometimes does not understand the impact of AI on their communities. The validity group wrote that researchers should be clear with participants about research project lifespans and that participants should evaluate the relevancy of research findings when work is completed. The impact group said that systematically studying individual cultures (known as ethnography) can help uncover the nuances in marginalized groups, which can be key to identifying benefits and challenges of AI solutions.

When Kuo and his research team studied homelessness, they considered barriers such as AI literacy, reading literacy, social stigmas, power imbalances, and suspicion about research motivation. To address these barriers, they presented research participants with illustrations, easily understandable captions, and comic boards that allowed people to determine whether generic personas were applicable to their own lives. Researchers also ran one-on-one studies with participants to mitigate power imbalances.

“One of our main contributions is a method to elicit feedback,” Kuo said. “We create a set of comic boards. We then ask open-ended questions to get participants’ thoughts on what they think the ideal solution is or what their suggestions might be. Then, we present the current design in reality. So, after they share their ideas, they're able to critique what they find appropriate or what could be improved in the current design. Altogether, this provides a scaffolding for people who may have lower AI or reading literacy.”

While Li has prior research experience in crowdsourcing, sensemaking, and human-AI collaboration, she is only beginning to work extensively with underserved populations. Her interest comes as she is seeking how to better support her aging parents. Li and her colleague are exploring how to volunteer at a residential community for older adults in West Lafayette, Indiana to understand their culture.

Li is also working on another project that aims to support high school students from marginalized communities. She is already learning things that she didn’t realize before.

“We are experiencing a lot of challenges communicating with teachers because they're busy with their own everyday work, and then they need to make time to talk to us,” she said. “It looks like they're not very accustomed to Zoom Meetings because of their nature of work. I feel like I am imposing some kind of biases and privileges when I get started because I thought, ‘Everybody has Zoom meetings. Everybody is good at emails.’ But then I realized not everybody's like that. I'm reflecting on my own limitations and biases when trying to get started in this field.”

Li plans to reach out to other researchers to learn from them.

The “Purposeful AI” collaborators are collating notes from the discussion and will post updates to their website. If you are interested in getting involved, reach out to them.