Engineering and social sciences are widely considered to be separate fields of practice. However, all technology we develop has the capacity to shape the society in which it is applied.
This spring semester, Jaylin Herskovitz joined Tufts’ computer science department as a new assistant professor, where she hopes to further integrate social and technical research. Herskovitz’s work revolves around designing personal assistive software using artificial intelligence, motivated by her studies on how blind and visually impaired people use the technology to accomplish specialized daily tasks that typical AI models cannot.
Professor Herskovitz began her journey into computer science in her second year of undergraduate studies at the University of Michigan when she joined a human-computer interaction lab. There, she experimented with programming software oriented around people, which steered her towards her accessibility research today. “I thought all of CS research must be extremely technical, like inventing new algorithms or models … which a lot of it is, but the element of getting to work with people was very interesting to me,” she said. She pursued doctoral studies in HCI, obtaining her Ph.D. in computer science in August 2025 and beginning her new role as an assistant professor at Tufts last month.
While working towards her Ph.D., Herskovitz studied how blind and visually impaired people utilized AI technology. She argues that these users offer a fruitful means of uncovering new practical applications of the technology. “We have people who are essentially extreme early adopters of this technology and actually using it in much more practical ways than the average person. Even when LLMs started to become more functional, your average person was asking it, ‘tell me a knock, knock joke’ ... and blind people [were] using it to complete tasks of high importance. Them being early adopters can illuminate a lot of what we eventually do.”
In her research, she uncovered techniques of “hacking, switching and combining” that BVI users apply across multiple AI assistive programs to navigate specific tasks in their lives. These techniques inspired her to develop ProgramAlly and AllyExtensions. With ProgramAlly, users can program AI to quickly identify specific pieces of visual information in their environment. AllyExtensions is another app that allows users to program their own shortcuts to bridge gaps between the accessibility software they rely on.
What sets Herskovitz’s discipline apart from other computer science is the way she applies social science concepts and research in the process of designing and critically analyzing software. “HCI is sort of this conglomeration of … anthropology, psychology, cognitive science, engineering, human factors,” Hershovitz says. Hershovitz wasn’t formally trained in the social sciences. However, she explained that she has intentionally broadened her interdisciplinary perspective. “As I went through my Ph.D., I’ve tried to … expand and learn from those other disciplines more. There are HCI researchers that draw much more deeply than me from social science theories. Actually, a couple years ago, I had one of them tell me, ‘You use theories in your work — you just don’t know they’re there.’”
Using those theories, she refines the purpose and mission behind her assistive technology development. When reflecting on ProgramAlly, she emphasized the importance of interdependence in disability communities.
“People with disabilities are like these domain experts in their own lives and their own ways of doing things, and so my work … looks at how we can help them — not take advantage of that knowledge … but how they can use that knowledge more effectively to build something for themselves,” she said. She added that there are sometimes misconceptions about how BVI people prefer to live. “Sometimes in accessibility work, it’s framed as, ‘we want people to be independent and just be able to do everything on their own.’ And a lot of people with disabilities actually like living in community with other people.”
Herskovitz is currently building the Tufts AI, Design and Accessibility Lab and actively seeking new student members. Some of the projects she hopes to mentor students in involve AI hallucinations and developing programs with newer generative models. “I have a student now, and she’s working on a project relating to errors in AI and how to help [BVI] people reason about the sources of errors. If they can’t see the input to an AI system … how do they reason about what’s happening, especially when something goes wrong? I’m very excited about that.”
As her lab continues to develop, Herskovitz hopes to further integrate the social science perspectives that have shaped her approach to assistive AI. Her career serves as a testament to the ways social science can deepen engineers’ understanding of the technology they develop.
