This Augmented Reality Tool Could Change Communication for Some Deaf and Hearing Impaired People

Madhav Lavakare had spent days searching both the internet and hardware stores in his home city of Delhi for a one-millimeter-thick sheet of acrylic when he found the crucial piece for his budding invention right under his nose. A CD case! Holed up in his bedroom, the teenager cut the clear front of the storage case to the perfect dimensions.

In that moment, five years ago, Lavakare was certain of two things: First, he would need to improvise wherever necessary. And secondly, with an effective beam splitter to reflect a text image, he was one step closer to building his vision for an assistive hearing device that attaches to any pair of glasses and projects real-time subtitles in the user’s field of vision—a product he dubbed TranscribeGlass.

“I started working on TranscribeGlass when I was in 11th grade,” says Lavakare, now 23 years old and pursuing a degree in computer science from Yale University. “One of my friends has hearing loss, and he doesn’t wear hearing aids. He had to drop out of school, as he found it difficult to understand what teachers and even his friends were saying. He thought he was better off just being homeschooled.”

A 2021 report by the World Health Organization (WHO) estimates that by 2050, around 2.5 billion people could have some degree of hearing loss, with over 700 million requiring hearing rehabilitation. Today, over 1 billion young people are at risk of permanent, avoidable hearing loss due to unsafe listening practices.

While there have been substantial advances in assistive technology in recent years, measures like hearing aids, cochlear implants, middle ear implants, software providing subtitle or captioning services, and alerting and telecommunication devices still fall short in various ways. No one device is a complete solution.

TranscribeGlass 90-second Demo

Watch on

Hearing aids aren’t dependable in a lot of situations, especially ones with a lot of background noise or where multiple people are speaking. Moreover, hearing aids and cochlear implants are expensive, often costing thousands of dollars, making them inaccessible to most in need. According to WHO’s report on hearing, nearly 80 percent of people with moderate or higher levels of hearing loss live in low-income and middle-income countries.

Assistive technology like captioning is an important means of communication for people with hearing loss. According to a 2013 survey conducted by Collaborative for Communication Access via Captioning, 70 percent of the 220 respondents “felt included, less stressed by their hearing loss and more able to participate when they had captioning.”

Cellphone subtitling software, meanwhile, allows users to understand what is being said, but at the expense of users then being stuck looking at their screens at all times, missing facial expressions, lip reading and other nonverbal communication cues from the speaker, which are all important aspects of communication for the deaf and hard of hearing.

“I had the idea of having subtitles in the field of view so that you could see all the nonverbal communication cues, look at the person speaking to you and also be able to see the actual content of what is being said at the same time,” says Lavakare.

Transcribe Glass isn’t the first assistive device providing real-time captions in the user’s view. Similar attempts at assistive technology are available as smart glasses, or attachment devices that require the user to carry a microphone and require the speaker to speak into a microphone or a smartphone. But unlike its predecessors, TranscribeGlass is an attachment device that can pick up speech or audio from across a lecture hall.

Lavakare struggled to raise funds for his first prototypes. He launched a campaign on Ketto, an Indian crowdfunding platform, and received donations from friends and family, but an assistive-device startup doesn’t exactly fall into a space scouted by financial investors seeking commercial returns. Despite his troubles, Lavakare was able to raise about $7,000 in grant funding from Pfizer, and through the United States-India Science & Technology Endowment Fund, a program that funds scientific technology innovation that has an impact in both countries.

In 2021, Tom Pritsky, then a Stanford University postgraduate studying biomedical informatics, teamed up with Lavakare to help turn the device into a product that could be delivered to users.

As a sophomore, Pritsky had worked as a research assistant in Stanford professor Stefan Heller’s lab, studying the genetic background of chickens that enables them to never permanently lose their hearing. Chickens, as well as certain other birds, fish, reptiles and amphibians, can indefinitely regenerate their hearing by producing new hair cells inside the cochlea, the cavity of the inner ear responsible for interpreting sound. Pritsky and his colleagues wondered if this particular gene was turned off in mammals and could be re-instigated in humans with drugs.

“I wanted to work on the hearing loss problem because I’ve had it all my life,” says Pritsky, who has had moderate to severe bilateral hearing loss since the age of 3.

As the founder of Stanford’s first club for the deaf and hard of hearing, Pritsky would invite speakers to campus. When he became interested in wearable captions, the club brought in the CEO of the captioning app Ava. “I would have companies like Epson send me their [Epson Moverio AR] hardware, and I would put subtitles into them, then walk around with these bulkier headsets and read captions,” Pritsky says. But these modified augmented reality headsets were expensive and heavy, with low battery life, making them impractical for regular use.

When Pritsky first saw TranscribeGlass in 2021, he knew it was revolutionary. “This device gives you critical information in your field of view,” he says. “It was the first time that seemed like a distinct possibility. None of these AR companies were trying to do that for me.”

Pritsky adds: “You can build an expensive device, but if people can’t afford it then it doesn’t really solve the problem. Part of the vision of our company is to make accessibility truly accessible. We want to give people who don’t have access to hearing aids something that they can use.”

The beta version of TranscribeGlass will cost $55, and the final version is expected to sell for around $95.

Weighing less than 15 grams, the AR attachment device can be affixed to any glasses. The TranscribeGlass companion app runs on a mobile device; it connects to a caption source and transfers the caption via phone microphone or Bluetooth microphone to the hardware, which then projects text in the user’s field of view. The text appears at a distance set by the user—anywhere from ten inches to hundreds of feet. The wearer can choose any captioning service from the mobile app, including Automatic Speech Recognition, live captioning for events and business meetings, and subtitle files for movies at the theater. The app also allows users to adjust the font size, text-scrolling speed and other preferences, such as default languages.

“The core application right now is transcription of speech to text,” says Pritsky. “We currently support English for our early users and are expanding support for other languages in the very near future.”

The distance at which the device can pick up audio depends on the source of the audio. When using a Bluetooth microphone as the audio source, the distance is limited to Bluetooth’s range. However, for a longer range, explains Pritsky, “You can integrate, for example, with StreamText, which is a software solution designed for conferences and lecture venues.” TranscribeGlass is using StreamText at Stanford for its pilot trial. “You can sit anywhere in the lecture hall, distance doesn’t matter, because it goes directly to the phone through our app and the Wi-Fi connection,” he says.

The device is in the beta testing phase, with 20 units in circulation, including among some users who paid to be part of the trial.

Parita Dholakia, a resident of Mumbai, India, enrolled as a tester in 2021 and started using TranscribeGlass regularly in March 2022. “As a late-deafened adult with no hearing aid or cochlear implant helping in my case, I rely completely on live transcription and captions to get on with my daily life,” she says. “This magical little device has changed my life. It has worked fabulously in group conversations for me, especially in office environments and closed meeting rooms.”

Dholakia could see the device holding a lot of promise in various scenarios, from doctor visits, to classroom lessons when a teacher is using a blackboard, to movie screenings lacking subtitles.

“Since the app needs the speaker to converse in English, I have not been able to use it much in social settings,” adds Dholakia, who comes from a non-English-speaking family. “I await the day when Indian languages can work well with the device.”

Raja Kushalnagar, who teaches and studies accessible computing at Gallaudet University, finds the augmented reality device impressive. “TranscribeGlass would benefit people who are hard of hearing to be able to read what they misheard or missed, especially when there is background noise, music or when the speakers have accents,” he says. “It would also benefit deaf people who cannot follow speech at all by providing a transcript of the speech, plus some non-speech sounds for environmental awareness.”

However, Kushalnagar isn’t shy to point out the issue of inaccurate transcription in these scenarios. Background noise and accents “are not obvious to speakers or people around the TranscribeGlass user,” he says. These factors can lead to communication frustrations and shift the burden of communication repair to the TranscribeGlass user.

As a regular user of the device, Pritsky believes reading captions of even a few words that are spoken can help the user fill in the blanks and “be better involved in the conversation.”

With a patent pending in India, and thousands of people on the wait list, the duo hopes to release the beta version of TranscribeGlass by the end of 2023.

Ramsha Zubairi | READ MORE

Ramsha Zubairi is a freelance writer. Her work has appeared in BBC Travel, Dwell, Atlas Obscura and other media outlets.