When it comes to American Sign Language (ASL), artificial intelligence (AI) is fluent.
Engineers at Florida Atlantic University (FAU) have developed a cutting-edge ASL interpretation system that uses a built-in webcam to accurately recognize hand gestures in real time, even tackling the tricky task of telling apart similar signs like A and T, or M and N. ASL consists of distinct hand gestures that represent letters, words and phrases, but previous systems were limited in accuracy and speed. The webcam captures live video and turns it into digital frames.
Developed by FAU, the system achieved an impressive 98.2% accuracy. That high mark is due to a massive training dataset of more than 130,000 images of hands signing ASL letters and phrases.
To ensure the AI could interpret signs reliably, the images were captured from different angles, under varying lighting conditions, against diverse backgrounds. Each image was meticulously annotated with 21 key points, providing a skeletal representation of the hand that helped the system learn the subtle distinctions between similar gestures — a major hurdle in ASL recognition technology.
“This project is a great example of how cutting-edge AI can be applied to serve humanity,” said Imad Mahgoub, Ph.D., co-author and Tecore Professor in FAU’s Department of Electrical Engineering and Computer Science. “By fusing deep learning with hand landmark detection, our team created a system that not only achieves high accuracy but also remains accessible and practical for everyday use. It’s a strong step toward inclusive communication technologies.”
The deaf population in the United States is approximately 11 million, or 3.6% of the population. Another 37.5 million American adults experience hearing difficulties. The World Health Organization (WHO) estimates that there are approximately 466 million people worldwide with disabling hearing loss, about 6.1% of the world’s population.
“What makes this system especially notable is that the entire recognition pipeline — from capturing the gesture to classifying it — operates seamlessly in real time, regardless of varying lighting conditions or backgrounds,” said Bader Alsharif, a principal developer of the system and a doctoral student in FAU’s Department of Electrical Engineering and Computer Science. “And all of this is achieved using standard, off-the-shelf hardware. This underscores the system’s practical potential as a highly accessible and scalable assistive technology, making it a viable solution for real-world applications.”
Stella Batalama, dean of FAU’s College of Engineering and Computer Science, said the system highlights the transformative power of AI-driven assistive technologies. “By bridging the communication gap through real-time ASL recognition, this system plays a key role in fostering a more inclusive society. It allows individuals with hearing impairments to interact more seamlessly with the world around them, whether they are introducing themselves, navigating their environment, or simply engaging in everyday conversations.”
There is no universal sign language; other countries and regions have their own sign languages. In the United Kingdom, for example, there is British Sign Language, and Brazil has Brazilian Sign Language. The words, phrases and sentences differ. With a tool such as the one developed by FAU, there could be obvious advantages for the deaf when traveling abroad.
According to the National Institute on Deafness and Other Communication Disorders, sign language also has regional accents. “Just as with other languages, specific ways of expressing ideas in ASL vary as much as ASL users themselves,” the institute states. “In addition to individual differences in expression, ASL has regional accents and dialects; just as certain English words are spoken differently in different parts of the country, ASL has regional variations in the rhythm of signing, pronunciation, slang and signs used. Other sociological factors, including age and gender, can affect ASL usage and contribute to its variety, just as with spoken languages.”
