Chinese TV Uses AI To Translate Broadcasts Into Sign Language. Its Not Going Well

China’s efforts to employ AI as a means of improving access to media for its deaf population aren’t going well, according to a professor at Beijing Normal University’s Faculty of Education.

Writing at an outlet called Sixth Tone, professor Zheng Xuan notes that China is home to 20.5 million people with hearing disabilities, and that the nation’s government has encouraged use of avatars and virtual presenters to provide real-time translation of some television programs.

Techno-optimists believe these flaws will be solved, but we shouldn’t ignore the ethical harm

Those efforts started at the 2022 Beijing Winter Olympic Games, which Zheng studied to assess the quality of the sign language translations – and came away unimpressed.

“We transcribed and back-translated the sign language created by the avatars, then compared the results with the original audio, finding that a significant amount of key information was lost or distorted in the AI-generated version,” she wrote.

Deaf people struggled to understand the AI-generated sign language.

“On closer inspection, the movements of the avatars differed considerably from everyday sign language in terms of hand shape, position, direction, and movement,” she wrote. “Other issues were even more prominent — the avatars’ facial expressions and body language were off, and their mouth movements were distorted.”

In interviews, viewers reported “they generally couldn’t understand the avatars’ movements and noted that they seemed to have a limited vocabulary, while struggling to handle words with multiple meanings.”

Zheng thinks the AIs did badly because “Chinese words cannot be found for the meanings expressed by 50 percent of gestures in Chinese sign language.”

Developers overlooked “the difference between signed and spoken language. In particular, many perceive sign language as an accessory to spoken language, or else believe that translating between the two is similar to translating between two spoken languages.”

“But the modalities of spoken language and sign language are quite different,” she wrote. “The former is an oral-auditory language, while the latter is a visual-gestural or visual-spatial language. The term ‘gestural’ is a relatively broad concept that includes not just hand movements, but also facial expressions and body language. Full utilization of the body in space allows sign language users to express the meaning of an entire sentence – such as ‘a person walks into a room’ – with just one action.”

The diversity of Chinese sign language is also problematic. Zheng wrote that Chinese sign language includes “‘natural sign language’,” which originates from the daily lives of Deaf people, and ‘signed Chinese language’, which is an expression of Chinese characters using signs.”

“The language used by most Chinese Deaf people lies somewhere on a spectrum between the two,” she wrote, adding that dialects of sign language are another complication for developers.

Building a translation avatar to handle all those nuances is hard and made more difficult by the scanty supply of useful data to inform AIs.

• Top AI models - even American ones - parrot Chinese propaganda, report finds
• AI-hosted infomercial shifts $7.5 million worth of product in China
• Canada orders Chinese CCTV biz Hikvision to quit the country ASAP
• China spawns an x86 supercomputing monster, with an AMD connection

Zheng also criticized Chinese technology companies, which she said “have not involved sign language linguists or Deaf people in any great depth. Even in the cases where sign language teachers or interpreters are included, developers often only slot them into supporting roles, instead of taking the opinions of Deaf users as the final arbiter of their products’ effectiveness.”

That’s not just a jab: Zheng worked as consultant for a team working on translation avatars and found them poorly equipped for the task.

“They seemed to underestimate the difficulty involved, overestimated the power of tech to solve problems, and lacked the necessary experience, resources, and ability to judge the quality of work done by third-party companies,” she wrote. “By the time I joined the project, these shortcomings had already become apparent. Although the development team welcomed my participation, I felt that this respect was more for my technical knowledge as a university professor rather than my identity as a Deaf person.”

Zheng said that when she pointed out the product wouldn’t satisfy users, “my feedback was not fully embraced, as the developers seemed unable to fully empathize with my frustrations.”

She now feels there are “fundamental issues with the way tech approaches the problem of sign language translation” because “Tech companies are used to first launching a version that has a lot of bugs, then optimizing it through a large amount of user feedback.”

The products she’s seen were so bad, she fears they harm the deaf community’s faith in technological solutions.

“That’s not to mention the fact that some companies mislead users by promoting their products using real humans rather than avatars, and then release an immature generative AI version,” she wrote. “Techno-optimists may believe that these flaws will all be solved with time, but we shouldn’t ignore the irreversible ethical harm: If the real needs of Deaf users are not responded to, they’ll feel that they’re being treated as guinea pigs.” ®