[DigitalToday reporter Chi-gyu Hwang (황치규)] Google has unveiled SL2T, an AI model that converts sign language into text, and applied it first to Pixel 11.

SiliconANGLE reported on Aug. 12 that SL2T converts sign language into text when deaf or hard-of-hearing users sign toward a smartphone camera.

SL2T is a multilingual sign-language translation model. The initial version translates American Sign Language into English text.

Users can search the web in sign language without typing, or draft emails and text messages. It can also be used for document editing. Users can also ask Gemini questions to get answers or request tasks to be carried out. In Live Transcribe, users can input responses in sign language during in-person conversations.

Google estimates the global deaf and hard-of-hearing population at about 70,000,000. It said users of more than 200 sign languages have not been sufficiently addressed in the AI industry. SL2T was designed to let them interact with smartphones in sign language.

Training for SL2T used more than 100,000 hours of data across more than 50 sign languages. About a quarter of that was American Sign Language data. At the initial launch stage, however, the only sign language it can understand is American Sign Language. Google explained it trained multiple sign languages together so the model could learn structural patterns shared across sign languages.

SL2T uses an on-device computer-vision model, MediaPipe Holistic, to track the positions of the face, hands, arms and torso, then sends only coordinate information to cloud servers rather than actual video. Google said this is faster and safer because it does not upload actual video.

SL2T converts sign-language motions directly into text without going through an intermediate text-annotation step known as gloss. It includes optimisations to reduce latency, safeguards to prevent hallucinations from movements that are not sign language, and support for left-handed users and one-handed signing.

SL2T scored 70 on BLEURT on the FLEURS-ASL benchmark. Google has also formed an AI sign-language advisory committee with deaf organisations and sign-language experts. The committee plans to support responsible application of the technology and co-author a report outlining SL2T's capabilities and current limitations. Google plans to expand supported sign languages and develop sign-language generation models.