Brain-computer interface makes breakthrough by deciphering Chinese speech in brain

SHANGHAI -- Chinese startup NeuroXess on Thursday reported two significant clinical-trial milestones: its flexible brain-computer interface (BCI) device successfully decoded the precise intended movements of one patient with a brain injury in real time, and decoded Chinese speech in real time for another.

The patients have demonstrated the prowess of BCI technology, using their minds to control software, pick up objects, operate a digital avatar through speech, and converse with an AI model.

In August 2024, neurosurgeons at Huashan Hospital, which is affiliated with Fudan University, implanted a 256-channel, high-throughput flexible BCI device designed by the Shanghai-based NeuroXess into a 21-year-old female patient with epilepsy who had a space-occupying lesion in the motor area of her brain.

The team extracted electrocorticogram (ECoG) features from the high-gamma band of the patient's brain signals and trained a neural network model to decode in real time, achieving a system delay under 60 milliseconds and accurately mapping brain function areas within minutes of the surgery, according to NeuroXess.

The high-gamma band (70-150 hertz) of ECoG signals tends to correlate with complex cognition and neural synchronization, offering detailed brain activity data, especially movement and sensory information.

In implantable BCI trials, extracting this band of signals helps to decode brain intentions more accurately, said Tao Hu, founder and chief scientist of NeuroXess.

Within the 48 hours following the procedure, the patient successfully engaged in table tennis and snake computer games through brain control, without any physical movement.

Following two weeks of training, the patient was able to use XessOS, NeuroXess' brain-computer operating system, adeptly to operate common smartphone apps like WeChat and Taobao, and to control smart home and wheelchair systems through brain control, significantly enhancing her daily-life capabilities.

CHINESE SPEECH DECODING

Language is humanity's most sophisticated high-level cognitive function, and deciphering speech from brain signals represents an exciting frontier in the evolution of BCI technology.

Chinese — a monosyllabic, tonal and logographic language — differs from alphabetic languages like English, and its processing utilizes more brain regions, thus demanding specialized neural mechanisms and methods, Tao said.

In December 2024, the collaborative team carried out the nation's first clinical trial of a high-throughput, flexible BCI to synthesize Chinese speech.

They implanted the 256-channel BCI in a female epilepsy patient with a tumor in the language area of the brain. The patient recovered well and achieved 71 percent speech decoding accuracy using 142 common Chinese syllables within five days, with a single-character decoding latency of under 100 milliseconds.

This result was the highest-level real-time decoding of Chinese speech that has been achieved in China, according to NeuroXess. A picture released by the company shows that with the prompt "Glad to meet you," the device decoded the patient's thoughts, displaying the Chinese characters for "Glad to meet" on a screen.

A video the company also released shows that the patient's brain was able to directly control robotic hands to grasp an apple and a pear, and to perform sign language for "Happy New Year, 2025."

The device also enabled the patient to operate a digital avatar to say "Hello" and interact with the AI model, which can be seen in the video. The company has proclaimed this breakthrough as the world's first "mind-to-AI large model" dialogue.

BCI, a technology that heralds the future, is swiftly emerging as a focal point for global innovation. Scientists and engineers are eager to integrate this pioneering tech into the realm of medical practice, embracing a new era of therapeutic possibilities.

Last year, Elon Musk announced that his Neuralink startup had implanted a device in a human and that the patient recovered well, with promising neuron spike detection.

"Neuralink's strategy requires implantation into the cerebral cortex, which can cause damage to brain tissue," Tao said. "Our approach involves the cerebral cortex, which can effectively avoid potential damage to brain tissue."

The NeuroXess scientists aim to use their BCI device to enhance the lives of individuals with speech or motor function disabilities resulting from conditions such as Amyotrophic Lateral Sclerosis (ALS), high-level paraplegia and stroke.