![](\"https:\/\/static01.nyt.com\/images\/2023\/08\/23\/autossell\/23avatar-vid1\/23avatar-vid1-videoSixteenByNine1050.jpg\"){kind=avatar}
<\/p>\nAt Ann Johnson\u2019s wedding reception 20 years ago, her gift for speech was vividly evident. In an ebullient 15-minute toast, she joked that she had run down the aisle, wondered if the ceremony program should have said \u201cflutist\u201d or \u201cflautist\u201d and acknowledged that she was \u201chogging the mic.\u201d<\/p>\n
Just two years later, Mrs. Johnson \u2014 then a 30-year-old teacher, volleyball coach and mother of an infant \u2014 had a cataclysmic stroke that paralyzed her and left her unable to talk.<\/p>\n
On Wednesday, scientists reported a remarkable advance toward helping her, and other patients, speak again. In a milestone of neuroscience and artificial intelligence, implanted electrodes decoded Mrs. Johnson\u2019s brain signals as she silently tried to say sentences. Technology converted her brain signals into written and vocalized language, and enabled an avatar on a computer screen to speak the words and display smiles, pursed lips and other expressions.<\/p>\n
The research, published in the journal Nature, demonstrates the first time spoken words and facial expressions have been directly synthesized from brain signals, experts say. Mrs. Johnson chose the avatar, a face resembling hers, and researchers used her wedding toast to develop the avatar\u2019s voice.<\/p>\n
\u201cWe\u2019re just trying to restore who people are,\u201d said the team\u2019s leader, Dr. Edward Chang, the chairman of neurological surgery at the University of California, San Francisco.<\/p>\n
\u201cIt let me feel like I was a whole person again,\u201d Mrs. Johnson, now 48, wrote me.<\/p>\n
The goal is to help people who cannot speak because of strokes or conditions like cerebral palsy and amyotrophic lateral sclerosis. To work, Mrs. Johnson\u2019s implant must be connected by cable from her head to a computer, but her team and others are developing wireless versions. Eventually, researchers hope, people who have lost speech may converse in real time through computerized pictures of themselves that convey tone, inflection and emotions like joy and anger.<\/p>\n
\u201cWhat\u2019s quite exciting is that just from the surface of the brain, the investigators were able to get out pretty good information about these different features of communication,\u201d said Dr. Parag Patil, a neurosurgeon and biomedical engineer at the University of Michigan, who was asked by Nature to review the study before publication.<\/p>\n
<\/p>\n
Mrs. Johnson\u2019s experience reflects the field\u2019s fast-paced progress. Just two years ago, the same team published research in which a paralyzed man, who went by the nickname Pancho, used a simpler implant and algorithm to produce 50 basic words like \u201chello\u201d and \u201chungry\u201d that were displayed as text on a computer after he tried to say them.<\/p>\n
Mrs. Johnson\u2019s implant has nearly twice as many electrodes, increasing its ability to detect brain signals from speech-related sensory and motor processes linked to the mouth, lips, jaw, tongue and larynx. Researchers trained the sophisticated artificial intelligence to recognize not individual words, but phonemes, or sound units like \u201cow\u201d and \u201cah\u201d that can ultimately form any word.<\/p>\n
\u201cIt\u2019s like an alphabet of speech sounds,\u201d David Moses, the project manager, said.<\/p>\n
While Pancho\u2019s system produced 15 to 18 words per minute, Mrs. Johnson\u2019s rate was 78 using a much larger vocabulary list. Typical conversational speech is about 160 words per minute.<\/p>\n
When researchers began working with her, they didn\u2019t expect to try the avatar or audio. But the promising results were \u201ca huge green light to say, \u2018OK, let\u2019s try the harder stuff, let\u2019s just go for it,\u2019\u201d Dr. Moses said.<\/p>\n
They programmed an algorithm to decode brain activity into audio waveforms, producing vocalized speech, said Kaylo Littlejohn, a graduate student at the University of California, Berkeley, and one of the study\u2019s lead authors, along with Dr. Moses, Sean Metzger, Alex Silva and Margaret Seaton.<\/p>\n
\u201cSpeech has a lot of information that is not well preserved by just text, like intonation, pitch, expression,\u201d Mr. Littlejohn said.<\/p>\n
Working with a company that produces facial animation, researchers programmed the avatar with data on muscle movements. Mrs. Johnson then tried to make facial expressions for happy, sad and surprised, each at high, medium and low intensity. She also tried to make various jaw, tongue and lip movements. Her decoded brain signals were conveyed on the avatar\u2019s face.<\/p>\n
Through the avatar, she said, \u201cI think you are wonderful\u201d and \u201cWhat do you think of my artificial voice?\u201d<\/p>\n
\u201cHearing a voice similar to your own is emotional,\u201d Mrs. Johnson told the researchers.<\/p>\n
She and her husband, William, a postal worker, even engaged in conversation. She said through the avatar: \u201cDo not make me laugh.\u201d He asked how she was feeling about the Toronto Blue Jays\u2019 chances. \u201cAnything is possible,\u201d she replied.<\/p>\n
The field is moving so quickly that experts believe federally approved wireless versions might be available within the next decade. Different methods might be optimal for certain patients.<\/p>\n
On Wednesday, Nature also published another team\u2019s study involving electrodes implanted deeper in the brain, detecting activity of individual neurons, said Dr. Jaimie Henderson, a professor of neurosurgery at Stanford and the team\u2019s leader, who was motivated by his childhood experience of watching his father lose speech after an accident. He said their method might be more precise but less stable because specific neurons\u2019 firing patterns can shift.<\/p>\n
Their system decoded sentences at 62 words per minute that the participant, Pat Bennett, 68, who has A.L.S., tried to say from a large vocabulary. That study didn\u2019t include an avatar or sound decoding.<\/p>\n
Both studies used predictive language models to help guess words in sentences. The systems don\u2019t just match words but are \u201cfiguring out new language patterns\u201d as they improve their recognition of participants\u2019 neural activity, said Melanie Fried-Oken, an expert in speech-language assistive technology at Oregon Health & Science University, who consulted on the Stanford study.<\/p>\n
![](\"https:\/\/static01.nyt.com\/images\/2023\/08\/23\/autossell\/23avatar-vid2cover\/23avatar-vid2cover-videoSixteenByNine1050.jpg\"){kind=avatar}
<\/p>\n
Neither approach was completely accurate. When using large vocabulary sets, they incorrectly decoded individual words about a quarter of the time.<\/p>\n
For example, when Mrs. Johnson tried to say, \u201cMaybe we lost them,\u201d the system decoded, \u201cMaybe we that name.\u201d But in nearly half of her sentences, it correctly deciphered every word.<\/p>\n
Researchers found that people on a crowdsourcing platform could correctly interpret the avatar\u2019s facial expressions most of the time. Interpreting what the voice said was harder, so the team is developing a prediction algorithm to improve that. \u201cOur speaking avatar is just at the starting point,\u201d Dr. Chang said.<\/p>\n
Experts emphasize that these systems aren\u2019t reading people\u2019s minds or thoughts. Rather, Dr. Patil said, they resemble baseball batters who \u201care not reading the mind of the pitcher but are kind of interpreting what they see the pitcher doing\u201d to predict pitches.<\/p>\n
Still, mind reading may ultimately be possible, raising ethical and privacy issues, Dr. Fried-Oken said.<\/p>\n
Mrs. Johnson contacted Dr. Chang in 2021, the day after her husband showed her my article about Pancho, the paralyzed man the researchers had helped. Dr. Chang said he initially discouraged her because she lived in Saskatchewan, Canada, far from his lab in San Francisco, but \u201cshe was persistent.\u201d<\/p>\n
Mr. Johnson, 48, arranged to work part time. \u201cAnn\u2019s always supported me to do what I\u2019ve wanted,\u201d including leading his postal union local, he said. \u201cSo I just thought it was important to be able to support her in this.\u201d<\/p>\n
She started participating last September. Traveling to California takes them three days in a van packed with equipment, including a lift to transfer her between wheelchair and bed. They rent an apartment there, where researchers conduct their experiments to make it easier for her. The Johnsons, who raise money online and in their community to pay for travel and rent for the multiyear study, spend weeks in California, returning home between research phases.<\/p>\n
\u201cIf she could have done it for 10 hours a day, seven days a week, she would have,\u201d Mr. Johnson said.<\/p>\n
Determination has always been part of her nature. When they began dating, Mrs. Johnson gave Mr. Johnson 18 months to propose, which he said he did \u201con the exact day of the 18th month,\u201d after she had \u201calready gone and picked out her engagement ring.\u201d<\/p>\n
Mrs. Johnson communicated with me in emails composed with the more rudimentary assistive system she uses at home. She wears eyeglasses affixed with a reflective dot that she aims at letters and words on a computer screen.<\/p>\n
It\u2019s slow, allowing her to generate only 14 words per minute. But it\u2019s faster than the only other way she can communicate at home: using a plastic letter board, a method Mr. Johnson described as \u201cher just trying to show me which letter she\u2019s trying to try to look at and then me trying to figure out what she\u2019s trying to say.\u201d<\/p>\n
The inability to have free-flowing conversations frustrates them. When discussing detailed matters, Mr. Johnson sometimes says something and receives her response by email the next day.<\/p>\n
\u201cAnn\u2019s always been a big talker in life, an outgoing, social individual who loves talking, and I don\u2019t,\u201d he said, but her stroke \u201cmade the roles reverse, and now I\u2019m supposed to be the talker.\u201d<\/p>\n
Mrs. Johnson was teaching high school math, health and physical education, and coaching volleyball and basketball when she had her brainstem stroke while warming up to play volleyball. After a year in a hospital and a rehabilitation facility, she came home to her 10-year-old stepson and her 23-month-old daughter, who has now grown up without any memory of hearing her mother speak, Mr. Johnson said.<\/p>\n
\u201cNot being able to hug and kiss my children hurt so bad, but it was my reality,\u201d Mrs. Johnson wrote. \u201cThe real nail in the coffin was being told I couldn\u2019t have more children.\u201d<\/p>\n
For five years after the stroke, she was terrified. \u201cI thought I would die at any moment,\u201d she wrote, adding, \u201cThe part of my brain that wasn\u2019t frozen knew I needed help, but how would I communicate?\u201d<\/p>\n
Gradually, her doggedness resurfaced. Initially, \u201cmy face muscles didn\u2019t work at all,\u201d she wrote, but after about five years, she could smile at will.<\/p>\n
She was entirely tube-fed for about a decade, but decided she wanted to taste solid food. \u201cIf I die, so be it,\u201d she told herself. \u201cI started sucking on chocolate.\u201d She took swallowing therapy and now eats finely chopped or soft foods. \u201cMy daughter and I love cupcakes,\u201d she wrote.<\/p>\n
When Mrs. Johnson learned that trauma counselors were needed after a fatal bus crash in Saskatchewan in 2018, she decided to take a university counseling course online.<\/p>\n
\u201cI had minimal computer skills and, being a math and science person, the thought of writing papers scared me,\u201d she wrote in a class report. \u201cAt the same time, my daughter was in grade 9 and being diagnosed with a processing disability. I decided to push through my fears and show her that disabilities don\u2019t need to stop us or slow us down.\u201d<\/p>\n
Helping trauma survivors remains her goal. \u201cMy shot at the moon was that I would become a counselor and use this technology to talk to my clients,\u201d she told Dr. Chang\u2019s team.<\/p>\n
At first when she started making emotional expressions with the avatar, \u201cI felt silly, but I like feeling like I have an expressive face again,\u201d she wrote, adding that the exercises also enabled her to move the left side of her forehead for the first time.<\/p>\n
She has gained something else, too. After the stroke, \u201cit hurt so bad when I lost everything,\u201d she wrote. \u201cI told myself that I was never again going to put myself in line for that disappointment again.\u201d<\/p>\n
Now, \u201cI feel like I have a job again,\u201d she wrote.<\/p>\n
Besides, the technology makes her imagine being in \u201cStar Wars\u201d: \u201cI have kind of gotten used to having my mind blown.\u201d<\/p>\n