Teaching Alexa to sound like she means it

Neural TTS · Amazon Alexa · 2019

Ask Alexa for a sports score in 2017 and you got the facts, read flat. Fans noticed. One piece of user research said it plainly:

"Alexa needs to be more colorful and use more words like thrashed the opponent, or crushed them, instead of just saying that they defeated them. Alexa should be as excited as me when my teams win big games, and remorseful if my teams should lose."— Sports superfan research

Fans on a couch erupting mid-celebration — From the research sessions: this is the energy a score update is competing with.

I led the design of Alexa's move to neural text-to-speech for expressive responses — taking the work from research and concept through prototype, testing, and launch.

What the research said

The need for contextual variation. The same score should not sound the same in a blowout and a nail-biter.
Emotional connection drives satisfaction. As of August 2017, product reviews that mention "Alexa" or a pronoun rated +0.104 stars higher — people who related to Alexa as a someone rated the product better.
A strong preference for sentiment. In Alexa Language Tech research, 74% of users preferred audio samples with sentiment; 69% found speechcons enjoyable and 82% found them engaging.

74%

of users preferred speech samples with sentiment over neutral ones

+0.104★

higher product ratings when reviews mentioned Alexa with emotional connection

82%

found speechcons — expressive interjections — engaging

A model for feeling

We mapped responses on the circumplex model of affect (Posner et al., 2005): two axes — sentiment (positive/negative) and intensity (high/low). A 10–0 win is elated; a 1–0 win is relieved; a loss is upset or bored depending on how it happened. The model gave the team a shared, designable vocabulary instead of vibes:

"Wow … we demolished the Mets, 10–0."
"Whew … our Cubs squeaked by the Mets, 1–0."
"They lost to the Nationals, 6 to 3 — but they'll re-match at Wrigley on Monday."

The circumplex model of affect, redrawn in the site's dark theme from the original deck: the flat — The circumplex model of affect (Posner et al., 2005), from the deck's back-to-back slides: today's delivery lives in the flat middle — the "Current Experience" — and the work was expanding it out to the full range.

Tenets

Adaptation and variation are key — vary responses to the customer's context to surprise and delight; avoid generating lasting negative reactions.
Align sentiment with affinities — favor the customer's team, their likes, their anticipated emotion.
Purposeful, readable decisions — emotion chosen deliberately, to sound authentic — never random.
Dynamic, scalable output — techniques that generalize across emotions and use cases. Go beyond sports.

Techniques

Two levers, layered over time. Adaptive prompting: curating specific word, phrase, and sentence choices in frequently requested prompts to match the content — "Earlier today, the Cubs crushed the Mets, 10–0…" And full-prompt emotional expression: carrying an emotion across a whole clause or answer with neural TTS markup, so the delivery itself — not just the words — does the feeling.

The Echo light ring: the other half of an expressive response

The work was prototyped as complete multimodal moments — voice and screen designed together. This concept film paired the updated prompt and prosody with the GUI: a Cubs blowout, delivered the way a fan would want to hear it.

The matchup as audio alone — first the standard delivery, then the expressive one the film below demonstrates:

The Cubs card — standard read

The score update in Alexa's standard delivery

The Cubs card — expressive read

The same update with sentiment: a 10–0 win that sounds like one

Concept film from the deck: "Neural TTS samples — updated prompt and prosody, w/GUI." The multimodal sports card builds as Alexa delivers the line.

Process

Proposal + prototype → test + tune → approval → audit → launch. We prototyped with real audio early (including a feedback form that let the wider org rate Alexa's personality traits), iterated on prompt and prosody with the science teams, and audited output before launch — the same loop of prototype, evaluate, and tune I now run on LLM behavior at Meta.

Alexa personality traits table: extraversion, emotional stability, agreeableness, conscientiousness, and openness, with the language cues for each — Alexa's personality targets — the traits we designed to, with the language cues for each.

The TTS feedback form: naturalness and emotion-rendering rating grids across anger, fear, happy, and sad, with an acceptable score of 8 — The test-and-tune feedback form: every sample rated for naturalness and for how well the emotion landed. The bar for "acceptable" was 8.

Outcome

The work shipped as part of Alexa's neural TTS rollout on AWS Polly's neural engine — sports answers first, with the strategy written to layer expression across more of Alexa over time.

AWS Polly neural TTS use cases: contact center, training materials, content creation, education and e-learning, internet of things, gaming and entertainment — Where neural TTS went next — the "go beyond sports" map from the deck: contact centers, training, content creation, education, IoT, and gaming. This slide in the deck linked out to the announcement coverage — tap through for The Verge on the neural voice launch.

Its most public debut came that fall, when the same neural voice technology got a famous first user:

Samuel L. Jackson at a studio microphone, in front of a backdrop reading — Samuel L. Jackson — announced in September 2019 as Alexa's first celebrity voice, built on neural text-to-speech. Tap through for The Verge's coverage of the launch.