Amazon Polly performs text-to-speech using machine learning, specifically, generative AI and TTS technologies. Its neural TTS models contain hundreds of millions of parameters, while its generative models leverage billion-plus-parameter transformers that convert raw text into speech codes. These codes are then processed by a convolution-based decoder, which generates waveforms in an incremental, streamable manner. Amazon Polly is available pursuant to the AWS Customer Agreement or other relevant agreements with AWS.

We say that an Amazon Polly model exhibits a particular "behavior" when it generates the same kind of output for the same kinds of inputs and configuration (for example, pause, emphasis, volume, pronunciation, etc.). For a given model architecture, the control levers that we have over the behaviors are primarily:

Our development process exercises these control levers as follows:

Performance of Polly will differ between applications, even if they support the same use case. Consider two applications A and B: Application A delivers real-time transport announcements in public spaces, requiring clear enunciation and audibility in noisy environments. Application B uses the same voice to read bedtime stories, where softness, pacing, and emotional tone are more critical. Even though the voice and system are identical, the differing demands of each use case led to different perceived performance. Developers using the Amazon Polly service should consider using different TTS engines as well as SSML tags to make sure that the resulting output is effective for the use case they are empowering.

Now consider two versions of Application B, each using a different voice. One voice is neutral and flat; the other is expressive and warm. Although the task—reading bedtime stories—remains the same, the listener experience and engagement can vary significantly depending on the voice's characteristics. This illustrates how both use case context and voice selection influence the effectiveness and quality of TTS output, even within the same deployment.

As a result, the overall utility of Amazon Polly will depend both on the selected voice and on the workflows it enables. Since performance results depend on a variety of factors including Amazon Polly's engine itself, the customer workflow, and the evaluation dataset, we recommend that customers test Amazon Polly using their own content.

With AI/ML services, performance results vary with the evaluation dataset used to generate the results. We recommend that customers test all AI/ML services from any vendor (not just AWS) on their own content. We measure the overall performance of the service along four effectiveness criteria: intelligibility, naturalness, prosody, and contextual appropriateness. The following sections contain information related to the methodology, metrics, datasets, and results used to evaluate Amazon Polly.

Amazon Polly's performance is evaluated using multiple complementary approaches to help provide consistent, high-quality synthetic speech across a wide range of use cases. Each voice is assessed individually against quality criteria including intelligibility, naturalness, prosody, and contextual appropriateness.

This test-driven methodology helps make the voices clear, natural, expressive, and robust and automated evaluation helps ensure all deployed voices meet stringent criteria for clarity, naturalness, expressiveness, and robustness.

Voice Quality Assurance (VoiceQA): For each voice/locale we release, we synthesize 1,000+ audio samples from a range of text types (e.g., customer service, fiction, non-fiction) which were evaluated by 50+ native listeners to check for various kinds of errors, such as mispronunciations, missing text (cutoffs), and extra text (hallucinations).

Customer Satisfaction (CSAT): A sample of 50 or more native listeners evaluate 15 audio samples split into three genres and score how much they like the voice across a range of characteristics in each genre. They also provide an overall score based on the complete set of audio samples.

Amazon Polly frequently launches new TTS voices in existing languages and new languages for AWS customers to use in their speech workflows. Amazon Polly maintains a high-quality bar for voices with Standard TTS and Neural TTS engines, with most of the TTS voices scoring a voice CSAT of 5.5 (on a scale of 1-7) in human listening evaluations and a critical pronunciation error rate of less than 5%. The Generative TTS/GTTS engine, launched in May 2024, has a voice CSAT of 5.8 (on a scale of 1-7) and critical pronunciation errors less than 3%. We continue to maintain and enhance the quality bar for all voices available on Amazon Polly with every model update.

For each voice, the CSAT is calculated by having at least 50 human evaluators listen to 15 audio samples and rate their satisfaction on a 1 to 7 scale, where 1 means very dissatisfied and 7 means very satisfied. The CSAT score is the average of the ratings from all evaluators.

For Voice QA, human evaluators listen to between 1,000 and 2,000 prompts and flag any critical errors. These are categorized as inconsistent speaker identity, cutoff, audio glitch, hallucination, inappropriate persona, intonation, pausing, pronunciation, and text normalization. The Voice QA Error% is the proportion of prompts with critical errors.

Voice Quality results by Gender

Voice Quality results by Engine

We use multiple datasets and human teams to evaluate the performance of Amazon Polly models. No single evaluation dataset suffices to completely capture performance. This is because evaluation datasets vary based on use case, intrinsic and confounding variation, the quality of ground truth available, and other factors. Our development testing involves automated testing against proprietary speech datasets; human evaluation of generated speech for intelligibility, naturalness, and expressiveness; manual red teaming; and more. Our development process examines Amazon Polly's performance using all these tests and takes steps to improve the model and the suite of evaluation datasets.

Automated testing provides apples-to-apples comparisons between candidate Polly models by substituting an automated "assessor" mechanism for human judgment, which can vary. Automated assessments can take several forms. One form is to evaluate generated speech using multiple speech datasets to assess intelligibility, naturalness, prosody, and contextual appropriateness. One industry standard measurement of this form is Mean Opinion Score (MOS).

Amazon Polly synthesizes speech from text using underlying linguistic representations such as phonemes and prosody markers. The system is designed to render input text accurately according to these representations. Customers can use SSML to control pronunciation, speaking rate, pitch, emphasis, and language for specific words to improve alignment with intended content. Evaluations of generated speech consider intelligibility, naturalness, prosody, and contextual appropriateness across voices and languages using multiple licensed datasets and listener studies. Performance is measured using metrics such as Mean Opinion Score (MOS), naturalness ratings, error rates, and listener preference studies, along with additional internal benchmarks, to provide high-quality synthetic speech.

Our goal is for Amazon Polly to accurately generate audio from text for diverse range of human voices for different speaker groups. We define speaker groups by demographic attributes such as age, gender and locales. We test on dataset with those demographic labels. As an example, on a 1,000 to 2,000 prompts per local dataset across 29 locales, male voices show slightly higher Voice QA percentage (0.27 percentage points higher) and marginally better customer satisfaction (0.07 points higher on the 7-point scale). Both genders have similar performance with high CSAT scores (>5.9) and low Voice QA percentages (<1.5%).

Robustness refers to a model's ability to perform reliably across a wide range of inputs, variations of spellings, or context-dependent pronunciation (e.g. 'live' a life vs 'live' telecast of a game), conditions, and speaker variations. Amazon Polly maximizes robustness by training on large and diverse datasets, including recordings from speakers of different ages, genders, accents, and speaking styles. Training on both formal and conversational speech helps the voices perform well across a variety of content types. Rigorous evaluation across voices, languages, and scenarios helps provide consistent, high-quality performance in real-world applications. We recommend that customers use SSML controls, voice and language selection, and speech parameter customization, along with maintaining workflow consistency and conducting periodic testing, to help support robustness in their own applications. In our test on a dataset of 1,000 to 2,000 prompts per local dataset across 29 locales. The Generative engine achieved 1.79% Voice QA Error rate and 6.15 CSAT score. The Long-form engine achieved 1.53% error rate and 6.09 CSAT score. The Neural engine achieved 0.77% error rate and 5.75 CSAT score.

Amazon Polly is a managed service and prompts and generations are never shared across customers. AWS does not store nor use inputs or outputs generated through the Amazon Polly service to train Amazon Polly service. For more information, see Section 50.3 of the AWS Service Terms and the AWS Data Privacy FAQs. For service-specific privacy information, see Data Privacy in the Amazon Polly FAQs.

Customers can use AWS PrivateLink to establish private connectivity between customized Amazon Polly models and on-premises networks without exposing customer traffic to the internet. For example, Amazon CloudWatch can help track usage metrics that are required for audit purposes, and AWS CloudTrail can help monitor API activity and troubleshoot issues as Amazon Polly is integrated with other AWS systems. Customer data is always encrypted in transit and at rest. Amazon Polly is not storing inputs nor outputs for text-to-speech synthesis functionality. The only customer provided content that is stored in the service is "pronunciation lexicons" encrypted with service owned encryption keys. Additionally, customers can also choose to store output generated through asynchronous synthesis functionality in their own encrypted Amazon Simple Storage Service (Amazon S3) bucket.

Amazon Polly provides information to customers through the following locations: the Amazon Polly AWS page, Polly FAQs, AWS educational channels (for example, blogs, developer classes), the AWS Console, and this Service Card. We accept feedback through customer support mechanisms such as account managers. Where appropriate for their use case, customers who incorporate Amazon Polly in their workflow should consider disclosing their use of ML to end users and other individuals impacted by the application, and customers should give their end users the ability to provide feedback to improve workflows. In their documentation, customers can also reference this Service Card.

We have rigorous methodologies to build our AWS AI services responsibly, including a working backwards product development process that incorporates Responsible AI at the design phase, design consultations, and implementation assessments by dedicated Responsible AI science and data experts, routine testing, reviews with customers, best practice development, dissemination, and training.