Understanding Amazon Chime SDK live transcription messages

The Amazon Chime SDK service shares transcription information with attendees by sending TranscriptEvent objects in data messages. A TranscriptEvent delivers a Transcript or a TranscriptionStatus.

A Transcript includes results with timestamped, user-attributed words and punctuation. A result may be “partial”, in which case the system usually updates it in a subsequent TranscriptEvent. This allows you to see transcriptions quickly and apply inline updates later as necessary.

A TranscriptStatus may deliver one of the TranscriptionStatusType events, listed in the example in the next section.

Newer versions of the Amazon Chime SDKs include additional data types and helper functions for common processing a TranscriptEvent.

TranscriptEvent

This example shows a typical transcription event.

type TranscriptEvent = Transcript | TranscriptionStatus;

export class TranscriptEventConverter {
  static from(dataMessage: DataMessage): TranscriptEvent[] {
    // convert DataMessage to TranscriptEvents
    return ...
  }
}

export default class TranscriptionStatus {
    type: TranscriptionStatusType;
    eventTimeMs:                   number;
    transcriptionRegion:           string;
    transcriptionConfiguration:    string;
    message?:                      string;
}

enum TranscriptionStatusType {
    STARTED        =    'started',
    INTERRUPTED    =    'interrupted',
    RESUMED        =    'resumed',
    STOPPED        =    'stopped',
    FAILED         =    'failed',
}

export default class Transcript {
    results: TranscriptResult[];    // at least one
}

export class TranscriptResult {
    resultId:        string;
    isPartial:       boolean;
    startTimeMs:     number;
    endTimeMs:       number;
    alternatives:    TranscriptAlternative[];    // most confident first
    }

export default class TranscriptAlternative {
    items: TranscriptItem[];    // in start time order
    transcript: string; //concatenated transcript items
    entities?: TranscriptEntity[];
}

export default class TranscriptItem {
    type:                      TranscriptItemType;
    startTimeMs:               number;
    endTimeMs:                 number;
    attendee:                  Attendee;
    content:                   string;
    vocabularyFilterMatch?:    boolean;
    confidence?:               number;  
    stable?:                   boolean;
}

enum TranscriptItemType {
    PRONUNCIATION    =    'pronunciation',// content is a word
    PUNCTUATION      =    'punctuation',// content is punctuation
}

export default class TranscriptEntity {  
    category:       string;  
    confidence:     number;  
    content:        string;  
    endTimeMs:      number;  
    startTimeMs:    number;  
    type?:          string;
}

// This is an existing SDK model
export default class Attendee {
    attendeeId:        string;
    externalUserId:    string;
}

Data guidelines

Keep these guidelines in mind as you go.

Registering event handlers for TranscriptEvents

The following examples use the Amazon Chime SDK client library for JavaScript. However, the pattern is consistent across all Amazon Chime SDKs.

The TranscriptionController in the RealtimeController and RealtimeControllerFacade includes specific functions for adding a handler that processes TranscriptionEvents:

/** 
 * Returns the [[TranscriptionController]] for this real-time controller. 
 */
readonly transcriptionController?: TranscriptionController; 

The TranscriptionController has two functions to manage subscribing and unsubscribing to TranscriptionEvent callbacks:

import TranscriptEvent from './TranscriptEvent';

export default interface TranscriptionController {
  /**
   * Subscribe a callback to handle received transcript event
   */
  subscribeToTranscriptEvent(callback: (transcriptEvent: TranscriptEvent) => void): void;

  /** 
   * Unsubscribe a callback from receiving transcript event 
   */
  unsubscribeFromTranscriptEvent(callback: (transcriptEvent: TranscriptEvent) => void): void;
}

Using the optional TranscriptionController

We provide a default implementation of TranscriptionController interface named DefaultTranscriptionController. The default implementation in DefaultRealtimeController and DefaultAudioVideoFacade returns a DefaultTranscriptionController object:

/** 
get transcriptionController(): TranscriptionController {
   return this.realtimeController.transcriptionController;
}

DefaultRealtimeController also takes an optional TranscriptionController object in its constructor. That allows you to override the DefaultTranscriptionController behavior. Developer applications subscribe and unsubscribe to one or more callbacks through the TranscriptionController object of the AudioVideoFacade object:

// Subscribe
this.audioVideo.transcriptionController?.subscribeToTranscriptEvent(this.transcriptEventHandler);

// Unsubscribe
this.audioVideo.transcriptionController?.unsubscribeFromTranscriptEvent(this.transcriptEventHandler););