Speech, Talking Avatar, and Lip Sync

Use API key authentication for all operations on this page. These mutations return Content!; use tasks and results to follow their output.

Text to speech

TextToSpeech

mutation TextToSpeech($metadata: SimpleText2SpeechContentMetadata!) {
  textSpeechSimplify(metadata: $metadata) {
    id
    task { id status progress { finishCode } }
  }
}

Variables

{
  "metadata": {
    "prompt": "Welcome to the studio.",
    "voiceModelId": "<VOICE_MODEL_ID>"
  }
}

prompt: String! supplies the speech and voiceModelId: String! selects a ready voice model. Optional emotion: Text2SpeechEmotion and speed: Text2SpeechSpeed use enums; this speed is different from the numeric voiceSpeed used below.

Talking Avatar with existing audio

Animate a source image with an accessible audio media reference:

TalkingAvatar

mutation TalkingAvatar($metadata: SimpleAudio2VideoContentMetadata!) {
  talkingAvatarAudioVideoSimplify(metadata: $metadata) {
    id
    task { id status progress { finishCode } }
  }
}

Variables

{
  "metadata": {
    "modelId": "<IMAGE_MODEL_ID>",
    "srcImageMediaId": "<IMAGE_MEDIA_ID>",
    "srcAudioMediaId": "<AUDIO_MEDIA_ID>",
    "outputQuality": "best"
  }
}

Lip Sync with existing audio

Provide a video source and the audio to synchronize:

LipSync

mutation LipSync($metadata: SimpleAudio2VideoContentMetadata!) {
  videoLipSyncSimplify(metadata: $metadata) {
    id
    task { id status progress { finishCode } }
  }
}

Variables

{
  "metadata": {
    "modelId": "<IMAGE_MODEL_ID>",
    "srcVideoMediaId": "<VIDEO_MEDIA_ID>",
    "srcAudioMediaId": "<AUDIO_MEDIA_ID>"
  }
}

Use a script instead of existing audio

Talking Avatar and Lip Sync support script synthesis with a selected voice. Request a duration quote for the exact script and speed before submission:

QuoteAvatarScript

query QuoteAvatarScript($script: String!, $voiceSpeed: Float) {
  calculateTalkingAvatarScriptCredit(script: $script, voiceSpeed: $voiceSpeed) {
    id
    estimatedAudioDurationMs
    expiresAt
  }
}

Variables

{
  "script": "Welcome to the studio.",
  "voiceSpeed": 1.0
}

Submit the avatar or lip-sync mutation with the same script and speed, a ready voice model, and the returned id as voiceScriptQuoteId. Omit existing-audio inputs when choosing script synthesis. For example, the variables for TalkingAvatar become:

TalkingAvatar variables

{
  "metadata": {
    "modelId": "<IMAGE_MODEL_ID>",
    "srcImageMediaId": "<IMAGE_MEDIA_ID>",
    "voiceModelId": "<VOICE_MODEL_ID>",
    "voiceScript": "Welcome to the studio.",
    "voiceSpeed": 1.0,
    "voiceScriptQuoteId": "<QUOTE_ID>",
    "outputQuality": "best"
  }
}

The quote estimates speech duration; it is not generation completion or a payment receipt. If the script or speed changes or the quote expires, obtain a matching quote. When voiceScriptQuoteId is omitted, the generation flow calculates a quote during submission.

Shared avatar and lip-sync metadata

Both operations use SimpleAudio2VideoContentMetadata:

Field	Type	Meaning
`modelId`	`String!`	Required ready image model.
`srcImageMediaId`	`ID`	Existing image for Talking Avatar.
`srcVideoMediaId`	`ID`	Existing video for Lip Sync.
`srcAudioMediaId`	`ID`	Existing audio input.
`inputImage`, `inputVideo`, `inputAudio`	Uploaded file inputs	Alternatives for integrations that provide verified file references.
`remixImage`, `remixVideo`, `remixAudio`	Remix file inputs	Alternative source references.
`voiceModelId`	`ID`	Required for script synthesis.
`voiceScript`	`String`	Script synthesis input for these two workflows.
`voiceScriptQuoteId`	`ID`	Optional existing quote for the exact script, speed, and account. Must still be valid when supplied.
`voiceSpeed`	`Float`	Script speed multiplier, `0.5` to `2.0`, default `1.0`.
`emotion`	`Text2SpeechEmotion`	Optional overall synthesis emotion.
`inputPrompt`	`String`	Optional style instructions.
`promptModifierIds`	`[ID!]`	Supported prompt modifiers.
`outputQuality`	`OutputQuality`	Quality supported by the chosen operation. Talking Avatar defaults to `best` when omitted.
`resolution`	`VideoResolution`	Target resolution where supported.

Ordinary image to video and Extend place dialogue in prompt text. Their input contracts differ from the script synthesis contract on this page.

Get started

Core concepts

Workflows

GraphQL API

Usage and troubleshooting

Speech, Talking Avatar, and Lip Sync

Text to speech

Talking Avatar with existing audio

Lip Sync with existing audio

Use a script instead of existing audio

Shared avatar and lip-sync metadata

​Text to speech

​Talking Avatar with existing audio

​Lip Sync with existing audio

​Use a script instead of existing audio

​Shared avatar and lip-sync metadata

Text to speech

Talking Avatar with existing audio

Lip Sync with existing audio

Use a script instead of existing audio

Shared avatar and lip-sync metadata