Avatoon: 3D Talking Avatar with Lip-Sync in React

Avatoon is a free, open-source React Three Fiber component for rendering an animated 3D talking avatar with real-time lip-sync that stays in sync with audio. It works on the web and in React Native (Expo) using the same API, and it’s built for voice assistants, interactive characters, and storytelling apps. This guide explains what Avatoon is, how it works, how to install it, and when to reach for it — with copy-paste code you can run today.

TL;DR: Install with npm install avatoon, drop in the <Avatoon /> component, feed it a GLB model plus viseme timing data, and you get a talking 3D character with lip-sync, subtle head motion, and automatic eye-blinking. It’s MIT-licensed and maintained by Khaled Alam. Repository: github.com/khaledalam/avatoon.

What is Avatoon?

Avatoon is a TypeScript library that renders an animated 3D talking avatar inside any React application. Under the hood it uses React Three Fiber (the React renderer for Three.js) and drei to draw a GLTF/GLB avatar and drive its mouth in real time from viseme data — the visual counterparts of speech sounds. The result is a character whose lips move naturally while it “talks,” complete with small idle motions that make it feel alive.

The core idea is simple: instead of hand-animating every frame, you give Avatoon a stream of phoneme-to-viseme mappings with timestamps, and it interpolates between mouth shapes so your 3D talking avatar appears to speak your audio. Because it’s built on React Three Fiber, it slots directly into any modern React or Next.js 3D scene, and the same component API also runs on React Native with Expo.

Why build a 3D talking avatar into your app?

Voice is everywhere — AI assistants, language-learning apps, customer-support bots, games, and interactive stories. But voice alone can feel flat. A 3D talking avatar gives users something to look at, builds trust, and dramatically improves engagement and retention. Avatoon exists to make that avatar layer trivial to add, so you can focus on your product instead of writing a custom Three.js animation pipeline.

• AI voice assistants — put a face on your LLM or TTS pipeline so responses feel conversational.
• Interactive characters — NPCs, guides, tutors, or brand mascots that speak to the user.
• Storytelling and edutainment apps — narrated 3D characters for kids’ apps, language learning, or explainers.
• Customer support and onboarding — a friendly animated agent that walks users through flows.
• Accessibility — visible mouth movement supports lip-reading and comprehension.

Key features of Avatoon

• Real-time lip-sync using phoneme–viseme mapping that stays aligned with your audio track.
• Subtle head motion while the avatar is talking, so it never looks frozen.
• Automatic eye-blinking when the model supports it, for lifelike idle behavior.
• Cross-platform — one API for both web (DOM canvas) and React Native (Expo, via expo-av).
• Plug-and-play with React Three Fiber and drei — it fits inside your existing 3D scene.
• Imperative playback control through a ref exposing play(), stop(), and toggle().
• Goal-based gestures — presets like Normal, Muscle, or Sleep to set the avatar’s mood/pose.
• Two avatar tiers — T1 (static, photorealistic) and T2 (expressive, morph-target enabled).
• TTS converters built in for Azure Speech, AWS Polly, and Rhubarb Lip Sync.
• MIT licensed and fully open source.

How Avatoon works: phonemes, visemes, and morph targets

Human speech is made of phonemes (distinct sound units). Each phoneme has a corresponding mouth shape called a viseme. Lip-sync animation is the art of showing the right viseme at the right moment. Avatoon represents visemes with compact single-letter codes — for example, A for an open mouth, B for closed lips, and X for silence — and it also understands the Oculus / Ready Player Me viseme names used by many avatar models.

For expressive T2 avatars, those visemes are mapped to morph targets (a.k.a. blend shapes) baked into the GLB model — the same ARKit/Oculus blend shapes used across the industry. Avatoon blends between them over time so transitions look smooth rather than snapping between poses. T1 avatars are static and photorealistic; they’re ideal when you want a lifelike still face with head motion but don’t need full mouth morphing.

You supply the timing as a viseme JSON object that maps timestamps to viseme codes, and can optionally embed a base64-encoded WAV so the audio and animation travel together. Avatoon reads that timeline, drives the mouth, and layers in head movement and blinking automatically.

Installation

Avatoon ships on npm. Install it together with its peer dependencies:

npm install avatoon react react-dom three @react-three/fiber @react-three/drei

Version requirements: React ≥ 18, three ≥ 0.153.0, @react-three/fiber ≥ 8.0.0, and @react-three/drei ≥ 9.0.0. For React Native, Avatoon uses expo-av for audio playback.

Quick start: your first 3D talking avatar

The fastest way to see Avatoon in action is the standalone LipSyncAvatoon component. It renders a self-contained avatar with a built-in Start/Stop button and procedural mouth animation — no audio file required, which is perfect for a first test:

import { Canvas } from "@react-three/fiber";
import { LipSyncAvatoon } from "avatoon";

export default function Demo() {
  return (
    <Canvas camera={{ position: [0, 1.5, 3] }}>
      <LipSyncAvatoon glbUrl="/models/avatar.glb" />
    </Canvas>
  );
}

When you want real, audio-synced speech, reach for the full Avatoon component and feed it viseme timing data:

import { Canvas } from "@react-three/fiber";
import { Avatoon } from "avatoon";
import visemeJson from "./speech.visemes.json";

export default function TalkingAvatar() {
  return (
    <Canvas camera={{ position: [0, 1.5, 3] }}>
      <Avatoon
        glbUrl="/models/avatar.glb"
        goal="Normal"
        visemeJson={visemeJson}
        showPlayVoiceButton
      />
    </Canvas>
  );
}

Prefer to control playback yourself — say, when your assistant finishes generating a response? Grab a ref and call the imperative API:

import { useRef } from "react";
import { Avatoon } from "avatoon";

function Assistant() {
  const avatarRef = useRef(null);

  const speak = () => avatarRef.current?.play();
  const hush  = () => avatarRef.current?.stop();

  return (
    <Avatoon ref={avatarRef} glbUrl="/models/avatar.glb" visemeJson={data} />
  );
}

The ref exposes play(), stop(), and toggle() — enough to wire the avatar to any event in your app, from a chat response arriving to a user pressing a button.

Understanding the viseme JSON format

The visemeJson prop expects a VisemeData object: a timeline that maps timestamps (in seconds or milliseconds) to viseme codes. Conceptually it looks like a list of “at this time, show this mouth shape” cues. Avatoon supports the single-letter codes (A, B, C, D, E, F, G, I, J, K, and X for silence) as well as the standard Oculus / Ready Player Me viseme names. You can optionally include a base64-encoded WAV so the component owns both the sound and the animation.

You rarely have to author this by hand. Avatoon includes converters that turn output from popular text-to-speech engines straight into its format:

• fromAzureVisemes — convert Azure Speech SDK viseme events.
• fromPollySpeechMarks — convert AWS Polly speech marks.
• fromRhubarb — convert output from Rhubarb Lip Sync, a popular offline analyzer.

This means you can plug Avatoon into an existing TTS pipeline: generate speech with Azure, Polly, or your own audio processed by Rhubarb, run the matching converter, and pass the result to the component. The avatar does the rest.

T1 vs. T2 avatars: which should you use?

If you’re building a talking assistant or interactive character, choose a T2 model with morph targets — for example, an avatar exported from Ready Player Me, which ships the standard viseme blend shapes Avatoon expects. If you only need a realistic face with idle head motion, T1 keeps things simple.

Using Avatoon in React Native and Expo

One of Avatoon’s biggest advantages is that the same component API works on mobile. On React Native with Expo, Avatoon renders through React Three Fiber’s native bindings and plays audio with expo-av, while on the web it uses the DOM canvas and HTMLAudioElement. You write your 3D talking avatar once and ship it to iOS, Android, and the browser — no separate animation code per platform. That’s a rare property in the 3D-avatar space and a major reason to pick Avatoon if you’re building cross-platform.

Goal-based gestures and idle life

Beyond talking, Avatoon supports goal-based gestures through the goal prop — presets such as Normal, Muscle, and Sleep that set the avatar’s overall pose and behavior. Combined with automatic eye-blinking and subtle head motion during speech, these small details cross the line from “3D model on screen” to “character that feels present.” Idle liveliness is what makes users comfortable talking to an avatar, and Avatoon handles it for you by default — the difference between a static mesh and a 3D talking avatar that feels present rather than robotic.

How does Avatoon compare to building it yourself?

You could wire up Three.js, load a GLTF, parse viseme timing, interpolate morph targets, add blinking timers, and reconcile web vs. native audio APIs by hand. Teams routinely spend weeks on exactly this. Avatoon packages those hard parts — phoneme–viseme mapping, morph-target blending, idle animation, cross-platform audio, and TTS converters — behind a single React component. Because it’s built on the popular React Three Fiber and drei stack, it also composes cleanly with lighting, environments, cameras, and any other 3D content you already render — the fastest path to a production 3D talking avatar without writing your own engine.

Getting started and contributing

Ready to add a 3D talking avatar to your product? Avatoon is MIT licensed and open source, authored by Khaled Alam. Install it with npm install avatoon, star or fork the repository, and open issues or pull requests on GitHub:

• Repository: github.com/khaledalam/avatoon
• Install: npm install avatoon
• License: MIT © Khaled Alam

Frequently asked questions

What is Avatoon used for?

Avatoon is used to add animated 3D talking avatars with real-time lip-sync to React and React Native apps — voice assistants, interactive characters, tutors, storytelling apps, and support bots. You give it a 3D model and viseme timing data, and it renders a character that speaks in sync with your audio.

Is Avatoon free?

Yes. Avatoon is free and open source under the MIT license, so you can use it in personal and commercial projects.

Does Avatoon work with React Native?

Yes. Avatoon exposes the same component API on web and React Native (Expo). On mobile it uses expo-av for audio; on the web it uses the standard DOM canvas and HTMLAudioElement.

What avatar models does Avatoon support?

Avatoon loads GLTF/GLB models. For full lip-sync, use a T2 model with ARKit/Oculus morph-target blend shapes (for example, avatars from Ready Player Me). Static T1 models are supported for photorealistic, non-morphing faces.

How do I generate lip-sync data for Avatoon?

Use the built-in converters. fromAzureVisemes, fromPollySpeechMarks, and fromRhubarb turn output from Azure Speech, AWS Polly, and Rhubarb Lip Sync into Avatoon’s viseme JSON format automatically.

What is a viseme?

A viseme is the visual mouth shape that corresponds to a speech sound (phoneme). Lip-sync works by showing the right viseme at the right time. Avatoon represents visemes with compact single-letter codes and also supports Oculus / Ready Player Me viseme names.

Related reading: if you’re exploring AI-powered creative tools beyond avatars, see how ArtifyMaster’s AI face-swap feature transforms photos.

Conclusion

Talking avatars used to mean a custom Three.js animation project. Avatoon collapses that into a single React component: install it, point it at a GLB model and some viseme timing, and you get real-time lip-sync, idle head motion, and automatic blinking on both web and mobile. Whether you’re putting a face on an AI assistant, building an interactive character, or narrating a story, Avatoon gives you a production-friendly, MIT-licensed starting point. Try it with npm install avatoon and explore the source at github.com/khaledalam/avatoon.