Avatar, Animation, Lipsync ‐ General

Animation System

This page details the animation system used in our application for avatar animations, including full-body animations and lip-sync. It covers the technical aspects of the implementation, how animations are loaded and applied, and how to add custom animations.

Table of Contents

• Overview
• Key Components
• Animation Loading and Application
• Blending and Transitioning
• Lip-Sync Animation
• Performance Considerations
• Adding Custom Animations
• Known Issues and Solutions

Overview

Our animation system primarily uses Three.js for handling 3D animations. It supports both full-body animations and lip-sync, with the ability to blend between different animations smoothly. The system is designed to be performant, allowing for multiple animation previews simultaneously.

Key Components

1. Avatar.tsx: Main component for avatar rendering and animation.
2. AnimationPreview.tsx: Component for previewing individual animations.
3. LipSyncControlPanel.tsx: Handles lip-sync animations based on audio input.

Animation Loading and Application

Animations are loaded using the GLTFLoader from Three.js. Here's a basic example of how animations are loaded and applied:

import { GLTFLoader } from 'three/examples/jsm/loaders/GLTFLoader';

const loader = new GLTFLoader();
loader.load(animationUrl, (gltf) => {
  const animationClip = gltf.animations[0];
  const action = mixer.clipAction(animationClip);
  action.play();
});

Blending and Transitioning

The system supports smooth blending between animations using Three.js' built-in crossFadeTo method:

if (currentActionRef.current && currentActionRef.current !== newAction) {
  newAction.reset();
  newAction.play();
  currentActionRef.current.crossFadeTo(newAction, 0.5, true);
  currentActionRef.current = newAction;
}

This allows for seamless transitions between different animations, such as from an action back to an idle state.

Lip-Sync Animation

Lip-sync is implemented using blend shapes (morph targets). The system uses a mapping from visemes to blend shapes:

const visemeToBlendShape: { [key: string]: string[] } = {
  'X': ['viseme_sil', 'viseme_sil'],
  'A': ['viseme_PP', 'viseme_PP'],
  // ... other mappings ...
};

Lip-sync animation is performed by interpolating between different visemes based on audio cues:

const animateLipsync = (delta: number) => {
  // ... find current viseme based on audio time ...
  const shapes = Object.keys(targetValuesRef.current);
  const values = shapes.map(shape => {
    const current = currentValuesRef.current[shape] || 0;
    const target = targetValuesRef.current[shape] || 0;
    return THREE.MathUtils.lerp(current, target, lerpFactorRef.current);
  });
  setBlendShapes(shapes, values);
};

Performance Considerations

The animation system is designed to be performant, allowing for multiple animation previews to run simultaneously. This is achieved by:

1. Using efficient Three.js methods for animation mixing and updating.
2. Updating animations in the useFrame hook to sync with the render loop.
3. Optimizing blend shape calculations and applications.

Adding Custom Animations

Custom animations can be added to the system in two formats:

1. .glb: Preferred format for Ready Player Me avatars. Higher compatibility and fewer issues.
2. .fbx: Can be directly added from Mixamo, but may cause compatibility issues.

To add a new animation:

1. Place the animation file in the appropriate directory (e.g., /public/animations/).
2. Load the animation in your component:

loader.load('/animations/your-new-animation.glb', (gltf) => {
  const newAnimation = gltf.animations[0];
  const action = mixer.clipAction(newAnimation);
  action.play();
});

Known Issues and Solutions

1. Rotation Issue with Mixed Formats:

   • Issue: Mixing .fbx and .glb animations can cause brief undesired rotations.
   • Solution: Fully reset the avatar to an idle or neutral state before transitioning to the next animation.

   Example implementation:

   const resetAndPlayAnimation = (newAnimationUrl: string) => {
     // First, crossfade to idle animation
     idleActionRef.current!.play();
     currentActionRef.current!.crossFadeTo(idleActionRef.current!, 0.5, true);
     
     // After reset, load and play new animation
     setTimeout(() => {
       loader.load(newAnimationUrl, (gltf) => {
         const newAnimation = gltf.animations[0];
         const newAction = mixer.clipAction(newAnimation);
         newAction.play();
         idleActionRef.current!.crossFadeTo(newAction, 0.5, true);
       });
     }, 500); // Adjust timeout as needed
   };

2. Teeth Animation:

   • The Wolf3D_Teeth mesh uses a separate 'mouthOpen' morph target instead of individual visemes.
   • Ensure the 'mouthOpen' morph target is properly set in the setBlendShapes function.
   • As the teeth visemes are somewhat missing some of the animations, it is recommended to target them separately and applying desired amplitudes to 'mouthOpen'