Unreal Engine Avatar Driven by Data Streams

Quick Summary of Technical Details: We have an existing “black box” (our code) that sends four data streams in parallel. Your job is to receive these streams and use them to drive a Metahuman (or similarly realistic) avatar in Unreal Engine, ultimately packaging the result as an executable for MacOS. Specifically: 1. Gestures – Data specifying the avatar’s hand/body movements (e.g., waving, pointing, bowing, etc.). 2. Real-Time Audio Input – Audio stream (no file-based audio) for lip-sync. Preferably leveraging real-time solutions like NVIDIA’s Audio2Face Live Link (or another robust method) for natural speech animation. 3. X,Y Location – Coordinates to dictate where the avatar looks. 4. Animation Toggle (Idle On/Off) – Indicates whether the avatar should be in idle mode or actively animating (talking, gesturing, etc.). Swappable Avatar Requirement: • We want to ensure easy swapping of the avatar if we create or import a custom avatar in the future. The system should allow us to drop in a replacement with minimal effort. The data format for these streams is flexible; we’re open to WebSockets or any recommended method that ensures real-time performance and is fully compatible with Unreal Engine. (Note: Our “black box” software will run on the same machine as the Unreal app.) Full Project Requirements 1. Unreal Engine Metahuman Character (or Custom Avatar) • Design & Development: Create a realistic avatar using Unreal Engine’s Metahuman framework or an equally detailed approach. • Easy Replacement: The avatar pipeline should allow us to swap in a different character model with minimal effort. • Clarification on Realism: While the system should be capable of high-fidelity output, we will likely design our own Metahuman using official Unreal tools, so ultra-realistic character design is not the primary focus. 2. Real-Time Lip-Syncing • Real-Time Input: Use the provided audio stream for immediate lip-sync without file-based processing. • Preferred Technology: Potentially leverage NVIDIA’s Audio2Face Live Link or an equivalent solution to ensure accurate mouth movement. • Accuracy: Aim for natural, accurate mouth movements matching live speech, with low latency for real-time interactivity. 3. Animations • Idle State: An animation state for when the avatar is not speaking. • Smooth Transitions: Seamless shifts between idle, speaking, and gesturing. • Multiple Gestures: Incorporate two to three dozen full-body gestures (e.g., waving hello, pointing, bowing) to add realism and variety. 4. Visual Quality • High Fidelity: Match or exceed visual quality seen on platforms like [login to view URL], [login to view URL], or digitalhumans.com. • Facial Expressions: Realistic micro-expressions and detailed facial rigging. • Textures & Rendering: High-resolution textures and lifelike rendering for a polished final result. 5. Customization • Scene & Backgrounds: Allow for a few swappable background options and lighting setups. • User Configuration: Settings to tweak lighting, environment, and other visual elements. 6. Platform Compatibility • MacOS Executable: The final output must run smoothly on Apple Silicon (M2 or better). • Performance Optimization: Heavy optimization isn’t our top priority given the power of Apple Silicon, but the system should still maintain real-time interaction without sacrificing too much quality. 7. Integration Support • Collaboration: Work with our development team to integrate the avatar into our AI assistant application. • Documentation: Provide clear instructions for implementing the avatar, handling the data streams, and swapping out the character model. Who We’re Looking For • Unreal Engine Expertise: Strong background in animation, Metahumans, and real-time animation. • Lip-Sync Know-How: Familiarity with real-time solutions like NVIDIA Audio2Face or similar. • Animation & Rigging Skills: Deep knowledge of facial and body rigging, blending animations, and expression mapping. • Optimization Experience: Capable of delivering high-quality visuals on MacOS Apple Silicon without performance bottlenecks. • Good Communicator: Comfortable collaborating and iterating with our team. To Bid on This Project 1. Portfolio Samples: Share work that demonstrates relevant experience—especially real-time lip-sync or Metahuman projects. 2. Process Overview: Outline how you’d approach building this avatar system and integrating it with our code/data streams. 3. Tech & Approach: Offer recommendations for data handling (e.g., WebSockets) and real-time animation technologies (Audio2Face, etc.). 4. Timeline & Cost: Provide an estimated schedule and budget.