Acoustic Primitives

Modeling and Driving Human Body Soundfields through Acoustic Primitives

1University of Rochester, Rochester, NY     2Codec Avatars Lab, Meta, Pittsburgh, PA

ECCV 2024

tl;dr: We propose Acoustic Primitives -- smaller and compact soundfield representations similar to volumetric primitives in computer graphics

Abstract

While rendering and animation of photorealistic 3D human body models have matured and reached an impressive quality over the past years, modeling the spatial audio associated with such full body models has been largely ignored so far. In this work, we present a framework that allows for high-quality spatial audio generation, capable of rendering the full 3D soundfield generated by a human body, including speech, footsteps, hand-body interactions, and others. Given a basic audio-visual representation of the body in form of 3D body pose and audio from a head-mounted microphone, we demonstrate that we can render the full acoustic scene at any point in 3D space efficiently and accurately. To enable near-field and realtime rendering of sound, we borrow the idea of volumetric primitives from graphical neural rendering and transfer them into the acoustic domain. Our acoustic primitives result in an order of magnitude smaller soundfield representations and overcome deficiencies in near-field rendering compared to previous approaches.

More contents coming soon...

BibTeX

If you find our work helpful, please consider citing:
@inproceedings{huang2024modeling,
      author    = {Huang, Chao and
                   Markovic, Dejan and
                   Xu, Chenliang and
                   Richard, Alexander},
      title     = {Modeling and Driving Human Body Soundfields through Acoustic Primitives},
      booktitle = {European Conference on Computer Vision},
      year      = {2024},
    }