iiinova — where your digital self keeps evolving

Ziyang Chen

@czyang.bsky.social

Ph.D. Student @ UMich EECS. Multimodal learning, audio-visual learning and computer vision. Prev research Intern @Adobe and @Meta ificl.github.io

Posts Replies Media

Ziyang Chen @czyang.bsky.social · 27/11/2024

This work is done during my internship at Adobe Research. Big thanks to all my collaborators @pseeth.bsky.social, Bryan Russell, @urinieto.bsky.social, David Bourgin, @andrewowens.bsky.social, and @justinsalamon.bsky.social!

050

Ziyang Chen @czyang.bsky.social · 27/11/2024

We jointly train our model on high-quality text-audio pairs as well as videos, enabling our model to generate full-bandwidth professional audio with fine-grained creative control and synchronization.

130

Ziyang Chen @czyang.bsky.social · 27/11/2024

MultiFoley is a unified framework for video-guided audio generation leveraging text, audio, and video conditioning within a single model. As a result, we can do text-guided foley, audio-guided foley (e.g. sync your favorite sample with the video), and foley audio extension.

120

Ziyang Chen @czyang.bsky.social · 27/11/2024

🎥 Introducing MultiFoley, a video-aware audio generation method with multimodal controls! 🔊 We can ⌨️Make a typewriter sound like a piano 🎹 🐱Make a cat meow like a lion roars! 🦁 ⏱️Perfectly time existing SFX 💥 to a video. arXiv: arxiv.org/abs/2411.17698 website: ificl.github.io/MultiFoley/

24212