We present MotionStream — real-time, long-duration video generation that you can interactively control just by dragging your mouse.
All videos here are raw, real-time screen captures without any post-processing. Model runs on a single H100 at 29 FPS and 0.4s latency.
It feels totally different when video models actually become real-time interactive. Drag the mouse, and the cup instantly moves with it, and the water follows. You’re not just watching a video anymore, you’re playing with it.
We achieve this by first fine-tuning a video diffusion model conditioned on point track motions. We then apply Self-Forcing style causal distillation to post-train it into a real-time interactive model.
To reduce error accumulation of long video generation, we apply attention sink during the training and inference.
For maximum speed in real-time scenario, we incorporate several additional design optimizations:
- Efficient encoding of track conditions via simple sinusoidal encoding and input-channel concatenation
- Local, sliding-window causal attention
- Lightweight VAE decoder
Using point tracks as a unified motion representation allows us to control not only the object motion but also the camera, all in real time.
We use monocular depth estimation to lift the input image into 3D and uniformly sample 3D points. Then, as the virtual camera moves, we project each 3D point into the new view to create 2D motion trajectories.
It supports real-time video editing.
Given an input video stream, we first edit the initial frame. We can estimates point tracks from the incoming video in real time. MotionStream then generates the output on the fly, conditioned on the edited initial frame and tracked points.
Work led by Joonghyuk Shin (@jhshin2000), and done with wonderful collaborators @zhengqi_li @rzhang88 @junyanz89 Jaesik Park @elishechtman
Arxiv: https://arxiv.org/abs/2511.01266
Check out our project page for more details!
https://joonghyuk.com/motionstream-web/
