Skip to main content
Back to Blog

Research Preview: Pika Enact

Creating Scenes that Respond to You

November 18, 2026Research


At Pika, we believe the next generation of AI creation should go beyond the prompt box, and more closely resemble how we already make films. Pika Enact is an exciting step in that direction.

Pika Enact is designed to mimic the experience of directing on set. It’s an experimental live-streaming environment built on Pika H3 Flex that generates video and sound in sub-second chunks, carrying scene context forward while incorporating new direction into the ongoing generation.

This allows the creator to interact with “actors,” manually re-stage a scene, and manipulate the camera—all in real time.

Pika Enact enables three new modes of interaction:

  1. Live Staging. Place and reposition actors or objects within a running video with your mouse. The scene continues while you move them.
  2. Live Actor Direction: Speak directly to actors on set, and shape their performances. They’ll remember your notes, and deliver a fresh, continuous take when you call “Action.”
  3. Live Camera Control. Direct camera movement with your mouse around a live, playing scene.

Below, we walk through each one: the data they learn from, their technical architecture, and some demo videos.

1. Live Staging

Scene design and actor blocking are mainstays of a live directing experience. But AI video creation makes that process inexact and time-consuming. With Enact, rather than waiting for a generation to load in order to see the outcome of your direction, you get instantaneous feedback; you can manually shape the scene as it plays.

Lifting, moving and setting down people across several scenes. A red dot marks a held mouse button; a white ring marks release.

The live staging within Pika Enact lets the creator drag an object or actor into place with their mouse, which creates several key benefits:

  • Precise placement: Where a prompt can only suggest “a bit to the left,” a cursor drags subjects to the exact intended spot.
  • Consistency with environment: The model generates whatever you move—a person or a chair—together with the rest of the scene. As you drag, the subject responds naturally to the movement, while its pose, lighting, and shadows adapt to its surroundings. The result has no cut-out edges, no hole where it stood, and no compositing pass to clean up afterwards.
  • Real-time responsiveness: The video keeps playing while you drag, and on a single server the model generates it 1.8× faster than it plays. Additionally, the subject actively reacts to being moved, shifting weight and adjusting posture mid-air before settling naturally where you let go.

To keep up with the cursor movement, we distill the model to a few steps and stream the mouse input and decoding in small pieces. In our low-latency setting, the picture responds in under 0.4 seconds on H100, decoding included.

The Data Engine

Training interactive video dragging requires strictly consistent before-and-after video pairs synchronized with frame-level cursor trajectories, which real-world footage cannot provide.

We built a physics-driven synthetic data engine to solve this. Starting from an untouched video, we animate a 3D stand-in along a drag path to simulate natural weight and momentum. Our video models then render photorealistic target clips while automatically logging mouse positions and button states.

This produces scalable, zero-labeling training pairs where only the dragged subject changes, covering lifting, multi-step carries, and inserting new subjects from photos into live scenes. The dataset drives our entire pipeline, from full-clip training and real-time distillation to evaluation.

For data access or more detail on how the data was made, contact our team at enterprise@pika.art.

City square: bringing a person into the playing scene.
Left: source video, nobody touches the person. Right: training target, the person is lifted and carried.

Technical Architecture

To make Pika H3 Flex draggable in real time, it has to learn two things: follow a cursor, and generate a stream instead of a clip.

Live Staging: mouse in, video out. The system watches how the mouse moves and uses that to predict what the video should show next.
Live Staging: mouse in, video out. The system watches how the mouse moves and uses that to predict what the video should show next.

To follow a drag in real time, Pika H3 Flex injects three frame-aligned signals via lightweight, zero-initialized modules: 24 fps mouse states, a spatial grab heatmap, and the untouched source video to keep the background intact. Rather than generating full clips at once, the model streams short causal chunks (0.7 s standard, 0.17 s low-latency) through a sliding key-value cache that retains the prompt, source video, first chunk, and four most recent chunks, with re-based positional embeddings for infinite sessions at constant cost.

To cut latency and prevent autoregressive drift, we apply self-forcing distribution matching distillation (DMD). A few-step student generates chunks directly from its own past outputs, guided by a trainable critic and the full-clip teacher with emphasized cursor signals. This reduces sampling from twenty steps to just a few while keeping cursor tracking sharp and stable.

2. Live Actor Direction

Once everything has been staged, the next step is typically directing your cast. With Enact, the people in a video become actors you can talk to. Type or say a note, such as “move closer to her”, “tell him why you’re upset” or “make it a kitchen at night”, and they play it out while the video keeps running.

A rehearsal with Enact: greeting the actors, then running the scene line by line. Edited for length; subtitles and the director’s webcam inset were added afterwards.

This mimics the experience of directing on set in several important ways:

  • Communicate conversationally: Give a note and the actors act on it; talk to them and they answer you in character, in their own voices. The video generates picture and sound 1.75× faster than they play, so the interaction happens at a real-time pace.
  • Rehearse first, then shoot. Work through a performance iteratively. Give notes, have the actors try gain, and when it feels right, call Action. Everything you rehearsed becomes one clean take, shot fresh from the first frame.
  • The actors have minds and memories of their own. Each is an AI agent with a distinct personality, its own logic, and the ability to think and respond in character. They remember what you liked, what you didn't, and everything that's happened in the scene. You never have to repeat a note or start from scratch. Every interaction builds on the last, making the characters feel more natural, consistent, and alive.

Data

During distillation, the real-time model learns from Pika H3 Flex itself: it generates audio-video clips chunk by chunk from its own output, and the full Pika H3 Flex model’s judgment of those clips is what it learns toward. For more on the training data, data access or custom training, contact our team at enterprise@pika.art.

Technical Architecture

Live Direction is an agentic model within Pika Enact that thinks and performs at the same time. It understands what you mean, plans what should happen next, remembers your notes, and acts the scene out live, picture and voices together. A script model does the thinking and a real-time video model does the acting. And while you talk to the agent, the performance never stops.

The Live Actor Direction works as one agent: notes become beats, beats become chunks of picture and sound, and the actors’ answers come back to you while the take keeps running.
The Live Actor Direction works as one agent: notes become beats, beats become chunks of picture and sound, and the actors’ answers come back to you while the take keeps running.

A script language model reads your notes together with the opening frame and session memory. Through tool calls, it translates directorial input into structured beats: action, dialogue, emotion, gaze, and lighting. When no instruction is queued, the scene holds naturally with subtle breathing and blinking.

At its core, Pika H3 Flex is distilled into a few-step causal audio-video model. This model uses a sliding key-value cache and self-forcing distribution matching distillation (DMD). By attending to the opening anchor frame and recent seconds of context, it keeps characters, sets, and native voices consistent across long takes. The opening anchor can pull idle scenes back toward the opening state. To counter this, each beat specifies actor gaze. In addition, Real-Time Director silently injects periodic hold beats to preserve lighting or staging changes.

During rehearsal, actors acknowledge the camera and converse with you. Every line is logged verbatim. Calling Action prompts the script model to rewrite that exchange into a seamless, uninterrupted performance that starts fresh from the first frame.

3. Live Camera Control

Framing and camera movement are what turn a scene into something cinematic. Pika Enact lets you steer a virtual camera with your mouse while a scene is continuously generating—panning, dollying in for a close-up, or orbiting around the subjects.

A live session with Live Camera Control: starting from an appearance reference and a 3D motion proxy, moving the virtual camera on the fly while the characters and room stay consistent.
  • You direct the lens in real time. Move and drag your mouse while the stream is playing to adjust camera angles, perspective, and movement on the fly—shaping upcoming frames as the scene unfolds instead of waiting for offline renders.
  • No existing video needed: Unlike post-hoc video editing that modifies an existing clip, the system generates audio and video from scratch. An appearance image anchors the cast and set, while a lightweight 3D motion proxy (such as an untextured Blender viewport) guides the camera path.
  • It runs faster than real time. On an 8×NVIDIA B300 node (640×480 @ 24 fps), the model renders initial frames in 0.51 s and generates subsequent chunks at 1.50× real-time speed, leaving headroom for proxy capture and streaming.

Data

To teach Pika H3 Flex how free-roaming 3D camera moves affect both visual perspective and spatial acoustics, training data must be tightly synchronized down to the frame level.

  1. Virtual Soundstage Capture: We built an automated Blender-to-Video synthesis pipeline that simultaneously records multi-angle camera trajectories, untextured 3D motion reference streams, target RGB video frames, and spatial audio tracks in controllable 3D environments.
  2. Rigorous Quality Filtering: Using automated optical flow checks and cross-frame identity verification, we filter out clipping artifacts and subject inconsistencies, retaining only high-signal-to-noise pairs for training.

Technical Architecture

Two stages turn offline 3D supervision into a live, responsive camera generator: a full-reference teacher that sees the entire sequence, and a causal student that generates chunk by chunk as you move the mouse.

Two offline training stages: the full-reference teacher learns from paired sequences; DMD trains the causal student on its own rollouts. At inference, the student reads current Blender references and generated history to produce successive audio-video chunks.
Two offline training stages: the full-reference teacher learns from paired sequences; DMD trains the causal student on its own rollouts. At inference, the student reads current Blender references and generated history to produce successive audio-video chunks.

During streaming, the causal student reads live 3D camera motion, the appearance reference, and its own history. It generates about 0.71 s chunks of synchronized audio and video. These chunks respond immediately to virtual camera movement. The student is distilled from the full-reference H3 teacher with self-forcing distribution matching distillation (DMD) to few denoising steps per chunk. It attends to the opening anchor frame and a sliding history window. The window carries synchronized relative positions. This design preserves character likeness and room geometry across long orbits at bounded memory cost.

To exceed real-time playback speed, we cache static appearance prefixes across chunks. We reuse fixed memory layouts and routing metadata with zero numerical loss. We also decode latents through a streaming video decoder. Together, these optimizations cut steady-state compute to 0.473 s per 0.708 s chunk, or 1.50 x real time.

What’s Next

Enact is an early look at what happens when video listens to you—and responds—while it plays. In our own tests, we were startled by how much more real and visceral it feels to generate video this way. So, while it’s only in research preview right now, we think this is the beginning of a new paradigm for creation. It’s the immediacy of directing in real life paired with the expansive possibilities of generative media.

Contact Us

Currently, Enact is a research preview, unavailable for consumer use. If you’re interested in trying it or talking about integration opportunities, contact our team at enterprise@pika.art.