Parallel Adapter Composition · Real-Time Joint Audio

Method

One frozen backbone, two independent adapters, one addition.

Three routes to a few-step streaming model and what the ordering costs — **Three routes to a few-step streaming model, and what the order costs.** Distilling last keeps the chain viable, at two extra stages. The reverse order, trained with a plain regression loss, un-distils: the regression pulls the few-step jump field back toward instantaneous velocity, so four steps no longer suffice and the residual compounds through the cache.

01

Frozen backbone

MiniMax-H3 in its first-frame-conditioned audio–video configuration. Text, a first-frame condition, video and audio form one packed sequence with three-dimensional rotary positions. The backbone is never updated.

02

Causal adapter Δ_c

Trained against the frozen backbone with clean-context teacher forcing on a doubled token axis: each chunk appears twice at identical rotary positions, once clean and once noisy, so a noisy chunk reads only strictly earlier clean ground truth. One plain flow-matching regression — the only objective we ever train.

03

Few-step adapter Δ_d

Not trained by us. We take a public third-party 4-step distillation of the same backbone, learned under bidirectional attention where distillation is already solved. Each capability is acquired where it is cheapest; we never touch an ODE solver, an EMA target, or a critic.

04

Parallel composition

W + Δ_d + Δ_c, combined by weight arithmetic at inference. Δ_d is fused offline so it adds no runtime overhead; Δ_c loads on top. Neither stage ever sees a base the other has altered, and Δ_c stays re-composable across step counts.

Attention mask for clean-context teacher forcing on the doubled token axis — **The attention layout of step 02**, on a three-chunk packed sequence. Clean tokens attend same-or-earlier clean tokens; a noisy chunk attends its own noisy tokens and strictly earlier clean history, never its clean copy. Text is visible in both directions; the first-frame condition is chunk −1, visible as a key to every content row.

Beyond the adaptation length

A causal adapter trained on 5–10 s clips, streamed to 30 s with the history capped by a sliding window. Scripts are written for 40 s so every system is still mid-sentence when the video ends — falling silent is therefore a failure, not an ending. Turn the sound on: the separation is audible before it is visible.

One clip streamed to 30 seconds across four systems, with the four soundtracks on one time axis — **One clip streamed to 30 s.** Rows are systems, columns are wall-clock seconds; below them the generated soundtracks share one time axis. Chained-DA stops articulating well before the end while the composed routes speak throughout.

Closing 5 s against opening 5 s, 40 clips

System	Sync-C ↑	Identity ↑	CLIP ↑	IQA ↑	Voice ↑	RMS
Decoupled	5.69 (−29.6%)	0.710 (−18.3%)	0.844 (−9.7%)	0.871 (+2.6%)	0.521 (−41.0%)	0.086 (−10.6%)
Decoupled-50	6.23 (−19.1%)	0.687 (−18.9%)	0.856 (−8.2%)	0.848 (+3.0%)	0.682 (−22.3%)	0.109 (−16.6%)
Chained-AD	7.82 (−12.8%)	0.484 (−41.8%)	0.737 (−19.7%)	0.526 (−35.4%)	0.608 (−31.9%)	0.185 (+49.0%)
Chained-DA	3.75 (−43.7%)	0.683 (−21.4%)	0.841 (−10.5%)	0.656 (−16.2%)	0.603 (−32.3%)	0.047 (−37.9%)

Each cell is the score on the final 5 s and, in parentheses, its change from the opening 5 s. Bold marks the best closing window; for RMS, an absolute level, it marks the smallest relative change. Decoupled closes best on identity and image quality; Chained-AD keeps the best lip sync but loses over a third of identity and image quality; Chained-DA has largely fallen silent.

First-frame consistency as a function of generation time — **Long-horizon consistency as a function of generation time.** FaceNet similarity and CLIP image similarity to the first frame of the same video; means over 40 clips with 95% confidence intervals, terminating at each method's actual duration.

Identity drift over the 30 second long-horizon runs — **Identity drift over the 30 s runs** (40 clips, probes every 5 s). Face identity is retained equally well by Decoupled, Decoupled-50 and Chained-DA, and worst by Chained-AD. Speaker identity falls furthest for Decoupled, the limitation noted below.

Geometry of each causal update against each distillation update, over all shared (block, projection) pairs. Both interference forms vanish for orthogonal subspaces; the null column pairs each block's update with another depth's.
Measurement	Decoupled	Chained-DA	Chained-AD	null
Projection	+0.0006	−0.0022	−0.0008	+0.0000
Write interference	0.0147	0.0169	0.0451	0.0117
Read interference	0.1356	0.1061	0.1844	0.0237

Results

Held-out test set (211 clips), against the teacher's own generation

Metric	Teacher50 step	Whole-seg4 step	Decoupled4 step, ours	Decoupled-5050 step	Chained-DA4 step	Chained-AD4 step
Sync-C ↑	8.412	8.403	7.443	7.197	6.389	8.021
AV-IB ↑	0.414	0.414	0.403	0.423	0.363	0.419
IQA ↑	0.908	0.915	0.914	0.894	0.854	0.877
ASE ↑	5.048	5.110	5.106	5.023	5.019	5.008
CLIP ↑	0.212	0.212	0.212	0.210	0.212	0.212
CSIM-1st ↑	0.886	0.870	0.864	0.853	0.871	0.835
IQA drift	1.005	1.008	1.030	1.026	1.002	0.990
CSIM drift	0.981	0.981	0.950	0.948	0.945	0.916
WER (En) ↓	0.041	0.047	0.127	0.110	0.126	0.136
CER (Zh) ↓	0.014	0.008	0.051	0.055	0.061	0.061

Bold marks the best of the four autoregressive systems per row; for the drift rows, the closest to 1. Speech is transcribed with Qwen3-ASR-1.7B: WER for the English clips, CER for the Chinese ones.

Appearance is untouched

Whole-seg → Decoupled varies only the attention layout: IQA 0.915 → 0.914, ASE 5.110 → 5.106, CLIP unchanged, FID 66.18 → 66.15 — below the teacher's floor of 67.10.

What causalisation does cost

Speech: intelligibility (WER 0.047 → 0.127), lip–speech coupling (Sync-C 8.403 → 7.443) and the audio distribution. All three need the model to know where it is in the script; chunkwise attention leaves only the audio history to infer it from.

Chaining first, regressing second

Chained-DA falls behind on appearance, synchronisation and speech, and its FID rises above the teacher's floor (70.31 against 67.10) while Decoupled stays below it (66.15) — the un-distillation the method section predicts.

The other chain survives

Chained-AD distils last with sampler-aware objectives and buys back the best lip sync of any 4-step configuration, at the price of appearance and identity (IQA 0.877 against 0.914) and two extra training stages.

Public test sets, scored against real source video

100 held-out clips each. The teacher is a system under test here, not the reference.

System	IQA ↑	Sync-C ↑	AV-IB ↑	DeSync ↓	CSIM-r ↑	FID ↓	FVD ↓
AVSpeech (n = 100)
GT (source)	0.852	6.82	0.377	0.115	0.867	—	—
Teacher	0.857	8.49	0.333	0.028	0.839	32.2	128.5
Whole-seg	0.835	7.82	0.328	0.040	0.864	33.7	119.2
Decoupled	0.866	7.54	0.334	0.007	0.791	35.2	129.0
Decoupled-50	0.836	7.13	0.324	0.028	0.761	38.9	143.6
Chained-AD	0.804	8.39	0.305	0.012	0.741	38.8	150.0
Chained-DA	0.803	6.10	0.290	0.023	0.790	40.2	131.9
VidChatBench (n = 100)
GT (source)	0.770	5.23	0.404	0.081	0.797	—	—
Teacher	0.769	7.32	0.328	0.022	0.725	38.1	171.4
Whole-seg	0.712	6.69	0.319	0.022	0.752	40.5	178.9
Decoupled	0.866	7.84	0.353	0.024	0.774	39.6	184.0
Decoupled-50	0.829	7.65	0.348	0.036	0.758	41.3	191.5
Chained-AD	0.742	7.85	0.336	0.031	0.698	42.4	203.0
Chained-DA	0.776	6.70	0.302	0.034	0.775	43.5	193.8
HDTF (n = 100)
GT (source)	0.877	8.24	0.330	—	0.937	—	—
Teacher	0.874	8.94	0.297	—	0.919	13.0	63.6
Whole-seg	0.877	8.65	0.263	—	0.933	11.3	48.3
Decoupled	0.892	9.04	0.295	—	0.905	14.4	61.4
Decoupled-50	0.880	8.69	0.270	—	0.902	15.8	59.2
Chained-AD	0.853	9.28	0.293	—	0.869	22.3	98.9
Chained-DA	0.876	8.58	0.253	—	0.919	14.2	49.6

Cost

The extra wall-clock of streaming is the key/value prefill, not the denoising steps.

4.2–4.3 stime-to-first-frame, any clip length

11.2–54.3 sthe same, whole-segment — it must finish the clip

≈26 fps480×832 on six GPUs, no quantisation

1 objectiveone plain regression; no ODE solver, EMA target or critic

Citation

@inproceedings{anonymous2027parallel,
  title     = {Real-Time Joint Audio--Video Generation by
               Parallel Adapter Composition},
  author    = {Anonymous},
  booktitle = {Submitted to the International Conference on
               Learning Representations (ICLR)},
  year      = {2027},
  note      = {Under review}
}

Real-Time Joint Audio–Video Generation
by Parallel Adapter Composition

The claim

Method

Frozen backbone

Causal adapter Δ_c

Few-step adapter Δ_d

Parallel composition

Held-out clips, four systems

Beyond the adaptation length

Closing 5 s against opening 5 s, 40 clips

A predicted geometry, then measured

Results

Held-out test set (211 clips), against the teacher's own generation

Appearance is untouched

What causalisation does cost

Chaining first, regressing second

The other chain survives

Public test sets, scored against real source video

Cost

Limitations

Citation

The claim

Method

Frozen backbone

Causal adapter Δc

Few-step adapter Δd

Parallel composition

Held-out clips, four systems

Beyond the adaptation length

Closing 5 s against opening 5 s, 40 clips

A predicted geometry, then measured

Results

Held-out test set (211 clips), against the teacher's own generation

Appearance is untouched

What causalisation does cost

Chaining first, regressing second

The other chain survives

Public test sets, scored against real source video

Cost

Limitations

Citation

Causal adapter Δ_c

Few-step adapter Δ_d