AutoRegressive Diffusion Training Results

[nighting0le01]

Hey @zhuhz22 i was trying to implement causal forcing for video to video on a different model. i wanted to attach some visuals and loss curves for the 40 step AR Teacher Model i've trained. seems it is not able to condition/ work well after AR Diffusion stage.

can you please advise if the results look ok for AR Teacher? or do they seems broken? What about loss

https://github.com/user-attachments/assets/b8fd4bfc-693d-45e8-9bd2-6939713ddb5c
https://github.com/user-attachments/assets/97642410-d194-42a1-8bd3-514352fdc478
https://github.com/user-attachments/assets/db46e223-87a6-425f-89cd-e55a57d58172
https://github.com/user-attachments/assets/71414f75-6554-4df4-8bb5-a8e4daede6d7
https://github.com/user-attachments/assets/0349d6c8-7b31-4ec4-993f-ab32679b18fd
https://github.com/user-attachments/assets/fb00393a-618b-4dbf-8169-0d716f59f93a
https://github.com/user-attachments/assets/96bf0062-22ea-43d4-b393-538a0cd5c246
https://github.com/user-attachments/assets/97a0b72f-ce22-4fb9-b73f-c4d37e8e02af
https://github.com/user-attachments/assets/f2cee617-8349-4757-8e7f-b4a5fa66a88c

[nighting0le01]

also would like to add these are just trained for 800 iterations, i'm not sure if this flickering will go away, with chunk_size of 1 (meaning 4 RGB frame/1 latent frame).

i'm wondering if it is due to the nature of teacher forcing that it is so bad with dealing with error accumulation

[zhuhz22]

Hi @nighting0le01 ,
The 40-step AR diffusion is expected to learn—though it will be worse than the bidirectional model—but it should not result in this kind of complete collapse you are seeing. Please check your training and inference code, especially whether the inference code uses self-rollout with KV cache. To verify the generation performance of the AR model, you can run:
hf download zhuhz22/Causal-Forcing framewise/ar_diffusion.pt --local-dir checkpoints
and then test it with:

python inference.py \
  --config_path configs/ar_diffusion_tf_{framewise OR chunkwise}.yaml \
  --output_folder output/{framewise OR chunkwise}_ar_diffusion \
  --checkpoint_path  checkpoints/{framewise OR chunkwise}/ar_diffusion.pt \
  --data_path prompts/demos.txt

Additionally, 800 steps is too few for chunk_size=1. I would recommend training for at least 3k steps. You can check every 1k steps to see if it is improving.

[nighting0le01]

hey @zhuhz22 thanks for the reply. i did try both infering the ckpt you shared as well as launching a causal forcing framewise AR stage using the Code in the repo. i launched in on just a single node, so batchsize maybe playing a role but here are the

results at step 3000 (Wan T2V model)

https://github.com/user-attachments/assets/4d3dd6e1-56cc-47d1-98b3-dd0b2284092b
https://github.com/user-attachments/assets/2b11a38d-1a23-482a-ac34-260f3da04b30
https://github.com/user-attachments/assets/3af7ca39-3a2c-42da-b90f-211d6b899fe9
https://github.com/user-attachments/assets/00afd39d-0563-4c66-83c5-1d576da36d1c
https://github.com/user-attachments/assets/080f52d8-8b56-4633-b646-921440ab2a17
https://github.com/user-attachments/assets/659ba01d-1557-4b32-88db-7591dd7224d2
https://github.com/user-attachments/assets/4cc01bfa-7a9d-4884-84ac-7e96879e8ecd
https://github.com/user-attachments/assets/2421651c-2b3e-468b-9e48-5383228c8c80
https://github.com/user-attachments/assets/ac224946-ee56-4ca5-a2ba-efd46e8cf780
https://github.com/user-attachments/assets/a4a48828-5a19-4e57-b3c7-10aadff5a19e

I launched T2V with my model as well and here are the results, (these are with 8 nodes) (my Model)

https://github.com/user-attachments/assets/5d786ba7-ddaf-4a22-ac29-5c8b4c1a74a8
https://github.com/user-attachments/assets/ee8463ce-4b53-4672-9c47-d8b3546b7cf9
https://github.com/user-attachments/assets/c334ce33-13da-478c-8241-cc4bddd39730
https://github.com/user-attachments/assets/3cc42184-b025-4213-88bb-7422ecf1cb04
https://github.com/user-attachments/assets/7a7eb820-2d9c-498c-91f2-9fce3cdee080
https://github.com/user-attachments/assets/ca992720-2865-4545-a3c0-23c0f5eba28d
https://github.com/user-attachments/assets/59acf21e-bd01-4ead-aa36-943571554af0

I launched V2V with my model as well and here are the results, (these are with 8 nodes) (my Model)

https://github.com/user-attachments/assets/a25ce86b-9171-455f-8cb4-d06e9b927c3e
https://github.com/user-attachments/assets/66837b2c-66b7-46c9-b7c3-8577082af0af
https://github.com/user-attachments/assets/e0f9dad7-0867-488e-bc25-b6bfbddadb38
https://github.com/user-attachments/assets/bdd9d43e-47e1-437d-900a-b1ff808d59e5
https://github.com/user-attachments/assets/cc8d1cfe-b4b1-4aff-a375-15d9799ac09a
https://github.com/user-attachments/assets/7b7fdb82-fc26-44ce-9cd4-a6682fde4fbd
https://github.com/user-attachments/assets/c81fae7b-103b-4a48-8193-79fe1e6dee25
https://github.com/user-attachments/assets/a1182a75-0ff4-4c4c-9f97-26f80615f73e

*all are causal forcing framewise AR results

for converting a model to a causal Model

1. Add causal Attention.
2. per frame modulation and per frame modulation final layer

For AR training

1. seq wise concat clean + noisy input concat, use the same
2. RoPE handling: freq_cis duplication [clean| noisy]
3. clean|noisy global condition compute and concat. clean global condition computed with sigma=0
4. uniform timesampler + timeshift of 5 for sampling training sigmas (can you confirm if you used this as well?)

can you confirm if you made just these changes and comment on the results? is there something i'm missing in the AR stage implementation? i see the frame-wise results in the WAN model trained from your code i see artifacts as well. how many iterations should i train till?

[nighting0le01]

i also checked for the WAN training code you have shared, the results quality is sometimes worse with more training,like every 1000 steps i infered the results and saw improvements sometimes but regression other times. so it is hard to pick a ckpt

[zhuhz22]

Hi @nighting0le01 , I think the issue may lie in your inference. The T2V framewise model in our repo already yields good results, far better than what you shows. Could you please tell me the specific ckpt and inference script you use?

[zhuhz22]

If your results at step 3000 (Wan T2V model) means our open-sourced framewise ckpt, then there must be a bug in the inference, since the results are far worse than expected.

[nighting0le01]

python inference.py \
  --config_path configs/ar_diffusion_tf_framewise.yaml \
  --output_folder output/framewise__ar_diffusion \
  --checkpoint_path  checkpoints/{framewise OR chunkwise}/ar_diffusion.pt \
  --data_path prompts/demos.txt

i used the same command for inference @zhuhz22 , i also trained the chunkwise one, and the results don't seem as bad.

For the WAN model, i used your repo and training code. and the default framewise AR config

[zhuhz22]

Let us first determine whether there is a bug.

Could you tell me: when you run inference with our official checkpoint, are most of the results broken like this:

Along.a.quiet.canal.in.Amsterdam.a.cyclist.in.a.mustard.sweater.wool.coat.and.leather.gloves.domi.mp4

or are they more like this:

At.a.tranquil.desert.temple.in.Petra.an.explorer.in.a.sand.scarf.olive.jacket.and.leather.satchel.mp4

If broken results dominate, for example more than half, then there must be a bug. If the latter type is the majority, then this is within expectation and can be addressed by the subsequent DMD stage.

[zhuhz22]

Also, what you mean by RoPE handling: freq_cis duplication [clean| noisy]? The correct RoPE implementation is that the clean and noisy parts each independently start their positional encoding from 0. They are not encoded continuously as one single natural sequence. In other words, although the noisy part is concatenated after the clean part, its positional indices should not start from the end of the clean part.

[nighting0le01]

Hi @zhuhz22 let me answer your questions in sequence:

1. when i say i used WAN model, i mean i ran the code in the this repo according to README for doing the AR Training.

2. i think alot of the samples are like the second video you shared. there are lesser jumps but it seems to decay in some way like overexposure

3. here is a side by side comparison of the ckpt that i trained using your code in the repo:

https://github.com/user-attachments/assets/b27ba578-8eca-4ba7-96d9-5f4ffb910ee5
https://github.com/user-attachments/assets/1abce34f-10ba-4c26-9c20-2c970cf8d39f
https://github.com/user-attachments/assets/438bf5b4-f656-4ffb-b42f-2674d58b7c25
https://github.com/user-attachments/assets/d7670420-e368-4269-be5f-1684a7792e38
https://github.com/user-attachments/assets/333f6838-a473-45da-b315-c797adee1291
https://github.com/user-attachments/assets/a8ca9339-0f4e-4b03-b6ec-680dac04c8b7
https://github.com/user-attachments/assets/748f3e0e-ae7b-45e1-bc0a-6fb5a2d46fc5
https://github.com/user-attachments/assets/dca1677f-a8e6-4b2b-ad8e-6dd6b44b4bff
https://github.com/user-attachments/assets/dd6c5303-8549-4b9b-9e52-f8f635155c82
https://github.com/user-attachments/assets/017d4e74-639d-4aed-905a-e757aff69c98

4. for the RoPE handling yes i just duplicated the freqs_cis for the noisy part and re-use them for the clean part. so i think they are equivalent to what you said

[zhuhz22]

Ok so you mean the results you showed are not the official ckpt offered in zhuhz22/Causal-Forcing framewise/ar_diffusion.pt? If so, I recommend try inference with our official ckpt, and if it's better than your own ckpt, then the issue lies in your training setting rather than our official algorithm. Could you please give it a try?

Also, I find that in your " side by side comparison of the ckpt that you trained ", clearly the performance is improved as the training steps increase. So you can train for more steps such as 10K. Since you only use 1 node (8 GPUs), your batch size is 8 times less than our official setting, so you can train more steps.

Also, since there are lesser jumps but it seems to decay in some way like over-exposure, it's expected and can be solved by the following DMD stage. Nevertheless, this issue can be mitigated as the number of training steps increases， such as 10K steps.

[nighting0le01]

thanks @zhuhz22,

here is the official ckpt results:

https://github.com/user-attachments/assets/af9f564d-544c-462f-a8bb-77bd8265ddbe
https://github.com/user-attachments/assets/97dab926-4703-4790-a886-56c026799ba0
https://github.com/user-attachments/assets/04703b68-cdc8-4434-8e05-051ff97991c3
https://github.com/user-attachments/assets/4182cbbf-5242-4df2-a2d9-c257bf25f442

i think lower batchsize might contribute,so i did let it train till 20k, here are the results

  torchrun --nnodes=8 --nproc_per_node=8 --rdzv_id=5235 \
  --rdzv_backend=c10d \
  --rdzv_endpoint $MASTER_ADDR \
  train.py \
  --config_path configs/ar_diffusion_tf_framewise.yaml \
  --logdir logs/ar_diffusion_framewise

side by side comparison 1k,5k,10k,20k on the ckpt obtained with re-running your AR training code
https://github.com/user-attachments/assets/952926c1-8c1f-43e8-a91c-a10d34136502
https://github.com/user-attachments/assets/42716a3e-52a2-429e-9101-8a79c13aa941
https://github.com/user-attachments/assets/08e1280d-c4d0-4b1a-b694-e91feab3e15a
https://github.com/user-attachments/assets/01aa30c8-fc5e-4dfb-a9f3-cb0ef3e2dab6
https://github.com/user-attachments/assets/7904d35a-0ae8-46e4-a25d-199d917c7ccf
https://github.com/user-attachments/assets/a2cf7807-7a72-4a73-b32f-82a672c9fd23
https://github.com/user-attachments/assets/b64663c2-fb72-4970-b545-7130d4419965
https://github.com/user-attachments/assets/561adb0a-fbb7-45e6-85e1-c58b863bb4fb
https://github.com/user-attachments/assets/1833e4e4-9a37-4b24-8be2-fef533e8b0d5
https://github.com/user-attachments/assets/773d8e8a-e1ba-4407-8ddd-6ff1bb9fd0b2
https://github.com/user-attachments/assets/0b02eae2-3286-4700-912f-892ee4b0e801

do you think the T2V and V2V results i attached for my model look ok?

[nighting0le01]

do you have visual results after each stage? that i can use as reference?

[zhuhz22]

I think the V2V model is good enough, and you can ignore it for now and move on to the subsequent stages. DMD is expected to address these artifacts. If it does not, we can come back and investigate them together.

Also I find that your T2V model yields worse results:
https://github.com/user-attachments/assets/5d786ba7-ddaf-4a22-ac29-5c8b4c1a74a8
It cannot be ruled out that this is caused by a bug or an overly small batch size. If the results are still unsatisfactory after DMD, you may consider enabling gradient accumulation to increase the effective batch size.