Help setting up Wan 2.2 Time to Move

[MichaelWkTran]

Hi guys,

I am fairly inexperienced with video generation, but I am interested in experimenting with Time to Move. However I am having a lot of trouble trying to get it set up. I have made two attempts.

Below is by thought process that got me here:
I tested the wanvideo_2_2_5B_I2V_example_WIP workflow. I then modified it so that it would use Wan2.2‑TI2V‑5B‑Q4_K_M.gguf, Wan2.2_vae.safetensors, and umt5_xxl_fp8_e4m3fn.safetensors. These models were recommended to me by https://ksimply.vercel.app/en based on my device specifications, which include an RTX 3070 8GB laptop GPU and 16GB of RAM. I am open to trying other models if there are alternatives you would suggest.

That workflow worked, so I proceeded to test the wanvideo_2_2_I2V_A14B_TimeToMove workflow.

Here is the error:
cannot access local variable 'image_cond_mask' where it is not associated with a value

Here is the console log

FETCH ComfyRegistry Data: 30/123
Found quantization metadata version 1
Using MixedPrecisionOps for text encoder
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
Requested to load WanTEModel
FETCH ComfyRegistry Data: 35/123
FETCH ComfyRegistry Data: 40/123
FETCH ComfyRegistry Data: 45/123
FETCH ComfyRegistry Data: 50/123
loaded partially; 5694.80 MB usable, 0.00 MB loaded, 6419.10 MB offloaded, 6089.00 MB buffer reserved, lowvram patches: 0
FETCH ComfyRegistry Data: 55/123
loaded partially; 5642.68 MB usable, 0.00 MB loaded, 6419.10 MB offloaded, 6089.00 MB buffer reserved, lowvram patches: 0
FETCH ComfyRegistry Data: 60/123
FETCH ComfyRegistry Data: 65/123
FETCH ComfyRegistry Data: 70/123
WanVAE encoding frames: 100%|████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.60it/s]
WanVideoEncode: Encoded latents shape torch.Size([1, 48, 8, 12, 20])
WanVAE encoding frames:   0%|                                                                    | 0/8 [00:00<?, ?it/s]FETCH ComfyRegistry Data: 75/123
WanVAE encoding frames: 100%|████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 15.47it/s]
CUDA Compute Capability: 8.6
Detected model in_channels: 48
Model cross attention type: t2v, num_heads: 24, num_layers: 30
5B model detected, no Teacache or MagCache coefficients available, consider using EasyCache for this model
Model variant detected: 14B
model_type FLOW
Using GGUF to load and assign model weights to device...
FETCH ComfyRegistry Data: 80/123
FETCH ComfyRegistry Data: 85/123
FETCH ComfyRegistry Data: 90/123
FETCH ComfyRegistry Data: 95/123
Loading transformer parameters to cuda:0: 100%|███████████████████████████████████| 825/825 [00:00<00:00, 10130.52it/s]
-------------------------
Transformer weights loaded:
Device: cuda:0   | Memory: 1,391,626.12 MB
------- Scheduler info -------
Total timesteps: tensor([999, 988, 975, 959, 941, 918, 888, 851, 799, 727, 615, 421],
       device='cuda:0')
Using timesteps: tensor([999, 988, 975, 959, 941, 918, 888, 851, 799, 727, 615, 421],
       device='cuda:0')
Using sigmas: tensor([1.0000, 0.9887, 0.9756, 0.9600, 0.9412, 0.9180, 0.8889, 0.8510, 0.8000,
        0.7272, 0.6153, 0.4210, 0.0000])
------------------------------
!!! Exception during processing !!! cannot access local variable 'image_cond_mask' where it is not associated with a value
Traceback (most recent call last):
  File "F:\ComfyUI_windows_portable\ComfyUI\execution.py", line 518, in execute
    output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\ComfyUI_windows_portable\ComfyUI\execution.py", line 329, in get_output_data
    return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\ComfyUI_windows_portable\ComfyUI\execution.py", line 303, in _async_map_node_over_list
    await process_inputs(input_dict, i)
  File "F:\ComfyUI_windows_portable\ComfyUI\execution.py", line 291, in process_inputs
    result = f(**inputs)
  File "F:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\nodes_sampler.py", line 238, in process
    if image_cond_mask is not None:
       ^^^^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'image_cond_mask' where it is not associated with a value

Prompt executed in 68.93 seconds

I have resolved this by following this link: #1874.
Although, now I am getting the error:
Given groups=1, weight of size [3072, 48, 1, 2, 2], expected input[1, 96, 8, 12, 20] to have 48 channels, but got 96 channels instead

This is where I am at a dead end. Here is my workflow.
wanvideo2_2_I2V_A14B_TimeToMove_TEST.json

And here is the console log.

WanVAE encoding frames: 100%|████████████████████████████████████████████████████████████| 8/8 [00:00<00:00,  9.34it/s]
WanVideoEncode: Encoded latents shape torch.Size([1, 48, 8, 12, 20])
WanVAE encoding frames: 100%|████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 15.32it/s]
Using GGUF to load and assign model weights to device...
Loading transformer parameters to cuda:0: 100%|███████████████████████████████████| 825/825 [00:00<00:00, 10172.78it/s]
-------------------------
Transformer weights loaded:
Device: cuda:0   | Memory: 1,391,626.12 MB
------- Scheduler info -------
Total timesteps: tensor([999, 988, 975, 959, 941, 918, 888, 851, 799, 727, 615, 421],
       device='cuda:0')
Using timesteps: tensor([999, 988, 975, 959, 941, 918, 888, 851, 799, 727, 615, 421],
       device='cuda:0')
Using sigmas: tensor([1.0000, 0.9887, 0.9756, 0.9600, 0.9412, 0.9180, 0.8889, 0.8510, 0.8000,
        0.7272, 0.6153, 0.4210, 0.0000])
------------------------------
EasyCache: Using cache device: cpu
Rope function: comfy
Using Time-to-move (TTM)
TTM reference latents shape: torch.Size([48, 8, 12, 20])
TTM motion mask shape: torch.Size([1, 8, 12, 20])
Applying TTM from step 1 to 3
---------- Sampling start ----------
29 frames at 320x192 (Input sequence length: 480) with 11 steps
  0%|                                                                                           | 0/11 [00:00<?, ?it/s]Error during model prediction: Given groups=1, weight of size [3072, 48, 1, 2, 2], expected input[1, 96, 8, 12, 20] to have 48 channels, but got 96 channels instead
  0%|                                                                                           | 0/11 [00:00<?, ?it/s]
Error during sampling: Given groups=1, weight of size [3072, 48, 1, 2, 2], expected input[1, 96, 8, 12, 20] to have 48 channels, but got 96 channels instead
!!! Exception during processing !!! Given groups=1, weight of size [3072, 48, 1, 2, 2], expected input[1, 96, 8, 12, 20] to have 48 channels, but got 96 channels instead
Traceback (most recent call last):
  File "F:\ComfyUI_windows_portable\ComfyUI\execution.py", line 518, in execute
    output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\ComfyUI_windows_portable\ComfyUI\execution.py", line 329, in get_output_data
    return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\ComfyUI_windows_portable\ComfyUI\execution.py", line 303, in _async_map_node_over_list
    await process_inputs(input_dict, i)
  File "F:\ComfyUI_windows_portable\ComfyUI\execution.py", line 291, in process_inputs
    result = f(**inputs)
  File "F:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\nodes_sampler.py", line 2591, in process
    raise e
  File "F:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\nodes_sampler.py", line 2484, in process
    noise_pred, noise_pred_ovi, self.cache_state = predict_with_cfg(
                                                   ~~~~~~~~~~~~~~~~^
        latent_model_input,
        ^^^^^^^^^^^^^^^^^^^
    ...<4 lines>...
        wananim_face_pixels=wananim_face_pixels, wananim_pose_latents=wananim_pose_latents, uni3c_data = uni3c_data, latent_model_input_ovi=latent_model_input_ovi, flashvsr_LQ_latent=flashvsr_LQ_latent,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "F:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\nodes_sampler.py", line 1664, in predict_with_cfg
    raise e
  File "F:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\nodes_sampler.py", line 1511, in predict_with_cfg
    noise_pred_cond, noise_pred_ovi, cache_state_cond = transformer(
                                                        ~~~~~~~~~~~^
        context=positive_embeds,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        **base_params
        ^^^^^^^^^^^^^
    )
    ^
  File "F:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "F:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
  File "F:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\wanvideo\modules\model.py", line 2527, in forward
    x = [self.original_patch_embedding(u.unsqueeze(0).to(torch.float32)).to(x[0].dtype) for u in x]
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "F:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
  File "F:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\conv.py", line 723, in forward
    return self._conv_forward(input, self.weight, self.bias)
           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\conv.py", line 718, in _conv_forward
    return F.conv3d(
           ~~~~~~~~^
        input, weight, bias, self.stride, self.padding, self.dilation, self.groups
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
RuntimeError: Given groups=1, weight of size [3072, 48, 1, 2, 2], expected input[1, 96, 8, 12, 20] to have 48 channels, but got 96 channels instead

Prompt executed in 36.96 seconds

---

Here is then my second attempt.
I have used the wanvideo_2_2_5B_I2V_example_WIP workflow, which is proven to work, and added the nodes related to Time to Move into it. I then connected WanVideo Add TTMLatents between WanVideo Empy Embeds, and WanVideo Sampler.

Here is the resulting error:
The size of tensor a (18) must match the size of tensor b (4) at non-singleton dimension 1

Here is the workflow:
wanvideo_2_2_5B_TI2V_TTM_TEST.json

And here is the console log

WanVAE encoding frames: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 18/18 [00:02<00:00,  6.68it/s]
WanVideoEncode: Encoded latents shape torch.Size([1, 48, 18, 22, 16])
WanVAE encoding frames: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 27.60it/s]
WanVideoEncode: Encoded latents shape torch.Size([1, 48, 1, 22, 16])
Using GGUF to load and assign model weights to device...
Loading transformer parameters to cuda:0: 100%|███████████████████████████████████████████████████████████████████████| 825/825 [00:00<00:00, 13498.19it/s]
-------------------------
Transformer weights loaded:
Device: cuda:0   | Memory: 1,391,626.12 MB
------- Scheduler info -------
Total timesteps: tensor([1000.0000,  988.7640,  975.6098,  960.0000,  941.1764,  918.0328,
         888.8889,  851.0638,  800.0000,  727.2728,  615.3846,  421.0526])
Using timesteps: tensor([1000.0000,  988.7640,  975.6098,  960.0000,  941.1764,  918.0328,
         888.8889,  851.0638,  800.0000,  727.2728,  615.3846,  421.0526])
Using sigmas: tensor([1.0000, 0.9888, 0.9756, 0.9600, 0.9412, 0.9180, 0.8889, 0.8511, 0.8000,
        0.7273, 0.6154, 0.4211])
------------------------------
Adding extra samples to latent indices 0 to 0
EasyCache: Using cache device: cpu
Rope function: comfy
Using Time-to-move (TTM)
TTM reference latents shape: torch.Size([48, 18, 22, 16])
TTM motion mask shape: torch.Size([1, 18, 22, 16])
Applying TTM from step 1 to 3
!!! Exception during processing !!! The size of tensor a (18) must match the size of tensor b (4) at non-singleton dimension 1
Traceback (most recent call last):
  File "F:\ComfyUI_windows_portable\ComfyUI\execution.py", line 518, in execute
    output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\ComfyUI_windows_portable\ComfyUI\execution.py", line 329, in get_output_data
    return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\ComfyUI_windows_portable\ComfyUI\execution.py", line 303, in _async_map_node_over_list
    await process_inputs(input_dict, i)
  File "F:\ComfyUI_windows_portable\ComfyUI\execution.py", line 291, in process_inputs
    result = f(**inputs)
  File "F:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\nodes_sampler.py", line 1086, in process
    latent = add_noise(ttm_reference_latents, noise, timesteps[ttm_start_step].to(noise.device)).to(latent)
             ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\multitalk\multitalk.py", line 28, in add_noise
    return (1 - timesteps) * original_samples + timesteps * noise
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~
RuntimeError: The size of tensor a (18) must match the size of tensor b (4) at non-singleton dimension 1

Prompt executed in 20.68 seconds

[MichaelWkTran]

Another Update, so I have tried again downloading the models from the template as well as gguf models. These models are:

• Wan2_2-I2V-A14B-HIGH_fp8_e4m3fn_scaled_KJ.safetensors
• Wan2_2-I2V-A14B-LOW_fp8_e4m3fn_scaled_KJ
• Wan2.2-I2V-A14B-HighNoise-Q5_K_M.gguf
• Wan2.2-I2V-A14B-HighNoise-Q8_0.gguf
• Wan2.2-I2V-A14B-LowNoise-Q5_K_M.gguf
• Wan2.2-I2V-A14B-LowNoise-Q8_0.gguf
• Wan2_1_VAE_bf16.safetensors
• umt5-xxl-enc-bf16.safetensors
• umt5-xxl-encoder-Q4_K_M.gguf (not sure how to use this model just yet)

All of the tests resulted in this error:
torch.OutOfMemoryError: Allocation on device 0 would exceed allowed memory. (out of memory)

[MichaelWkTran]

I realized by previous message did not send? In summary. I had found some leads to resolving the problem. For this workflow, I did not properly set the frame count of I think the latents to the length of the videos leading to the error. Afterwards. I get this massively glitchy result.

WanVideoWrapper_5BI2V_00002.mp4

After trying to swap out the WanVideo Encode and WanVideo Empty Embeds to WanVideo ImageToVideo Encode, I get RuntimeError: Given groups=1, weight of size [3072, 48, 1, 2, 2], expected input[1, 96, 8, 22, 16] to have 48 channels, but got 96 channels instead. Which tells me for both workflows that the error is over there.

Sorry if this message is not as clear as the others. Let me know if you need further context.

[MichaelWkTran]

Ok, so the glitchy video of the monkey I sent is a result of the video resolution I set being too small. Reverting it have resolved the issue nicely and finally allows me to use Time to Move. But I wonder whether it is possible to use WanVideo ImageToVideo Encode as it seems to have features like first frame last frame that sounds like it could be useful.