PyTorch maDLC analysis crashes post-inference with `ValueError: need at least one array to stack`
Is there an existing issue for this?
- I have searched the existing issues
Operating System
AlmaLinux 9.6 (Sage Margay) (HPC)
DeepLabCut version
3.0rc10/rc12
What engine are you using?
pytorch
DeepLabCut mode
multi animal
Device type
Nvidia L40S 48GB
Bug description 🐛
After training up a PyTorch model, I ran the analysis step in order to extract outlier frames.
The analysis step run to 100% and then threw an error.
100%|██████████| 1296914/1296914 [15:18:47<00:00, 23.53it/s] Traceback (most recent call last): File "/home/ts/scripts/DLC_analyze_and_extract.py", line 71, in <module> sys.exit(main()) File "/home/ts/scripts/DLC_analyze_and_extract.py", line 37, in main deeplabcut.analyze_videos( File "/home/ts/miniforge3/envs/dlc/lib/python3.12/site-packages/deeplabcut/compat.py", line 953, in analyze_videos return analyze_videos( File "/home/ts/miniforge3/envs/dlc/lib/python3.12/site-packages/deeplabcut/pose_estimation_pytorch/apis/videos.py", line 689, in analyze_videos _generate_assemblies_file( File "/home/ts/miniforge3/envs/dlc/lib/python3.12/site-packages/deeplabcut/pose_estimation_pytorch/apis/videos.py", line 883, in _generate_assemblies_file unique_bpts = np.stack(keypoint_preds[num_bodyparts:]) File "/home/ts/miniforge3/envs/dlc/lib/python3.12/site-packages/numpy/core/shape_base.py", line 445, in stack raise ValueError('need at least one array to stack') ValueError: need at least one array to stack
I tried with both autotrack = true and false, tried stitching the tracklets after the analysis.
All resulted in the same error.
Even a short 2 minute clip caused the same error.
With some trial and error I figured out that the _full.pickle file is not written, but the _full.pickle.dat, _full.pickle.dir, _full.pickle.bak, and _meta.pickle exist.
Steps To Reproduce
- Train pytorch model
- run analysis
- see failure once 100% is reached
- No
_full.pickle
Relevant log output
Anything else?
Tried it both with rc10 and rc12, both with the same error.
Also both dekr_w48 and dlcrnet_stride32_ms5 show the same behavior.
Pytorch is 2.8.0+cu128
I can share a reproducible example via email with a 2 min video.
Code of Conduct
- I agree to follow this project's Code of Conduct