IndexError: list index out of range when using analyze_videos

Is there an existing issue for this?

I have searched the existing issues

Bug description

Hi!
This is my first time using python and DeepLabCut, so any suggestions or comments are greatly appreciated! When I was trying to analyze the video file, it was running fine until it failed halfway. Then I tried the suggestion in #1527 but the same error occured again. Due to the capacity of the gpu, I am running on batchsize=1.

Operating System

Ubuntu 20.04.6 LTS

DeepLabCut version

2.3.8

DeepLabCut mode

multi animal

Device type

gpu

Steps To Reproduce

Enter conda environment by conda activate DEEPLABCUTAUD
Switch to ipython
import deeplabcut
Analyze videos
Error appears after analyzing two videos

Relevant log output

In [28]: deeplabcut.analyze_videos('/home/fishlab4/Fish_audrey-Audrey-2023-10-31/config.yaml',['/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos'],videotype='.MP4',auto_track=True,allow_growth=True)
Using snapshot-500000 for model /home/fishlab4/Fish_audrey-Audrey-2023-10-31/dlc-models/iteration-0/Fish_audreyOct31-trainset95shuffle1
/home/fishlab4/anaconda3/envs/DEEPLABCUTAUD/lib/python3.9/site-packages/tensorflow/python/keras/engine/base_layer_v1.py:1694: UserWarning: `layer.apply` is deprecated and will be removed in a future version. Please use `layer.__call__` method instead.
  warnings.warn('`layer.apply` is deprecated and '
Activating extracting of PAFs
2023-11-15 11:33:07.319245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1371 MB memory:  -> device: 0, name: Quadro P620, pci bus id: 0000:18:00.0, compute capability: 6.1
Analyzing all the videos in the directory...
Starting to analyze %  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Non-Resident1.MP4
Loading  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Non-Resident1.MP4
Duration of video [s]:  712.71 , recorded with  59.94 fps!
Overall # of frames:  42720  found with (before cropping) frame dimensions:  2704 1520
Starting to extract posture from the video(s) with batchsize: 1
                                                                                                                                                                                                          2023-11-15 11:33:09.072472: W tensorflow/core/common_runtime/bfc_allocator.cc:290] Allocator (GPU_0_bfc) ran out of memory trying to allocate 592.19MiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available.
  0%|                                                                                                                                                                          | 0/42720 [4:56:08<?, ?it/s]
 34%|████████████████████████████████████████████████████▉                                                                                                       | 14493/42720 [4:50:29<9:24:09,  1.20s/it 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 42720/42720 [14:15:16<00:00,  1.20s/it]
Video Analyzed. Saving results in /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos...██████████████████████████████████████████████████████████████████████████| 42720/42720 [14:15:16<00:00,  1.20s/it]
Using snapshot-500000 for model /home/fishlab4/Fish_audrey-Audrey-2023-10-31/dlc-models/iteration-0/Fish_audreyOct31-trainset95shuffle1
Processing...  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Non-Resident1.MP4
Analyzing /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Non-Resident1DLC_dlcrnetms5_Fish_audreyOct31shuffle1_500000.h5
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 42720/42720 [00:22<00:00, 1871.29it/s]
42720it [00:51, 826.14it/s] 
The tracklets were created (i.e., under the hood deeplabcut.convert_detections2tracklets was run). Now you can 'refine_tracklets' in the GUI, or run 'deeplabcut.stitch_tracklets'.
Processing...  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Non-Resident1.MP4
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2595/2595 [00:22<00:00, 115.13it/s]
Starting to analyze %  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Non-Resident1.MP4
Loading  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Non-Resident1.MP4
Duration of video [s]:  711.71 , recorded with  59.94 fps!
Overall # of frames:  42660  found with (before cropping) frame dimensions:  2704 1520
Starting to extract posture from the video(s) with batchsize: 1
 63%|█████████████████████████████████████████████████████████████████████████████████████████████████▊                                                          | 26746/42660 [8:55:39<5:17:41,  1.20s/it]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 42660/42660 [14:16:36<00:00,  1.20s/it]
Video Analyzed. Saving results in /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos...
Using snapshot-500000 for model /home/fishlab4/Fish_audrey-Audrey-2023-10-31/dlc-models/iteration-0/Fish_audreyOct31-trainset95shuffle1
Processing...  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Non-Resident1.MP4
Analyzing /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Non-Resident1DLC_dlcrnetms5_Fish_audreyOct31shuffle1_500000.h5
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 42660/42660 [00:20<00:00, 2042.56it/s]
42660it [00:41, 1019.74it/s]
The tracklets were created (i.e., under the hood deeplabcut.convert_detections2tracklets was run). Now you can 'refine_tracklets' in the GUI, or run 'deeplabcut.stitch_tracklets'.
Processing...  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Non-Resident1.MP4
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1490/1490 [00:14<00:00, 100.95it/s]
Starting to analyze %  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Non-Resident1.MP4
Loading  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Non-Resident1.MP4
Duration of video [s]:  712.71 , recorded with  59.94 fps!
Overall # of frames:  42720  found with (before cropping) frame dimensions:  2704 1520
Starting to extract posture from the video(s) with batchsize: 1
  4%|██████▍                                                                                                                                                       | 1728/42720 [34:31<13:42:12,  1.20s/it]---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[28], line 1
----> 1 deeplabcut.analyze_videos('/home/fishlab4/Fish_audrey-Audrey-2023-10-31/config.yaml',['/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos'],videotype='.MP4',auto_track=True,allow_growth=True)

File ~/anaconda3/envs/DEEPLABCUTAUD/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/predict_videos.py:628, in analyze_videos(config, videos, videotype, shuffle, trainingsetindex, gputouse, save_as_csv, in_random_order, destfolder, batchsize, cropping, TFGPUinference, dynamic, modelprefix, robust_nframes, allow_growth, use_shelve, auto_track, n_tracks, calibrate, identity_only, use_openvino)
    623 from deeplabcut.pose_estimation_tensorflow.predict_multianimal import (
    624     AnalyzeMultiAnimalVideo,
    625 )
    627 for video in Videos:
--> 628     AnalyzeMultiAnimalVideo(
    629         video,
    630         DLCscorer,
    631         trainFraction,
    632         cfg,
    633         dlc_cfg,
    634         sess,
    635         inputs,
    636         outputs,
    637         destfolder,
    638         robust_nframes=robust_nframes,
    639         use_shelve=use_shelve,
    640     )
    641     if auto_track:  # tracker type is taken from default in cfg
    642         convert_detections2tracklets(
    643             config,
    644             [video],
   (...)
    651             identity_only=identity_only,
    652         )

File ~/anaconda3/envs/DEEPLABCUTAUD/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/predict_multianimal.py:184, in AnalyzeMultiAnimalVideo(video, DLCscorer, trainFraction, cfg, dlc_cfg, sess, inputs, outputs, destfolder, robust_nframes, use_shelve)
    172     PredicteData, nframes = GetPoseandCostsF(
    173         cfg,
    174         dlc_cfg,
   (...)
    181         shelf_path,
    182     )
    183 else:
--> 184     PredicteData, nframes = GetPoseandCostsS(
    185         cfg,
    186         dlc_cfg,
    187         sess,
    188         inputs,
    189         outputs,
    190         vid,
    191         nframes,
    192         shelf_path,
    193     )
    195 stop = time.time()
    197 if cfg["cropping"] == True:

File ~/anaconda3/envs/DEEPLABCUTAUD/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/predict_multianimal.py:511, in GetPoseandCostsS(cfg, dlc_cfg, sess, inputs, outputs, cap, nframes, shelf_path)
    503         frame = rgba2rgb(frame)
    504     dets = predict.predict_batched_peaks_and_costs(
    505         dlc_cfg,
    506         np.expand_dims(frame, axis=0),
   (...)
    509         outputs,
    510     )
--> 511     db[key] = dets[0]
    512     del dets
    513 elif counter >= nframes:

IndexError: list index out of range

In[30] deeplabcut.analyze_videos('/home/fishlab4/Fish_audrey-Audrey-2023-10-31/config.yaml',['/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Non-Resident1.MP4','/home/fishlab4/Fish_audr
    ...: ey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Predator1.MP4','/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Resident1.MP4','/home/fishlab4/Fish_audrey-Audrey-2023-10-
    ...: 31/videos/LD1_(Tank51)_Non-Resident2.MP4'],videotype='.MP4',auto_track=True)
Using snapshot-500000 for model /home/fishlab4/Fish_audrey-Audrey-2023-10-31/dlc-models/iteration-0/Fish_audreyOct31-trainset95shuffle1
/home/fishlab4/anaconda3/envs/DEEPLABCUTAUD/lib/python3.9/site-packages/tensorflow/python/keras/engine/base_layer_v1.py:1694: UserWarning: `layer.apply` is deprecated and will be removed in a future version. Please use `layer.__call__` method instead.
  warnings.warn('`layer.apply` is deprecated and '
Activating extracting of PAFs
2023-11-17 12:16:38.828896: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1371 MB memory:  -> device: 0, name: Quadro P620, pci bus id: 0000:18:00.0, compute capability: 6.1
Starting to analyze %  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Non-Resident1.MP4
Loading  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Non-Resident1.MP4
Duration of video [s]:  712.71 , recorded with  59.94 fps!
Overall # of frames:  42720  found with (before cropping) frame dimensions:  2704 1520
Starting to extract posture from the video(s) with batchsize: 1
                                                                                                                                                                                                          ---------------------------------------------------------------------------                                                                                         | 1728/42720 [34:33<13:36:07,  1.19s/it]
IndexError                                Traceback (most recent call last)
Cell In[30], line 1
----> 1 deeplabcut.analyze_videos('/home/fishlab4/Fish_audrey-Audrey-2023-10-31/config.yaml',['/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Non-Resident1.MP4','/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Predator1.MP4','/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Resident1.MP4','/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Non-Resident2.MP4'],videotype='.MP4',auto_track=True)

File ~/anaconda3/envs/DEEPLABCUTAUD/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/predict_videos.py:628, in analyze_videos(config, videos, videotype, shuffle, trainingsetindex, gputouse, save_as_csv, in_random_order, destfolder, batchsize, cropping, TFGPUinference, dynamic, modelprefix, robust_nframes, allow_growth, use_shelve, auto_track, n_tracks, calibrate, identity_only, use_openvino)
    623 from deeplabcut.pose_estimation_tensorflow.predict_multianimal import (
    624     AnalyzeMultiAnimalVideo,
    625 )
    627 for video in Videos:
--> 628     AnalyzeMultiAnimalVideo(
    629         video,
    630         DLCscorer,
    631         trainFraction,
    632         cfg,
    633         dlc_cfg,
    634         sess,
    635         inputs,
    636         outputs,
    637         destfolder,
    638         robust_nframes=robust_nframes,
    639         use_shelve=use_shelve,
    640     )
    641     if auto_track:  # tracker type is taken from default in cfg
    642         convert_detections2tracklets(
    643             config,
    644             [video],
   (...)
    651             identity_only=identity_only,
    652         )

File ~/anaconda3/envs/DEEPLABCUTAUD/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/predict_multianimal.py:184, in AnalyzeMultiAnimalVideo(video, DLCscorer, trainFraction, cfg, dlc_cfg, sess, inputs, outputs, destfolder, robust_nframes, use_shelve)
    172     PredicteData, nframes = GetPoseandCostsF(
    173         cfg,
    174         dlc_cfg,
   (...)
    181         shelf_path,
    182     )
    183 else:
--> 184     PredicteData, nframes = GetPoseandCostsS(
    185         cfg,
    186         dlc_cfg,
    187         sess,
    188         inputs,
    189         outputs,
    190         vid,
    191         nframes,
    192         shelf_path,
    193     )
    195 stop = time.time()
    197 if cfg["cropping"] == True:

File ~/anaconda3/envs/DEEPLABCUTAUD/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/predict_multianimal.py:511, in GetPoseandCostsS(cfg, dlc_cfg, sess, inputs, outputs, cap, nframes, shelf_path)
    503         frame = rgba2rgb(frame)
    504     dets = predict.predict_batched_peaks_and_costs(
    505         dlc_cfg,
    506         np.expand_dims(frame, axis=0),
   (...)
    509         outputs,
    510     )
--> 511     db[key] = dets[0]
    512     del dets
    513 elif counter >= nframes:

IndexError: list index out of range

Anything else?

Below is my config file

  # Project definitions (do not edit)
Task: Fish_audrey
scorer: Audrey
date: Oct31
multianimalproject: true
identity: true

    # Project path (change when moving around)
project_path: /home/fishlab4/Fish_audrey-Audrey-2023-10-31

    # Annotation data set configuration (and individual video cropping parameters)
video_sets:
  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Non-Resident1.MP4:
    crop: 0, 2704, 0, 1520
  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Predator1.MP4:
    crop: 0, 2704, 0, 1520
  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/L.bicolor1_(Tank23)_Resident1.MP4:
    crop: 0, 2704, 0, 1520
  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Non-Resident1.MP4:
    crop: 0, 2704, 0, 1520
  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Non-Resident2.MP4:
    crop: 0, 2704, 0, 1520
  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Predator1.MP4:
    crop: 0, 2704, 0, 1520
  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LD1_(Tank51)_Resident1.MP4:
    crop: 0, 2704, 0, 1520
  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Non-Resident1.MP4:
    crop: 0, 2704, 0, 1520
  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Predator1.MP4:
    crop: 0, 2704, 0, 1520
  /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Resident1.MP4:
    crop: 0, 2704, 0, 1520
individuals:
- Cleaner
- ClientPredator
- ClientResident
- ClientNonResident
uniquebodyparts: []
multianimalbodyparts:
- HeadTerminalMouth
- HeadBasisUp
- HeadBasisDown
- TailBottomCorner
- TailUpperCorner
- TailJunctionBody
- BodyMidUp
- BodyMidDown
bodyparts: MULTI!

    # Fraction of video to start/stop when extracting frames for labeling/refinement
start: 0
stop: 1
numframes2pick: 20

    # Plotting configuration
skeleton:
 - bodypart1
 - bodypart2
 - bodypart2
 - bodypart3
 - bodypart1
 - bodypart3
skeleton_color: black
pcutoff: 0.6
dotsize: 12
alphavalue: 0.7
colormap: rainbow

    # Training,Evaluation and Analysis configuration
TrainingFraction:
- 0.95
iteration: 0
default_net_type: dlcrnet_ms5
default_augmenter: multi-animal-imgaug
default_track_method: ellipse
snapshotindex: -1
batch_size: 1

    # Cropping Parameters (for analysis and outlier frame detection)
cropping: false
    #if cropping is true for analysis, then set the values here:
x1: 0
x2: 640
y1: 277
y2: 624

    # Refinement configuration (parameters from annotation dataset configuration also relevant in this stage)
corner2move2:
- 50
- 50
move2corner: true

Code of Conduct

I agree to follow this project's Code of Conduct

The index error refers to dets[0] which should be the detections for this frame. Since it's happening at some point during I would assume this is related to some video encoding issues.

Could you try reencoding the video and running it again?

@trexbone

Hi @trexbone! I think @KonradDanielewski's suggestion is very good. Try re-encoding the video that failed to run, and analyze it again. You can re-encode it using the command line and FFmpeg by running the command (in the same environment that you launched the DeepLabCut GUI):

ffmpeg -i /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Non-Resident1.MP4 -vcodec libx264 -crf 18 /home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Non-Resident1-reencoded.mp4

I guessed which video had the problem from the logs. This will create a new video called LQ1_Tank51_Non-Resident1-reencoded.mp4 in the same folder as the original one.

Let me know if this fixes your issue!

@n-poulsen

@n-poulsen @KonradDanielewski Hey guys thank you and sorry for the late reply! So I had three videos left and I re-encoded them as suggested, but the videos still failed to analyze with the same error as before. So I tried both codes below, because for some reason the code with allow_growth worked for other videos. I tried rebooting my computer to analyze again but I'm not sure if it would be of much help.
deeplabcut.analyze_videos('/home/fishlab4/Fish_audrey-Audrey-2023-10-31/config.yaml',['/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Non-Resident1-reencoded.mp4'],videotype='.mp4',auto_track=True,allow_growth=True)

deeplabcut.analyze_videos('/home/fishlab4/Fish_audrey-Audrey-2023-10-31/config.yaml',['/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Non-Resident1-reencoded.mp4'],videotype='.mp4
...: ',auto_track=True)

@n-poulsen

deeplabcut.analyze_videos('/home/fishlab4/Fish_audrey-Audrey-2023-10-31/config.yaml',['/home/fishlab4/Fish_audrey-Audrey-2023-10-31/videos/LQ1_Tank51_Non-Resident1-reencoded.mp4'],videotype='.mp4',auto_track=True,allow_growth=True)

@n-poulsen When I used this code on
LQ1_Tank51_Non-Resident1-reencoded.mp4
and other two re-encoded videos, the same error
IndexError: list index out of range occured. Do you think it would help if I reencoded the original video into another video format like .avi?

@trexbone

Hey @trexbone! Looking at the code, I think this might be failing because the model isn't making any predictions for a frame - do you have performance information (the RMSE for your model snapshot)?

And are the videos where it's failing any different to the ones where it's not (e.g., frames where there is nothing visible)?

I don't think switching to a format like .avi would help

@n-poulsen

Hi @n-poulsen thank you for your reply and belated Merry Christmas!
For the RMSE, do you mean those in the .csv file created after evaluation?
And for the 3 videos that failed to analyze, because they were filmed in a different location, the frames where the analysis stopped was because someone blocked the camera and the fishes were out of view. I wasn't aware of that since I thought it was a memory problem.
Would it be better if I edited the three videos and replace the old ones with the edited videos as new videos? Thank you again for your help!!

@trexbone

Hi @trexbone! Yes that's what I meant, however this issue should now have been fixed by the merge above!

The issue must have indeed been because someone stepped in front of the camera, and there was a bug in the code where it would crash if no prediction was made. So you should be able to analyze your videos with the new code. You can either reinstall deeplabcut locally if you already cloned the repository, or you can install the code including the fix by running

pip install --upgrade git+https://github.com/deeplabcut/deeplabcut.git

in your DeepLabCut conda environment. Let me know if you have any other issues!

Navigation Menu

Repository navigation

IndexError: list index out of range when using analyze_videos #2449

Is there an existing issue for this?

Bug description

Operating System

DeepLabCut version

DeepLabCut mode

Device type

Steps To Reproduce

Relevant log output

Anything else?

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

IndexError: list index out of range when using analyze_videos #2449

Description

Is there an existing issue for this?

Bug description

Operating System

DeepLabCut version

DeepLabCut mode

Device type

Steps To Reproduce

Relevant log output

Anything else?

Code of Conduct

Activity

KonradDanielewski commented on Nov 20, 2023

n-poulsen commented on Nov 22, 2023

trexbone commented on Nov 24, 2023

trexbone commented on Nov 26, 2023

n-poulsen commented on Dec 18, 2023

trexbone commented on Dec 26, 2023

n-poulsen commented on Jan 8, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions