The script now supports parallel execution. Set N_WORKERS to control how many experiments run simultaneously.

# In run_experiments_with_cv.py, line ~21
N_WORKERS = 4  # Change from 1 to desired number of parallel workers

python run_experiments_with_cv.py

That's it! The script will now run 4 experiments in parallel.

Rule of thumb: Start with N_WORKERS = 2, then increase gradually while monitoring GPU memory.

# Watch GPU memory usage in real-time
watch -n 1 nvidia-smi

# Or use this one-liner
nvidia-smi -l 1

For 4 folds × 2 seeds = 8 total experiments:

RuntimeError: CUDA out of memory

Solution: Reduce N_WORKERS by half

Cause: Parallel output is less visible Solution: Check GPU usage with nvidia-smi - if GPUs are active, it's working

Solution: Set N_WORKERS = 1 to see full error traces

• Before: Script modified config.yaml directly (caused conflicts)
• After: Each worker gets its own temporary config file (no conflicts)

• Before: Experiments ran one after another
• After: Multiple experiments run simultaneously

• Same format: Results are identical, just computed faster

If you have multiple GPUs, you can assign workers to specific GPUs:

# Add this at the start of run_single_fold() function
gpu_id = seed_idx % torch.cuda.device_count()
os.environ['CUDA_VISIBLE_DEVICES'] = str(gpu_id)

• config_seed0_fold0.yaml
• config_seed0_fold1.yaml
• etc.

These are automatically deleted after each experiment completes.

• results_{timestamp}_{uuid}.csv
• all_results_{timestamp}.csv
• Training models in dlc-models-pytorch/
• Evaluation results in evaluation-results-pytorch/

✅ Original config.yaml is never modified ✅ Temporary config files are auto-deleted ✅ Can switch back to sequential mode anytime (N_WORKERS = 1) ✅ Results format unchanged ✅ No changes to training or evaluation logic

==================== Running 8 tasks with 4 workers ====================

==================== FOLD 1/4 SEED 1/2 ====================
Train ratio: 0.75
  Shuffle 1: Training with 150 frames, testing with 50 frames.
  Creating training dataset for shuffle 1...

==================== FOLD 2/4 SEED 1/2 ====================
Train ratio: 0.75
  Shuffle 2: Training with 150 frames, testing with 50 frames.
  Creating training dataset for shuffle 2...

[... 4 experiments running in parallel ...]

==================== Cross-Validation Summary ====================

N_WORKERS = 1  # (or not set)

# Runs experiments one by one
# Total time: 8 hours (for 8 experiments)
# GPU utilization: 100% on 1 GPU

N_WORKERS = 4

# Runs 4 experiments simultaneously
# Total time: 2 hours (for 8 experiments)
# GPU utilization: 100% on 1 GPU (shared by 4 workers)

1. Start small: Begin with N_WORKERS = 1 to verify everything works
2. Increase gradually: Try 2, then 4, then 8
3. Monitor resources: Watch GPU memory with nvidia-smi
4. Adjust as needed: If you see OOM errors, reduce N_WORKERS
5. For debugging: Always use N_WORKERS = 1

See detailed documentation:

• PARALLEL_EXECUTION_README.md - Full documentation
• PARALLEL_EXECUTION_SUMMARY.md - Visual overview
• CHANGES_DETAILED.md - Code-level changes

To go back to sequential execution:

N_WORKERS = 1

That's it! The script will behave exactly as before.