Visitar URL original
Expose DataLoader kwargs (`TrainSettingsConfig.dataloader_kwargs`) + add validation helper `reject_reserved_keys` by deruyter92 · Pull Request #3554 · DeepLabCut/DeepLabCut · GitHub
Skip to content

Expose DataLoader kwargs (TrainSettingsConfig.dataloader_kwargs) + add validation helper reject_reserved_keys - #3554

Closed
deruyter92 wants to merge 6 commits into
devfrom
jaap/config-extra-kwargs
Closed

deruyter92 wants to merge 6 commits into
devfrom
jaap/config-extra-kwargs

Conversation

@deruyter92

@deruyter92 deruyter92 commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up PR on #3550 with alternative solution. Can be closed after merging into that PR.

Changes:

  • Add TrainSettingsConfig.dataloader_kwargs to pass-through non-exposed fields to torch Dataloader
  • Add reject_reserved_keys validation helper for mappings that may not contain some specific keys.

See also #3469 for a related case. The wandb_kwargs solution could also be simplified using the helper from this PR.

lu123bing and others added 6 commits October 13, 2026 11:16
…tings

Training on small datasets over hundreds of epochs respawns DataLoader
workers every epoch (persistent_workers=False) with a fixed prefetch depth
of 2, leaving GPU utilization on the table for CPU-heavy pipelines. These
knobs existed in PyTorch but were not reachable through TrainSettingsConfig,
pytorch_config.yaml, or pytorch_cfg_updates (strict schema rejects them).

- TrainSettingsConfig: add dataloader_persistent_workers (default False)
  and dataloader_prefetch_factor (default None = PyTorch default 2,
  validated >= 1).
- train(): forward both to the training DataLoader when
  dataloader_workers > 0; warn and ignore them for single-process loading
  (PyTorch requires workers for both options). .get() with defaults keeps
  old persisted configs without the keys working.
- base.yaml / base_detector.yaml defaults + pytorch_config.md docs.

Adds config roundtrip/validation tests and train() DataLoader-kwarg tests.

Addresses #3547

@Ayskin Ayskin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice generic design for additional DataLoader arguments! And to jump ahead to #3555 that suggests validating dataloader_kwargs against dataloader_workers, I would keep the design argument-agnostic for dataloader_kwargs with the cost of failure at training time. What do you think ?

@deruyter92

Copy link
Copy Markdown
Collaborator Author

merged in #3550

@deruyter92 deruyter92 closed this Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants