Visitar URL original
Metric path consistency for top-down, and safer snapshot selection when a metric is undefined · Issue #3521 · DeepLabCut/DeepLabCut · GitHub
Skip to content

Metric path consistency for top-down, and safer snapshot selection when a metric is undefined #3521

Open

Description

@C-Achard

Follow-up to: #3518, #3519.

There are (several) additional findings that matter for follow-up PRs and could improve metrics-based snapshot selection quite a lot.

snapshotindex: "best" raises instead of falling back

After #3519, no best snapshot is written when the key metric is never computable, e.g. a one-bodypart project, where mAP is permanently NaN. snapshotindex="best" then raises No best snapshot found in ....

Fix: in pose_estimation_pytorch/apis/utils.py, fall back to the latest snapshot with a loud warning rather than raising. The warning should say why no best snapshot exists and what to set instead.

Raise at config validation time, not during training

Several of the failures above are config mistakes, and can be validated against.

  • key_metric is an unvalidated free string. RunnerConfig.key_metric is typed str. Validate against the metric names the runner can actually produce.
  • key_metric_asc is not checked against key_metric. The two fields are independent, so key_metric: "test.rmse" with key_metric_asc: true silently selects the worst model. Error metrics must be descending, AP/AR metrics ascending. Probably a loud warning rather than a hard error, since existing configs may already have it wrong.
  • key_metric can be unusable for certain projects With fewer than two bodyparts, an OKS-based key_metric disables best-snapshot selection for the whole run. The validator can see metadata.bodyparts and runner.key_metric.
  • metadata.individuals and the head's num_animals can disagree. A hand-edited pytorch_config.yaml has one individual in metadata while a dlcrnet/dekr head still predicts several, the single-animal metric path raises. snapshotindex cannot express "best". ProjectConfig.snapshotindex is typed Literal["all"] | int, so snapshotindex: best in config.yaml fails validation even though get_model_snapshots supports "best" and documents it.

What cannot move to config validation:

Top-down and conditional top-down still use the multi-animal metric path

For TOP_DOWN and COND_TOP_DOWN, each training sample is a single cropped individual, and
epoch metrics are accumulated one crop at a time. Training-time metrics are therefore always
one-ground-truth-to-one-prediction, whatever the project's animal count.

#3519 derives the single-animal flag from max_num_animals == 1, so multi-animal top-down
projects keep the multi-animal path. That path matches predictions to ground truth via greedy
OKS before computing RMSE, and discards pairs that fail the threshold.

Expected: for a given snapshot, training-time RMSE is computed the same way as
evaluate_network computes it.

Actual: for multi-animal top-down projects, training-time RMSE silently omits badly-placed
predictions, so it disagrees with evaluate_network. No warning is emitted.

Fix: extend PoseTrainingRunner._compute_epoch_metrics
(pose_estimation_pytorch/runners/train.py) to treat top-down and conditional top-down as
single-animal, since the accumulated data is 1-vs-1 by construction.

Note training and evaluate_network cannot produce identical numbers for multi-animal top-down, because training aggregates per crop while evaluation aggregates per image with several individuals.

Misc

  • There is no training-side equivalent of evaluate()'s force_multi_animal override. Only
    matters if a hand-edited pytorch_config.yaml has metadata.individuals and a head's
    num_animals out of sync

Activity

  1. added theissue type on Sep 17, 2026
  2. added
    enhancementNew feature or request
    configRelated to config.yaml, ruamel, YAML parsing, ...
    metricsRelated to performance reporting and metrics calculation
    on Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

configRelated to config.yaml, ruamel, YAML parsing, ...enhancementNew feature or requestmetricsRelated to performance reporting and metrics calculationpytorch

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions