Repository navigation
Description
There are (several) additional findings that matter for follow-up PRs and could improve metrics-based snapshot selection quite a lot.
snapshotindex: "best" raises instead of falling back
After #3519, no best snapshot is written when the key metric is never computable, e.g. a one-bodypart project, where mAP is permanently NaN. snapshotindex="best" then raises No best snapshot found in ....
Fix: in pose_estimation_pytorch/apis/utils.py, fall back to the latest snapshot with a loud warning rather than raising. The warning should say why no best snapshot exists and what to set instead.
Raise at config validation time, not during training
Several of the failures above are config mistakes, and can be validated against.
key_metricis an unvalidated free string.RunnerConfig.key_metricis typedstr. Validate against the metric names the runner can actually produce.key_metric_ascis not checked againstkey_metric. The two fields are independent, sokey_metric: "test.rmse"withkey_metric_asc: truesilently selects the worst model. Error metrics must be descending, AP/AR metrics ascending. Probably a loud warning rather than a hard error, since existing configs may already have it wrong.key_metriccan be unusable for certain projects With fewer than two bodyparts, an OKS-basedkey_metricdisables best-snapshot selection for the whole run. The validator can seemetadata.bodypartsandrunner.key_metric.metadata.individualsand the head'snum_animalscan disagree. A hand-editedpytorch_config.yamlhas one individual inmetadatawhile adlcrnet/dekrhead still predicts several, the single-animal metric path raises.snapshotindexcannot express"best".ProjectConfig.snapshotindexis typedLiteral["all"] | int, sosnapshotindex: bestinconfig.yamlfails validation even thoughget_model_snapshotssupports"best"and documents it.
What cannot move to config validation:
- Whether a
-bestfile exists on disk depends on the training outcome, not the config - Whether ground truth keypoints are visible or degenerate is a property of the data, so the
warning added in Fix silent OKS metrics failure on invalid computation and problematic metrics propagation #3519 has to stay where it is. -1resolving to the best snapshot depends on which files are present
Top-down and conditional top-down still use the multi-animal metric path
For TOP_DOWN and COND_TOP_DOWN, each training sample is a single cropped individual, and
epoch metrics are accumulated one crop at a time. Training-time metrics are therefore always
one-ground-truth-to-one-prediction, whatever the project's animal count.
#3519 derives the single-animal flag from max_num_animals == 1, so multi-animal top-down
projects keep the multi-animal path. That path matches predictions to ground truth via greedy
OKS before computing RMSE, and discards pairs that fail the threshold.
Expected: for a given snapshot, training-time RMSE is computed the same way as
evaluate_network computes it.
Actual: for multi-animal top-down projects, training-time RMSE silently omits badly-placed
predictions, so it disagrees with evaluate_network. No warning is emitted.
Fix: extend PoseTrainingRunner._compute_epoch_metrics
(pose_estimation_pytorch/runners/train.py) to treat top-down and conditional top-down as
single-animal, since the accumulated data is 1-vs-1 by construction.
Note training and evaluate_network cannot produce identical numbers for multi-animal top-down, because training aggregates per crop while evaluation aggregates per image with several individuals.
Misc
- There is no training-side equivalent of
evaluate()'sforce_multi_animaloverride. Only
matters if a hand-editedpytorch_config.yamlhasmetadata.individualsand a head's
num_animalsout of sync
Activity