Visitar URL original
Migrate to `pathlib.Path` instead of `str` (first phase) (#3350) · DeepLabCut/DeepLabCut@30a8155 · GitHub
Skip to content

Commit 30a8155

Browse files
deruyter92C-AchardCopilot
authored
Migrate to pathlib.Path instead of str (first phase) (#3350)
* fix path resolution on network drives. Replace Path(..).resolve() with Path(..).absolute() * Add safe_resolve helper for path resolution * replace str with pathlib.Path * fix redundant resolution of resolved path in memory_replay.py * fix open -> os.stat * replace os.stat -> open for files, listdir for dirs. * fix TestReadConfigProjectPath * Update deeplabcut/utils/auxiliaryfunctions.py * additional replacements: str ->Path * Update function signatures str -> str | Path * Add ruff linting rule PTH * Open dataset file directly using Path.open Fix path handling when loading the dataset pickle. The code previously attempted Path(project_path) / file_name even though file_name already included the project path, resulting in an incorrect path. This changes the context manager to call file_name.open('rb') directly, removing the redundant join and preventing path errors when loading the pickle. * Use context manager when writing pose_cfg.yaml Replace direct ruamel_file.dump(...open('w')) call with a with-statement to open pose_cfg.yaml. This ensures the file handle is properly closed after writing and avoids potential resource leaks. * Pass string path to cv2.VideoCapture Convert video_path to str when constructing cv2.VideoCapture to ensure pathlib.Path objects are handled correctly and avoid type errors when opening videos. This makes VideoReader more robust when given Path instances. * Use explicit loop to unlink frame files Replace the list-comprehension side-effect used to delete temporary frame files with an explicit for-loop that checks p.is_file() before calling unlink. This avoids building an unused list, prevents errors from trying to unlink non-file entries (e.g., directories), and improves readability and safety. * Fix CircleCI failure * fix Matplotlib Path import shadowing ->MPLPath * use pathlib consistently in `metrics` * use safe_resolve in export * remove redundant defensive block for windows paths on python 3.6 * remove unused `get_immediate_subdirectories` * use Path in plot_edge_affinity_distributions * remove PTH enforcement in linting * use safe_resolve instead of resolve in `training` and `auxiliaryfunctions` * Update GUI assets import (#3370) * Add GUI asset helper utilities Introduce deeplabcut.gui.gui_assets module providing utilities to load bundled GUI assets. It locates the package assets directory via importlib.resources, and exposes resource_bytes, resource_text, get_assets_dir, get_style_qss, pixmap_from_resource and icon_from_resource helpers (using PySide6 QPixmap/QIcon). Includes error handling when image data cannot be loaded and uses type annotations for clarity. * Use gui_assets for icons and pixmaps Replace direct filesystem asset loading (BASE_DIR/assets and manual Path lookups) with centralized gui_assets helpers (icon_from_resource, pixmap_from_resource, get_style_qss/get_assets_dir) across multiple GUI modules. Updated imports and calls in components, launch_script, create_project, modelzoo, open_project, train_network, and window to use the new resource functions and remove redundant logo path handling, improving resource access and packaging robustness. * fix path concatenation in `make_labeled_images_from_dataframe` --------- Co-authored-by: Jaap de Ruyter van Steveninck <32810691+deruyter92@users.noreply.github.com> * Adjust config path tests for Path objects Update `test_auxiliaryfunctions.py` to assert `project_path` as a `pathlib.Path` instead of a string. The tests now cast to `str` only for the `"Volume{"` check, use `.exists()` for filesystem validation, and compare against `project_dir.absolute()` directly. * Fix os import in safe_resolve Import `os` inside `safe_resolve` before calling `os.listdir` so directory resolution no longer risks a `NameError`. This also removes an unused `os` import in `test_auxiliaryfunctions.py` and a stray blank line in `video_processor.py`. * address remaining `.resolve()` cases: prefer `.absolute()`, or otherwise `safe_resolve()` * update `safe_resolve`: add logging, use os.stat (more efficient) * update test safe_resolve: allow fallback and test open/closing instead Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix remaining Path.resolve() * fix reintroduced pathlib `resolve()` occurences (see #3349) * use pathlib Path in GUI codebase * use pathlib Path in memory replay * GUI rename field `config` to `config_path` * GUI use os.fspath for str conversions where required * use pathlib Path in create_project * use pathlib Path across pose_estimation_pytorch internally * use centralized yaml dumper (which normalizes Path -> str) * fix materialize.py downstream old str.split -> Path * fix testsscripts str usage * fix old str usage in trainingsetmanipulation * allow str type at TF boundary * allow str formatting in tensorflow code * fix config update in train_from_coco.py PoseConfig.model.backbone is currently a dict-type but freeze_bn_stats were set as attribute. This commit fixes that and treats backbone correctly as dict. --------- Co-authored-by: Cyril Achard <cyril.achard@epfl.ch> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
1 parent 89aa023 commit 30a8155

122 files changed

Lines changed: 1454 additions & 1474 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎deeplabcut/benchmark/__init__.py‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -13,12 +13,13 @@
1313
import json
1414
import os
1515
from collections.abc import Container
16+
from pathlib import Path
1617
from typing import Literal
1718

1819
from deeplabcut.benchmark.base import Benchmark, Result, ResultCollection
1920

20-
DATA_ROOT = os.path.join(os.getcwd(), "data")
21-
CACHE = os.path.join(os.getcwd(), ".results")
21+
DATA_ROOT = Path.cwd() / "data"
22+
CACHE = Path.cwd() / ".results"
2223

2324
__registry = []
2425

@@ -99,20 +100,20 @@ def evaluate(
99100

100101

101102
def get_filepath(basename: str):
102-
return os.path.join(DATA_ROOT, basename)
103+
return DATA_ROOT / basename
103104

104105

105106
def savecache(results: ResultCollection):
106-
with open(CACHE, "w") as fh:
107+
with Path(CACHE).open("w") as fh:
107108
json.dump(results.todicts(), fh, indent=2)
108109

109110

110111
def loadcache(cache=CACHE, on_missing: Literal["raise", "ignore"] = "ignore") -> ResultCollection:
111-
if not os.path.exists(cache):
112+
if not Path(cache).exists():
112113
if on_missing == "raise":
113114
raise FileNotFoundError(cache)
114115
return ResultCollection()
115-
with open(cache) as fh:
116+
with Path(cache).open() as fh:
116117
try:
117118
data = json.load(fh)
118119
except json.decoder.JSONDecodeError as e:

‎deeplabcut/benchmark/metrics.py‎

Lines changed: 19 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -11,9 +11,9 @@
1111

1212
"""Evaluation metrics for the DeepLabCut benchmark."""
1313

14-
import os
1514
import pickle
1615
from collections import defaultdict
16+
from pathlib import Path
1717

1818
import numpy as np
1919
import pandas as pd
@@ -33,7 +33,7 @@ def _format_gt_data(h5file: str, test_indices: list[int] | None = None):
3333
except KeyError:
3434
n_unique = 0
3535
guarantee_multiindex_rows(df)
36-
file_paths = [os.path.join(*row) for row in df.index.to_list()]
36+
file_paths = [Path(*row) for row in df.index.to_list()]
3737
temp = (
3838
df.stack("individuals", dropna=False)
3939
.reindex(animals, level="individuals")
@@ -98,19 +98,18 @@ def calc_prediction_errors(preds, gt):
9898
return errors
9999

100100

101-
def _map(strings, substrings):
101+
def _map(paths: list[Path], subpaths: list[Path]) -> dict[Path, Path]:
102102
"""Map image paths from predicted data to GT as the first are typically absolute
103103
whereas the latter are relative to the project path."""
104-
105-
lookup = dict()
106-
strings_ = strings.copy()
107-
substrings_ = substrings.copy()
108-
while strings_:
109-
string = strings_.pop()
110-
for s in substrings_:
111-
if string.endswith(s):
112-
lookup[string] = s
113-
substrings_.remove(s)
104+
lookup = {}
105+
paths_ = paths.copy()
106+
subpaths_ = subpaths.copy()
107+
while paths_:
108+
path = paths_.pop()
109+
for s in subpaths_:
110+
if path.parts[-len(s.parts) :] == s.parts:
111+
lookup[path] = s
112+
subpaths_.remove(s)
114113
break
115114
return lookup
116115

@@ -146,6 +145,7 @@ def calc_map_from_obj(
146145
drop_kpts=None,
147146
):
148147
"""Calculate mean average precision (mAP) based on predictions."""
148+
eval_results = {Path(k): v for k, v in eval_results_obj.items()}
149149
df = pd.read_hdf(h5_file)
150150
try:
151151
df.drop("single", level="individuals", axis=1, inplace=True)
@@ -157,7 +157,7 @@ def calc_map_from_obj(
157157
test_indices = _load_test_indices(metadata_file)
158158
df_test = df.iloc[test_indices]
159159
test_images = load_test_images(h5_file, metadata_file)
160-
missing_images = set(test_images) - set(eval_results_obj.keys())
160+
missing_images = set(test_images) - set(eval_results)
161161
if len(missing_images) > 0:
162162
raise ValueError(
163163
f"Failed to compute the test mAP: there are test images missing from theprediction object: {missing_images}"
@@ -184,7 +184,7 @@ def calc_map_from_obj(
184184
for ind in sorted(drop_kpts, reverse=True):
185185
kpts.pop(ind)
186186

187-
assemblies_pred = conv_obj_to_assemblies(eval_results_obj, kpts)
187+
assemblies_pred = conv_obj_to_assemblies(eval_results, kpts)
188188
with deeplabcut.benchmark.utils.DisableOutput():
189189
oks = inferenceutils.evaluate_assembly(
190190
assemblies_pred,
@@ -213,7 +213,7 @@ def calc_rmse_from_obj(
213213
for ind in sorted(drop_kpts, reverse=True):
214214
kpts.pop(ind)
215215

216-
test_objects = {k: v for k, v in eval_results_obj.items() if k in gt["annotations"].keys()}
216+
test_objects = {Path(k): v for k, v in eval_results_obj.items() if Path(k) in gt["annotations"]}
217217
if len(gt["annotations"]) != len(test_objects):
218218
gt_images = list(gt["annotations"].keys())
219219
missing_images = [img for img in gt_images if img not in test_objects]
@@ -239,22 +239,17 @@ def calc_rmse_from_obj(
239239
return np.nanmean(errors[..., 0])
240240

241241

242-
def load_test_images(h5file: str, metadata: str) -> list[str]:
242+
def load_test_images(h5file: str, metadata: str) -> list[Path]:
243243
"""Returns the names of the test images for the benchmark, in the order
244244
corresponding to the test indices."""
245245
df = pd.read_hdf(h5file)
246246
test_indices = _load_test_indices(metadata)
247247
df_test = df.iloc[test_indices]
248-
test_images = []
249-
for img_path in df_test.index:
250-
if not isinstance(img_path, str):
251-
img_path = os.path.join(*img_path)
252-
test_images.append(img_path)
253-
return test_images
248+
return [Path(img_path) if isinstance(img_path, str) else Path(*img_path) for img_path in df_test.index]
254249

255250

256251
def _load_test_indices(shuffle_metadata_path: str) -> list[int]:
257252
"""Returns the indices of test images in the training dataset dataframe."""
258-
with open(shuffle_metadata_path, "rb") as f:
253+
with Path(shuffle_metadata_path).open("rb") as f:
259254
test_indices = set([int(i) for i in pickle.load(f)[2]])
260255
return list(sorted(test_indices))

‎deeplabcut/benchmark/utils.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
import os
1919
import pkgutil
2020
import sys
21+
from pathlib import Path
2122

2223

2324
class RedirectStdStreams:
@@ -46,7 +47,7 @@ def __exit__(self, exc_type, exc_value, traceback):
4647

4748
class DisableOutput(RedirectStdStreams):
4849
def __init__(self):
49-
devnull = open(os.devnull, "w")
50+
devnull = Path(os.devnull).open("w")
5051
super().__init__(stdout=devnull, stderr=devnull)
5152

5253

0 commit comments

Comments
 (0)