Migration to Pandas 3.0 #3362

Activity

1. deruyter92

   self-assigned this

   on Jun 10, 2026
2. deruyter92

   added the

   Task

   issue type on Jun 10, 2026
3. deruyter92

   added

   dependenciesPull requests that update a dependency filePull requests that update a dependency file

   packagingPRs that modify metadata, dependency declarations, etc used to build and install the project.PRs that modify metadata, dependency declarations, etc used to build and install the project.

   on Jun 10, 2026
4. C-Achard

   added this to the Pandas 3.0 upgrade milestone on Jun 26, 2026
5. deruyter92

   added 2 commits that reference this issue on Aug 28, 2026

   Guard CoW read-only .to_numpy()/.values mutations (#3362)

   ...

   aafaa2e

   Guard remaining CoW read-only .to_numpy()/.values mutations

   ...

   18bc897
6. AxelNoun commented on Oct 7, 2026

   

   AxelNoun

   on Oct 7, 2026

   Contributor

   More actions

   Notes on the pandas 3 migration, checked on 2026-10-07 against dev at 350917d.

   Canary cell pandas>=3 (ubuntu, Python 3.12, pandas 3.0.6). The most recent run of this cell (job on #3544, against dev, 2026-10-06) reports 5 failed, 3587 passed, 41 skipped, 6 xfailed, and the functional scripts pass. The three runs of 2026-10-01 on #3540, #3541 and #3542 (against main) show the same five failures. All five are in tests/utils/test_pandas_cow.py, which @deruyter92 added to #3445. Each of these tests sets DLC_PANDAS_FUTURE=1, so the version gate in deeplabcut/utils/pandas_future_mode.py, which by design accepts only pandas 2.3.x, raises:

   RuntimeError: pandas future mode requires pandas 2.3.x, got 3.0.6
   

   These failures therefore come from the test setup, not from DeepLabCut behaviour under pandas 3.

   Two call sites that the canary does not exercise. No test under tests/ references either module, and neither functional script uses them. I ran the exact source of the code involved (_format_gt_data and its helpers; filter_low_prob and the groupby statement), extracted from dev at 350917d, on synthetic DLC-style frames:

   • deeplabcut/gui/tracklet_toolbox.py:886, df.groupby(level="bodyparts", axis=1, group_keys=False).apply(filter_low_prob, prob=pcutoff): under pandas 3.0.6, it raises TypeError: DataFrame.groupby() got an unexpected keyword argument 'axis'; under pandas 2.3.3, it runs with a FutureWarning.
   • deeplabcut/benchmark/metrics.py:38, df.stack("individuals", dropna=False) in _format_gt_data: under pandas 3.0.6, it raises ValueError: dropna must be unspecified as the new implementation does not introduce rows of NA values; under pandas 2.3.3, it runs with a FutureWarning.

   Two observations that may be relevant to the fix:

   • metrics.py: with stack("individuals", future_stack=True) instead, _format_gt_data runs on both versions, and under 2.3.3 it returns the same arrays and metadata as the current code on a synthetic two-animal file with missing values. This is a single synthetic case, not a proof of equivalence.
   • tracklet_toolbox.py: under 2.3.3, each bodyparts group contains the columns of every individual, and cols.iloc[:, 2] is the likelihood of the first individual only. On a synthetic two-animal frame, a low likelihood for animal1 also sets the keypoint of animal2 to NaN, whereas a low-likelihood keypoint of animal2 is kept. A port that preserves the grouping and iloc[:, 2] would preserve this behaviour; masking per individual would change what is exported. Is the current behaviour intended?

   HDF and the str dtype (issue description). On pandas 3.0.6 with PyTables 3.11.1, synthetic DLC-style frames whose four column levels have the new str dtype round-trip through to_hdf/read_hdf under a strict assert_frame_equal (dtypes included), with either the python or the pyarrow string storage. This holds in fixed format with a RangeIndex or a labeled-data/video/image MultiIndex as row index, and in table format with a RangeIndex. With a row MultiIndex, table format already fails on pandas 2.3.3, with a ValueError about data_columns. The NotImplementedError quoted in the description is raised by pandas 2.3.3 in fixed format once future.infer_string is enabled, as pandas_future_mode.py does; in pandas 3.0.0 and 3.0.6, pandas/io/pytables.py skips this check for StringDtype levels. For these layouts, the conversion back to object before saving may therefore not be needed under pandas 3; I have not tested this on real project files.

   @deruyter92, as the assignee of this issue (the #3362 TODOs in the code also carry your handle): are these sites already covered in your migration work? If not, would separate small PRs against dev be useful, first for test_pandas_cow.py under pandas >= 3, then for each call site with a regression test?