Repository navigation
Optimize pathlib path construction #101362
Description
Activity
I'd like to land #101363 before I put the first PR up for this issue.
- added 5 commits that reference this issue
on Feb 4, 2023 Does the PR cope with
WindowsPurePathandPosixPurePathand converting between them?Could you clarify? The PRs maintain the behaviour that attempting to instantiate
pathlib.PurePathwill give you aPureWindowsPathor aPurePosixPathdepending on your platform, but pathlib doesn't support converting between Windows and POSIX paths as I understand it.This behaviour should be preserved:
Python 3.10.10 (tags/v3.10.10:aad5f6a, Feb 7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from pathlib import * >>> p = PureWindowsPath("a/b/c") >>> p PureWindowsPath('a/b/c') >>> PurePosixPath(p) PurePosixPath('a/b/c') >>> PurePath(p) PureWindowsPath('a/b/c') >>> PosixPath(p) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__3847v3x7pw1km\lib\pathlib.py", line 962, in __new__ raise NotImplementedError("cannot instantiate %r on your system" NotImplementedError: cannot instantiate 'PosixPath' on your system >>> WindowsPath(p) WindowsPath('a/b/c') >>>Reacted by Barney GaleRight! That will be broken by #101667 as things stand:
>>> from os import fspath >>> from pathlib import * >>> p = PureWindowsPath("a/b/c") >>> p PureWindowsPath('a/b/c') >>> fspath(p) 'a\\b\\c' >>> PurePosixPath(fspath(p)) PurePosixPath('a\\b\\c') >>> PurePosixPath(p) PurePosixPath('a\\b\\c')
It doesn't appear to be documented or tested behaviour, and it feels odd to me that
PurePosixPath(p)can give a different result toPurePosixPath(fspath(p)). I'll try to find a good way forwards!Reacted by Steve DowerThis feature doesn't really work with drives or roots:
>>> PurePosixPath(PureWindowsPath('//server/share/dir')) PurePosixPath('\\\\server\\share\\/dir') >>> PurePosixPath(PureWindowsPath('c:/dir')) PurePosixPath('c:\\/dir') >>> PurePosixPath(PureWindowsPath('/dir')) PurePosixPath('\\/dir')
As far as I can tell, no one has ever logged a bug about it.
However, using
PurePosixPath(PureWindowsPath(...).as_posix())works everywhere:>>> PurePosixPath(PureWindowsPath('//server/share/dir').as_posix()) PurePosixPath('//server/share/dir') >>> PurePosixPath(PureWindowsPath('c:/dir').as_posix()) PurePosixPath('c:/dir') >>> PurePosixPath(PureWindowsPath('/dir').as_posix()) PurePosixPath('/dir')
So I'm tempted to conclude that converting with
PurePosixPath(PureWindowsPath(...))alone is not supported, and that it only happened to work in some limited circumstances. Users can add.as_posix()to signal their intent and make it work reliably. What do you reckon?I think the direct conversion should either be consistent with
fspath(p)orp.parts, but no strong preference which. As you say, neither makes a huge amount of sense, and the workaround (.as_posix()) has been there for a long time.Might be worth adding a short note to the docs warning that the constructor can't reliably convert from different
pathlibobjects, and maybe specify whether it willfspathor.partsthem.On another perf note, is it possible that parsing the path up front isn't necessary? Obviously it'll save the most time to keep a single string literal around and parse it later, but I don't personally have a good feel for whether that's common or not. (Obviously if it's available pre-parsed then keep it.)
Reacted by Barney Gale7 remaining items
- added 2 commits that reference this issue
on Mar 6, 2023 For anyone following along, I think this ticket can be resolved if/when this PR lands:
Resolving this issue. It's now much cheaper to construct
pathlib.PurePathandPathobjects. Some operations likestr()andjoinpath()are also cheaper.Future work:
Reacted by Alex Waygood and skshetry- added a commit that references this issue
on Nov 23, 2023
Pathlib is slow. One of the most obvious symptoms is that
pathlib.PurePathobjects are slow to construct. We should be able to speed construction up without making other parts of pathlib slower.Two possible approaches:
__new__(),_from_parts(),_parse_parts(),_parse_args().Linked PRs
pathlib.PurePath()._parts#102476