pathlib.Path.glob does not follow symlinks

@pfmoore

BPO	33428
Nosy	@pfmoore, @pitrou, @tjguk, @jreese, @zware, @zooba, @emilyemorehouse, @BrianMSheldon

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-05-05.06:23:00.558>
labels = ['type-bug', 'library', 'OS-windows']
title = 'pathlib.Path.glob does not follow symlinks'
updated_at = <Date 2020-04-29.11:12:49.344>
user = 'https://github.com/BrianMSheldon'

bugs.python.org fields:

activity = <Date 2020-04-29.11:12:49.344>
actor = 'Danya.Alexeyevsky'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2018-05-05.06:23:00.558>
creator = 'brianmsheldon'
dependencies = []
files = []
hgrepos = []
issue_num = 33428
keywords = []
message_count = 4.0
messages = ['316197', '316724', '316880', '367640']
nosy_count = 9.0
nosy_names = ['paul.moore', 'pitrou', 'tim.golden', 'jreese', 'Danya.Alexeyevsky', 'zach.ware', 'steve.dower', 'emilyemorehouse', 'brianmsheldon']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33428'
versions = ['Python 3.6']

Linked PRs

Given a pathlib.Path that contains symlinked subfolders, Path.glob (and .rglob) do not follow symlinks. This is not consistent with glob.glob which does.

For example given the following:
C:\Folder
C:\Folder\Subfolder -> D:\Subfolder
D:\Subfolder\File.txt

pathlib.Path('C:/Folder').glob('**/*') yields the following paths:
WindowsPath('C:/Folder/Subfolder')

glob.glob('C:/Folder/**/*') yields the following paths:
'C:/Folder\Subfolder'
'C:/Folder\Subfolder\File.txt'

Notice how the contents of Subfolder are present in the glob.glob results but not for Path.glob.

I would expect Path.glob to be consistent with glob.glob. This is not the only inconsistency (e.g. bpo-22276, bpo-31202) and perhaps Path.glob should be re-implemented using glob.glob.

This looks like an issue specific to Windows? I can't replicate on Mac, and given Windows' method of implementing "symlinks" as junctions.

Windows does not implement symlinks as junctions. Windows has hardlinks, symlinks and junctions which are all distinctly different in behaviour.

I don't doubt that this is a Windows-specific issue, although I have not tested other platforms. Path.glob and .rglob does work for junctions and hardlinks but glob.glob works consistently for all three.

I can reproduce the bug with Linux and python 3.7.5:

Python 3.7.5 (default, Apr 19 2020, 20:18:17) 
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pathlib import Path
>>> Path('a/b').mkdir(parents=True)
>>> Path('c/d').mkdir(parents=True)
>>> Path('a/c').symlink_to('../c')
>>> Path('e').symlink_to('c')
>>> list(Path('.').rglob('*'))
[PosixPath('e'), PosixPath('c'), PosixPath('a'), PosixPath('c/d'), PosixPath('a/c'), PosixPath('a/b')]

Expected result:

[PosixPath('e'), PosixPath('e/d'), PosixPath('c'), PosixPath('a'), PosixPath('c/d'), PosixPath('a/c'), PosixPath('a/c/d'), PosixPath('a/b')]

Following symlinks was disabled on purpose as a fix for #70200:

cpython/Lib/pathlib.py

Line 319 in 69f6cc7

if entry_is_dir and not entry.is_symlink():

To re-enable it, we'd have to come up with a different mitigation for the symlink loops problem.

Or possibly, if hidden behind a follow_symlinks arg which would be False by default, it might be enough to just document it as a limitation when follow_symlinks=True? I don't know.

Or possibly, if hidden behind a follow_symlinks arg which would be False by default, it might be enough to just document it as a limitation when follow_symlinks=True? I don't know.

glob() still follows symlinks when matching segments like foo/, */ and so forth; it only refuses to follow symlinks when it encounters a ** wildcard. So the current behaviour is roughly follow_symlinks=Sometimes.

I realised this because I've been trying to implement an iterative version of rglob(), building upon walk() and filtering paths with a compiled re.Pattern object. walk() has its own follow_symlinks argument that accepts True and False, but neither option aligns with the existing behaviour, so it seems like a dead end unless this "bug" is "fixed".

zsh has a *** wildcard that recurses into symlinks to directories, see Recursive Globbing here: https://linux.die.net/man/1/zshexpn

Having thought about this some more, I'd like to propose that we add follow_symlinks arguments to glob() and rglob(), where:

follow_symlinks=False treats symlinks as files (default)
follow_symlinks=True follows symlinks to directories

Note that the default behaviour would therefore change: a pattern like foo/bar or */bar would no longer match foo/bar if foo is a symlink. This is consistent with how **/bar works today.

With that in place, we could implement glob() iteratively atop walk(), which would address #89727 and substantially improve performance!

Thoughts?

Merge branch 'main' into pythongh-77609-glob-follow-symlinks

GH-77609: Add follow_symlinks argument to `pathlib.Path.glob()` (GH-1…

@zooba

I've added a keyword-only follow_symlinks argument, which will be available in Python 3.13. See 9700dd1 / #102616

Thank you everyone, and especially @zooba, for the help in getting this through. Resolving!

Lets change the new follow_symlinks=None to follow_symlinks=NotRecursive (or some other explicitly named sentinel value defined in the pathlib module).

People reading code that explicitly specifies follow_symlinks=None as would be needed if we ever decide to change the default would not be able to reason about what that does and why =None isn't the same behavior as =False or =0. Seeing a named constant makes it much more clear to those not intimately familiar with the API.

This is good even if we never decide to work towards a default change.

pythonGH-77609: Add recurse_symlinks argument to `pathlib.Path.glob()`

GH-77609: Add recurse_symlinks argument to `pathlib.Path.glob()` (#11…

Re-resolving - the argument is now recurse_symlinks and accepts a boolean.

pythonGH-77609: Add recurse_symlinks argument to `pathlib.Path.glob()` (

Navigation Menu

Repository navigation

pathlib.Path.glob does not follow symlinks #77609

Linked PRs

24 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

pathlib.Path.glob does not follow symlinks #77609

Description

Linked PRs

Activity

BrianMSheldon commented on May 5, 2018

jreese commented on May 15, 2018

BrianMSheldon commented on May 17, 2018

DanyaAlexeyevsky commented on Apr 29, 2020

dlukes commented on Nov 24, 2022

barneygale commented on Jan 28, 2023

barneygale commented on Jan 29, 2023

barneygale commented on Feb 6, 2023

24 remaining items

barneygale commented on May 29, 2023

gpshead commented on Mar 23, 2024

barneygale commented on Apr 7, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions