Visitar URL original
Path.rglob fails with *.* on Python 3.11.0a7 · Issue #91616 · python/cpython · GitHub
Skip to content

Path.rglob fails with *.* on Python 3.11.0a7 #91616

Description

@domdfcoding

Bug report

On Python 3.11.0a7, calling Path.rglob("*.*") returns an iterator over no files, whereas on previous versions (including alpha 6) it would correctly return an iterator over all files in the directory and all of its children.

The following code illustrates this problem:

import pathlib, tempfile, pprint

with tempfile.TemporaryDirectory() as tmpdir:
	tmp_path = pathlib.Path(tmpdir)
	(tmp_path / "a.txt").touch()
	(tmp_path / "b.txt").touch()
	(tmp_path / "c").mkdir()
	(tmp_path / "c" / "d.txt").touch()
	pprint.pprint(list(tmp_path.rglob("*.*")))
	pprint.pprint(list(tmp_path.rglob("*.txt")))

Save this as pathlib_rglob.py.

Output:

$ python3.11 pathlib_rglob.py 
[]
[PosixPath('/tmp/tmpngjo2cyn/a.txt'),
 PosixPath('/tmp/tmpngjo2cyn/b.txt'),
 PosixPath('/tmp/tmpngjo2cyn/c/d.txt')]
$ python3.10 pathlib_rglob.py 
[PosixPath('/tmp/tmpodfbqyhx/a.txt'),
 PosixPath('/tmp/tmpodfbqyhx/b.txt'),
 PosixPath('/tmp/tmpodfbqyhx/c/d.txt')]
[PosixPath('/tmp/tmpodfbqyhx/a.txt'),
 PosixPath('/tmp/tmpodfbqyhx/b.txt'),
 PosixPath('/tmp/tmpodfbqyhx/c/d.txt')]

$ # This is Python 3.11.0a6 built from source
$ ~/python311a6/bin/python3.11 pathlib_rglob.py 
[PosixPath('/tmp/tmpobellcrw/a.txt'),
 PosixPath('/tmp/tmpobellcrw/b.txt'),
 PosixPath('/tmp/tmpobellcrw/c/d.txt')]
[PosixPath('/tmp/tmpobellcrw/a.txt'),
 PosixPath('/tmp/tmpobellcrw/b.txt'),
 PosixPath('/tmp/tmpobellcrw/c/d.txt')]

After much debugging I have tracked the issue to #32029
With that change, fnmatch.translate("*.*") gives (?s:(?>.*?\.).*)\Z.
With the change reverted (or on 3.11.0a6) its (?s:(?=(?P<g0>.*?\.))(?P=g0).*)\Z.
I don't know what the difference between those amounts to, but the former doesn't work while the latter does:

>>> import re
>>> re.fullmatch('(?s:(?>.*?\\.).*)\\Z', "a.txt")
>>> re.fullmatch('(?s:(?=(?P<g0>.*?\\.))(?P=g0).*)\\Z', "a.txt")
<re.Match object; span=(0, 5), match='a.txt'>

Your environment

  • CPython versions tested on: 3.10.4, 3.11.0a6, 3.11.0a7, built from git checkouts of the relevant tag.
  • Operating system and architecture: Ubuntu 20.04 amd64

Activity

  1. domdfcoding commented on Apr 16, 2022

    @domdfcoding
    ContributorAuthor

    @tim-one as you made the original change in #32029 could you take a look at this?

  2. added
    stdlibStandard Library Python modules in the Lib/ directory
    3.11only security fixes
    on Apr 16, 2022
  3. serhiy-storchaka commented on Apr 17, 2022

    @serhiy-storchaka
    Member

    Ah, I see.

    >>> re.match(r'(?s:(?>.*?\.).*)\Z', 'a.txt')
    <re.Match object; span=(0, 5), match='a.txt'>
    >>> re.fullmatch(r'(?s:(?>.*?\.).*)\Z', 'a.txt')
    

    I thought that issues with fullmatch() and atomic grouping were solved, but it is not so.

    Use the following tests in your PR:

    diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py
    index 959582e2f1..a5aa1b358d 100644
    --- a/Lib/test/test_re.py
    +++ b/Lib/test/test_re.py
    @@ -2173,6 +2173,10 @@ def test_fullmatch_possessive_quantifiers(self):
             self.assertIsNone(re.fullmatch(r'a*+', 'ab'))
             self.assertIsNone(re.fullmatch(r'a?+', 'ab'))
             self.assertIsNone(re.fullmatch(r'a{1,3}+', 'ab'))
    +        self.assertTrue(re.fullmatch(r'a++b', 'ab'))
    +        self.assertTrue(re.fullmatch(r'a*+b', 'ab'))
    +        self.assertTrue(re.fullmatch(r'a?+b', 'ab'))
    +        self.assertTrue(re.fullmatch(r'a{1,3}+b', 'ab'))
     
             self.assertTrue(re.fullmatch(r'(?:ab)++', 'ab'))
             self.assertTrue(re.fullmatch(r'(?:ab)*+', 'ab'))
    @@ -2182,6 +2186,10 @@ def test_fullmatch_possessive_quantifiers(self):
             self.assertIsNone(re.fullmatch(r'(?:ab)*+', 'abc'))
             self.assertIsNone(re.fullmatch(r'(?:ab)?+', 'abc'))
             self.assertIsNone(re.fullmatch(r'(?:ab){1,3}+', 'abc'))
    +        self.assertTrue(re.fullmatch(r'(?:ab)++c', 'abc'))
    +        self.assertTrue(re.fullmatch(r'(?:ab)*+c', 'abc'))
    +        self.assertTrue(re.fullmatch(r'(?:ab)?+c', 'abc'))
    +        self.assertTrue(re.fullmatch(r'(?:ab){1,3}+c', 'abc'))
     
         def test_findall_possessive_quantifiers(self):
             self.assertEqual(re.findall(r'a++', 'aab'), ['aa'])
    @@ -2217,6 +2225,10 @@ def test_fullmatch_atomic_grouping(self):
             self.assertIsNone(re.fullmatch(r'(?>a*)', 'ab'))
             self.assertIsNone(re.fullmatch(r'(?>a?)', 'ab'))
             self.assertIsNone(re.fullmatch(r'(?>a{1,3})', 'ab'))
    +        self.assertTrue(re.fullmatch(r'(?>a+)b', 'ab'))
    +        self.assertTrue(re.fullmatch(r'(?>a*)b', 'ab'))
    +        self.assertTrue(re.fullmatch(r'(?>a?)b', 'ab'))
    +        self.assertTrue(re.fullmatch(r'(?>a{1,3})b', 'ab'))
     
             self.assertTrue(re.fullmatch(r'(?>(?:ab)+)', 'ab'))
             self.assertTrue(re.fullmatch(r'(?>(?:ab)*)', 'ab'))
    @@ -2226,6 +2238,10 @@ def test_fullmatch_atomic_grouping(self):
             self.assertIsNone(re.fullmatch(r'(?>(?:ab)*)', 'abc'))
             self.assertIsNone(re.fullmatch(r'(?>(?:ab)?)', 'abc'))
             self.assertIsNone(re.fullmatch(r'(?>(?:ab){1,3})', 'abc'))
    +        self.assertTrue(re.fullmatch(r'(?>(?:ab)+)c', 'abc'))
    +        self.assertTrue(re.fullmatch(r'(?>(?:ab)*)c', 'abc'))
    +        self.assertTrue(re.fullmatch(r'(?>(?:ab)?)c', 'abc'))
    +        self.assertTrue(re.fullmatch(r'(?>(?:ab){1,3})c', 'abc'))
     
         def test_findall_atomic_grouping(self):
             self.assertEqual(re.findall(r'(?>a+)', 'aab'), ['aa'])
  4. serhiy-storchaka commented on Apr 17, 2022

    @serhiy-storchaka
    Member

    Make the same change in POSSESSIVE_REPEAT.

  5. added a commit that references this issue on Apr 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.11only security fixesstdlibStandard Library Python modules in the Lib/ directorytopic-regextype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions