Allow setting timestamp in gzip-compressed tarfiles

@jonashaag

BPO	31526
Nosy	@jonashaag, @vadmium, @randombit, @madebr, @FFY00

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-09-20.05:11:11.655>
labels = ['3.7', 'type-feature', 'library']
title = 'Allow setting timestamp in gzip-compressed tarfiles'
updated_at = <Date 2021-05-04.00:32:57.741>
user = 'https://github.com/randombit'

bugs.python.org fields:

activity = <Date 2021-05-04.00:32:57.741>
actor = 'FFY00'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-09-20.05:11:11.655>
creator = 'randombit'
dependencies = []
files = []
hgrepos = []
issue_num = 31526
keywords = []
message_count = 5.0
messages = ['302590', '305915', '306065', '375849', '375850']
nosy_count = 5.0
nosy_names = ['jonash', 'martin.panter', 'randombit', 'maarten', 'FFY00']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue31526'
versions = ['Python 3.7']

Linked PRs

gh-75707: tarfile: Add optional open() argument "mtime" #138117

Context: I have a script which checks out a software release (tagged git revision) and builds an archive to distribute to end users. One goal of this script is that the archive is reproducible, ie if the script is run twice (at different times, on different machines, by different people) it produces bit-for-bit identical output, and thus also has the same SHA-256 hash.

Mostly this works great, using the TarInfo feature of tarfile.py to set the uid/gid/mtime to fixed values. Except I also want to compress the archive, and tarfile calls time.time() to find out the timestamp that will be embedded in the gzip header. This breaks my carefully deterministic output.

I would like it if tarfile accepted an additional keyword that allowed overriding the time value for the gzip header. As it is I just hack around it with

def null_time():
    return 0
time.time = null_time

which does work but is also horrible.

Alternately, tarfile could just always set the timestamp header to 0 and avoid having its output depend on the current clock. I doubt anyone would notice.

The script in question is here
https://github.com/randombit/botan/blob/master/src/scripts/dist.py

My script uses Python2 for various reasons, but it seems the same problem affects even the tarfile.py in latest Python3. I would be willing to try writing a patch for this, if anything along these lines might be accepted.

Thanks.

This affects me too.

Perhaps you can compress the tar file using the “gzip.GzipFile” class. It accepts a custom “mtime” parameter (see bpo-4272, added in 2.7 and 3.1+):

>>> gztar = BytesIO()
>>> tar = GzipFile(fileobj=gztar, mode="w", mtime=0)
>>> tarfile.open(fileobj=tar, mode="w|").close()
>>> tar.close()
>>> gztar.getvalue().hex()
'1f8b08000000000002ffedc1010d000000c2a0f74f6d0e37a00000000000000000008037039ade1d2700280000'

However, “tarfile.open” accepts a “compresslevel” argument for two of the compressors, so you could argue it is okay to add another argument to pass to the gzip compressor.

I have the same issue.
The timestamp is inserted here:

cpython/Lib/tarfile.py

Lines 419 to 420 in 802726a

    
           timestamp = struct.pack("<L", int(time.time())) 
        
           self.__write(b"\037\213\010\010" + timestamp + b"\002\377")

Because I noticed the timestamp was not included in the timestamp,
I could zero it by doing:

with open(gzipped_tarball,"r+b") as f:
    f.seek(4, 0)
    f.write(b"\x00\x00\x00\x00")

My previous comment should have contained:

Because I noticed the timestamp was not included in the CRC,
...

Also see #120036

(although the simplest current resolution to this reproducibility issue is to use a wrapper compression format other than gzip that doesn't add an extra timestamp in the compression header, such as xz)

pythongh-75707: tarfile: Add optional open() argument "reprodicuble"

pythongh-75707: tarfile: Add optional open() argument "reproducible"

pythongh-75707: tarfile: Add optional open() argument "mtime"

gh-75707: tarfile: Add optional open() argument "mtime" (GH-138117)

Please add a versionchanged directive in the docs.

Navigation Menu

Repository navigation

Allow setting timestamp in gzip-compressed tarfiles #75707

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	timestamp = struct.pack("<L", int(time.time()))
	self.__write(b"\037\213\010\010" + timestamp + b"\002\377")

Allow setting timestamp in gzip-compressed tarfiles #75707

Description

Linked PRs

Activity

randombit commented on Sep 20, 2017

jonashaag commented on Nov 8, 2017

vadmium commented on Nov 10, 2017

madebr commented on Aug 24, 2020

madebr commented on Aug 24, 2020

ncoghlan commented on Aug 13, 2024

serhiy-storchaka commented on May 22, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions