Repository navigation
[Python][Parquet] Errno 14 Error reading bytes from file with pyarrow.parquet.ParquetFile(), but consistently succeeds on second try #51674
Description
Activity
I really don't understand the minimal case to reproduce this. The example contains three attempts to open a parquet file. One says working, one is on a while loop and is not clear at all what is the problem. Please, could you try to share the minimal reproducer? Is it opening the parquet file on a while loop?
Is it after doing something specific?
Is it for a specific Parquet file or is reproducible with any Parquet file?
Was PyArrow installed from PyPI, from conda, was built from source?What would be helpful is a snippet with something like this, btw I can't reproduce with the following:
import pyarrow as pa import pyarrow.parquet as pq table = pa.table({'a': range(10), 'b': [10] * 10, 'c': [1, 2] * 5}) pq.write_table(table, 'output.parquet') i = 0 while i < 10: parquet_file = pq.ParquetFile('output.parquet') i += 1
Please try to use plain languange, AI tends to be overly verbose and it's difficult to grasp the real issue. Thanks!
Apologies. It's not AI, I just have a tendency to write too verbosely and haven't submitted many bug reports... please definitely let me know if what I wrote below is not clear.
See minimal case below. This is analogous to the code I used to create the problematic parquet file in the first place. This script actually fails earlier, on
scanner.to_batches(), with the same error message... But the created parquet file can be read withParquetFile()just fine:import pyarrow as pa import pyarrow.parquet as pq import pyarrow.dataset as ds table = pa.table({'a': range(500000), 'b': [10] * 500000, 'c': [1, 2] * 250000}) pq.write_table(table, 'example.parquet') dataset = ds.dataset( "example.parquet", format="parquet" ) scanner = dataset.scanner() writer = None try: for batch in scanner.to_batches(): if writer is None: writer = pq.ParquetWriter( "example2.parquet", batch.schema ) writer.write_batch(batch) finally: if writer is not None: writer.close()The script above consistently fails on
for batch in scanner.to_batches():with the same errno 14 error message.However, it consistently does not fail if I pre-emptively call
print(scanner.to_batches())prior to the while loop. It also does not fail if the parquet is much smaller (table = pa.table({'a': range(10), 'b': [10] * 10, 'c': [1, 2] * 5})works).Core issue: errno 14 seems to occur whenever I interact with any sufficiently large parquet file on this system, not necessarily via
pq.ParquetFile().The issue can be bypassed by invoking some type of call to the file... In the above snippet, pre-emptively calling
print(scanner.to_batches())prior to the for loop solves the issue. In my production code, simply callingpq.ParqeutFile()twice bypasses the issue, as the second pq.ParquetFile() command no longer returns an error.This seems to work with any sufficiently large parquet file, but smaller parquets don't have this issue.
PyArrow was installed via pip onto a
venvby first runningpython -m pip install --upgrade pip setuptools wheel, and then runningpython -m pip install -r requirements.txt.requirements.txtcontains at least the following:numpy pandas pyarrow dask[array,dataframe,distributed] zarr datasetsThanks for the response!
I've tried to reproduce you example locally on Debian Forky with Python 3.14.7 and PyArrow 25.0.1 and I have not been able to reproduce. If someone can try to set up a Python 3.13 and Rocky Docker environment and can validate whether the example above is reproducible it would be of great help.
From my 🤖 :
On parallel filesystems like Lustre, GPFS or BeeGFS, large reads can bypass the usual kernel copy and pin the destination memory directly. If that memory was just allocated and the program hasn't written to it yet, the pinning can fail with "Bad address" (Errno 14). A normal local filesystem would handle that memory without complaint, and that would explain why only large files fail and why a retry works once the memory has been touched. To test this, copy the same file to a local disk such as /tmp or /dev/shm and run the failing script against that copy in a fresh process: if it never fails there but fails every time on /globalscratch (with df -T naming the filesystem type), the filesystem client is to blame.
So could you please try testing on
/globalscratchand say/tmp?Thanks a ton! From what I can tell, it is most likely indeed a filesystem issue. Specifically, I ran the same snippet I replied above on a few locations:
- Run on HPC
/globalscratch/which isbeegfs: fails, produces behavior as I wrote in the bug report. - Run on HPC
/tmp/which isxfs: no issue - Run on HPC home dir
/home/<institutional_affiliations>/which isnfs4: no issue - Run on my own computer which is
btrfs: no issue
- Run on HPC
This is likely due to Beegfs allocating memory differently than pyarrow.
Another suggestion from my 🤖 :
Could you run your failing snippet again on /globalscratch, this time with Arrow's system memory allocator:
ARROW_DEFAULT_MEMORY_POOL=system python your_script.py
To confirm the setting took effect, add print(pyarrow.default_memory_pool().backend_name) at the top of the script. It should print system. Does it still fail with Errno 14?
Thanks. I tried with both
ARROW_DEFAULT_MEMORY_POOL=systemandARROW_DEFAULT_MEMORY_POOL=jemalloc, both produce the exact same Errno 14. The default memory pool on this system ismimalloc.I've previously tried toggling
pq.ParquetFile(memory_map=False)(on the non-snippet code with a parquet that was built, but cannot be read without the workaround), which didn't seem to work either.What happens if
memory_map=True?
Perhaps try reporting to https://github.com/ThinkParQ/beegfs/issues to see if they have an idea what could be going wrong?
Describe the bug, including details regarding any error messages, version, and platform.
The following bug happens specifically on a Python
venvI created on an HPC. Python version is 3.13.5 with pyarrow version 25.0.1. This is a bioinformatics environment aimed to use single-cell packages (such as scanpy/anndata), so possibly other contributing packages also includenumpy==2.5.3,pandas==3.0.6,anndata==0.13.4.The following error reproducibly occurs whenever I run
pyarrow.parquet.ParquetFile(filename). However, this error no longer occurs if I 1) runpyarrow.parquet.ParquetFile(filename)directly from the interpreter, or 2) if I specifically do a try/except loop and re-runParquetFile()on the except (but it doesn't work with a while loop).Error trace:
See the following script for reproducing this issue. This is guaranteed to fail on my setup; if I comment out
pq_example(), the other two functions behave exactly as described.The exact OS information:
Component(s)
Python