Input and Output¶

Vortex arrays support reading and writing to local and remote file systems, including plain-old HTTP, S3, Google Cloud Storage, and Azure Blob Storage.

`open`	Lazily open a Vortex file located at the given path or URL.
`open_readable`	Lazily open a Vortex file through a Python object that performs the IO itself.
`ReadAt`	A positional byte source that Vortex reads from, for storage only reachable from Python.
`ReadBytesAt`	A positional byte source that returns its own buffers, for storage only reachable from Python.
`SegmentCache`	A segment cache that many Vortex files share, capped by total bytes.
`VortexFile`
`Footer`	The parsed footer of a Vortex file: its layout, segment map and dtype.
`RepeatedScan`	A prepared scan that is optimized for repeated execution.
`read_url`	Read a vortex struct array from a URL.
`write`	Write an array to a Vortex file.

Lazily open a Vortex file located at the given path or URL.

Parameters:

path (str) – A local path or URL to the Vortex file.
store – An object store created from the vortex.store package. By default the store is inferred based on the path
footer (vortex.file.Footer | None) – The VortexFile.footer of an earlier open of the same file. Opening then does not read the footer. Vortex cannot check that it belongs to this file, so the file must be the same one and must not have changed since.
without_segment_cache (bool) – If true, disable the segment cache for this file, useful when memory is constrained.
segment_cache (vortex.SegmentCache | None) – A cache shared with other files, used in place of this file’s own segment cache. Requires cache_key.
cache_key (str | None) – Identifies this file’s contents within segment_cache. Files opened with the same key share cached segments, so the key must change whenever the file does.

Examples

Open a Vortex file and perform a scan operation:

>>> import vortex as vx
>>> vxf = vx.open("data.vortex")
>>> array_iterator = vxf.scan()

Lazily open a Vortex file through a Python object that performs the IO itself.

Use this for storage only reachable from Python. Storage with a native object store should be opened with vortex.open() instead, which does its IO without taking the GIL.

Parameters:

reader (vortex.io.ReadBytesAt | vortex.io.ReadAt | binary file object) – An object implementing vortex.io.ReadBytesAt or vortex.io.ReadAt, or a binary file object with seek and readinto (or read), such as open(path, "rb"), io.BytesIO or an fsspec file. Vortex does not close it; keep it open for as long as the returned file, or anything scanned from it, is in use.
footer (vortex.file.Footer | None) – The VortexFile.footer of an earlier open of the same file. Opening then does no IO, which saves the footer read when a file is opened again and again. Vortex checks only that the footer fits within the size of reader, so the file must not have changed since.
concurrency (int | None) – The most reads to have in flight at once through a vortex.io.ReadBytesAt or vortex.io.ReadAt, 192 by default. Not accepted for a file object, whose reads are serialized because each one has to seek first.
without_segment_cache (bool) – If true, disable the segment cache for this file, useful when memory is constrained.
segment_cache (vortex.SegmentCache | None) – A cache shared with other files, used in place of this file’s own segment cache. Requires cache_key.
cache_key (str | None) – Identifies this file’s contents within segment_cache. Files opened with the same key share cached segments, so the key must change whenever the file does.

Examples

Open a Vortex file through an fsspec file object:

>>> import fsspec
>>> import vortex as vx
>>> with fsspec.open("memory://data.vortex", "rb") as f:
...     table = vx.open_readable(f).to_arrow().read_all()

Open the same file again without reading its footer:

>>> with fsspec.open("memory://data.vortex", "rb") as f:
...     footer = vx.open_readable(f).footer
...     vxf = vx.open_readable(f, footer=footer)

class vortex.io.ReadAt(*args, **kwargs)¶

A positional byte source that Vortex reads from, for storage only reachable from Python.

Pass one to vortex.open_readable(). Binary file objects (open(path, "rb"), io.BytesIO, fsspec and pyarrow files) are accepted too without implementing this protocol, but their reads are serialized because each one has to seek first. Implement ReadAt when the underlying storage supports positional reads, such as os.pread() or HTTP range requests, so that Vortex can issue many reads at once: up to 192 by default, set with the concurrency argument of vortex.open_readable().

Reads are called from Vortex worker threads, potentially many concurrently, so implementations must be safe for concurrent use. Releasing the GIL while waiting on IO lets those reads overlap.

Examples

A reader over an open file descriptor:

>>> import os
>>> class PReadFile:
...     def __init__(self, path):
...         self._fd = os.open(path, os.O_RDONLY)
...
...     def size(self) -> int:
...         return os.fstat(self._fd).st_size
...
...     def read_into(self, offset: int, buffer: memoryview) -> int:
...         return os.preadv(self._fd, [buffer], offset)
>>> vxf = vx.open_readable(PReadFile("data.vortex"))

read_into(offset: int, buffer: memoryview) → int¶

Read bytes starting at absolute offset into the writable buffer.

Returns the number of bytes written. A short read is retried for the remainder, and returning 0 before buffer is full is an error. buffer is valid only for the duration of the call: do not keep it, or any view derived from it, after returning.

size() → int¶: Total length of the source in bytes. Called once, when the file is opened.

class vortex.io.ReadBytesAt(*args, **kwargs)¶

A positional byte source that returns its own buffers, for storage only reachable from Python.

Pass one to vortex.open_readable(). Use it instead of ReadAt when the storage already produces a buffer, such as bytes from os.pread() or the result of an obstore range request. Vortex keeps that buffer without copying it when it is read-only, C-contiguous, the full requested length, and aligned as Vortex needs. It copies any other buffer once. An object with both read_at and read_into is read through read_at.

Reads are called from Vortex worker threads, potentially many concurrently, so implementations must be safe for concurrent use. Up to 192 reads run at once by default, set with the concurrency argument of vortex.open_readable().

Examples

A reader over an open file descriptor:

>>> import os
>>> class PReadBytes:
...     def __init__(self, path):
...         self._fd = os.open(path, os.O_RDONLY)
...
...     def size(self) -> int:
...         return os.fstat(self._fd).st_size
...
...     def read_at(self, offset: int, length: int) -> bytes:
...         return os.pread(self._fd, length, offset)
>>> vxf = vx.open_readable(PReadBytes("data.vortex"))

read_at(offset: int, length: int) → Buffer¶

Return up to length bytes starting at absolute offset, as any buffer-protocol object.

A shorter result is retried for the remainder, and an empty result before length bytes is an error. Vortex may keep the returned object for as long as it uses the bytes. Do not change its contents after returning it, even through a read-only view.

size() → int¶: Total length of the source in bytes. Called once, when the file is opened.

class vortex.SegmentCache(max_bytes)¶

A segment cache that many Vortex files share, capped by total bytes.

Pass it to vortex.open() or vortex.open_readable() with a cache_key. Files opened with the same key share cached segments, so a file opened again, for example once per batch, does not read them again. The key must identify the file’s contents, not only its location: a file that has changed must get a new key.

It is safe to share between threads, but it is not picklable: create one in each process.

Parameters:: max_bytes (int) – The most bytes of segments to hold. The least recently used segments are evicted first.

clear()¶: Remove every cached segment.

entry_count¶

The number of cached segments.

Return type:: int

size_bytes¶

The total bytes of the cached segments.

Return type:: int

final class vortex.VortexFile(file: VortexFile)¶

property dtype: DType¶: The dtype of the file.

property footer: Footer¶: The parsed footer, to pass to vortex.open() or vortex.open_readable() to open this file again.

property path: str¶: The path or URL this file was opened from.

Scan the Vortex file returning a vortex.ArrayIterator.

Parameters:

projection (vortex.Expr | list[str] | None) – The projection expression to read, or else read all columns.
expr (vortex.Expr | None) – The predicate used to filter rows. The filter columns do not need to be in the projection.
limit (int | None) – The maximum number of rows to read after filtering. If None, read all rows.
indices (vortex.Array | None) – The indices of the rows to read. Must be sorted and non-null.
batch_size (int | None) – The number of rows to read per chunk.

Examples

Scan a file with a structured column and nulls at multiple levels and in multiple columns.

>>> import vortex as vx
>>> import vortex.expr as ve
>>> a = vx.array([
...     {'name': 'Joseph', 'age': 25},
...     {'name': None, 'age': 31},
...     {'name': 'Angela', 'age': None},
...     {'name': 'Mikhail', 'age': 57},
...     {'name': None, 'age': None},
... ])
>>> vx.io.write(a, "a.vortex")
>>> vxf = vx.open("a.vortex")
>>> vxf.scan().read_all().to_arrow_array()
<pyarrow.lib.StructArray object at ...>
-- is_valid: all not null
-- child 0 type: string_view
  [
    "Joseph",
    null,
    "Angela",
    "Mikhail",
    null
  ]
-- child 1 type: int64
  [
    25,
    31,
    null,
    57,
    null
  ]

Read just the age column:

>>> vxf.scan(['age']).read_all().to_arrow_array()
<pyarrow.lib.StructArray object at ...>
-- is_valid: all not null
-- child 0 type: int64
  [
    25,
    31,
    null,
    57,
    null
  ]

Keep rows with an age above 35. This will read O(N_KEPT) rows, when the file format allows.

>>> vxf.scan(expr=ve.column("age") > 35).read_all().to_arrow_array()
<pyarrow.lib.StructArray object at ...>
-- is_valid: all not null
-- child 0 type: string_view
  [
    "Mikhail"
  ]
-- child 1 type: int64
  [
    57
  ]

Scan the Vortex file as a pyarrow.RecordBatchReader.

Parameters:

projection (vortex.Expr | list[str] | None) – Either an expression over the columns of the file (only referenced columns will be read from the file) or an explicit list of desired columns.
expr (vortex.Expr | None) – The predicate used to filter rows. The filter columns need not appear in the projection.
indices (vortex.Array | None) – The indices of the rows to read. Must be strictly increasing and non-null.
batch_size (int | None) – The number of rows to read per chunk.
schema (pyarrow.Schema | None) – The Arrow schema to return. Use pyarrow.string() for StringArray fields. Use pyarrow.binary() for BinaryArray fields.

to_dataset() → VortexDataset¶: Scan the Vortex file using the pyarrow.dataset.Dataset API.

to_polars() → LazyFrame¶: Read the Vortex file as a pl.LazyFrame, supporting column pruning and predicate pushdown.

Prepare a scan of the Vortex file for repeated reads, returning a vortex.RepeatedScan.

Parameters:

projection (vortex.Expr | list[str] | None) – The projection expression to read, or else read all columns.
expr (vortex.Expr | None) – The predicate used to filter rows. The filter columns do not need to be in the projection.
indices (vortex.Array | None) – The indices of the rows to read. Must be sorted and non-null.
batch_size (int | None) – The number of rows to read per chunk.

class vortex.file.Footer¶

The parsed footer of a Vortex file: its layout, segment map and dtype.

Pass it to vortex.open() or vortex.open_readable() to open the same file again without reading the footer. It holds no IO state, but it is not picklable.

row_count¶

The number of rows in the file.

Return type:: int

final class vortex.RepeatedScan(scan: RepeatedScan)¶

A prepared scan that is optimized for repeated execution.

execute(*, row_range: tuple[int, int] | None = None) → ArrayIterator¶

Execute the scan returning a vortex.ArrayIterator.

Parameters:: row_range (tuple[int, int] | None) – Tuple is interpreted as [start, stop).

Examples

Scan a file with a structured column and nulls at multiple levels and in multiple columns.

>>> import vortex as vx
>>> import vortex.expr as ve
>>> a = vx.array([
...     {'name': 'Joseph', 'age': 25},
...     {'name': None, 'age': 31},
...     {'name': 'Angela', 'age': None},
...     {'name': 'Mikhail', 'age': 57},
...     {'name': None, 'age': None},
... ])
>>> vx.io.write(a, "a.vortex")
>>> scan = vx.open("a.vortex").to_repeated_scan()
>>> scan.execute(row_range=(1, 3)).read_all().to_arrow_array()
<pyarrow.lib.StructArray object at ...>
-- is_valid: all not null
-- child 0 type: string_view
  [
    null,
    "Angela"
  ]
-- child 1 type: int64
  [
    31,
    null
  ]

scalar_at(index: int) → Scalar¶

Fetch a scalar from the scan returning a vortex.Scalar.

Parameters:: index (int) – The row index to fetch. Raises an IndexError if out of bounds or if the given row index was not included in the scan.

Examples

Scan a file with a structured column and nulls at multiple levels and in multiple columns.

>>> import vortex as vx
>>> import vortex.expr as ve
>>> a = vx.array([
...     {'name': 'Joseph', 'age': 25},
...     {'name': None, 'age': 31},
...     {'name': 'Angela', 'age': None},
...     {'name': 'Mikhail', 'age': 57},
...     {'name': None, 'age': None},
... ])
>>> vx.io.write(a, "a.vortex")
>>> scan = vx.open("a.vortex").to_repeated_scan()
>>> scan.scalar_at(1)
<vortex.StructScalar object at ...>

class vortex.io.VortexWriteOptions¶

Write Vortex files with custom configuration.