Visitar URL original
feat: Add OnlineStore for Aerospike by vkagamlyk · Pull Request #6532 · feast-dev/feast · GitHub
Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
d82dcf2
feat: scaffold Aerospike online store
vkagamlyk Apr 20, 2026
2fa7ba0
feat: implement Aerospike online_write_batch
vkagamlyk Apr 20, 2026
4173e0a
feat: implement Aerospike online_read
vkagamlyk Apr 20, 2026
534b935
feat: implement Aerospike update and teardown
vkagamlyk Apr 20, 2026
fb4dff7
test: add Aerospike unit and integration tests
vkagamlyk Apr 20, 2026
53d5972
feat: add async online_read/write and lifecycle hooks for Aerospike
vkagamlyk Apr 20, 2026
17b1dcb
docs: add Aerospike online store reference and tuning guide
vkagamlyk Apr 20, 2026
f552a94
fix: use bytearray keys and zip-based batch mapping for Aerospike reads
vkagamlyk Apr 20, 2026
ebb3d60
docs: clarify Aerospike auth and TLS sections are Enterprise-only
vkagamlyk Apr 20, 2026
d86dcbf
feat: add aerospike to feast-operator supported online stores
vkagamlyk Apr 20, 2026
5d9e38f
fix(aerospike): project requested_features server-side and surface pe…
vkagamlyk Apr 22, 2026
5553367
feat(aerospike)!: rename total_timeout_ms -> batch_total_timeout_ms a…
vkagamlyk Apr 22, 2026
953b9d1
refactor(aerospike): use MAP_KEY_ORDERED, KEY_DIGEST, and instance-sc…
vkagamlyk Apr 22, 2026
0c8c184
feat(aerospike): add per-FV namespace/set overrides and prewriting hook
vkagamlyk Apr 30, 2026
6628607
test: update aerospike image tag
vkagamlyk Jun 16, 2026
9e76c2f
chore: sync README template and secrets baseline after master merge
vkagamlyk Jun 16, 2026
ea9da18
chore: fix secrets baseline line number for v1 operator types
vkagamlyk Jun 16, 2026
2c37c0f
docs: update aerospike docs
vkagamlyk Jun 16, 2026
e70ba39
Merge branch 'master' into feat/aerospike-online-store
vkagamlyk Jun 22, 2026
1d016f6
fix(aerospike): wire batch max_retries and fix empty projection handling
vkagamlyk Jun 22, 2026
666cb60
style(aerospike): format online_read docs assignment for ruff
vkagamlyk Jun 22, 2026
7cd0bd3
Merge branch 'master' into feat/aerospike-online-store
vkagamlyk Jun 25, 2026
dbc45fc
Merge branch 'master' into feat/aerospike-online-store
vkagamlyk Jun 26, 2026
8dd84e3
Merge branch 'master' into feat/aerospike-online-store
vkagamlyk Jun 26, 2026
4d5954d
Merge branch 'master' into feat/aerospike-online-store
vkagamlyk Jun 28, 2026
07738ab
Merge branch 'master' into feat/aerospike-online-store
vkagamlyk Jun 29, 2026
b4ccaec
Merge branch 'master' into feat/aerospike-online-store
vkagamlyk Jun 29, 2026
10a1fb3
Merge branch 'master' into feat/aerospike-online-store
vkagamlyk Jun 29, 2026
b2b98bc
chore: update pixi.lock for aerospike optional extra
vkagamlyk Jun 30, 2026
256d65d
fix(aerospike): add client init lock and batch chunking
vkagamlyk Jun 30, 2026
3766212
Merge branch 'master' into feat/aerospike-online-store
vkagamlyk Jun 30, 2026
d3ef432
Merge branch 'master' into feat/aerospike-online-store
vkagamlyk Jul 1, 2026
33f5d37
Merge branch 'master' into feat/aerospike-online-store
vkagamlyk Jul 1, 2026
d3dc67e
Merge branch 'master' into feat/aerospike-online-store
vkagamlyk Jul 2, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
docs: add Aerospike online store reference and tuning guide
Signed-off-by: Valentyn Kahamlyk <valentin.kagamlyk@gmail.com>
  • Loading branch information
vkagamlyk committed Jun 16, 2026
commit 17b1dcbe5ea4a6dcad4144dc331f77a831715faa
1 change: 1 addition & 0 deletions docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,7 @@
* [SingleStore](reference/online-stores/singlestore.md)
* [Milvus](reference/online-stores/milvus.md)
* [MongoDB](reference/online-stores/mongodb.md)
* [Aerospike](reference/online-stores/aerospike.md)
* [Elasticsearch](reference/online-stores/elasticsearch.md)
* [Qdrant](reference/online-stores/qdrant.md)
* [Faiss](reference/online-stores/faiss.md)
Expand Down
31 changes: 31 additions & 0 deletions docs/how-to-guides/online-server-performance-tuning.md
Original file line number Diff line number Diff line change
Expand Up @@ -278,6 +278,7 @@ The online store is the single largest factor in `get_online_features()` latency
| **DynamoDB** | 2–5 ms | Yes | Serverless, auto-scaling on AWS | Pay-per-request cost; batch API limits (100 items) |
| **PostgreSQL** | 3–10 ms | No (threadpool) | Teams with existing Postgres infra | Connection pooling needed at scale |
| **MongoDB** | 2–5 ms | Yes | Flexible schema, async-native | Requires index tuning for large datasets |
| **Aerospike** | < 1 ms | No (threadpool) | Ultra-low latency, hybrid memory (RAM + SSD), large datasets | Namespace must be pre-configured on the cluster |
| **Bigtable** | 3–8 ms | No (threadpool) | Large-scale GCP workloads | Row-key design affects read performance |
| **Cassandra / ScyllaDB** | 2–5 ms | No (threadpool) | Multi-region, write-heavy | Tunable consistency; requires DC-aware routing |
| **Remote** | Varies | No (threadpool) | Centralized feature server architecture | Adds an HTTP hop; tune connection pool |
Expand Down Expand Up @@ -305,6 +306,7 @@ The feature server can read from the online store using either an **async** or *
| **MongoDB** | Yes | Yes | Uses `motor` (async MongoDB driver) |
| **PostgreSQL** | Implemented | No | Has `online_read_async` but does not yet advertise via `async_supported`; uses sync/threadpool path |
| **Redis** | Implemented | **Yes** | `online_read_async` and `online_write_batch_async` both implemented; uses sync/threadpool path for `get_online_features` (overridden with batched single pipeline) |
| **Aerospike** | Implemented | No | Async methods wrap the blocking C client via `run_in_executor`; does not yet advertise via `async_supported`, so the server still uses the threadpool path |
| All others | No | No | Fall back to sync with `run_in_threadpool()` |

**When async matters most:**
Expand Down Expand Up @@ -467,6 +469,34 @@ online_store:
- **`connectTimeoutMS` / `socketTimeoutMS`**: Tighter timeouts improve p99 by failing fast on slow connections.
- MongoDB is one of the stores with **full async support** (read and write), so it benefits from concurrent feature view reads via `asyncio.gather()`.

### Aerospike tuning

Aerospike offers sub-millisecond reads thanks to its hybrid-memory architecture (primary index in RAM, data on SSD or RAM). Tune the per-call policies in the Feast config and rely on the Aerospike cluster's own tuning for everything else:

```yaml
online_store:
type: aerospike
hosts:
- ["aerospike-1.internal", 3000]
- ["aerospike-2.internal", 3000]
namespace: feast
read_timeout_ms: 50
write_timeout_ms: 200
total_timeout_ms: 500
max_retries: 2
ttl_seconds: 86400 # record-level TTL; omit to use the namespace default
client_kwargs: # escape hatch for any client-config field not surfaced above
policies:
batch:
concurrent_nodes: 0 # 0 = parallel to every node (lowest latency on multi-node clusters)
```

- **`read_timeout_ms` / `write_timeout_ms` / `total_timeout_ms`**: Per-call deadlines (in milliseconds). Tighter deadlines improve p99 by failing fast on misbehaving nodes; pair with `max_retries`.
- **`hosts`**: List every seed node. The Aerospike client discovers the rest of the cluster automatically and opens one connection pool per node.
- **`ttl_seconds: 0`** means "never expire"; omit the key to inherit the namespace's `default-ttl`. Expiry is enforced by the server's `nsup` thread — nothing to delete on the client side.
- Co-locate the feature server in the **same availability zone / rack** as the Aerospike cluster; sub-millisecond reads are bandwidth- and RTT-sensitive.
- On the Aerospike side: increase `proto-fd-max` if the feature server reports "max fd" errors, and turn on `partition-tree-sprigs` for namespaces with billions of records to spread index contention.

### Remote online store tuning

The Remote online store connects to a Feast feature server over HTTP. Connection pooling is critical:
Expand Down Expand Up @@ -700,6 +730,7 @@ This applies to every connection-oriented online store:
| **DynamoDB** | `max_pool_connections` (HTTP pool) | 10 | No hard limit, but AWS SDK has per-process pool caps; monitor throttling |
| **Redis** | Connection per worker | 1 | `maxclients` on the Redis server (default: 10,000) |
| **MongoDB** | `maxPoolSize` (in `client_kwargs`) | 100 | Server's `net.maxIncomingConnections` |
| **Aerospike** | Driver manages pool per seed node | Auto | `proto-fd-max` (default 15000) on each Aerospike node |
| **Cassandra** | Driver manages pool per node | Auto | `native_transport_max_threads` on each Cassandra node |
| **Remote** | `connection_pool_size` (HTTP pool) | 50 | The target feature server's worker capacity |

Expand Down
4 changes: 4 additions & 0 deletions docs/reference/online-stores/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,10 @@ Please see [Online Store](../../getting-started/components/online-store.md) for
[mongodb.md](mongodb.md)
{% endcontent-ref %}

{% content-ref url="aerospike.md" %}
[aerospike.md](aerospike.md)
{% endcontent-ref %}

{% content-ref url="hazelcast.md" %}
[hazelcast.md](hazelcast.md)
{% endcontent-ref %}
Expand Down
198 changes: 198 additions & 0 deletions docs/reference/online-stores/aerospike.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,198 @@
# Aerospike online store (Preview)

## Description

The [Aerospike](https://aerospike.com/) online store provides support for materializing feature values into an Aerospike cluster for serving online features.

{% hint style="warning" %}
The Aerospike online store is currently in **preview**. Some functionality may be unstable, and breaking changes may occur in future releases.
{% endhint %}

## Features

* Supports both synchronous and asynchronous read/write paths (`online_read` / `online_read_async`, `online_write_batch` / `online_write_batch_async`). Async methods wrap the blocking client in `run_in_executor`, keeping the event loop responsive in feature-server workloads.
* Partial, server-side upserts via Aerospike Map CDT operations — writing one feature view never clobbers another feature view stored on the same entity.
* Record-level TTL controlled by a single `ttl_seconds` config option (honours the namespace default, a "never expire" sentinel, or an explicit number of seconds).
* Authentication and TLS options for Aerospike Enterprise Edition passed straight through to the Aerospike Python client.
* `client_kwargs` escape hatch for any advanced client-config field not surfaced on `AerospikeOnlineStoreConfig`.
* Baseline: Aerospike Server **≥ 6.0** (uses batch-write / batch-operate APIs). The store has been developed against CE 8.x.

## Getting started

Install the Aerospike extra (alongside the dependency for the offline store of choice):

```bash
pip install 'feast[aerospike]'
```

You can start from any of the standard templates (e.g. `feast init -t local` or `feast init -t aws`) and then swap in Aerospike as the online store as shown below.

## Examples

### Basic configuration — local Aerospike CE

{% code title="feature_store.yaml" %}
```yaml
project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
type: aerospike
hosts:
- ["127.0.0.1", 3000]
namespace: feast
```
{% endcode %}

### Multi-node cluster

{% code title="feature_store.yaml" %}
```yaml
project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
type: aerospike
hosts:
- ["aerospike-1.internal", 3000]
- ["aerospike-2.internal", 3000]
- ["aerospike-3.internal", 3000]
namespace: feast
ttl_seconds: 86400 # 24h record-level TTL
read_timeout_ms: 50
write_timeout_ms: 200
total_timeout_ms: 500
max_retries: 2
```
{% endcode %}

### Aerospike Enterprise with authentication

{% code title="feature_store.yaml" %}
```yaml
project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
type: aerospike
hosts:
- ["aerospike.internal", 3000]
namespace: feast
user: feast_user
password: ${AEROSPIKE_PASSWORD} # pragma: allowlist secret
auth_mode: internal # internal | external | pki
```
{% endcode %}

### TLS-enabled cluster

{% code title="feature_store.yaml" %}
```yaml
project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
type: aerospike
hosts:
- ["aerospike-1.internal", 4333, "aerospike-tls"]
namespace: feast
tls:
enable: true
cafile: /etc/aerospike/certs/ca.pem
certfile: /etc/aerospike/certs/client.pem
keyfile: /etc/aerospike/certs/client.key
```
{% endcode %}

The full set of configuration options is available in [`AerospikeOnlineStoreConfig`](https://rtd.feast.dev/en/latest/#feast.infra.online_stores.aerospike_online_store.aerospike.AerospikeOnlineStoreConfig).

## Data Model

The Aerospike online store uses a **single set per project** with entity-key collocation. Features from multiple feature views for the same entity are stored together on a single Aerospike record, analogous to the MongoDB online store's "one document per entity" layout.

| Aerospike concept | Feast mapping |
| :---------------- | :---------------------------------------------------------------------------- |
| Namespace | `online_store.namespace` (must be pre-configured on the cluster) |
| Set | `online_store.set_name_template` → `"{project}_{collection_suffix}"` by default |
| Key | `serialize_entity_key(entity_key)` (raw bytes) |
| Bin `features` | Map CDT keyed by feature-view name, each value a map of `feature → native` |
| Bin `event_ts` | Map CDT keyed by feature-view name, each value an int64 epoch-ms timestamp |
| Bin `created_ts` | Top-level int64 epoch-ms timestamp (last `feast materialize`) |

### Example record

For a single entity carrying features from two feature views (`driver_stats` and `pricing`):

```text
key: (ns="feast", set="my_feature_repo_latest", user_key=<serialized_entity_key bytes>)
bins:
features:
driver_stats:
rating: 4.91
trips_last_7d: 132
pricing:
surge_multiplier: 1.2
event_ts:
driver_stats: 1737374400000 # 2025-01-20T12:00:00Z
pricing: 1737447000000 # 2025-01-21T08:30:00Z
created_ts: 1737460805000 # 2025-01-21T12:00:05Z
```

### Key design decisions

* **Record per entity, bin per concept.** `features` and `event_ts` are Aerospike Map CDT bins, not dynamic bins, which keeps the store within the 15-byte Aerospike bin-name limit regardless of how many feature views a project has.
* **Partial upserts via Map CDT ops.** Writes use `batch_write` with `map_put_items("features", {<fv>: {...}})` and `map_put("event_ts", <fv>, <epoch_ms>)`. Concurrent writes to different feature views on the same entity never clobber each other — each write mutates only its own map keys.
* **Entity-key bytes as the Aerospike user key.** Feast's `serialize_entity_key` output is used directly as the user key, so lookups are O(1) and no secondary index is needed.
* **Timestamps as int64 epoch milliseconds.** Aerospike has no native datetime type; tz-naive timestamps are treated as UTC per the `OnlineStore` contract.

### TTL and expiry

`ttl_seconds` is written as record-level metadata on every `online_write_batch` call:

| `ttl_seconds` | Aerospike TTL | Effect |
| :------------ | :-------------------------------- | :---------------------------------------------------------- |
| not set / `null` | `TTL_NAMESPACE_DEFAULT` | Record inherits the namespace's configured `default-ttl`. |
| `0` | `TTL_NEVER_EXPIRE` | Record is kept until explicitly deleted. |
| `>0` | that many seconds | Record is evicted by the server's `nsup` thread. |

There is no per-feature-view TTL override in this version — the setting is applied uniformly for every write made by the online store.

### Indexes

No secondary indexes are created. All access goes through the primary key, which is the serialized entity key.

## Async support

Async read/write are provided by running the Aerospike Python client's blocking calls on the default thread-pool executor (`loop.run_in_executor`). The underlying C client releases the GIL during network I/O, so `await store.online_read_async(...)` keeps the event loop responsive. A native asyncio Aerospike client is not currently used.

Both sync and async methods are fully supported:

* `online_read` / `online_read_async`
* `online_write_batch` / `online_write_batch_async`
* `initialize` / `close` — `initialize(config)` eagerly opens the connection so feature servers pay the TCP/handshake cost at startup; `close()` releases the cached client.

## Functionality Matrix

The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
Below is a matrix indicating which functionality is supported by the Aerospike online store.

| | Aerospike |
| :-------------------------------------------------------- | :-------- |
| write feature values to the online store | yes |
| read feature values from the online store | yes |
| update infrastructure (e.g. tables) in the online store | yes |
| teardown infrastructure (e.g. tables) in the online store | yes |
| generate a plan of infrastructure changes | no |
| support for on-demand transforms | yes |
| readable by Python SDK | yes |
| readable by Java | no |
| readable by Go | no |
| support for entityless feature views | yes |
| support for concurrent writing to the same key | yes |
| support for ttl (time to live) at retrieval | yes |
| support for deleting expired data | yes |
| collocated by feature view | no |
| collocated by feature service | no |
| collocated by entity key | yes |

To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).
1 change: 1 addition & 0 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ The list below contains the functionality that contributors are planning to deve
* [x] [SingleStore](https://docs.feast.dev/reference/online-stores/singlestore)
* [x] [Couchbase](https://docs.feast.dev/reference/online-stores/couchbase)
* [x] [MongoDB](https://docs.feast.dev/reference/online-stores/mongodb)
* [x] [Aerospike](https://docs.feast.dev/reference/online-stores/aerospike)
* [x] [Qdrant (vector store)](https://docs.feast.dev/reference/online-stores/qdrant)
* [x] [Milvus (vector store)](https://docs.feast.dev/reference/online-stores/milvus)
* [x] [Faiss (vector store)](https://docs.feast.dev/reference/online-stores/faiss)
Expand Down