Visitar URL original
Add comprehensive integration tests for dbt import feature · feast-dev/feast@3b5a3f8 · GitHub
Skip to content

Commit 3b5a3f8

Browse files
YassinNouh21YassinNouh21
committed
Add comprehensive integration tests for dbt import feature
- Add 82 integration tests covering dbt manifest parsing, mapper, codegen, and CLI - Add real dbt project with DuckDB for testing - Add GitHub Actions workflow for dbt integration tests - Fix entity column handling: codegen now includes entity columns in schema to match mapper behavior (FeatureView.__init__ expects them) - Fix type mapping: Int32→ValueType.INT32, Float32→ValueType.FLOAT (instead of widening to INT64/DOUBLE) - Add multi-entity column support tests - Add comprehensive test coverage for edge cases, validation errors, special characters, and CLI commands Signed-off-by: yassinnouh21 <yassinnouh21@gmail.com>
1 parent 19f9bb8 commit 3b5a3f8

18 files changed

Lines changed: 1637 additions & 3 deletions

‎.github/workflows/dbt-integration-tests.yml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
name: dbt-integration-tests
2+
3+
# Run dbt integration tests on PRs
4+
on:
5+
pull_request:
6+
paths:
7+
- 'sdk/python/feast/dbt/**'
8+
- 'sdk/python/tests/integration/dbt/**'
9+
- 'sdk/python/tests/unit/dbt/**'
10+
- '.github/workflows/dbt-integration-tests.yml'
11+
12+
jobs:
13+
dbt-integration-test:
14+
runs-on: ubuntu-latest
15+
strategy:
16+
matrix:
17+
python-version: ["3.11", "3.12"]
18+
env:
19+
PYTHON: ${{ matrix.python-version }}
20+
steps:
21+
- uses: actions/checkout@v4
22+
23+
- name: Setup Python
24+
uses: actions/setup-python@v5
25+
with:
26+
python-version: ${{ matrix.python-version }}
27+
architecture: x64
28+
29+
- name: Install the latest version of uv
30+
uses: astral-sh/setup-uv@v5
31+
with:
32+
enable-cache: true
33+
34+
- name: Install dependencies
35+
run: make install-python-dependencies-ci
36+
37+
- name: Install dbt and dbt-duckdb
38+
run: |
39+
uv pip install --system dbt-core dbt-duckdb
40+
41+
- name: Run dbt integration tests
42+
run: make test-python-integration-dbt
43+
44+
- name: Minimize uv cache
45+
run: uv cache prune --ci

‎Makefile‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -187,6 +187,28 @@ test-python-integration-container: ## Run Python integration tests using Docker
187187
python -m pytest -n 8 --integration sdk/python/tests \
188188
) || echo "This script uses Docker, and it isn't running - please start the Docker Daemon and try again!";
189189

190+
test-python-integration-dbt: ## Run dbt integration tests
191+
@echo "Running dbt integration tests..."
192+
@cd sdk/python/tests/integration/dbt/test_dbt_project && \
193+
echo "Installing dbt dependencies..." && \
194+
dbt deps && \
195+
echo "Building dbt models..." && \
196+
dbt build
197+
@cd sdk/python/tests/integration/dbt && \
198+
echo "Setting up Feast project..." && \
199+
mkdir -p feast_repo/data && \
200+
echo "project: feast_dbt_test\nregistry: data/registry.db\nprovider: local\nonline_store:\n type: sqlite\n path: data/online_store.db" > feast_repo/feature_store.yaml
201+
@cd sdk/python/tests/integration/dbt/feast_repo && \
202+
echo "Testing feast dbt import..." && \
203+
feast dbt import -m ../test_dbt_project/target/manifest.json -e driver_id -d file --tag feast && \
204+
echo "Verifying Feast objects..." && \
205+
feast feature-views list && \
206+
feast entities list
207+
@cd sdk/python && \
208+
echo "Running pytest integration tests..." && \
209+
python -m pytest tests/integration/dbt/test_dbt_integration.py -v --tb=short
210+
@echo "✓ dbt integration tests completed successfully!"
211+
190212
test-python-universal-spark: ## Run Python Spark integration tests
191213
PYTHONPATH='.' \
192214
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.offline_stores.contrib.spark_repo_configuration \

‎sdk/python/feast/dbt/codegen.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -248,7 +248,9 @@ def generate(
248248
if not entity_cols:
249249
raise ValueError("At least one entity column must be specified")
250250

251-
excluded = set(entity_cols) | {self.timestamp_field}
251+
# Note: entity columns should NOT be excluded - FeatureView.__init__
252+
# expects entity columns to be in the schema and will extract them
253+
excluded = {self.timestamp_field}
252254
if exclude_columns:
253255
excluded.update(exclude_columns)
254256

‎sdk/python/feast/dbt/mapper.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,9 +29,9 @@
2929
# Mapping from FeastType to ValueType for entity value inference
3030
FEAST_TYPE_TO_VALUE_TYPE: Dict[FeastType, ValueType] = {
3131
String: ValueType.STRING,
32-
Int32: ValueType.INT64,
32+
Int32: ValueType.INT32,
3333
Int64: ValueType.INT64,
34-
Float32: ValueType.DOUBLE,
34+
Float32: ValueType.FLOAT,
3535
Float64: ValueType.DOUBLE,
3636
Bool: ValueType.BOOL,
3737
Bytes: ValueType.BYTES,

‎sdk/python/tests/integration/dbt/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# Integration tests for dbt import functionality

‎sdk/python/tests/integration/dbt/conftest.py‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
"""
2+
Conftest for dbt integration tests.
3+
4+
This is a standalone conftest that doesn't depend on the main Feast test infrastructure.
5+
"""
6+
7+
from pathlib import Path
8+
9+
import pytest
10+
11+
# This conftest is minimal and doesn't import the main feast conftest
12+
# to avoid complex dependency chains for dbt-specific tests
13+
14+
# Path to the test dbt project manifest
15+
TEST_DBT_PROJECT_DIR = Path(__file__).parent / "test_dbt_project"
16+
TEST_MANIFEST_PATH = TEST_DBT_PROJECT_DIR / "target" / "manifest.json"
17+
18+
19+
def pytest_collection_modifyitems(config, items): # noqa: ARG001
20+
"""
21+
Skip dbt integration tests if manifest.json doesn't exist.
22+
23+
These tests require running 'dbt build' first to generate the manifest.
24+
The dbt-integration-test workflow handles this, but regular unit test
25+
runs don't, so we skip them to avoid failures.
26+
"""
27+
if not TEST_MANIFEST_PATH.exists():
28+
skip_marker = pytest.mark.skip(
29+
reason="dbt manifest.json not found - run 'dbt build' first or use dbt-integration-test workflow"
30+
)
31+
for item in items:
32+
item.add_marker(skip_marker)

‎sdk/python/tests/integration/dbt/pytest.ini‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
[pytest]
2+
# Prevent loading parent conftest.py files which may import heavy dependencies
3+
# like ray that are not needed for dbt integration tests
4+
norecursedirs = ..
5+
asyncio_mode = auto
6+
7+
# Test markers
8+
markers =
9+
dbt: dbt integration tests

0 commit comments

Comments
 (0)