Rental Market Dynamics — Dubai

ETL and analytics pipeline for Dubai Ejari rent transactions — registered tenancy contracts from the configured Rent Transaction Details endpoint.

Sales, title deeds, and short-term stay data are out of scope. Analysis favors medians over means, with guards for bulk registrations and unrealistic unit areas so headline rent figures stay trustworthy.

The pipeline treats data as a bronze → silver → gold flow. Bronze is one file per day; Silver and Gold live together in a single cumulative store that grows every day:

One release per data date (release-YYYY-MM-DD) holds the raw CSV, etl_status.json and the combined rents_layers.duckdb together.

The Silver tables and the Gold views are in the same DuckDB file, so a plain duckdb.connect("rents_layers.duckdb") resolves every view — no ATTACH, no alias. The store is cumulative (seeded empty, grown daily), so the time-series marts (AggMonthlyRegistrations, and the per-area medians) span all history rather than a single week.

• The daily job is fail-open, and every run is marked. An empty incremental window is a success, not an error, but it is never silent: the run writes output/etl_status.json and logs a greppable NO_NEW_DATA line. That status is also published to the day's release, so a release with no CSV is a quiet day and a missing release is a run that never happened — the two are distinguishable from the releases list alone. A missing file after a successful download is not treated as "no data" — it proceeds and fails.
• Idempotent by primary key. The extract uses a 2-day window, so a registration can appear in two consecutive daily files. FctContract is keyed on contract_id, so ingesting a day again (a re-run, or the next day's overlapping window) replaces the row instead of duplicating it — no cross-file dedup code, just the DB constraint.
• A stalled feed fails. Ingest keeps one trust check: the newest registration may trail the data date its filename claims by at most MAX_REGISTRATION_LAG_DAYS (1). A quiet day is fine; a feed that stops advancing is not. The store's freshness is readable from _meta.data_through.

The _meta table is one row describing the current store — built_at, data_from, data_through, total_contracts, last_ingested_window.

EJARI_URL (paginated)
        │
        ▼
   Extract (Scrapy / downloader)
        │
        ▼
   Bronze: rent_contracts_YYYYMMDD.csv ──► daily release (empty window → NO_NEW_DATA, exit 0)
        │
        ▼
   Ingest ──► enrich (Polars) ──► INSERT OR REPLACE (key: contract_id)
        │
        ▼
   rents_layers.duckdb   (cumulative, one file)
        ├── Silver tables: DimArea, DimPropertyType, DimMetro, FctContract, _meta
        └── Gold views:    gold_*, Agg*   (resolve in-file, no ATTACH)

Orchestration is Make-driven locally and via GitHub Actions (daily ETL, daily layers ingest, push builds). See ADR-10 for why the layers are cumulative, keyed and combined in one file (it supersedes ADR-07 and ADR-08).

• Python 3.9+ (CI uses 3.12)
• Polars / PyArrow for transform
• Pydantic v2 for the Silver contract
• DuckDB for the cumulative layers store
• Scrapy for Ejari extraction
• uv (or pip) for dependencies

git clone https://github.com/dataengineergaurav/rental-market-dynamics-dubai
cd rental-market-dynamics-dubai
uv sync   # or: pip install -r requirements.txt && pip install lib/
cp .env.example .env

Required environment variables:

• EJARI_URL — Ejari rent transactions endpoint
• GH_TOKEN — GitHub token (for publishing releases)

make all              # build → daily ETL → tests
make layers           # ingest today's CSV into the cumulative layers DuckDB
make layers-publish   # publish the layers DuckDB into the day's release
make test             # pytest
make lint             # ruff check + format-check
make coverage         # pytest with a coverage floor
make scrapy-rents     # Scrapy extract only

Quality gates run in CI on every push (build_and_deploy.yml): make lint (high-signal ruff — undefined names, unused imports, real-bug lints) and make coverage (a ratchet floor in the Makefile). Dependencies are locked — uv sync reproduces the environment from uv.lock. See ADR-11.

Daily entry point: run_etl_pipeline.py. Layers ingest: python -m lib.analysis.build_layers_duckdb --csv output/rent_contracts_YYYYMMDD.csv (repeat --csv to backfill several days at once). Publishing: python -m lib.workspace.publish_layers --artifact output/rents_layers.duckdb.

lib/              Extract, transform, enrichment, analytics, release helpers
rents_scraper/    Scrapy spider for Ejari rents
output/           Daily CSVs, Parquet, the cumulative layers DuckDB
tests/            Pipeline and data-quality gates
docs/             Documentation hub: architecture, data dictionary, analyst cookbook, operations, ADRs
.github/          Daily ETL / daily layers / push workflows

One release per data date (release-YYYY-MM-DD) carries the raw CSV, etl_status.json and the combined rents_layers.duckdb — see GitHub Releases. Each day's release holds a full snapshot of the cumulative DuckDB as of that date, so any historical layer file is also directly retrievable.

Start at the documentation hub, or jump straight to the part you need:

• Architecture — the pipeline end to end, the medallion contracts, and the invariants the tests pin
• Data dictionary — every table, view, column, code and constant
• Analyst cookbook — real market questions answered with SQL against the Gold views
• Operations — schedules, releases, commands, triage, release verification
• Library usage guide — MarketAnalytics / enrichment APIs

Decisions and history:

• ADRs — 06 (pydantic Silver contract), 09 (release hardening), 10 (cumulative combined layers), 11 (reproducible deps + CI gate). ADR-07/08 are superseded by ADR-10.
• Silver contract design — field measurements and the market-health gate
• CHANGELOG · RELEASE_NOTES — what changed, and what each release contains

See CONTRIBUTING.md. Licensed under MIT.