Visitar URL original
GitHub - motherduckdb/metabase_duckdb_driver: Metabase DuckDB Driver shipped as 3rd party plugin · GitHub
Skip to content

About

Metabase DuckDB Driver shipped as 3rd party plugin

Resources

Stars

174 stars

Watchers

3 watching

Forks

Repository files navigation

Metabase DuckDB Driver

The Metabase DuckDB driver allows Metabase (GitHub) to use the embedded DuckDB (GitHub) database.

This driver is supported by MotherDuck. If you would like to open a GitHub issue to report a bug or request new features, or would like to open a pull requests against it, please do so in this repository, and not in the core Metabase GitHub repository.

DuckDB

DuckDB is an in-process SQL OLAP database management. It does not run as a separate process, but completely embedded within a host process. So, it embedds to the Metabase process like SQLite.

Obtaining the DuckDB Metabase driver

Where to find it

Click here to view the latest release of the Metabase DuckDB driver; click the link to download duckdb.metabase-driver.jar.

You can find past releases of the DuckDB driver here, and releases earlier than 0.2.6 (corresponding to DuckDB v0.10.0) here.

How to Install it

Metabase will automatically make the DuckDB driver available if it finds the driver in the Metabase plugins directory when it starts up. All you need to do is create the directory plugins (if it's not already there), move the JAR you just downloaded into it, and restart Metabase.

By default, the plugins directory is called plugins, and lives in the same directory as the Metabase JAR.

For example, if you're running Metabase from a directory called /app/, you should move the DuckDB driver to /app/plugins/:

# example directory structure for running Metabase with DuckDB support
/app/metabase.jar
/app/plugins/duckdb.metabase-driver.jar

If you're running Metabase from the Mac App, the plugins directory defaults to ~/Library/Application Support/Metabase/Plugins/:

# example directory structure for running Metabase Mac App with DuckDB support
/Users/you/Library/Application Support/Metabase/Plugins/duckdb.metabase-driver.jar

If you are running the Docker image or you want to use another directory for plugins, you should specify a custom plugins directory by setting the environment variable MB_PLUGINS_DIR.

Configuring

Once you've started up Metabase, go to add a database and select "DuckDB". Provide the path to the DuckDB database file. To use DuckDB in the in-memory mode without any database file, you can specify :memory: as the database path.

Parquet

Does it make sense to start DuckDB Database in-memory mode without any data in system like Metabase? Of Course yes! Because of feature of DuckDB allowing you to run SQL queries directly on Parquet files. So, you don't need a DuckDB database.

For example (somewhere in Metabase SQL Query editor):

# DuckDB selected as source

SELECT originalTitle, startYear, genres, numVotes, averageRating from '/Users/you/movies/title.basics.parquet' x
JOIN (SELECT * from '/Users/you/movies/title.ratings.parquet') y ON x.tconst = y.tconst
ORDER BY averageRating * numVotes DESC

Ducklake

Starting from driver version 1.4.1.0, you can configure the DuckDB data source to point to a ducklake database by setting the database file field to ducklake:/path/to/db_name.ducklake. This will also create a folder /path/to/db_name.ducklake.files, where the parquet files are stored.

Right now, specifying alternative data path for a brand new ducklake database, like ATTACH 'ducklake:my_other_ducklake.ducklake' AS my_other_ducklake (DATA_PATH '/some/other/path/'); is not natively supported. But you can first initialize the ducklake in SQL, using another duckdb client or within the Metabase SQL interface, with the target data path, then create the data source attaching the ducklake database already initialized with the target data path.

Always give DATA_PATH an absolute path (or an object-store URI). DuckLake stores it verbatim in the catalog and resolves a relative path against the process working directory, which for Metabase is its own container or service directory — not wherever you initialised the lake. Files written from another client are then unreadable, reporting IO Error: Cannot open file "relative/path/….parquet": No such file or directory even though the file is there, and the table still lists in the data browser because its metadata comes from the catalog. Under Docker the absolute path also has to be inside a mounted volume, or the parquet is written into the container and disappears when it is recreated.

To repair a lake that already recorded a relative path, re-attach it once with (DATA_PATH '/absolute/path/', OVERRIDE_DATA_PATH true).

MotherDuck-hosted Ducklake

If you're using a ducklake database on MotherDuck, it can be attached like a regular MotherDuck database, e.g. md:my_ducklake_database.

Ducklake with an external catalog

For a catalog that isn't a local file — Postgres, MySQL, or one behind a ducklake secret — put the INSTALL/LOAD/ATTACH statements in the Init SQL connection field (see below), not in the SQL editor:

INSTALL ducklake; LOAD ducklake;
INSTALL postgres; LOAD postgres;
ATTACH IF NOT EXISTS 'ducklake:postgres:dbname=catalog user=me password=secret host=pg port=5432'
  AS my_lake (DATA_PATH 's3://my-bucket/my_lake/');
USE my_lake;

Init SQL

The Init SQL connection field runs on every new DuckDB connection Metabase opens. Use it for whatever a connection needs before it can answer queries: installing and loading extensions, creating secrets, attaching catalogs.

This matters because Metabase keeps a pool of connections and opens new ones as it goes. Running ATTACH once in the SQL editor only affects the connection that happened to serve that query. The attached tables can then show up in the data browser — a sync saw them — but fail with table does not exist when a later query lands on a connection that never ran the ATTACH, or after a restart. Init SQL is what gives every connection the same setup.

Statements run as one batch, so keep them ordered and idempotent (CREATE OR REPLACE SECRET, ATTACH IF NOT EXISTS, ...).

Init SQL cannot rescue a :memory: database

The data source should point to a real database file or lake with ducklake:/path/to/catalog.ducklake. Each connection to :memory: is a separate DuckDB database, so anything a connection creates — a table, a secret, an attached catalog — is invisible to the others. Init SQL gives them all the same statements, but not the same state: a table created by one query is missing from the next, depending on which connection serves it. Worse, two connections cannot attach the same file catalog, because DuckDB allows one handle per file per process — so a file-based DuckLake catalog attached from Init SQL fails with Unique file handle conflict, which IF NOT EXISTScannot avoid. Therefore, use :memory: only for stateless work, such as querying parquet by path.

Attached catalogs need a search_path

Metabase records a table's schema without its catalog, so a table in an attached catalog is only reachable if DuckDB can resolve it from the search path. Set one in Init SQL for every catalog you attach, or queries fail with Catalog Error: Table with name <table> does not exist! even though the table is listed in the data browser:

ATTACH IF NOT EXISTS '/data/second.duckdb' AS second;
SET search_path='second.main,main';

Extensions on a restricted network

Some extensions are compiled into the driver's duckdb_jdbc and work with no network at all: icu, json and parquet report install_mode = STATICALLY_LINKED. Everything else — httpfs, ducklake, iceberg, postgres, motherduck — is downloaded from extensions.duckdb.org on first use, into a directory DuckDB must be able to write.

Two DuckDB settings control where that happens, and both can be set today in the Additional DuckDB connection string options field under Advanced options, &-separated:

home_directory=/var/lib/metabase/duck&extension_directory=/opt/duckdb-extensions
Setting What it does
home_directory Where DuckDB resolves ~; extensions land in <home_directory>/.duckdb/extensions/. Set this when the Metabase process has no writable home, otherwise installs fail with IO Error: Can't find the home directory at '...'
extension_directory The extension tree itself, independent of the home directory. Use it to point at a directory you pre-populated

Pre-seeding, when extensions.duckdb.org is blocked

Install the extensions once somewhere with network access, on the same driver version and platform, then ship the directory to the restricted host and point extension_directory at it:

SET extension_directory='/opt/duckdb-extensions';
INSTALL httpfs; INSTALL ducklake; INSTALL iceberg; INSTALL motherduck;

The layout is pinned to version and platform, e.g. /opt/duckdb-extensions/v1.5.6/linux_arm64/httpfs.duckdb_extension, so re-seed it whenever the driver's bundled DuckDB version changes.

Pre-seeding only removes the need to download an extension. It does not make MotherDuck work offline: motherduck loads from disk, but its initialisation still has to reach the MotherDuck service. So this helps where the extension repository is blocked and MotherDuck itself is reachable.

Changing the MotherDuck token

DuckDB refuses to open the same database under a different configuration while connections to it are still open:

Connection Error: Can't open a connection to same database file with a
different configuration than existing connections

Metabase validates a data source by connecting before it saves. On an existing source its pool is still holding connections open with the old token, so that validation connection is refused and the save fails with the error above — the new token is never stored, and queries carry on using the old one.

Restart Metabase, then change the token. A fresh process has no pool for that database, so the validation connection is the only one and it succeeds.

Pivot tables and if() formulas

A custom aggregation whose condition tests one of the question's own breakout columns renders fine as a table, then breaks when the visualisation is switched to pivot. With breakouts on Sales Rep and Region:

if([Region] = "EMEA", SumIf([Revenue], [Channel] = "Online") / SumIf([Revenue], [Channel] = "Retail"), Sum(0))

Metabase builds a pivot by running one query per grouping level. The level that rolls Region up drops it from the GROUP BY while the aggregation still references it, which standard SQL forbids:

Binder Error: column "region" must appear in the GROUP BY clause or must be
part of an aggregate function.

Depending on the surface, the pivot either shows that error or just renders empty — the pivot API reports the query as completed with zero rows while carrying the failure in its response details. This comes from Metabase's pivot rewrite rather than the driver — the same query fails identically on Metabase's own H2 sample database (H2 enforces the rule at run time, DuckDB at bind time), and no DuckDB version accepts it (reported upstream as metabase#73153).

Workaround: keep the condition inside the aggregations, so that nothing outside an aggregate refers to a breakout column:

SumIf([Revenue], [Channel] = "Online" AND [Region] = "EMEA")
  / SumIf([Revenue], [Channel] = "Retail" AND [Region] = "EMEA")

That pivots correctly at every grouping level.

Docker

Unfortunately, DuckDB plugin doesn't work in the default Alpine based Metabase docker container out of the box due to some glibc problems. But we provide a Debian-based Docker image of Metabase where the DuckDB plugin does work.

Pre-built images

Pre-built images are published to the GitHub Container Registry and are the easiest way to get started:

# Latest Metabase with the latest DuckDB driver
docker pull ghcr.io/motherduckdb/metabase-duckdb:latest

# Specific Metabase and driver version
docker pull ghcr.io/motherduckdb/metabase-duckdb:0.63.17-duckdb1.5.5.0

Tags follow the pattern <metabase_version>-duckdb<driver_version>. Browse all available tags at ghcr.io/motherduckdb/metabase-duckdb.

Start the container:

docker run --name metabase_duckdb -d -p 3000:3000 ghcr.io/motherduckdb/metabase-duckdb:latest
# Then open http://localhost:3000

Building locally

On Alpine the native DuckDB library fails to load — Error loading shared library libstdc++.so.6 on the first attempt, then Could not initialize class org.duckdb.DuckDBNative on every attempt after. Installing packages does not fix it: duckdb_jdbc ships a glibc build and Alpine is musl. Use a glibc base image such as the one below.

See the included Dockerfile for a complete setup. You can build the container like so, optionally with specific Metabase or DuckDB driver versions:

# Build with default versions: the newest Metabase version in
# metabase_versions.json + the latest driver release
docker build . --tag metabase_duckdb:latest

# Build with a specific Metabase version and driver jar (a release URL or a
# path inside the build context)
docker build . --tag metabase_duckdb:latest \
  --build-arg METABASE_VERSION=0.59.12 \
  --build-arg DUCKDB_DRIVER_URL=https://github.com/motherduckdb/metabase_duckdb_driver/releases/download/1.5.2.0/duckdb.metabase-driver.jar

Publishing new images (maintainers)

Which Metabase versions the driver supports is recorded in metabase_versions.json, a plain list of versions.

Images are always built from a jar produced by CI, never from a release download:

  • Pull requests push pr-<number> (PRs from forks build without pushing).
  • Pushes to main push main and sha-<short-sha>.
  • The Release workflow pushes one image per listed Metabase version, tagged <metabase_version>-duckdb<driver_version>, with :latest going to the newest listed Metabase version.

Build Container Images is a manual backfill tool for combining an already-published driver release with another Metabase version (e.g. a new Metabase release for an existing driver). It pulls the driver jar from that GitHub release and takes these inputs:

Input Required Default Effect
metabase_version yes — Metabase version, no v prefix (e.g. 0.63.10)
driver_version yes — Driver version, which must equal its release tag (e.g. 1.5.5.0)
tag_latest no false Also push :latest
force_rebuild no false Overwrite the tag if it already exists
dry_run no false Build both platforms but push nothing

Releasing a new driver version

Don't create releases or tags from the GitHub UI. Merge any version bumps first: org.duckdb/duckdb_jdbc in deps.edn, and metabase_versions.json if the supported range changed. The Dockerfile defaults to the newest listed Metabase version and the newest driver release, so it needs no edits.

Then run the Release workflow with the following parameters:

  • sha: a main commit whose Build Metabase DuckDB Driver run is green (defaults to the latest).
  • version: the first three numbers must match duckdb_jdbc in deps.edn, and the fourth counts driver releases on that DuckDB version (1.5.6.0, then 1.5.6.1 for a driver-only fix).
  • dry_run: run every check and smoke test, then only print the release command.

The workflow fails if the tag already exists, the version does not match deps.edn, or the commit has no green main build. Otherwise it takes the jar from that build and smoke-tests it in an image for every version in metabase_versions.json. It then creates the tag <version> on the commit and publishes the release, titled Metabase <newest listed> + DuckDB <x.y.z>, with the jar attached and generated notes. gh release create publishes only after the jar is uploaded, so no release goes out without it. The images are pushed last; if a push fails, backfill it with Build Container Images.

The integration tests run with continue-on-error, so a green build does not mean they passed. Check their logs before releasing.

Both the target registry and the driver download URL are derived from the repository the workflow runs in, so the whole pipeline can be exercised on a fork without editing anything.

Using DB file with Docker

In order to use the DuckDB database file from your local host in the docker container you should mount folder with your DB file into docker container

docker run -v /dir_with_my_duck_db_file_in_the_local_host/:/container/directory ...

Next, in the settings page of DuckDB of Metabase Web UI you could set your DB file name like this

/container/directory/<you_duckdb_file>

The same way you could mount the dir with parquet files into container and make SQL queries to this files using directory in your container.

Development

Everything runs in Docker; no local JDK, Clojure or Node needed.

make dist   # build dist/duckdb.metabase-driver.jar from this working tree
make dev    # run Metabase with that jar on http://localhost:3000
make test   # run the driver test suite
make stop   # stop the dev Metabase

make dev keeps its Metabase application database in ./data, so connections and questions survive a restart. Drop .duckdb or .parquet files in ./data to reach them from Metabase as /home/metabase/data/<file>. If motherduck_token is set in your shell it is passed through, so md: works with an empty token field.

To iterate on the driver: edit src/, then run make dev again — it rebuilds the jar and restarts Metabase in about a minute. The Metabase checkout and the Maven cache live in ~/.cache/metabase-duckdb-driver and are shared by every worktree of this repo, so a second branch does not re-download them.

Variable Default Purpose
MB_VERSION newest in metabase_versions.json Metabase release make dev runs
MB_REF master Metabase revision the driver is built and tested against
DRIVER_VERSION (unset) Use a published driver release instead of a local build
DRIVERS duckdb make test DRIVERS=motherduck runs the suite against MotherDuck (needs motherduck_token)
TEST (unset) make test TEST=metabase.driver.duckdb-test for a single namespace
make repl                       # Metabase REPL with the driver on the classpath
make dev MB_VERSION=0.63.1      # reproduce a version-specific report
make dev DRIVER_VERSION=1.5.4   # ... against a released driver, to bisect a regression
make clean                      # drop dist/ and the dev container
make distclean                  # also drop the cached Metabase checkout

A VS Code DevContainer is also available if you prefer working inside the Metabase checkout directly.

Acknowledgement

Thanks @AlexR2D2 for originally authoring this connector.

About

Metabase DuckDB Driver shipped as 3rd party plugin

Resources

Stars

174 stars

Watchers

3 watching

Forks

Releases

Packages

Used by

Contributors

Languages