Visitar URL original
Failed to create datafusion memory table: Error during planning: Mismatch between schema and batches · Issue #408 · roapi/roapi · GitHub
Skip to content

Failed to create datafusion memory table: Error during planning: Mismatch between schema and batches #408

Open
Open
Failed to create datafusion memory table: Error during planning: Mismatch between schema and batches#408

Description

@amokfa
amokfa
opened on Aug 15, 2025
❯ cat config.yaml
addr:
  http: 0.0.0.0:8080
  postgres: 0.0.0.0:5432
tables:
  - name: "tbl"
    uri: "tbl"
    option:
      format: "deltalake"%
❯ ./roapi -c config.yaml
[2025-08-15T07:22:24Z INFO  roapi::context] loading `uri(tbl)` as table `tbl`
error: Failed to build context: Failed to load table: Table error: Failed to create datafusion memory table: Error during planning: Mismatch between schema and batches

The delta lake uses parquet and contains a date column which isn't listed as a supported data type for parquet. https://parquet.apache.org/docs/file-format/types/
Could this be the issue?

Activity

  1. houqp commented on Aug 15, 2025

    @houqp
    houqp
    on Aug 15, 2025
    Member

    it should work, can you share a dummy table file with me for reproduction? you can create one by deleting all the rows from the table.

  2. amokfa commented on Aug 18, 2025

    @amokfa
    amokfa
    on Aug 18, 2025
    Author

    The delta lake is backed by a bunch of parquet files, partitioned by date. If I delete all the data in those parquet files, roapi loads the delta lake without any errors. If I leave just one row in one of the parquet files (tbl_empty/when=2025-08-14/part-00001-05f89d9f-51a2-4a19-9bbb-6030aa23f385-c000.snappy.parquet in the shared DB), I get the message in the original post. roapi config:

    addr:
  http: 0.0.0.0:8080
  postgres: 0.0.0.0:5433
tables:
  - name: "tbl"
    uri: "tbl_empty"
    option:
      format: "delta"

    Delta lake directory attached.

    tbl.tar.gz

  3. houqp commented on Aug 21, 2025

    @houqp
    houqp
    on Aug 21, 2025
    Member

    thank you, I am able to repro on my end, will investigate this weekend

  4. amokfa commented on Sep 17, 2025

    @amokfa
    amokfa
    on Sep 17, 2025
    Author

    Hey @houqp , any updates?

  5. prabodh516 commented on Oct 22, 2025

    @prabodh516
    prabodh516
    on Oct 22, 2025
    Contributor

    @houqp @amokfa

    As per code, for memory table the parquet files are read but their schema doesn't include partition column which is part of delta table schema.

    raised a PR to fix this #411

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions