Visitar URL original
feat: Alternative table provider as spark data source by timsaucer · Pull Request #110 · apache/datafusion-java · GitHub
Skip to content
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
3cf0e81
build: consolidate Rust crates into a Cargo workspace + extract nativ…
timsaucer Jun 12, 2026
fd4dd74
build: inherit crate version/edition from workspace; publish datafusi…
timsaucer Jun 12, 2026
2e50d6e
build: drop forward-referencing RAT excludes from foundation split
timsaucer Jun 12, 2026
9d2096c
build: drop overbroad **/*.md RAT exclude
timsaucer Jun 12, 2026
2514f56
docs: add README for datafusion-jni-common
timsaucer Jun 12, 2026
3ff4fda
fix: scope foundation docs/excludes to what 1/6 ships
timsaucer Jun 12, 2026
bb023eb
build: point release scripts at workspace rust-target/ dir
timsaucer Jun 12, 2026
9e9de7d
build: mark datafusion-jni-common publish = false
timsaucer Jun 12, 2026
e87fd64
feat: add plain-C scan ABI crate over Arrow C Data/Stream
timsaucer Jun 15, 2026
2598322
feat: add scan_config / scan_request protobuf for the C scan ABI
timsaucer Jun 15, 2026
cfaac8d
build: compile scan protobufs into datafusion-scan-ffi
timsaucer Jun 15, 2026
4cded33
feat: add datafusion.listing provider decoding ScanConfig
timsaucer Jun 15, 2026
f95191d
feat: add JNI shim over the plain-C scan ABI
timsaucer Jun 15, 2026
bc91ff9
test: end-to-end JVM scan through the JNI shim
timsaucer Jun 15, 2026
436e2b3
test: projection and filter pushdown from Java
timsaucer Jun 15, 2026
d5eefee
feat: wire row limit through the scan ABI
timsaucer Jun 15, 2026
0b24d7c
feat: add Spark DataSourceV2 connector
timsaucer Jun 15, 2026
1a1d0ec
feat: columnar reader on Spark 4.0 with provided Arrow
timsaucer Jun 15, 2026
573b438
feat: push Spark limit into the scan (SupportsPushDownLimit)
timsaucer Jun 15, 2026
a378bb4
docs: add DataFusion-Spark DataSource design doc
timsaucer Jun 15, 2026
301fd13
docs: add ADBC reuse analysis to Spark design doc
timsaucer Jun 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
build: inherit crate version/edition from workspace; publish datafusi…
…on-jni-common

Add [workspace.package] (version, edition, license, repository) to the root
manifest and have both crates inherit it via `*.workspace = true`, so a single
bump re-versions the workspace in lock step.

Make datafusion-jni-common publishable: drop `publish = false` and add a
`description` (crates.io requires it). All its dependencies are registry
crates, so nothing blocks publish. datafusion-jni stays `publish = false`
since it is a JVM-loaded cdylib, not a crates.io library.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
  • Loading branch information
timsaucer and claude committed Jun 12, 2026
commit fd4dd74f3d8184a2002cd29131bc3d07fe9c525f
9 changes: 9 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,15 @@ members = [
"native-common",
]

# Shared package metadata so every crate moves in lock step. Members inherit
# via `version.workspace = true` / `edition.workspace = true` etc.; a single
# bump here re-versions the whole workspace.
[workspace.package]
version = "0.1.0"
edition = "2021"
license = "Apache-2.0"
repository = "https://github.com/apache/datafusion-java"

# Every dependency used by any workspace member is declared here so version
# bumps live in one place and the resolver picks a single version of each
# crate across the workspace. Members reference these via `{ workspace = true }`
Expand Down
8 changes: 5 additions & 3 deletions native-common/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,11 @@

[package]
name = "datafusion-jni-common"
version = "0.1.0"
edition = "2021"
publish = false
version.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
description = "Shared JNI plumbing for DataFusion Java native crates: error-to-exception mapping, the per-cdylib Tokio runtime singleton, and the async-stream-to-FFI_ArrowArrayStream bridge."

[features]
# `datafusion-jni` builds DataFusion with `avro`, which adds the
Expand Down
7 changes: 5 additions & 2 deletions native/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,11 @@

[package]
name = "datafusion-jni"
version = "0.1.0"
edition = "2021"
version.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
# cdylib JNI artifact loaded by the JVM, not a crates.io library.
publish = false

[lib]
Expand Down