Audit third-party code bundled in published JARs

Background

We need to identify which published JARs bundle third-party code. ASF release policy requires JARs to place LICENSE and NOTICE in META-INF and requires the LICENSE file to cover every bundled non-Apache component. Any required third-party notices also need to be included.

This issue documents the findings and provides a script to verify them.

Findings

Cross-checking the final JAR contents against the Maven Shade configuration found:

7 binary JARs with bundled third-party classes (these need to be reflected in their respective LICENSE and NOTICE files)
13 Javadoc JARs with JDK-generated assets
39 JARs with no bundled third-party code

Artifact	Maven configuration	Final JAR contents
`parquet-avro-*.jar`	Explicitly shades FastUtil	FastUtil
`parquet-column-*.jar`	Inherits FastUtil/OpenHFT Shade rules	FastUtil, OpenHFT
`parquet-format-structures-*.jar`	Explicitly includes `libthrift`	Apache Thrift
`parquet-hadoop-*.jar`	Shade processes Jackson and FastUtil; filters/minimization remove Jackson	FastUtil only
`parquet-jackson-*.jar`	Explicitly includes Jackson artifacts	Jackson
`parquet-hadoop-bundle-*.jar`	Includes other `org.apache.parquet:parquet-*` JARs	Carries their already-shaded third-party classes
`parquet-cli-*-runtime.jar`	`shadedArtifactAttached=true`, classifier `runtime`, includes all dependencies	Full runtime dependency set

A few edge cases:

parquet-variant, parquet-protobuf, and parquet-thrift run Shade, but filtering and minimization leave no third-party classes in the final JARs.
parquet-benchmarks.jar bundles the full dependency tree, but it is a local-only uber-JAR. The published versioned parquet-benchmarks-*.jar contains only project classes.
The 15 source and 15 test JARs contain project output rather than dependency code.
The 13 Javadoc JARs contain JDK-generated JavaScript and CSS. ASF policy explicitly permits these assets in Maven Javadoc JARs.

Setup and verification commands

Setup

Run the setup commands from the repository root with Java 17. Maven output is left visible so build progress and failures are easy to follow.

java -version
./mvnw clean package --batch-mode -DskipTests
./mvnw --batch-mode process-resources source:jar-no-fork javadoc:jar

Verification

After setup completes, run the verification separately:

bash <<'EOF'
set -euo pipefail
export LC_ALL=C

tmp_dir=$(mktemp -d)
trap 'rm -rf "$tmp_dir"' EXIT

candidate_jars="$tmp_dir/candidates"
third_party_jars="$tmp_dir/third-party"
javadoc_jars="$tmp_dir/javadocs"
clean_jars="$tmp_dir/clean"

: >"$third_party_jars"
: >"$javadoc_jars"
: >"$clean_jars"

find . -mindepth 3 -maxdepth 3 -type f \
  -path './*/target/*.jar' \
  ! -name 'original-*.jar' \
  ! -name 'parquet-benchmarks.jar' \
  -print0 >"$candidate_jars"

count_non_parquet_classes() {
  jar tf "$1" | awk '
    /\.class$/ {
      path=$0
      sub(/^META-INF\/versions\/[0-9]+\//, "", path)
      if (path !~ /^org\/apache\/parquet\//) count++
    }
    END { print count+0 }
  '
}

count_external_sources() {
  jar tf "$1" | awk '
    /\.java$/ && $0 !~ /^org\/apache\/parquet\// { count++ }
    END { print count+0 }
  '
}

count_extra_test_classes() {
  local jar_file=$1
  local module_dir=$2
  local jar_classes="$tmp_dir/jar-classes"
  local project_classes="$tmp_dir/project-classes"

  jar tf "$jar_file" | awk '/\.class$/' | sort -u >"$jar_classes"
  : >"$project_classes"

  if [[ -d "$module_dir/target/test-classes" ]]; then
    (
      cd "$module_dir/target/test-classes"
      find . -type f -name '*.class' -print |
        sed 's#^\./##' |
        sort -u
    ) >"$project_classes"
  fi

  comm -13 "$project_classes" "$jar_classes" | awk 'END { print NR+0 }'
}

printf 'Inspecting existing JARs...\n'

jar_count=0
while IFS= read -r -d '' jar_file; do
  jar_count=$((jar_count + 1))
  jar_name=$(basename "$jar_file")
  module_dir=${jar_file%/target/*}

  case "$jar_name" in
    *-javadoc.jar)
      unzip -p "$jar_file" script.js >"$tmp_dir/script.js"
      if grep -Fq 'Oracle and/or its affiliates' "$tmp_dir/script.js"; then
        printf '%s\tJDK-generated JavaScript/CSS\n' "$jar_file" >>"$javadoc_jars"
      else
        printf '%s\n' "$jar_file" >>"$clean_jars"
      fi
      continue
      ;;
    *-sources.jar)
      bundled_count=$(count_external_sources "$jar_file")
      ;;
    *-tests.jar)
      bundled_count=$(count_extra_test_classes "$jar_file" "$module_dir")
      ;;
    *)
      bundled_count=$(count_non_parquet_classes "$jar_file")
      ;;
  esac

  if [[ $bundled_count -gt 0 ]]; then
    printf '%s\t%d bundled classes or sources\n' \
      "$jar_file" "$bundled_count" >>"$third_party_jars"
  else
    printf '%s\n' "$jar_file" >>"$clean_jars"
  fi
done <"$candidate_jars"

if [[ $jar_count -eq 0 ]]; then
  printf 'No JARs found. Run the setup commands first.\n' >&2
  exit 1
fi

printf '\n=== JARs containing bundled third-party code ===\n\n'
sort "$third_party_jars"

printf '\n=== Javadoc JARs containing JDK-provided assets ===\n\n'
sort "$javadoc_jars"

printf '\n=== JARs with no bundled third-party code found ===\n\n'
sort "$clean_jars"

printf '\nInspected %d published JARs.\n' "$jar_count"
EOF

The Java 17 build completed successfully, and the legal-file check verified all 59 JARs.

I checked the final binary JARs on master. These are the third-party dependencies whose classes are physically present.

Based on these results, we’ll review and update each JAR’s LICENSE and NOTICE files so they accurately reflect the third-party code bundled in that artifact.

JAR	Bundled third-party dependencies
`parquet-avro-*.jar`	`it.unimi.dsi:fastutil:8.5.18`
`parquet-column-*.jar`	`it.unimi.dsi:fastutil:8.5.18` `net.openhft:zero-allocation-hashing:0.27ea1`
`parquet-format-structures-*.jar`	`org.apache.thrift:libthrift:0.23.0`
`parquet-hadoop-*.jar`	`it.unimi.dsi:fastutil:8.5.18`
`parquet-jackson-*.jar`	`com.fasterxml.jackson.core:jackson-annotations:2.22` `com.fasterxml.jackson.core:jackson-core:2.22.1` `com.fasterxml.jackson.core:jackson-databind:2.22.1`
`parquet-hadoop-bundle-*.jar`	`it.unimi.dsi:fastutil:8.5.18` `net.openhft:zero-allocation-hashing:0.27ea1` `org.apache.thrift:libthrift:0.23.0` `com.fasterxml.jackson.core:jackson-annotations:2.22` `com.fasterxml.jackson.core:jackson-core:2.22.1` `com.fasterxml.jackson.core:jackson-databind:2.22.1`
`parquet-cli-*-runtime.jar`	`com.beust:jcommander:1.82` `com.fasterxml.jackson.core:jackson-annotations:2.22` `com.fasterxml.jackson.core:jackson-core:2.22.1` `com.fasterxml.jackson.core:jackson-databind:2.22.1` `com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.22.1` `com.github.luben:zstd-jni:1.5.7-11` `commons-codec:commons-codec:1.11` `commons-pool:commons-pool:1.6` `io.airlift:aircompressor:2.0.3` `it.unimi.dsi:fastutil:8.5.18` `javax.annotation:javax.annotation-api:1.3.2` `net.openhft:zero-allocation-hashing:0.27ea1` `net.sf.opencsv:opencsv:2.3` `org.apache.avro:avro:1.11.5` `org.apache.commons:commons-compress:1.26.2` `org.apache.commons:commons-lang3:3.20.0` `org.apache.commons:commons-text:1.15.0` `org.apache.thrift:libthrift:0.23.0` `org.locationtech.jts:jts-core:1.20.0` `org.slf4j:slf4j-api:1.7.33` `org.tukaani:xz:1.12` `org.xerial.snappy:snappy-java:1.1.10.8`

The check removes Parquet’s own classes, matches and removes every expected dependency, and fails if any expected dependency is missing or unexplained classes remain. All seven JARs ended with zero unmatched classes.

Verification command

Run from the repository root after building the JARs:

bash <<'EOF'
set -euo pipefail
export LC_ALL=C

tmp_dir=$(mktemp -d)
trap 'rm -rf "$tmp_dir"' EXIT

catalog="$tmp_dir/dependencies"
failures=0
verified_jars=0

cat >"$catalog" <<'CATALOG'
fastutil;it.unimi.dsi:fastutil:8.5.18;^(META-INF/versions/[0-9]+/)?(shaded/parquet/)?it/unimi/dsi/fastutil/
openhft;net.openhft:zero-allocation-hashing:0.27ea1;^(META-INF/versions/[0-9]+/)?(shaded/parquet/)?net/openhft/hashing/
thrift;org.apache.thrift:libthrift:0.23.0;^(META-INF/versions/[0-9]+/)?shaded/parquet/org/apache/thrift/
jackson-annotations;com.fasterxml.jackson.core:jackson-annotations:2.22;^(META-INF/versions/[0-9]+/)?(shaded/parquet/)?com/fasterxml/jackson/annotation/
jackson-core;com.fasterxml.jackson.core:jackson-core:2.22.1;^(META-INF/versions/[0-9]+/)?(shaded/parquet/)?com/fasterxml/jackson/core/|^META-INF/versions/9/module-info\.class$
jackson-databind;com.fasterxml.jackson.core:jackson-databind:2.22.1;^(META-INF/versions/[0-9]+/)?(shaded/parquet/)?com/fasterxml/jackson/databind/
jackson-jsr310;com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.22.1;^(META-INF/versions/[0-9]+/)?com/fasterxml/jackson/datatype/jsr310/
javax-annotation;javax.annotation:javax.annotation-api:1.3.2;^(META-INF/versions/[0-9]+/)?javax/annotation/
jts;org.locationtech.jts:jts-core:1.20.0;^(META-INF/versions/[0-9]+/)?org/locationtech/jts/
snappy;org.xerial.snappy:snappy-java:1.1.10.8;^(META-INF/versions/[0-9]+/)?org/xerial/snappy/
aircompressor;io.airlift:aircompressor:2.0.3;^(META-INF/versions/[0-9]+/)?io/airlift/compress/
commons-pool;commons-pool:commons-pool:1.6;^(META-INF/versions/[0-9]+/)?org/apache/commons/pool/
avro;org.apache.avro:avro:1.11.5;^(META-INF/versions/[0-9]+/)?shaded/parquet/org/apache/avro/
commons-compress;org.apache.commons:commons-compress:1.26.2;^(META-INF/versions/[0-9]+/)?org/apache/commons/compress/
xz;org.tukaani:xz:1.12;^(META-INF/versions/[0-9]+/)?org/tukaani/xz/
zstd;com.github.luben:zstd-jni:1.5.7-11;^(META-INF/versions/[0-9]+/)?com/github/luben/zstd/
slf4j;org.slf4j:slf4j-api:1.7.33;^(META-INF/versions/[0-9]+/)?org/slf4j/
opencsv;net.sf.opencsv:opencsv:2.3;^(META-INF/versions/[0-9]+/)?au/com/bytecode/opencsv/
commons-text;org.apache.commons:commons-text:1.15.0;^(META-INF/versions/[0-9]+/)?org/apache/commons/text/
commons-lang3;org.apache.commons:commons-lang3:3.20.0;^(META-INF/versions/[0-9]+/)?org/apache/commons/lang3/
jcommander;com.beust:jcommander:1.82;^(META-INF/versions/[0-9]+/)?com/beust/(jcommander|ah)/
commons-codec;commons-codec:commons-codec:1.11;^(META-INF/versions/[0-9]+/)?org/apache/commons/codec/
CATALOG

check_jar() {
  local jar_file=$1
  local jar_name
  local expected
  local all_classes="$tmp_dir/all-classes"
  local remaining="$tmp_dir/remaining"
  local next="$tmp_dir/next"
  local total_classes
  local project_classes
  local matched_classes
  local remaining_classes
  local processed_dependencies=0
  local expected_dependencies

  jar_name=$(basename "$jar_file")

  case "$jar_name" in
    parquet-avro-*.jar)
      expected="fastutil"
      ;;
    parquet-column-*.jar)
      expected="fastutil openhft"
      ;;
    parquet-format-structures-*.jar)
      expected="thrift"
      ;;
    parquet-hadoop-bundle-*.jar)
      expected="fastutil openhft thrift jackson-annotations jackson-core jackson-databind"
      ;;
    parquet-hadoop-*.jar)
      expected="fastutil"
      ;;
    parquet-jackson-*.jar)
      expected="jackson-annotations jackson-core jackson-databind"
      ;;
    parquet-cli-*-runtime.jar)
      expected="fastutil openhft thrift jackson-annotations jackson-core
        jackson-databind jackson-jsr310 javax-annotation jts snappy
        aircompressor commons-pool avro commons-compress xz zstd slf4j
        opencsv commons-text commons-lang3 jcommander commons-codec"
      ;;
    *)
      return
      ;;
  esac

  expected=$(printf '%s' "$expected" | tr '\n' ' ' | tr -s ' ')
  verified_jars=$((verified_jars + 1))

  if ! jar tf "$jar_file" |
    awk '/\.class$/ { print }' >"$all_classes"; then
    printf 'FAIL %s: unreadable JAR\n' "$jar_file" >&2
    failures=$((failures + 1))
    return
  fi

  total_classes=$(wc -l <"$all_classes" | tr -d ' ')

  grep -Ev \
    '^(META-INF/versions/[0-9]+/)?org/apache/parquet/' \
    "$all_classes" >"$remaining" || true

  remaining_classes=$(wc -l <"$remaining" | tr -d ' ')
  project_classes=$((total_classes - remaining_classes))

  printf '\n%s\n' "$jar_file"
  printf '  %-68s %6d classes\n' \
    'Parquet project classes' "$project_classes"

  while IFS=';' read -r dependency_id coordinate pattern; do
    case " $expected " in
      *" $dependency_id "*)
        ;;
      *)
        continue
        ;;
    esac

    processed_dependencies=$((processed_dependencies + 1))
    matched_classes=$(grep -Ec "$pattern" "$remaining" || true)

    printf '  %-68s %6d classes\n' "$coordinate" "$matched_classes"

    if [[ $matched_classes -eq 0 ]]; then
      printf '  FAIL: expected dependency matched no classes\n' >&2
      failures=$((failures + 1))
      continue
    fi

    grep -Ev "$pattern" "$remaining" >"$next" || true
    mv "$next" "$remaining"
  done <"$catalog"

  expected_dependencies=$(printf '%s\n' "$expected" | wc -w | tr -d ' ')

  if [[ $processed_dependencies -ne $expected_dependencies ]]; then
    printf '  FAIL: dependency catalog is incomplete\n' >&2
    failures=$((failures + 1))
  fi

  remaining_classes=$(wc -l <"$remaining" | tr -d ' ')

  printf '  %-68s %6d classes\n' \
    'Unmatched after filtering' "$remaining_classes"

  if [[ $remaining_classes -ne 0 ]]; then
    printf '  FAIL: unexplained classes remain:\n' >&2
    sed 's/^/    /' "$remaining" >&2
    failures=$((failures + 1))
  fi
}

while IFS= read -r -d '' jar_file; do
  check_jar "$jar_file"
done < <(
  find parquet-avro parquet-column parquet-format-structures \
    parquet-hadoop parquet-jackson parquet-hadoop-bundle parquet-cli \
    -type f -path '*/target/*.jar' \
    ! -name 'original-*.jar' \
    ! -name '*-tests.jar' \
    ! -name '*-sources.jar' \
    ! -name '*-javadoc.jar' \
    -print0
)

if [[ $verified_jars -ne 7 ]]; then
  printf '\nFAIL: expected 7 JARs, found %d. Run a clean package build first.\n' \
    "$verified_jars" >&2
  failures=$((failures + 1))
fi

if [[ $failures -ne 0 ]]; then
  printf '\nVerification failed with %d error(s).\n' "$failures" >&2
  exit 1
fi

printf '\nVerified %d JARs; every dependency matched and 0 classes remain.\n' \
  "$verified_jars"
EOF

Output

parquet-avro/target/parquet-avro-1.18.0-SNAPSHOT.jar
  Parquet project classes                                                 106 classes
  it.unimi.dsi:fastutil:8.5.18                                            943 classes
  Unmatched after filtering                                                 0 classes

parquet-column/target/parquet-column-1.18.0-SNAPSHOT.jar
  Parquet project classes                                                 653 classes
  it.unimi.dsi:fastutil:8.5.18                                           1074 classes
  net.openhft:zero-allocation-hashing:0.27ea1                              60 classes
  Unmatched after filtering                                                 0 classes

parquet-format-structures/target/parquet-format-structures-1.18.0-SNAPSHOT.jar
  Parquet project classes                                                 431 classes
  org.apache.thrift:libthrift:0.23.0                                       73 classes
  Unmatched after filtering                                                 0 classes

parquet-hadoop/target/parquet-hadoop-1.18.0-SNAPSHOT.jar
  Parquet project classes                                                 304 classes
  it.unimi.dsi:fastutil:8.5.18                                            371 classes
  Unmatched after filtering                                                 0 classes

parquet-jackson/target/parquet-jackson-1.18.0-SNAPSHOT.jar
  Parquet project classes                                                   0 classes
  com.fasterxml.jackson.core:jackson-annotations:2.22                      77 classes
  com.fasterxml.jackson.core:jackson-core:2.22.1                          221 classes
  com.fasterxml.jackson.core:jackson-databind:2.22.1                      812 classes
  Unmatched after filtering                                                 0 classes

parquet-hadoop-bundle/target/parquet-hadoop-bundle-1.18.0-SNAPSHOT.jar
  Parquet project classes                                                1795 classes
  it.unimi.dsi:fastutil:8.5.18                                           1074 classes
  net.openhft:zero-allocation-hashing:0.27ea1                              60 classes
  org.apache.thrift:libthrift:0.23.0                                       73 classes
  com.fasterxml.jackson.core:jackson-annotations:2.22                      77 classes
  com.fasterxml.jackson.core:jackson-core:2.22.1                          221 classes
  com.fasterxml.jackson.core:jackson-databind:2.22.1                      812 classes
  Unmatched after filtering                                                 0 classes

parquet-cli/target/parquet-cli-1.18.0-SNAPSHOT-runtime.jar
  Parquet project classes                                                2032 classes
  it.unimi.dsi:fastutil:8.5.18                                          14280 classes
  net.openhft:zero-allocation-hashing:0.27ea1                             120 classes
  org.apache.thrift:libthrift:0.23.0                                       73 classes
  com.fasterxml.jackson.core:jackson-annotations:2.22                     154 classes
  com.fasterxml.jackson.core:jackson-core:2.22.1                          441 classes
  com.fasterxml.jackson.core:jackson-databind:2.22.1                     1624 classes
  com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.22.1            64 classes
  javax.annotation:javax.annotation-api:1.3.2                              15 classes
  org.locationtech.jts:jts-core:1.20.0                                    732 classes
  org.xerial.snappy:snappy-java:1.1.10.8                                   40 classes
  io.airlift:aircompressor:2.0.3                                          124 classes
  commons-pool:commons-pool:1.6                                            55 classes
  org.apache.avro:avro:1.11.5                                             450 classes
  org.apache.commons:commons-compress:1.26.2                              571 classes
  org.tukaani:xz:1.12                                                     129 classes
  com.github.luben:zstd-jni:1.5.7-11                                       37 classes
  org.slf4j:slf4j-api:1.7.33                                               34 classes
  net.sf.opencsv:opencsv:2.3                                               10 classes
  org.apache.commons:commons-text:1.15.0                                  168 classes
  org.apache.commons:commons-lang3:3.20.0                                 421 classes
  com.beust:jcommander:1.82                                                73 classes
  commons-codec:commons-codec:1.11                                         96 classes
  Unmatched after filtering                                                 0 classes

Verified 7 JARs; every dependency matched and 0 classes remain.

Thanks for generating the list. I just checked, and all the included binaries are ASF compatible 👍.

apacheGH-3690: Update third-party NOTICE and LICENSE

Thanks for checking! Compatibility looks good. We still need to review each bundled dependency for any required LICENSE or NOTICE updates. ASF release policy requires those files to account for the exact contents of each artifact, including non-Apache license terms and relevant upstream notices.

For example, we were pretty meticulous about this in iceberg: apache/iceberg#16196

Good news is, we only need to check the 3rd party dependencies listed above for each jar.

EDIT: i see #3694, thanks! i'll take a look

GH-3690: Update third-party NOTICE and LICENSE (#3694)

Navigation Menu

Repository navigation

Audit third-party code bundled in published JARs #3690

Background

Findings

Setup

Verification

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Audit third-party code bundled in published JARs #3690

Description

Background

Findings

Setup

Verification

Activity

kevinjqliu commented on Jul 26, 2026

Fokko commented on Jul 27, 2026

kevinjqliu commented on Jul 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions