Repository navigation
Unable to use file.path.pattern in an ingestionConfig #19696
Description
Activity
Thanks for the report. The leading \ and the alternatives you tried all have the same root cause.
Table config values are resolved against environment variables and system properties when the config is read: a string value that is entirely of the form ${NAME:default} is replaced by the value of NAME, or by default when NAME is not set. ${filePathPattern:\1} has exactly that shape, so it is replaced by \1 before the segment name generator sees it:
segment.name.template in the table config (JSON) |
Value the generator receives | Segment name |
|---|---|---|
"\\${filePathPattern:\\1}" |
\${filePathPattern:\1} (not resolved, does not start with ${) |
\<file name> |
"${filePathPattern:\\1}" |
\1 |
\1 for every file |
"${filePathPattern:1}" |
1 |
1 for every file |
"${filePathPattern:$1}" |
$1 |
$1 for every file |
The \${filePathPattern:\1} form in the job specification docs is specific to the ingestion job spec YAML, which is rendered with Groovy's SimpleTemplateEngine where \$ is the escape. Table configs don't go through Groovy, so the backslash ends up in the segment name.
Workarounds:
- Released versions (up to 1.5.1): make sure the value is not entirely a
${...}expression by adding a literal prefix or suffix, e.g."segmentNameGenerator.configs.segment.name.template": "myTable_${filePathPattern:\\1}". - Current master (not yet released): a leading
$$skips the resolution (Support Kafka config providers in realtime table configs #19202), so"$${filePathPattern:\\1}"reaches the generator as${filePathPattern:\1}.
I'll also make the input file segment name generator reject a template that doesn't reference any match group of file.path.pattern, so this fails with a clear error instead of silently giving every segment the same name.
Hello,
I am unable to name my segments based on the file name.
In the table's ingestion config I set:
The segment name is generated with the expected name but with a leading '' character which is refused when writing the Blob on ADLS.
I tried some alternatives to be sure this is an issue with Pinot:
Thanks