Repository navigation
Unable to upload existing segments to a table #19384
Description
Activity
Do you want to help debug the code? Seems a valid case to me
Yep, here is what I saw:
SegmentUriPushJobRunner extends BaseSegmentPushJobRunner.
The method called to run the job is BaseSegmentPushJobRunner.run().
In this method, there is a call to SegmentPushUtils.getSegmentUriToTarPathMap(_outputDirURI, _spec.getPushJobSpec(), _files); to get the segments to upload via uploadSegments(segmentsUriToTarPathMap);.
This util method return only the URIs ending with .tar.gz:
if (uri.getPath().endsWith(Constants.TAR_GZ_FILE_EXT)) {
URI updatedURI = SegmentPushUtils.generateSegmentTarURI(outputDirURI, uri, pushSpec.getSegmentUriPrefix(),
pushSpec.getSegmentUriSuffix());
segmentUriToTarPathMap.put(updatedURI.toString(), file);
}
Ref:
This might be too restricted. Do you want to help contribute a fix?
Hello,
I try to upload some old segments produced by an REALTIME table to an OFFLINE table.
It is working as expected using the API as explained here: https://docs.pinot.apache.org/build-with-pinot/ingestion/batch-ingestion/segment-upload#uri-push
But the API is for 1 segment and I want to upload an entire "directory" with many segments. For this a job using Pinot Admin seems to be the solution.
I tried using "SegmentUriPush" but it does not work because my segment are named "myTable__0__0__20260601T1557Z" with no tar.gz extension.
But this extension seems to be required when the job is used (this is not the case using the API).
Am I doing something wrong or is it a limitation that can be fixed in a future version ?
I use Azure Data Lake Storage as a deep store.
Thanks for your help.