Segmentation fault in rnaseq_collapse

Dear EGAPx Team,

I have recently been using EGAPx v0.5.2 for genome annotation with approximately 300 RNA-seq datasets. Most steps run well, but we consistently encounter an error during the rnaseq_collapse step.

I am not sure what is causing this issue and would appreciate your advice on how to troubleshoot or resolve it.

Could you please let me know:

what the possible reasons for this error might be,
whether this could be related to the large number of RNA-seq samples,
and what information or log files would be most useful for debugging?

Thank you very much for your help.

Erro like this:
76063dc00000-76063dc04000 r-xp 00000000 00:73 357133357 76063dc04000-76063de03000 ---p 00004000 00:73 357133357 76063de03000-76063de04000 r--p 00003000 00:73 357133357 76063de04000-76063de05000 rw-p 00004000 00:73 357133357 76063de05000-76063de06000 rw-p 00005000 00:73 357133357 76063de07000-76063de11000 rw-p 00000000 00:00 0
76063de11000-76063de12000 r--p 00000000 00:73 356079471 76063de12000-76063de13000 r-xp 00001000 00:73 356079471 76063de13000-76063de14000 r--p 00002000 00:73 356079471 76063de14000-76063de15000 r--p 00002000 00:73 356079471 76063de15000-76063de16000 rw-p 00003000 00:73 356079471 76063de16000-76063de18000 rw-p 00000000 00:00 0
76063de18000-76063de19000 r--p 00000000 00:73 356079473 76063de19000-76063de1a000 r-xp 00001000 00:73 356079473 76063de1a000-76063de1b000 r--p 00002000 00:73 356079473 76063de1b000-76063de1c000 r--p 00002000 00:73 356079473 76063de1c000-76063de1d000 rw-p 00003000 00:73 356079473 76063de1d000-76063de1e000 r--p 00000000 00:73 356079406 76063de1e000-76063de1f000 r-xp 00001000 00:73 356079406 76063de1f000-76063de20000 r--p 00002000 00:73 356079406 76063de20000-76063de21000 r--p 00002000 00:73 356079406 76063de21000-76063de22000 rw-p 00003000 00:73 356079406 76063de23000-76063de50000 rw-p 00000000 00:00 0
76063de50000-76063de52000 r--p 00000000 00:00 0 76063de52000-76063de54000 r--p 00000000 00:00 0 76063de54000-76063de56000 r-xp 00000000 00:00 0 76063de56000-76063de57000 r--p 00000000 00:73 356079372 76063de57000-76063de82000 r-xp 00001000 00:73 356079372 76063de82000-76063de8c000 r--p 0002c000 00:73 356079372 76063de8c000-76063de8e000 r--p 00036000 00:73 356079372 76063de8e000-76063de90000 rw-p 00038000 00:73 356079372 7ffce5f4b000-7ffce5f6c000 rw-p 00000000 00:00 0 ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 .command.sh: line 8100: 164 Segmentation fault ++ kill 28 /img/root/lib/libSegFault.so
/img/root/lib/libSegFault.so
/img/root/lib/libSegFault.so
/img/root/lib/libSegFault.so
/img/root/lib/libSegFault.so
/usr/lib/x86_64-linux-gnu/libpthread.so.0
/usr/lib/x86_64-linux-gnu/libpthread.so.0
/usr/lib/x86_64-linux-gnu/libpthread.so.0
/usr/lib/x86_64-linux-gnu/libpthread.so.0
/usr/lib/x86_64-linux-gnu/libpthread.so.0
/usr/lib/x86_64-linux-gnu/librt.so.1
/usr/lib/x86_64-linux-gnu/librt.so.1
/usr/lib/x86_64-linux-gnu/librt.so.1
/usr/lib/x86_64-linux-gnu/librt.so.1
/usr/lib/x86_64-linux-gnu/librt.so.1
/usr/lib/x86_64-linux-gnu/libdl.so.2
/usr/lib/x86_64-linux-gnu/libdl.so.2
/usr/lib/x86_64-linux-gnu/libdl.so.2
/usr/lib/x86_64-linux-gnu/libdl.so.2
/usr/lib/x86_64-linux-gnu/libdl.so.2
[vvar]
[vvar_vclock]
[vdso]
/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
[stack]
[vsyscall]
(core dumped) rnaseq_collapse -backlog 1 -max-jobs 1 -support-non-sra -O tmp/interim -nogenbank -lds2 tmp/genome_lds -sorted-vols align.mft -scaffold-list scaffold_list.mft -sra-metadata-manifest metadata.mft -start-job-id $start_job_id -input-jobs job.003 -workers 32

Work dir:
/data/lidong/81.egapx/workdir/03/41743b9ff99534979b8d214fe16c72

Container:
ncbi/egapx:0.5.2

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run

-- Check '/data/lidong/81.egapx/TJ.300sample/nextflow/nextflow.log' file for details

I located this erro here:
00164/001/0078/RB 04AD00A4A055FF51 0264/0007 2026-05-14T05:39:10.790054 b58912a6c587 UNK_CLIENT UNK_SESSION rnaseq_collapse request-start internal_thread=monitor&cpu_count=512&normalized_load=24&total_ram=2434550063104&effective_ram=2432402579456&pct_ram_used=0&mem_total=666701824&mem_peak=666701824&mem_self=666701824&mem_available=2431735877632&ncbi_app_version=0.0.1975&ncbi_app_sc_version=29&ncbi_app_vcs_revision=709142&ncbi_app_revision=709142
00164/001/0078/RE 04AD00A4A055FF51 0265/0008 2026-05-14T05:39:10.790128 b58912a6c587 UNK_CLIENT UNK_SESSION rnaseq_collapse request-stop 200 0.000108957 0 0
00164/032/0069/R 04AD00A4A055FF51 0266/0003 2026-05-14T05:39:12.625093 b58912a6c587 UNK_CLIENT UNK_SESSION rnaseq_collapse extra num_skipped=172
00164/031/0066/R 04AD00A4A055FF51 0267/0003 2026-05-14T05:39:12.625419 b58912a6c587 UNK_CLIENT UNK_SESSION rnaseq_collapse extra num_skipped=172
00164/033/0071/R 04AD00A4A055FF51 0268/0003 2026-05-14T05:39:12.625911 b58912a6c587 UNK_CLIENT UNK_SESSION rnaseq_collapse extra num_skipped=458
00164/034/0073/R 04AD00A4A055FF51 0269/0003 2026-05-14T05:39:12.627176 b58912a6c587 UNK_CLIENT UNK_SESSION rnaseq_collapse extra num_skipped=458
00164/022/0047/R 04AD00A4A055FF51 0270/0003 2026-05-14T05:39:12.627924 b58912a6c587 UNK_CLIENT UNK_SESSION rnaseq_collapse extra num_skipped=617
00164/025/0054/R 04AD00A4A055FF51 0271/0003 2026-05-14T05:39:12.628333 b58912a6c587 UNK_CLIENT UNK_SESSION rnaseq_collapse extra num_skipped=1267
00164/023/0049/R 04AD00A4A055FF51 0272/0003 2026-05-14T05:39:12.630205 b58912a6c587 UNK_CLIENT UNK_SESSION rnaseq_collapse extra num_skipped=617
00164/005/0010/R 04AD00A4A055FF51 0273/0003 2026-05-14T05:39:12.631312 b58912a6c587 UNK_CLIENT UNK_SESSION rnaseq_collapse extra num_skipped=1637
00164/025/0054/R 04AD00A4A055FF51 0274/0004 2026-05-14T05:39:12.633139 b58912a6c587 UNK_CLIENT UNK_SESSION rnaseq_collapse Error: SERIAL(CSerialException::eFormatError) "objistrasnb.cpp", line 197: ncbi::CObjectIStreamAsnBinary::UnexpectedContinuation() --- byte 928528: EndOfContentsByte expected ( at Seq-align.segs)
00164/025/0054/R 04AD00A4A055FF51 0275/0005 2026-05-14T05:39:12.633170 b58912a6c587 UNK_CLIENT UNK_SESSION rnaseq_collapse Error: LIB "wn_worker_thread.cpp", line 267: ncbi::CWorkerThread::x_DoJob() --- error processing job: (CSerialException::eFormatError) byte 928528: EndOfContentsByte expected ( at Seq-align.segs)
00164/025/0054/RE 04AD00A4A055FF51 0276/0006 2026-05-14T05:39:12.633757 b58912a6c587 UNK_CLIENT UNK_SESSION rnaseq_collapse request-stop 500 10.840983152 0 40
00164/025/0079/RB 04AD00A4A055FF51 0277/0007 2026-05-14T05:39:12.633793 b58912a6c587 UNK_CLIENT UNK_SESSION rnaseq_collapse request-start ncbi_app_version=0.0.1975&ncbi_app_sc_version=29&ncbi_app_vcs_revision=709142&ncbi_app_revision=709142
00164/025/0079/R 04AD00A4A055FF51 0278/0008 2026-05-14T05:39:12.633839 b58912a6c587 UNK_CLIENT UNK_SESSION rnaseq_collapse extra job_id=6321&affinity=lcl|chr14&input=<job+offs%3D"30180608"+len%3D"121696"+subject%3D"lcl|chr14">
*** Segmentation fault

command.log This is the log file.

Just some update: If I resume the pipeline and I have changed the memory in config to:

params.threads = 16
params.nodes = 1
params.num_cpus_per_node = 512

process {

}
process {
memory = 60.GB
cpus = 7
ext.align_mem = '13'
time = 6.h
errorStrategy = 'retry'
maxRetries = 3

withLabel: 'long_job' {
    time = 36.h
}
withLabel: 'small_mem' { 
    memory = 16.GB
}
withLabel: 'med_mem' {
    memory = 512.GB
}
withLabel: 'large_mem' { 
    memory = 1024.GB
}

Now the erro is like the structure file is wrong. I am not sure why this happend. This is the latest log file here. Thank you.

Best regards,
Dong

command2.log

Can you zip the outdir/nextflow directory and post it here? Is that possible?

Did you download the RNASeq data from SRA, or elsewhere? Or was it local? What kind of id's are in those files?

Dear EGAPx Team,

I am Dong's collaborator, and I would like to provide details of the error.

We used 300 pre-downloaded RNA-seq samples stored locally for annotation. Each time the command ran to the step "egapx:rnaseq_short_plane:rnaseq_collapse:run_rnaseq_collapse", the error message

.command.sh: line 8100: 164 Segmentation fault (core dumped) rnaseq_collapse -backlog 1 -max-jobs 1 -support-non-sra -O tmp/interim -nogenbank -lds2 tmp/genome_lds -sorted-vols align.mft -scaffold-list scaffold_list.mft -sra-metadata-manifest metadata.mft -start-job-id $start_job_id -input-jobs job.002 -workers 16 ++ kill 28

We initially thought the error was caused by the excessive computational resources required for annotating 300 transcriptomes using version 0.5.2. So we tested with three transcriptomes and obtained the same error.

It's worth mentioning that we used version 0.3.2 and didn't have this problem at all.

I've attached the incorrect file below. Looking forward to your advice.

Best regards
yulong

RNA.Protein.yaml

3sample.tar.gz

300sample.tar.gz

Thanks for sending the files.
What do the seqid's look like in the read file?
/home/lidong/lidong/81.egapx/RNA/1824262_reads_R1.fastq.gz

@pstrope

Hi, @pstrope. The file looks like:

@ERR1824262.1 HISEQ:38:H7E1RADXX:1:1101:1095:2087/1
AGCANGGAGAAAAGGTCGTTAGCCCTTCCTCAGAAAGGGTGCCTGGGATGCAGGGCAGCTGAAACAGCAACAGATGTTCAAAGATTGCTTGCATCTCCGT
+
CCCF#2ADHHHHHJJEGIIJIIIJJJJJJJJJJJJJHHHDGHIJJJJJJJJJJJJJJHHHHFFFFFFEEEEDDDDDDDEEEDCDDDDDDDDDDEDDDDDD
@ERR1824262.3 HISEQ:38:H7E1RADXX:1:1101:1211:2193/1
GGCTATTCACAGGCGCGATCCCACTACTGATCAGCACGGGAGTTTTGACCTGCTCCGTTTCCGACCTGGGCCGGTTCACCCCTCCTTAGGCAACCTGGTG
+
CCCFFFFFHHHGHJJJJJJJJJJJJJJJJJJJJJJJJJJJGJFHIIGGIJIJHGHHGFFFFFDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDCC
@ERR1824262.4 HISEQ:38:H7E1RADXX:1:1101:1086:2221/1
CAATNGTAGAGGGTTATTTGTGCGCGATACAAGCTTTGTAAGGCATGTAGCTGCCAGTAACTGGGAGTAAGATGAACTTCCTCTTTCGAGGAGTAGCTGG
+
CCCF#2=BFHHHHEGHIJIIHIGGGGHIIJIHHJJJJJIGIGHIJJJJH@FHIHJJJHHHHHHHHFFBCEEEEEEDDDDDDDDDDDDDDDDDDDCCDDDD
@ERR1824262.5 HISEQ:38:H7E1RADXX:1:1101:1106:2228/1
CTGTAATGGTGGGTACGTGACATTGTGTTTTCTGTCAAAATCCAATAAACGTAAACTATGAACTTTAGTATTGATGTATGAATACTGATCCAATTAAAAT
+
@CCDDFFFHDFHDCFHEEHHHIHIHHEFGHHIGFIIIIJJGGGIJJJJJIDHGIIGGEGFHIJJGJIH=DGHA=EEHFEEBDDCFFBCACEEDCDDDDDE
@ERR1824262.6 HISEQ:38:H7E1RADXX:1:1101:1348:2063/1
NGTCACCTGATGATAACCAAAAAAACCCCACAAAGTTTCCTAGTAAGTGGTAAAGGCTTTAACTAGATAAAAAGCAACGTGGTTCACTAGCGTTAAAGAT
+
#1:BDFFFGHHHHJJJJJJJJIIJJJJJJGIJJJJFHIHGJJIFHGIIHJ@FEGHIJJJHGHHHGFFFFFFFEDEEDDDBBDBBDDDDDDDD>BBDDDDD
@ERR1824262.7 HISEQ:38:H7E1RADXX:1:1101:1451:2063/1
NCAGGTATTTTATTTACATAGCTTCTCATTTATTCCTAACAAACCTGTGAAGCAGACAGTATTCTCTCCATTACACAAGTGAAGAAACGGAAGCTCAACA
+
#1=DDBDFHHHHHJJJJJJJIJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJHIJJJJJIJJJJJJJIJJJJHGHHHHHHFFFDDDDDDDDDD
@ERR1824262.8 HISEQ:38:H7E1RADXX:1:1101:1468:2068/1
NCCTACACCATAGCTCATGGCAACGCCAGATCCCTAAGCCACTGTTTGAGGCCAGGGATCAAACCTGCATCCTCATGGATACTAGTCAGATTGGTTTCTG
+
#1:BDDFEHHGHGIGGIGGEIIEIGHIJJIJIIGEHGEEHIIFEDGBC=CEHIIJIIIEHHEEHFB@BCCEEDECCCDCDDDCDC@CDCAACDDCBBDCC
@ERR1824262.9 HISEQ:38:H7E1RADXX:1:1101:1451:2089/1
CCATACATGACCTCCATGAACTTCTCATCGGCATCCTGAAAGCGCTGATCCAGGAAGGAGCTTGTCTTCCGCACCAGGTTCCAGATCATGTAGTTGTTCA
+
CCCFFFFFHHHHHJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJIJJIIJJJJJJJIJIJJHHHHHFFFFDDDDDDDDCDDEDCDCDDDDEEDDDCDDE
@ERR1824262.10 HISEQ:38:H7E1RADXX:1:1101:1267:2114/1
CTATGGTGCACAAGTACCCAGGAGAGTATCTTTCCATTGCGCAGCTTGAACTCTGACCACAGGTAAAGAGCCCCTGAGATTGTCAGAGCTGACCCAACTT
+
CCCFFFDFHHHHHJIIJJJJJJJJJJHHIJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJEEIHHHHHFFFFDEEEEDDDEEDDDDDDDDDDDDDDD
@ERR1824262.11 HISEQ:38:H7E1RADXX:1:1101:1336:2124/1
CTGCAATCATTTAACATCTCCTCTGAGCACAATGATCTACAAGCAGAAAATGAACAAACCTCCTTCCACAAAGTCCTTGGTTCTAACCCAGAAGCACTCT
+
CCCFFFFFHHHHHJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJIIJJJJJJJJJJJJJJJJJJJJJHHHHHHHFFFFFFEEDDDDDDDDDDDD
@ERR1824262.12 HISEQ:38:H7E1RADXX:1:1101:1431:2153/1
CGGGCGGGGAGGCGCCCGCAGAGGGGCGGGAAGGGCGAGAACCGGGAGAGAAGCCCAGGCCCCCGACCCGGCAGCGGCGATGTCCTTAAGCCCAGAGTCT
+
CCCFFFFFHHHHHFFDDDDDDDDDDDDDDDBD@DDBBB@BDCD@DD8@8<?C?CDDBBDDDDDD;;BDD>B@9>B>BB@39@ACD>CACAACD?A@A:AC
@ERR1824262.13 HISEQ:38:H7E1RADXX:1:1101:1256:2163/1
TGTAGTTCTAGTGGAAGCAATGCTTTCTCTTCTCGGGTCTTCTGAGAGCTACCAGCTGGGGTGCTGACCTTTCGAGGCCTCAAGCTCTTAAGAGGCTCCT
+
@@@DDDDDHABCFE?<EDHHHE;FHCHHGIIIIDGB@G>BFD@<?BFCEI>FHDHFEIGHH-5?;=CEHDEEE?;>>BBCBAACC@@@C:,:>@@?@?BC
@ERR1824262.14 HISEQ:38:H7E1RADXX:1:1101:1258:2237/1
TCCCACAGCTGGTTGGAGATGGAGCGAGTCAGTGATGTGACTGCAAGCCTATCTCAACACACATCCCATGCTCATTTTCCCTTTTGTTTTCATTGAAGTA

@ld9866

@ld9866 - Are you downloading your read files from EBI ERA?

With EGAP running here at NCBI, we've always run with data pulled from NCBI SRA, using the reads as FASTA with seq-ids formatted like gnl|SRA|ERR1824262.14. With user-provided files without the gnl|SRA| prefix, the seq-ids are handled a bit differently, which is fine for seq-ids from the machine but based on what you're seeing here and also the case on #245, it looks like SRR/ERR/DRR seq-ids without the gnl|SRA| are triggering an unexpected condition in the rnaseq_collapse code.

For your earlier run with v0.3.2, was that also with RNA-seq formatted like this?

Please try adding gnl|SRA| prefixes on a run. The user on #245 got their ERA data to work by adding an arbitrary string S01 which would be an alternative. Those two options will behave slightly differently in a technical sense, and there may be some slight advantages to using gnl|SRA| so I would try that first with just a run or two.

@murphyte

@murphyte Yes, the fastq file are download from EBI.

Thank you for the suggestion. We are currently testing the issue by modifying the FASTA headers following the approach described in #245. We will also try adding the gnl|SRA| prefix and compare the behavior across runs.

Our earlier run with v0.3.2 used RNA-seq data without the gnl|SRA| prefix as well.

We’ll update you once we complete the tests.

Best regards,
Dong

Navigation Menu

Repository navigation

Segmentation fault in rnaseq_collapse #246

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Segmentation fault in rnaseq_collapse #246

Description

Activity

ld9866 commented on May 14, 2026

ld9866 commented on May 14, 2026

ld9866 commented on May 14, 2026

pstrope commented on May 14, 2026

abcyulongwang commented on May 14, 2026

pstrope commented on May 14, 2026

ld9866 commented on May 14, 2026

murphyte commented on May 15, 2026

ld9866 commented on May 15, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions