This document describes the pipeline entry points, runtime configuration, commands, checkpoints, and generated files. Project motivation and benchmark design belong in the repository-level README.

• pipeline.sh: runs one staged local input through a selected pipeline. It executes the configured steps, writes per-step timing metrics, and maintains restart checkpoints.
• rnaseq-pipeline.sh: runs one remotely located RNA-seq input. It downloads the input, verifies its MD5 checksum, runs the same step implementation as pipeline.sh, and starts process-I/O sampling.
• rnaseq.sh: GCP batch wrapper. It reads split accession manifests, launches one Docker container per input, samples container and cgroup metrics, uploads result archives, and removes transient data.
• rnaseq.steps.sh: ordered RNA-seq step implementation sourced by both single-input runners.
• rnaseq.env: reference paths and runtime tool/version checks for direct execution.
• rnaseq.docker.env: reference paths used in the RNA-seq image.
• steps-io.sh: samples /proc/<pid>/io while a remote-input pipeline is running and writes JSON Lines output.
• scripts/rnaseqSummary.py: combines HISAT2 and featureCounts results into one QC summary JSON object.
• scripts/cgMetrics2Json.py: converts live cgroup CPU, memory, and I/O files into JSON.
• scripts/cgFinalMetrics2Json.py: converts final cgroup counters into JSON after a containerized pipeline finishes.
• ../build.sh: builds the Docker image defined by ../build/Dockerfile.rnaseq.

The pipeline currently supports rnaseq. Input type is inferred from the input path:

• a path containing lite is classified as lite;
• a path containing fastq is classified as fastq;
• every other path is classified as normalized SRA, named etl internally.

Set REFHOME before running locally if the reference set is not mounted at the default Docker path. rnaseq.env checks required programs and their versions when it is sourced.

For FASTQ input, provide a directory containing gzip-compressed FASTQ files. The expected layout is the output of fasterq-dump --split-3, followed by gzip compression. The first two files are treated as paired reads; additional files are supplied to HISAT2 as unpaired reads.

Use pipeline.sh when the SRA file or FASTQ directory is already available on the host. The script creates <work>/rnaseq/<accession>.<etl|lite|fastq>/ and runs the steps declared in rnaseq.steps.sh.

Normalized SRA:

./pipelines/pipeline.sh \
  --pipeline rnaseq \
  --work ./work \
  --input ./input/etl/SRR29735851.sra \
  --threads 8

SRA Lite:

./pipelines/pipeline.sh \
  --pipeline rnaseq \
  --work ./work \
  --input ./input/lite/SRR29735851.lite.sra \
  --threads 8

FASTQ directory:

./pipelines/pipeline.sh \
  --pipeline rnaseq \
  --work ./work \
  --input ./input/fastq/SRR29735851 \
  --threads 8

In these examples, pipeline.sh is the single-input local runner; rnaseq.steps.sh supplies the RNA-seq commands, and rnaseq.env supplies the reference paths and validates the installed tools.

Use rnaseq-pipeline.sh for an object in S3 or Google Cloud Storage. The current argument parser requires --md5 for every input type. Downloaded data is written below --dock.

Normalized SRA from S3:

./pipelines/rnaseq-pipeline.sh \
  --pipeline rnaseq \
  --work ./work \
  --dock ./dock \
  --rlocation s3://sra-pub-run-odp/sra/SRR29735851/SRR29735851 \
  --md5 2a71bd052e563db5c52968166e5253b7 \
  --threads 8

SRA Lite from Google Cloud Storage:

./pipelines/rnaseq-pipeline.sh \
  --pipeline rnaseq \
  --work ./work \
  --dock ./dock \
  --rlocation gs://sra-pub-zq-100/SRR29735851/SRR29735851.lite.1 \
  --md5 a908115f7176efae89a0eb95ba87a5d0 \
  --threads 8

rnaseq-pipeline.sh is the remote-input runner. It downloads and verifies the object before executing rnaseq.steps.sh; steps-io.sh runs alongside it to sample process I/O.

Each successful step creates <work>/rnaseq/<accession>.<kind>/<step>.done. Rerunning the same command skips completed steps and resumes at the first step without a checkpoint.

To force a completed step to run again, remove its .done file and the checkpoints for all dependent downstream steps. Existing outputs are not automatically invalidated.

Per-input files are written under <work>/rnaseq/<accession>.<kind>/. The primary outputs are:

• gene counts and the featureCounts gene summary;
• exon counts, the featureCounts exon summary, and exon-junction counts;
• 5-prime/3-prime decay-signature counts and summary;
• HISAT2 alignment summary and metrics;
• ordered, count-only vectors named gene.<accession>.<kind>.<take>.numbers and exon.<accession>.<kind>.<take>.numbers;
• <accession>.fc.summary, one JSON object containing RNA-seq QC metrics;
• <accession>.geo-fc.tar.gz, the packaged pipeline results.

Inspect QC summaries for all formats of an accession with:

grep -n acc work/rnaseq/SRR29735851.*/SRR29735851.fc.summary

grep is used only to display the one-line JSON summaries; it does not run or modify the pipeline.

Both single-input runners execute external steps through /usr/bin/time with the format %e %U %S %M %x. The tab-separated metrics include pipeline, accession, input kind, step, start time, elapsed time, user CPU time, system CPU time, maximum resident set size, exit status, and command.

pipeline.sh writes <work>/rnaseq/<accession>.<take>.metrics. rnaseq-pipeline.sh writes <work>/rnaseq/<accession>.<kind>.metrics and also starts steps-io.sh, which writes <accession>.<kind>.steps-io.jsonline.

Build a local image with:

./build.sh --pipeline rnaseq --version v0.0.2 --test

build.sh records the current Git branch, source revision, and build ID as image build arguments. --test stops after the local image is built.

Run one remote input in the image with:

docker run --rm \
  --name SRR29735851-lite \
  --env REFHOME=/static/refset/GCF_000001635.27.v2 \
  -v $PWD/work:/work \
  -v $PWD/dock:/input \
  -v <host-reference-root>:/static:ro \
  rnaseq \
  --work /work \
  --dock /input \
  --rlocation gs://sra-pub-zq-100/SRR29735851/SRR29735851.lite.1 \
  --md5 a908115f7176efae89a0eb95ba87a5d0 \
  --threads 8

The image entry point is rnaseq-pipeline.sh --pipeline rnaseq; arguments after the image name are passed to that script. The three mounts provide the work directory, transient input directory, and read-only reference data.

rnaseq.sh is intended to process a set of accessions on a single GCP VM. It expects:

• an active gcloud project;
• Docker and cgroup v2 files;
• reference data mounted at /mnt/static;
• split manifests under input/rnaseq-<kind>-split/;
• tab-separated manifest rows containing kind, accession, remote location, and MD5 checksum.

Run a selected range of manifest chunks with:

./pipelines/rnaseq.sh \
  --kind lite \
  --wdir /mnt/chunks \
  --take v0 \
  --threads 10 \
  --bgn 0 \
  --end 5

rnaseq.sh launches each chunk as a detached background job. It runs rnaseq-pipeline.sh through the Docker image, uses cgMetrics2Json.py and cgFinalMetrics2Json.py for cgroup measurements, and uploads archives to gs://<gcp-project>-sralitebenchmarks/rnaseq/.

Batch execution also writes:

• docker.stat, sampled Docker statistics in JSON Lines format;
• cgroup.metrics.jsonline, live cgroup samples;
• cgroup.final.metrics.jsonline, final cgroup counters;
• fc.summary.jsonline, accumulated QC summaries;
• collected gene and exon .numbers files under the chunk result directory.

Unless --keep is specified, successfully processed transient inputs are removed after their results are uploaded.

The split manifests are headerless, tab-separated files with four columns: kind accession remote_location md5 Columns:

• kind: etl, lite, or fastq
• accession: SRA accession
• remote_location: S3/GCS object path
• md5: expected checksum