Version
main branch
Describe what's wrong
LocalJobExecutor.cancelJob stops a running job with Process.destroy(), which sends SIGTERM, and then marks the job CANCELLING. Two things can go wrong after that.
A job whose process ignores SIGTERM never exits. runJob keeps blocking in process.waitFor(), the job stays CANCELLING, and its worker thread is never released. The worker pool is fixed at maxRunningJobs, so each such job takes a slot for good; once every slot is held this way, new jobs stay queued until the server restarts.
A job whose process traps SIGTERM and exits 0 is reported SUCCEEDED. runJob checks the exit code before it checks whether the job was being cancelled, so a clean exit taken in response to the cancel wins over the cancel.
Error message and/or stacktrace
No error is raised. In the first case the job stays CANCELLING indefinitely; in the second it ends SUCCEEDED instead of CANCELLED.
How to reproduce
- Submit a shell job running
trap '' TERM; while :; do sleep 1; done, wait until it is STARTED, and cancel it. It never leaves CANCELLING.
- Submit a shell job running
trap 'exit 0' TERM; while :; do sleep 1; done, wait until it is STARTED, and cancel it. It ends SUCCEEDED.
Version
main branch
Describe what's wrong
LocalJobExecutor.cancelJobstops a running job withProcess.destroy(), which sends SIGTERM, and then marks the job CANCELLING. Two things can go wrong after that.A job whose process ignores SIGTERM never exits.
runJobkeeps blocking inprocess.waitFor(), the job stays CANCELLING, and its worker thread is never released. The worker pool is fixed atmaxRunningJobs, so each such job takes a slot for good; once every slot is held this way, new jobs stay queued until the server restarts.A job whose process traps SIGTERM and exits 0 is reported SUCCEEDED.
runJobchecks the exit code before it checks whether the job was being cancelled, so a clean exit taken in response to the cancel wins over the cancel.Error message and/or stacktrace
No error is raised. In the first case the job stays CANCELLING indefinitely; in the second it ends SUCCEEDED instead of CANCELLED.
How to reproduce
trap '' TERM; while :; do sleep 1; done, wait until it is STARTED, and cancel it. It never leaves CANCELLING.trap 'exit 0' TERM; while :; do sleep 1; done, wait until it is STARTED, and cancel it. It ends SUCCEEDED.