<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">If you're on Linux and using Slurm cgroups, your job processes should be contained in a memory cgroup. The /proc/<pid>/cgroup file indicates to which cgroups a process is assigned, so:<div class=""><br class=""></div><div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class="">$ srun [...] /bin/bash -c "grep memory: /proc/\$\$/cgroup | sed 's%^[0-9]*:memory:%/sys/fs/cgroup/memory%'"</div><div class="">/sys/fs/cgroup/memory/slurm/uid_1001/job_459890/step_0/task_0</div></blockquote><div class=""><br class=""></div><div class=""><br class=""></div><div class="">The 'sed' above assumes cgroups are mounted under /sys/fs/cgroup. Check the usage by examining the "memory.usage_in_bytes" file; here's a script that includes a Bash function to backtrack to the "job_#" directory in order to get the overall usage for all steps/tasks on the node for the job:</div><div class=""><br class=""></div><div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class="">#!/bin/bash</div><div class=""><br class=""></div><div class="">function cgroup_job_dir() {</div><div class=""> local CGROUP_DIR="$1"</div><div class=""> while [[ $CGROUP_DIR != '/sys/fs/cgroup' && ! $CGROUP_DIR =~ /job_[0-9]+$ ]]; do</div><div class=""> CGROUP_DIR="$(dirname "$CGROUP_DIR")"</div><div class=""> done</div><div class=""> if [[ $CGROUP_DIR != '/sys/fs/cgroup' ]]; then</div><div class=""> echo "$CGROUP_DIR"</div><div class=""> return 0</div><div class=""> fi</div><div class=""> return 1</div><div class="">}</div><div class=""><br class=""></div><div class="">CGROUP_MEMORY="$(grep memory: /proc/$$/cgroup | sed 's%^[0-9]*:memory:%/sys/fs/cgroup/memory%')"</div><div class="">printf "%s: %s\n" "${CGROUP_MEMORY}/memory.usage_in_bytes" "$(cat "${CGROUP_MEMORY}/memory.usage_in_bytes")"</div><div class=""><br class=""></div><div class="">CGROUP_MEMORY="$(cgroup_job_dir "$CGROUP_MEMORY")"</div><div class="">if [ $? -eq 0 ]; then</div><div class=""> printf "%s: %s\n" "${CGROUP_MEMORY}/memory.usage_in_bytes" "$(cat "${CGROUP_MEMORY}/memory.usage_in_bytes")"</div><div class="">fi</div></blockquote><div class=""><br class=""></div><div class=""><br class=""></div><div class="">E.g. when using this with srun:</div><div class=""><br class=""></div><div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class="">$ srun [...] check_mem.sh </div><div class="">/sys/fs/cgroup/memory/slurm/uid_1001/job_459900/step_0/task_0/memory.usage_in_bytes: 462848</div><div class="">/sys/fs/cgroup/memory/slurm/uid_1001/job_459900/memory.usage_in_bytes: 610304</div></blockquote><div class=""><br class=""></div><div class=""><br class=""></div><div class="">or with sbatch:</div><div class=""><br class=""></div><div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class="">$ sbatch [...] check_mem.sh </div><div class="">Submitted batch job 459903</div><div class=""> :</div><div class="">$ cat slurm-459903.out </div><div class="">/sys/fs/cgroup/memory/slurm/uid_1001/job_459903/step_batch/task_0/memory.usage_in_bytes: 466944</div><div class="">/sys/fs/cgroup/memory/slurm/uid_1001/job_459903/memory.usage_in_bytes: 614400</div></blockquote><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class="">On Apr 1, 2019, at 2:28 AM, Bjørn-Helge Mevik <<a href="mailto:b.h.mevik@usit.uio.no" class="">b.h.mevik@usit.uio.no</a>> wrote:<br class=""><br class="">If the job is alone on its node(s), you can use "scontrol show node <nodes>" and<br class="">look at "RealMemory minus FreeMem".<br class=""><br class="">-- <br class="">B/H<br class=""></blockquote><br class=""><div class=""><br class="">::::::::::::::::::::::::::::::::::::::::::::::::::::::<br class="">Jeffrey T. Frey, Ph.D.<br class="">Systems Programmer V / HPC Management<br class="">Network & Systems Services / College of Engineering<br class="">University of Delaware, Newark DE 19716<br class="">Office: (302) 831-6034 Mobile: (302) 419-4976<br class="">::::::::::::::::::::::::::::::::::::::::::::::::::::::<br class=""><br class=""><br class=""><br class=""></div><br class=""></div></body></html>