<div dir="ltr">Hi everyone,<div><br></div><div><p style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);margin:0px 0px 1.2em">I’m running into out-of-memory errors when I specify an array job. Needless to say, 100M should be more than enough, and increasing the allocated memory to 1G doesn't solve the problem. I call my script as follows:<span> </span><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">sbatch --array=100-199 run_batch_job</code>.<span> </span><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">run_batch_job</code><span> </span>c<wbr>ontains</p><pre style="white-space:pre-wrap;color:rgb(34,34,34);font-size:1em;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);font-family:Consolas,Inconsolata,Courier,monospace;line-height:1.2em;margin:1.2em 0px"><code class="gmail-m_4897992182702385426gmail-m_6636225074262047266hljs gmail-m_4897992182702385426gmail-m_6636225074262047266language-bash" style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;white-space:pre-wrap;overflow:auto;border-radius:3px;border:1px solid rgb(204,204,204);padding:0.5em;color:rgb(51,51,51);background:rgb(248,248,248);display:block"><span class="gmail-m_4897992182702385426gmail-m_6636225074262047266hljs-shebang" style="color:rgb(153,153,153);font-weight:bold">#!/bin/env bash</span>
<span class="gmail-m_4897992182702385426gmail-m_6636225074262047266hljs-comment" style="color:rgb(153,153,136);font-style:italic">#SBATCH --partition=lln</span>
<span class="gmail-m_4897992182702385426gmail-m_6636225074262047266hljs-comment" style="color:rgb(153,153,136);font-style:italic">#SBATCH --output=/home/user/outs/%x.<wbr>out.%a</span>
<span class="gmail-m_4897992182702385426gmail-m_6636225074262047266hljs-comment" style="color:rgb(153,153,136);font-style:italic">#SBATCH --error=/home/user/outs/%x.err<wbr>.%a</span>
<span class="gmail-m_4897992182702385426gmail-m_6636225074262047266hljs-comment" style="color:rgb(153,153,136);font-style:italic">#SBATCH --cpus-per-task=1</span>
<span class="gmail-m_4897992182702385426gmail-m_6636225074262047266hljs-comment" style="color:rgb(153,153,136);font-style:italic">#SBATCH --mem-per-cpu=100M</span>
<span class="gmail-m_4897992182702385426gmail-m_6636225074262047266hljs-comment" style="color:rgb(153,153,136);font-style:italic">#SBATCH --time=2-00:00:00</span>

srun my_program.out <span class="gmail-m_4897992182702385426gmail-m_6636225074262047266hljs-variable" style="color:rgb(0,128,128)">$SLURM_ARRAY_TASK_ID</span>
</code></pre><p style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);margin:0px 0px 1.2em">Instead of using<span> </span><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">--mem-per-cpu</code><span> </span>and<span> </span><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">--<wbr>cpus-per-task</code>, I’ve also tried the following:</p><pre style="white-space:pre-wrap;color:rgb(34,34,34);font-size:1em;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);font-family:Consolas,Inconsolata,Courier,monospace;line-height:1.2em;margin:1.2em 0px"><code class="gmail-m_4897992182702385426gmail-m_6636225074262047266hljs gmail-m_4897992182702385426gmail-m_6636225074262047266language-bash" style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;white-space:pre-wrap;overflow:auto;border-radius:3px;border:1px solid rgb(204,204,204);padding:0.5em;color:rgb(51,51,51);background:rgb(248,248,248);display:block"><span class="gmail-m_4897992182702385426gmail-m_6636225074262047266hljs-comment" style="color:rgb(153,153,136);font-style:italic">#SBATCH --mem=100M</span>
<span class="gmail-m_4897992182702385426gmail-m_6636225074262047266hljs-comment" style="color:rgb(153,153,136);font-style:italic">#SBATCH --ntasks=1  # Number of cores</span>
<span class="gmail-m_4897992182702385426gmail-m_6636225074262047266hljs-comment" style="color:rgb(153,153,136);font-style:italic">#SBATCH --nodes=1  # All cores on one machine</span>
</code></pre><p style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);margin:0px 0px 1.2em">But in both cases for some of the runs, I get the error:</p><pre style="white-space:pre-wrap;color:rgb(34,34,34);font-size:1em;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);font-family:Consolas,Inconsolata,Courier,monospace;line-height:1.2em;margin:1.2em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;background-color:rgb(248,248,248);white-space:pre-wrap;overflow:auto;border-radius:3px;border:1px solid rgb(204,204,204);padding:0.5em 0.7em;display:block">slurmstepd: error: Exceeded job memory limit at some point.
srun: error: obelix-cn002: task 0: Out Of Memory
slurmstepd: error: Exceeded job memory limit at some point.
</code></pre><p style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);margin:0px 0px 1.2em">I’ve also posted the question on<span> </span><a href="https://stackoverflow.com/questions/48763851/how-to-specify-memory-per-process-in-an-array-job-in-slurm" target="_blank" style="color:rgb(17,85,204)">stackoverflow</a>. Does anyone know what is happening here?</p><p style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);margin:0px 0px 1.2em">Kind regards,<br>Geert Kapteijns</p><div class="gmail-yj6qo" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"></div><br class="gmail-Apple-interchange-newline"></div><br></div>