<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><meta http-equiv="Content-Type" content="text/html; charset=us-ascii" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Could be this quote from the srun man page:<blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class="">-u, --unbuffered<br class="">By default the connection between slurmstepd and the user launched application is over a pipe. The stdio output written by the application is buffered by the glibc until it is flushed or the output is set as unbuffered. See setbuf(3). If this option is specified the tasks are executed with a pseudo terminal so that the application output is unbuffered. This option applies to step allocations.</div></blockquote><div class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Feb 10, 2021, at 7:14 PM, Maria Semple <<a href="mailto:maria@rstudio.com" class="">maria@rstudio.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div dir="ltr" class="">The larger cluster is using NFS. I can see how that could be related to the difference of behaviours between the clusters.<div class=""><br class=""></div><div class=""> The buffering behaviour is the same if I tail the file from the node running the job. The only thing that seems to change the behaviour is whether I use <font face="monospace" class="">srun</font> to create a job step or not.</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 10, 2021 at 4:09 PM Aaron Jackson <<a href="mailto:Aaron.Jackson@nottingham.ac.uk" class="">Aaron.Jackson@nottingham.ac.uk</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Is it being written to NFS? You say on your local dev cluster it's a<br class="">
single node. Is it also the login node as well as compute? In that case<br class="">
I guess there is no NFS. Larger cluster will be using some sort of<br class="">
shared storage, so whichever shared file system you are using likely has<br class="">
caching.<br class="">
<br class="">
If you are able to connect directly to the node which is running the<br class="">
job, you can try tailing from there. It'll likely update immediately if<br class="">
what I said above is the case.<br class="">
<br class="">
Cheers,<br class="">
Aaron<br class="">
<br class="">
<br class="">
On 9 February 2021 at 23:47 GMT, Maria Semple wrote:<br class="">
<br class="">
> Hello all,<br class="">
><br class="">
> I've noticed an odd behaviour with job steps in some Slurm environments.<br class="">
> When a script is launched directly as a job, the output is written to file<br class="">
> immediately. When the script is launched as a step in a job, output is<br class="">
> written in ~30 second chunks. This doesn't happen in all Slurm<br class="">
> environments, but if it happens in one, it seems to always happen. For<br class="">
> example, on my local development cluster, which is a single node on Ubuntu<br class="">
> 18, I don't experience this. On a large Centos 7 based cluster, I do.<br class="">
><br class="">
> Below is a simple reproducible example:<br class="">
><br class="">
> loop.sh:<br class="">
> #!/bin/bash<br class="">
> for i in {1..100}<br class="">
> do<br class="">
> echo $i<br class="">
> sleep 1<br class="">
> done<br class="">
><br class="">
> withsteps.sh:<br class="">
> #!/bin/bash<br class="">
> srun ./loop.sh<br class="">
><br class="">
> Then from the command line running sbatch loop.sh followed by tail -f<br class="">
> slurm-<job #>.out prints the job output in smaller chunks, which appears to<br class="">
> be related to file system buffering or the time it takes for the tail<br class="">
> process to notice that the file has updated. Running cat on the file every<br class="">
> second shows that the output is in the file immediately after it is emitted<br class="">
> by the script.<br class="">
><br class="">
> If you run sbatch withsteps.sh instead, tail-ing or repeatedly cat-ing the<br class="">
> output file will show that the job output is written in a chunk of 30 - 35<br class="">
> lines.<br class="">
><br class="">
> I'm hoping this is something that is possible to work around, potentially<br class="">
> related to an OS setting, the way Slurm was compiled, or a Slurm setting.<br class="">
<br class="">
<br class="">
-- <br class="">
Research Fellow<br class="">
School of Computer Science<br class="">
University of Nottingham<br class="">
<br class="">
<br class="">
<br class="">
This message and any attachment are intended solely for the addressee<br class="">
and may contain confidential information. If you have received this<br class="">
message in error, please contact the sender and delete the email and<br class="">
attachment. <br class="">
<br class="">
Any views or opinions expressed by the author of this email do not<br class="">
necessarily reflect the views of the University of Nottingham. Email<br class="">
communications with the University of Nottingham may be monitored <br class="">
where permitted by law.<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class="gmail_signature"><div dir="ltr" class=""><div class="">Thanks,<br class=""></div>Maria<br class=""></div></div>
</div></blockquote></div><br class=""><div class="">
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div class="" style="font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; line-height: normal; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div class="" style="color: rgb(0, 0, 100); font-size: 14px;"><font face="Arial" class=""><div ;="" margin-left:="" 0px;="" margin-right:="" 0px"="" class="" style="text-align: center; display: inline-block; width: 100px;"><br class="Apple-interchange-newline">____________</div><div class="" style="text-align: right; display: inline-block; width: 1px; margin-left: -3px;"></div><div class=""></div><div class="" style="display: inline-block; width: 1px; margin-right: 0px;">|</div><div ;="" margin-left:="" 0px;="" margin-right:="" 0px"="" class="" style="text-align: center; display: inline-block; width: 100px;"></div><div class="" style="text-align: right; display: inline-block; width: 1px; margin-left: -3px;">|</div><div class=""></div><div class="" style="display: inline-block; width: 1px; margin-right: 0px;">|</div><div ;="" margin-left:="" 0px;="" margin-right:="" 0px"="" class="" style="text-align: center; display: inline-block; width: 100px;"><b class="">U.S. NAVAL</b></div><div class="" style="text-align: right; display: inline-block; width: 1px; margin-left: -3px;">|</div><div class=""></div><div class="" style="display: inline-block; width: 1px; margin-right: 0px;">|</div><div ;="" margin-left:="" 0px;="" margin-right:="" 0px"="" class="" style="text-align: center; display: inline-block; width: 100px;">_<b class="">RESEARCH</b>_</div><div class="" style="text-align: right; display: inline-block; width: 1px; margin-left: -3px;">|</div><div class=""></div><div class="" style="display: inline-block; width: 1px; margin-right: 0px;"></div><div ;="" margin-left:="" 0px;="" margin-right:="" 0px"="" class="" style="text-align: center; display: inline-block; width: 100px;"><span class="" style="font-size: 12px;">LABORATORY</span></div><div class="" style="text-align: right; display: inline-block; width: 1px; margin-left: -3px;"></div><div class=""></div></font></div><div class=""><br class="webkit-block-placeholder"></div><div class="">Noam Bernstein, Ph.D.</div></div><div class="" style="font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; line-height: normal;">Center for Materials Physics and Technology</div><div class="" style="font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; line-height: normal;">U.S. Naval Research Laboratory</div><div class="" style="font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; line-height: normal;">T +1 202 404 8628 F +1 202 404 7546</div><div class="" style="font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; line-height: normal;"><a href="https://www.nrl.navy.mil/" class="">https://www.nrl.navy.mil</a></div></div>
</div>
<br class=""></div></div></div></body></html>