<div style="font-family: Arial, sans-serif; font-size: 14px;"><span>RE: Placing the full pathname of the job stdout in an environment variable</span></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><span><br></span></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><span>Would others find it useful if new variables were added that contained the full pathnames of the standard input, error and input files of batch jobs?<br></span><div><br></div><div><span>## SYNOPSIS</span></div><div><br></div><div><span>Proposed new environment variables SLURM_STDOUT,SLURM_STDERR, and</span></div><div><span>SLURM_STDIN pointing to the full pathnames of the output, error output,</span></div><div><span>and input files of a task would be a useful feature for batch jobs</span></div><div><span>in particular.</span></div><div><br></div><div><span>## PROBLEM</span></div><div><br></div><div><span>There are cases where it is desirable to have a job process the standard</span></div><div><span>files (stdin, stdout, stderr) from a batch job from within the job</span></div><div><span>instead of later via an epilog or via post-processing.</span></div><div><br></div><div><span>Just a few examples where the job might want to reference the full</span></div><div><span>pathname of stdout:</span></div><div><br></div><div><span> * copy the file to a remote machine (via scp(1), for example)</span></div><div><span> * move the file from local or scratch space to permanent globally</span></div><div><span> accessible storage</span></div><div><span> * remove the file depending on completion status</span></div><div><span> * mail the output </span></div><div><span> * archive it </span></div><div><span> * post-process it</span></div><div><br></div><div><span>Slurm commands like scontrol(1) and squeue(1) can help you locate the</span></div><div><span>file names. But the names returned do not always expand all the macros</span></div><div><span>allowed when specifying the names, and require calling a Slurm command</span></div><div><span>(which might overload Slurm if millions of jobs are submitted or cause</span></div><div><span>problems if Slurm is not responding for any reason(?)). For</span></div><div><span>example:</span></div><div><br></div><div><span> #!/bin/bash</span></div><div><span> # Minimalist submit script for sbatch(1)</span></div><div><span> #SBATCH --nodes 1-1 --ntasks=1 --time 0-0:1:00 --chdir=/tmp</span></div><div><span> #SBATCH --output=stdout.%j </span></div><div><span> #SBATCH --error=stderr.%A:%a:%J:%j:%N:%n:%s:%t:%u.out</span></div><div><span> # query filenames via squeue(1)</span></div><div><span> export SLURM_STDOUT=$(squeue --noheader --Format=STDOUT: --job=$SLURM_JOBID)</span></div><div><span> export SLURM_STDERR=$(squeue --noheader --Format=STDERR: --job=$SLURM_JOBID)</span></div><div><span> # PS: Current documentation for squeue(1) does not explicitly tell the</span></div><div><span> # user a null size works.</span></div><div><span> # query filenames via scontrol(1)</span></div><div><span> declare -x $(scontrol show job=$SLURM_JOBID|grep StdOut)</span></div><div><span> declare -x $(scontrol show job=$SLURM_JOBID|grep StdErr)</span></div><div><span> cat <<EOF</span></div><div><span> SLURM_STDOUT=$SLURM_STDOUT</span></div><div><span> SLURM_STDERR=$SLURM_STDERR</span></div><div><span> StdOut=$StdOut</span></div><div><span> StdErr=$StdErr</span></div><div><span> EOF</span></div><div><span> ls stdout.* stderr.*</span></div><div><br></div><div><span>The resulting output shows either none or some macros are expanded by</span></div><div><span>the commands (and I am not sure all commands will always return a full pathname(?))</span></div><div><br></div><div><span> SLURM_STDOUT=/home/urbanjs/venus/V600/stdout.%j</span></div><div><span> SLURM_STDERR=/home/urbanjs/venus/V600/stderr.%A:%a:%J:%j:%N:%n:%s:%t:%u.out</span></div><div><span> StdOut=/home/urbanjs/venus/V600/stdout.96</span></div><div><span> StdErr=/home/urbanjs/venus/V600/stderr.96:4294967294:%J:96:%N:%n:%s:%t:urbanjs.out</span></div><div><span> stderr.96:4294967294:96:96:mercury:0:4294967294:0:urbanjs.out</span></div><div><span> stdout.96</span></div><div><br></div><div><span>One currently available work-around would be that the user just avoid</span></div><div><span>the filename macros and always specify the filenames using a standard</span></div><div><span>convention (which would obviously have to be specific to a specific</span></div><div><span>platform). This is error-prone as it requires the user to strictly follow</span></div><div><span>a protocol where the pathnames are specified that can be overridden by</span></div><div><span>command-line options on sbatch(1) and so-on.</span></div><div><br></div><div><span>Alternatively instead of the new environment variables Slurm could have</span></div><div><span>a command option that returns the fully expanded names always expanding the</span></div><div><span>macros.</span></div><div><br></div><div><span>But if you query Slurm you have to worry about scaleability. Is it</span></div><div><span>OK for a 100 000, jobs to query Slurm simulataneously via squeue(1)</span></div><div><span>or scontrol(1)? What happens if for external reasons the Slurm daemons</span></div><div><span>are down or not responding?</span></div><div><br></div><div><span>As a current work-around using commands like realpath(1), stat(1),</span></div><div><span>ls(1),find(1),and getting pids with fuser(1), pidof(1), and ps(1) become</span></div><div><span>attractive. I find using the realpath(1) command at the top of the job</span></div><div><span>and $SLURM_TASK_ID works the best.</span></div><div><br></div><div><span>realpath(1) may not be available on all platforms(?) as well as the</span></div><div><span>/proc/$PID/fd/1 file, in which case the other commands might be used to</span></div><div><span>the same effect but it would be nice to have a simple standard way that</span></div><div><span>does not require calling a command. The proposed environment variables</span></div><div><span>seem the most obvious solution.</span></div><div><br></div><div><span> # do this early in the job in case the user changes or closes </span></div><div><span> # file descriptor 1 ...</span></div><div><span> export SLURM_STDOUT=$(realpath /proc/$SLURM_TASK_PID/fd/1)</span></div><div><span> # create softlink OUTPUT to stdout of job</span></div><div><span> ln -s -f $SLURM_STDOUT OUTPUT</span></div><div><span> # now user can do things like mailx(1) or scp(1) the OUTPUT file.</span></div><div><br></div><div><span>Other reasons I might want to know the stdout pathname are to minimize</span></div><div><span>network traffic and filespace on central servers. I might specify the</span></div><div><span>output of a job goes to scratch or local storage or a memory-resident file</span></div><div><span>system and move it all to a location on another system or a long-term</span></div><div><span>archive instead of a central area that might be filled by other jobs or</span></div><div><span>that I have a quota on, etc.</span></div><div><br></div><div><span>So since the stdout might not be easily accessible from my other sessions</span></div><div><span>also having something like "scontrol write batch_stdout=$SLURM_JOBID"</span></div><div><span>that shows the stdout of the job would also be useful. Note the LSF job</span></div><div><span>scheduler bpeek(1) command allows for a tail(1)-like interface to the</span></div><div><span>job stdout to support the stdout not being accessible from all system</span></div><div><span>access points, just as described.</span></div><div><br></div><div><span>The current situation where in typical usage the output file is often in</span></div><div><span>a globally mounted area and I can just access it from other nodes with</span></div><div><span>grep(1), tail(1), and so on covers many users needs in simple cluster</span></div><div><span>configurations; but there are several scenarios as described above</span></div><div><span>primarily encountered by people running many thousands of single-node jobs</span></div><div><span>where assigning output to a local scratch device and then selectively</span></div><div><span>processing at job termination is preferable. As mentioned, those users</span></div><div><span>can specify such a local name with "#SBATCH --output" and then, knowing</span></div><div><span>it, process it as desired but having something like $SLURM_STDOUT being</span></div><div><span>the full pathname to the file is much more generic.</span></div><div><br></div><div><span>## SUMMARY</span></div><div><br></div><div><span>So adding the variables creates an ability for users to reliably access</span></div><div><span>the standard files of a job with a low-load scalable method; and being</span></div><div><span>a SLURM_* module it could be much more reliably accessed via prolog and</span></div><div><span>epilog scripts and module(1) scripts as well.</span></div><div><br></div><div><span>## PS:</span></div><div><br></div><div><span>Related to this would be instead of just --chdir a --mkdir and --scratch</span></div><div><span>#SBATCH option would be useful that expanded file macros that would be</span></div><div><span>similar to --chdir but would make sure the directory existed (using only</span></div><div><span>user privelages) and a --scratch that would do the same as --mkdir but</span></div><div><span>remove the directory at job termination.</span></div><div><span>urbanjs@mercury:~/SLURM/proposals$ </span></div><div><br></div><div><br></div><div><br></div><span></span><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div>
<div class="protonmail_signature_block" style="font-family: Arial, sans-serif; font-size: 14px;">
<div class="protonmail_signature_block-user protonmail_signature_block-empty">
</div>
<div class="protonmail_signature_block-proton">
Sent with <a target="_blank" href="https://proton.me/" rel="noopener noreferrer">Proton Mail</a> secure email.
</div>
</div>