[slurm-users] Is sacct not handling quotes properly?

[Michael Jennings]

On Wednesday, 04 May 2022, at 10:00:57 (-0700),
David Henkemeyer wrote:

>I am seeing what I think might be a bug with sacct.  When I do the
>following:
>
>
>*> sbatch --export=NONE --wrap='uname -a' --exclusive*
>*Submitted batch job 2869585*
>
>Then, I ask sacct for the SubmitLine, as such:
>
>
>
>
>*> sacct -j 2869586 -o
>"SubmitLine%-70"SubmitLine----------------------------------------------------------------------sbatch
>--export=NONE --wrap=uname -a --exclusive*
>
>As you can see, the quotes around 'uname -a' are gone.  Hence, that submit
>line is not a valid sbatch commandline:
>
>
>*> sbatch --export=NONE --wrap=uname -a --exclusivesbatch: error: Batch job
>submission failed: Invalid job array specification*
>
>Like I said, I suspect this is a bug with sacct.  But I want to make sure.
>Can I somehow "peek" inside the accounting DB to see if the quotes are
>there?
>
>Perhaps there is an interaction with my bash shell, that's stripping the
>quotes?  This seems unlikely to me.

It's not actually unlikely; that's exactly what's happening.  Check
out the BASH man page section on EXPANSION and look for the term
"quote removal."  It's part of shells' normal operations. :-)

The real question is in how the command is being stored; are you
seeing 4 words or 5?  Shells like BASH internally track token sets in
terms of "words," and those words can have embedded spaces.  But
embedded spaces and word-separation spaces are indistinguishable
without additional metadata.

In other words, what you're really asking are two questions:
  - Does the accounting DB store the submit line in a way that
    preserves embedded spaces in arguments?
  - If so, can that value be retrieved in a way that quotes those
    arguments with embedded spaces (possibly by quoting *all*
    arguments) in a way that can be reused as shell input?

We can see this in action with the following example using BASH's
support for arrays.  Just like with `$*` vs. `$@`, array expansion
unquoted or inside double-quotes expands to a single word with
elements separated by the first character of `$IFS` (space by default)
when `$*` or `${ARRAYNAME[*]}` is used but to individual separate
words instead when `$@` or `${ARRAYNAME[@]}` is used.  However, it's
impossible to see the difference between embedded single spaces inside
a word/element and the single spaces BASH puts between them when
they're displayed:

   $ (declare -ax TESTME=( 1 2 '3 4 5' 6 7 8 ) ; echo ${TESTME[*]} ; echo "${TESTME[@]}" ; declare -p TESTME)
   1 2 3 4 5 6 7 8
   1 2 3 4 5 6 7 8
   declare -ax TESTME=([0]="1" [1]="2" [2]="3 4 5" [3]="6" [4]="7" [5]="8")

Notice that the 1st 2 lines look identical to one another, even though
internally they're not.  As you can see in the final line, there are
still spaces in the 3rd array element (`[2]="3 4 5"`).

The distinction becomes clearer if more than one space at a time is
embedded into the 3rd element, like so:

   $ (declare -ax TESTME=( 1 2 '3    4    5' 6 7 8 ) ; echo ${TESTME[*]} ; echo "${TESTME[@]}" ; declare -p TESTME)
   1 2 3 4 5 6 7 8
   1 2 3    4    5 6 7 8
   declare -ax TESTME=([0]="1" [1]="2" [2]="3    4    5" [3]="6" [4]="7" [5]="8")

So if you want to find the answers to the above questions (at least
the 1st one), pass 'uname    -a' instead of 'uname -a' and see what
you get back from your `sacct` query! :-)

HTH,
Michael

--
Michael E. Jennings <mej at lanl.gov> - [PGPH: he/him/his/Mr]  --  hpc.lanl.gov
HPC Systems Engineer   --   Platforms Team   --  HPC Systems Group (HPC-SYS)
Strategic Computing Complex, Bldg. 03-2327, Rm. 2341    W: +1 (505) 606-0605
Los Alamos National Laboratory,  P.O. Box 1663,  Los Alamos, NM   87545-0001