How do I get the information available from `scontrol show job` in a format I can safely parse?
It was recently pointed out to me that `scontrol show job` will include jobs' comments, and those comments seem to be able to contain arbitrary text, including newlines and "=". The comments don't seem to be delimited or escaped in any way within the `scontrol show job` output. This means I can't actually parse the output of `scontrol show job`, because once a comment starts I can never tell that it has ended. Someone can put a whole valid-looking job record inside their job comment, or something that looks like the tail end of one job record and the beginning of another. Is there a way to adjust the output format or access that data, as a normal cluster user, through another interface? To illustrate the problem, I ran: sbatch --wrap "sleep 10" --comment="$(printf "\n\nJobID=12345\n\n")" And I found this in my `scontrol show job` output: JobId=38807436 JobName=wrap UserId=anovak1(30065) GroupId=prismuser(600) MCS_label=N/A Priority=4202 Nice=0 Account=standard QOS=normal JobState=RUNNING Reason=None Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:09 TimeLimit=00:10:00 TimeMin=N/A SubmitTime=2026-09-22T13:47:15 EligibleTime=2026-09-22T13:47:15 AccrueTime=2026-09-22T13:47:15 StartTime=2026-09-22T13:47:17 EndTime=2026-09-22T13:57:17 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-09-22T13:47:17 Scheduler=Backfill Partition=short AllocNode:Sid=phoenix:1297984 ReqNodeList=(null) ExcNodeList=(null) NodeList=phoenix-06 BatchHost=phoenix-06 NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=1,mem=4000M,node=1,billing=1 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryCPU=4000M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=(null) WorkDir=/private/home/anovak1/workspace/cwltool Comment= JobID=12345 StdErr=/private/home/anovak1/workspace/cwltool/slurm-38807436.out StdIn=/dev/null StdOut=/private/home/anovak1/workspace/cwltool/slurm-38807436.out Power= -- Adam Novak (He/Him) Senior Software Engineer Computational Genomics Lab UC Santa Cruz Genomics Institute "Revealing life’s code." Personal Feedback: https://forms.gle/UXZhZc123knF65Dw5
Adam, If your cluster is running Slurm 21.08 or later, scontrol show job <jobid> --json (or --yaml) produces structured output that properly escapes all fields, including the comment. That is the cleanest path for a normal user — no delimiter ambiguity, no newline injection. If the cluster is older, squeue --Format with a custom separator is the practical fallback. Something like squeue -j <jobid> --Format=jobid,comment --separator='|' gives you a delimited record where the comment is a single field. It won't contain the full scontrol field set, but it covers most of what you'd want to parse. Jim On Tue, Sep 22, 2026 at 2:49 PM, Adam Novak via slurm-users < slurm-users@lists.schedmd.com> wrote:
It was recently pointed out to me that `scontrol show job` will include jobs' comments, and those comments seem to be able to contain arbitrary text, including newlines and "=". The comments don't seem to be delimited or escaped in any way within the `scontrol show job` output.
This means I can't actually parse the output of `scontrol show job`, because once a comment starts I can never tell that it has ended. Someone can put a whole valid-looking job record inside their job comment, or something that looks like the tail end of one job record and the beginning of another.
Is there a way to adjust the output format or access that data, as a normal cluster user, through another interface?
To illustrate the problem, I ran:
sbatch --wrap "sleep 10" --comment="$(printf "\n\nJobID=12345\n\n")"
And I found this in my `scontrol show job` output:
JobId=38807436 JobName=wrap UserId=anovak1(30065) GroupId=prismuser(600) MCS_label=N/A Priority=4202 Nice=0 Account=standard QOS=normal JobState=RUNNING Reason=None Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:09 TimeLimit=00:10:00 TimeMin=N/A SubmitTime=2026-09-22T13:47:15 EligibleTime=2026-09-22T13:47:15 AccrueTime=2026-09-22T13:47:15 StartTime=2026-09-22T13:47:17 EndTime=2026-09-22T13:57:17 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-09-22T13:47:17 Scheduler=Backfill Partition=short AllocNode:Sid=phoenix:1297984 ReqNodeList=(null) ExcNodeList=(null) NodeList=phoenix-06 BatchHost=phoenix-06 NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=1,mem=4000M,node=1,billing=1 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryCPU=4000M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=(null) WorkDir=/private/home/anovak1/workspace/cwltool Comment=
JobID=12345 StdErr=/private/home/anovak1/workspace/cwltool/slurm-38807436.out StdIn=/dev/null StdOut=/private/home/anovak1/workspace/cwltool/slurm-38807436.out Power=
-- Adam Novak (He/Him) Senior Software Engineer Computational Genomics Lab UC Santa Cruz Genomics Institute "Revealing life’s code."
Personal Feedback: https://forms.gle/UXZhZc123knF65Dw5
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Try —yaml or —json option Sent from my iPhone
On Sep 22, 2026, at 5:07 PM, Adam Novak via slurm-users <slurm-users@lists.schedmd.com> wrote:
It was recently pointed out to me that `scontrol show job` will include jobs' comments, and those comments seem to be able to contain arbitrary text, including newlines and "=". The comments don't seem to be delimited or escaped in any way within the `scontrol show job` output.
This means I can't actually parse the output of `scontrol show job`, because once a comment starts I can never tell that it has ended. Someone can put a whole valid-looking job record inside their job comment, or something that looks like the tail end of one job record and the beginning of another.
Is there a way to adjust the output format or access that data, as a normal cluster user, through another interface?
To illustrate the problem, I ran:
sbatch --wrap "sleep 10" --comment="$(printf "\n\nJobID=12345\n\n")"
And I found this in my `scontrol show job` output:
JobId=38807436 JobName=wrap UserId=anovak1(30065) GroupId=prismuser(600) MCS_label=N/A Priority=4202 Nice=0 Account=standard QOS=normal JobState=RUNNING Reason=None Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:09 TimeLimit=00:10:00 TimeMin=N/A SubmitTime=2026-09-22T13:47:15 EligibleTime=2026-09-22T13:47:15 AccrueTime=2026-09-22T13:47:15 StartTime=2026-09-22T13:47:17 EndTime=2026-09-22T13:57:17 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-09-22T13:47:17 Scheduler=Backfill Partition=short AllocNode:Sid=phoenix:1297984 ReqNodeList=(null) ExcNodeList=(null) NodeList=phoenix-06 BatchHost=phoenix-06 NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=1,mem=4000M,node=1,billing=1 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryCPU=4000M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=(null) WorkDir=/private/home/anovak1/workspace/cwltool Comment=
JobID=12345 StdErr=/private/home/anovak1/workspace/cwltool/slurm-38807436.out StdIn=/dev/null StdOut=/private/home/anovak1/workspace/cwltool/slurm-38807436.out Power=
-- Adam Novak (He/Him) Senior Software Engineer Computational Genomics Lab UC Santa Cruz Genomics Institute "Revealing life’s code."
Personal Feedback: https://forms.gle/UXZhZc123knF65Dw5
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
One correction to my own answer: --separator delimits fields but doesn't escape newlines, so a comment containing one still breaks the record. On older Slurm the safe route is sacct --parsable2 rather than the CLI job view, or restricting what can go in a comment. The --json / --yaml path remains the right answer on 21.08+. Jim
On Wednesday, September 23rd, 2026 at 05:04, Adam Novak via slurm-users <slurm-users@lists.schedmd.com> wrote:
To illustrate the problem, I ran:
sbatch --wrap "sleep 10" --comment="$(printf "\n\nJobID=12345\n\n")"
And I found this in my `scontrol show job` output:
JobId=38807436 JobName=wrap UserId=anovak1(30065) GroupId=prismuser(600) MCS_label=N/A Priority=4202 Nice=0 Account=standard QOS=normal JobState=RUNNING Reason=None Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:09 TimeLimit=00:10:00 TimeMin=N/A SubmitTime=2026-09-22T13:47:15 EligibleTime=2026-09-22T13:47:15 AccrueTime=2026-09-22T13:47:15 StartTime=2026-09-22T13:47:17 EndTime=2026-09-22T13:57:17 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-09-22T13:47:17 Scheduler=Backfill Partition=short AllocNode:Sid=phoenix:1297984 ReqNodeList=(null) ExcNodeList=(null) NodeList=phoenix-06 BatchHost=phoenix-06 NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=1,mem=4000M,node=1,billing=1 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryCPU=4000M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=(null) WorkDir=/private/home/anovak1/workspace/cwltool Comment=
JobID=12345 StdErr=/private/home/anovak1/workspace/cwltool/slurm-38807436.out StdIn=/dev/null StdOut=/private/home/anovak1/workspace/cwltool/slurm-38807436.out Power=
Using slurm 25.11.7 $ scontrol show job --oneline <jobid> JobId=<jobid> JobName=wrap ... ... SubmitLine=sbatch --wrap sleep 10 -p <partition> -A <account> -N 1 -n 1 --comment= JobID=12345 WorkDir=/path/to/work/dir Comment= JobID=12345 StdErr= StdIn=/dev/null ... $ so, if you have "SubmitLine=" in the output (which I don't see in yours?), then you, or your parsing tools, should be able to "see" both the CLI comment, the bit after "--comment=", and the "stored" comment, the bit after "Comment=". Still not as trivial as parsing the "marked-up" outputs in json or yaml format, but at least you can identify a duplication.
If none of the excellent options presented by the others work for you, one other possibilities is to invoke "scontrol show job NNNNN" (and maybe loop around various values of NNNNN from the outside) so you know that only *one* job output will be printed, and therefore you have less to parse/assume. Yet another possibility is to use sacct instead of scontrol. At the cost of a uglier default and more complicated utilization, you have the advantage of being able to specify only the fields you care with the format AND of being able to run the query also one the jobs get out of the queue, when their records disappear from the scontrol output (the latter is a huge plus for me, but it could be useless and/or a distraction for your use case) HTH On Tue, Sep 22, 2026 at 9:16 PM Kevin Buckley (Pawsey) via slurm-users < slurm-users@lists.schedmd.com> wrote:
On Wednesday, September 23rd, 2026 at 05:04, Adam Novak via slurm-users < slurm-users@lists.schedmd.com> wrote:
To illustrate the problem, I ran:
sbatch --wrap "sleep 10" --comment="$(printf "\n\nJobID=12345\n\n")"
And I found this in my `scontrol show job` output:
JobId=38807436 JobName=wrap UserId=anovak1(30065) GroupId=prismuser(600) MCS_label=N/A Priority=4202 Nice=0 Account=standard QOS=normal JobState=RUNNING Reason=None Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:09 TimeLimit=00:10:00 TimeMin=N/A SubmitTime=2026-09-22T13:47:15 EligibleTime=2026-09-22T13:47:15 AccrueTime=2026-09-22T13:47:15 StartTime=2026-09-22T13:47:17 EndTime=2026-09-22T13:57:17 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-09-22T13:47:17
Scheduler=Backfill
Partition=short AllocNode:Sid=phoenix:1297984 ReqNodeList=(null) ExcNodeList=(null) NodeList=phoenix-06 BatchHost=phoenix-06 NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=1,mem=4000M,node=1,billing=1 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryCPU=4000M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=(null) WorkDir=/private/home/anovak1/workspace/cwltool Comment=
JobID=12345 StdErr=/private/home/anovak1/workspace/cwltool/slurm-38807436.out StdIn=/dev/null StdOut=/private/home/anovak1/workspace/cwltool/slurm-38807436.out Power=
Using slurm 25.11.7
$ scontrol show job --oneline <jobid> JobId=<jobid> JobName=wrap ... ... SubmitLine=sbatch --wrap sleep 10 -p <partition> -A <account> -N 1 -n 1 --comment=
JobID=12345 WorkDir=/path/to/work/dir Comment=
JobID=12345 StdErr= StdIn=/dev/null ... $
so, if you have "SubmitLine=" in the output (which I don't see in yours?), then you, or your parsing tools, should be able to "see" both the CLI comment, the bit after "--comment=", and the "stored" comment, the bit after "Comment=".
Still not as trivial as parsing the "marked-up" outputs in json or yaml format, but at least you can identify a duplication.
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Hi, since I haven't seen it being mentioned so far: You could also just sanitize the job comment field from any unwanted characters (e.g. newlines) at submit time by filtering them in job_submit.lua. I do that for both job comments and job names (otherwise you can apparently do some pretty wild stuff like creating colored squeue output via ANSI color codes): function filter_specialchars(string, string_desc) string_clean = string:gsub("[^%w_%-%.%+%^%⁰-⁹]", "_") slurm.log_info("%s: Cleaned-up %s: %s.", log_prefix, string_desc, string_clean) return string_clean end This function will replace all characters with underscores except alphanumeric character, '_', '-', '.', '+', '^', and numeric superscripts ('string' parameters is either job_desc.name or job_desc.comment; the other two variables are for logging only: 'string_desc' is either 'job name' or 'job comment' and 'log_prefix' is 'slurm_job_submit'). The superscripts are allowed since our chemistry users sometimes use them for chemical formulae (not quite IUPAC compliant, but subscripts are much more of a hassle to type). Kind regards René Sitt Am 22.09.26 um 22:49 schrieb Adam Novak via slurm-users:
It was recently pointed out to me that `scontrol show job` will include jobs' comments, and those comments seem to be able to contain arbitrary text, including newlines and "=". The comments don't seem to be delimited or escaped in any way within the `scontrol show job` output.
This means I can't actually parse the output of `scontrol show job`, because once a comment starts I can never tell that it has ended. Someone can put a whole valid-looking job record inside their job comment, or something that looks like the tail end of one job record and the beginning of another.
Is there a way to adjust the output format or access that data, as a normal cluster user, through another interface?
To illustrate the problem, I ran:
sbatch --wrap "sleep 10" --comment="$(printf "\n\nJobID=12345\n\n")"
And I found this in my `scontrol show job` output:
JobId=38807436 JobName=wrap UserId=anovak1(30065) GroupId=prismuser(600) MCS_label=N/A Priority=4202 Nice=0 Account=standard QOS=normal JobState=RUNNING Reason=None Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:09 TimeLimit=00:10:00 TimeMin=N/A SubmitTime=2026-09-22T13:47:15 EligibleTime=2026-09-22T13:47:15 AccrueTime=2026-09-22T13:47:15 StartTime=2026-09-22T13:47:17 EndTime=2026-09-22T13:57:17 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-09-22T13:47:17 Scheduler=Backfill Partition=short AllocNode:Sid=phoenix:1297984 ReqNodeList=(null) ExcNodeList=(null) NodeList=phoenix-06 BatchHost=phoenix-06 NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=1,mem=4000M,node=1,billing=1 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryCPU=4000M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=(null) WorkDir=/private/home/anovak1/workspace/cwltool Comment=
JobID=12345 StdErr=/private/home/anovak1/workspace/cwltool/slurm-38807436.out StdIn=/dev/null StdOut=/private/home/anovak1/workspace/cwltool/slurm-38807436.out Power=
-- Dipl.-Chem. René Sitt Abteilung Zentrale Systeme - Hochleistungsrechnen Hochschulrechenzentrum (HRZ) Hessisches Kompetenzzentrum für Hochleistungsrechnen (HKHLR) Philipps-Universität Marburg Hans-Meerwein-Straße 6 35032 Marburg Büro: Gebäude H|04, Raum 04A32 T +49 6421 28-23523 E sittr@hrz.uni-marburg.de M @sittr:matrix.uni-marburg.de https://www.uni-marburg.de/de/hrz https://www.hkhlr.de
Thanks for all the help! I will look at using sacct and at --oneline and --json. do have Slurm 21.08 and it appears to *not* actually have --json; it looks like that option first appears in the documentation around version 24.05: https://web.archive.org/web/20240817232557/https://slurm.schedmd.com/scontro... On Wed, Sep 23, 2026 at 9:46 AM René Sitt via slurm-users < slurm-users@lists.schedmd.com> wrote:
Hi,
since I haven't seen it being mentioned so far: You could also just sanitize the job comment field from any unwanted characters (e.g. newlines) at submit time by filtering them in job_submit.lua. I do that for both job comments and job names (otherwise you can apparently do some pretty wild stuff like creating colored squeue output via ANSI color codes):
function filter_specialchars(string, string_desc) string_clean = string:gsub("[^%w_%-%.%+%^%⁰-⁹]", "_") slurm.log_info("%s: Cleaned-up %s: %s.", log_prefix, string_desc, string_clean) return string_clean end
This function will replace all characters with underscores except alphanumeric character, '_', '-', '.', '+', '^', and numeric superscripts ('string' parameters is either job_desc.name or job_desc.comment; the other two variables are for logging only: 'string_desc' is either 'job name' or 'job comment' and 'log_prefix' is 'slurm_job_submit'). The superscripts are allowed since our chemistry users sometimes use them for chemical formulae (not quite IUPAC compliant, but subscripts are much more of a hassle to type).
Kind regards René Sitt
Am 22.09.26 um 22:49 schrieb Adam Novak via slurm-users:
It was recently pointed out to me that `scontrol show job` will include jobs' comments, and those comments seem to be able to contain arbitrary text, including newlines and "=". The comments don't seem to be delimited or escaped in any way within the `scontrol show job` output.
This means I can't actually parse the output of `scontrol show job`, because once a comment starts I can never tell that it has ended. Someone can put a whole valid-looking job record inside their job comment, or something that looks like the tail end of one job record and the beginning of another.
Is there a way to adjust the output format or access that data, as a normal cluster user, through another interface?
To illustrate the problem, I ran:
sbatch --wrap "sleep 10" --comment="$(printf "\n\nJobID=12345\n\n")"
And I found this in my `scontrol show job` output:
JobId=38807436 JobName=wrap UserId=anovak1(30065) GroupId=prismuser(600) MCS_label=N/A Priority=4202 Nice=0 Account=standard QOS=normal JobState=RUNNING Reason=None Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:09 TimeLimit=00:10:00 TimeMin=N/A SubmitTime=2026-09-22T13:47:15 EligibleTime=2026-09-22T13:47:15 AccrueTime=2026-09-22T13:47:15 StartTime=2026-09-22T13:47:17 EndTime=2026-09-22T13:57:17 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-09-22T13:47:17 Scheduler=Backfill Partition=short AllocNode:Sid=phoenix:1297984 ReqNodeList=(null) ExcNodeList=(null) NodeList=phoenix-06 BatchHost=phoenix-06 NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=1,mem=4000M,node=1,billing=1 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryCPU=4000M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=(null) WorkDir=/private/home/anovak1/workspace/cwltool Comment=
JobID=12345 StdErr=/private/home/anovak1/workspace/cwltool/slurm-38807436.out StdIn=/dev/null StdOut=/private/home/anovak1/workspace/cwltool/slurm-38807436.out Power=
-- Dipl.-Chem. René Sitt Abteilung Zentrale Systeme - Hochleistungsrechnen Hochschulrechenzentrum (HRZ) Hessisches Kompetenzzentrum für Hochleistungsrechnen (HKHLR)
Philipps-Universität Marburg Hans-Meerwein-Straße 6 35032 Marburg Büro: Gebäude H|04, Raum 04A32
T +49 6421 28-23523 <+49%206421%202823523> E sittr@hrz.uni-marburg.de M @sittr:matrix.uni-marburg.de
https://www.uni-marburg.de/de/hrz https://www.hkhlr.de
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
-- Adam Novak (He/Him) Senior Software Engineer Computational Genomics Lab UC Santa Cruz Genomics Institute "Revealing life’s code." Personal Feedback: https://forms.gle/UXZhZc123knF65Dw5
participants (6)
-
Adam Novak -
Davide DelVento -
Feng Zhang -
James Jardine -
Kevin Buckley (Pawsey) -
René Sitt