- slurm-users - lists.schedmd.com

Reserving resources for use by non-slurm stuff
by Shooktija S N 17 Apr '24

17 Apr '24

Hi, I am running Slurm (v22.05.8) on 3 nodes each with the following specs: OS: Proxmox VE 8.1.4 x86_64 (based on Debian 12) CPU: AMD EPYC 7662 (128) GPU: NVIDIA GeForce RTX 4070 Ti Memory: 128 Gb This is /etc/slurm/slurm.conf on all 3 computers without the comment lines: ClusterName=DlabCluster SlurmctldHost=server1 GresTypes=gpu ProctrackType=proctrack/linuxproc ReturnToService=1 SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=root StateSaveLocation=/var/spool/slurmctld TaskPlugin=task/affinity,task/cgroup InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=120 SlurmdTimeout=300 Waittime=0 SchedulerType=sched/backfill SelectType=select/cons_tres JobCompType=jobcomp/none JobAcctGatherFrequency=30 SlurmctldDebug=debug3 SlurmctldLogFile=/var/log/slurmctld.log SlurmdDebug=debug3 SlurmdLogFile=/var/log/slurmd.log NodeName=server[1-3] RealMemory=128636 Sockets=1 CoresPerSocket=64 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:1 PartitionName=mainPartition Nodes=ALL Default=YES MaxTime=INFINITE State=UP I want to reserve a few cores and a few gigs of RAM for use only by the OS which cannot be accessed by jobs being managed by Slurm. What configuration do I need to do to achieve this? Is it possible to reserve in a similar fashion a 'percent' of the GPU which Slurm cannot exceed so that the OS has some GPU resources? Is it possible to have these configs be different for each of the 3 nodes? Thanks!

3 2

Inconsistencies in CPU time Reporting by sreport and sacct Tools
by KK 17 Apr '24

17 Apr '24

I wish to ascertain the CPU core time utilized by user dj1 and dj. I have tested with sreport cluster UserUtilizationByAccount, sreport job SizesByAccount, and sacct. It appears that sreport cluster UserUtilizationByAccount displays the total core hours used by the entire account, rather than the individual user's cpu time. Here are the specifics: Users dj and dj1 are both under the account mehpc. In 2024-04-12 ~ 2024-04-15, dj1 used approximately 10 minutes of core time, while dj used about 4 minutes. However, "sreport Cluster UserUtilizationByAccount user=dj1 start=2024-04-12 end=2024-04-15" shows 14 minutes of usage. Similarly, "sreport job SizesByAccount Users=dj start=2024-04-12 end=2024-04-15" hows about 14 minutes. Using "sreport job SizesByAccount Users=dj1 start=2024-04-12 end=2024-04-15" or "sacct -u dj1 -S 2024-04-12 -E 2024-04-15 -o "jobid,partition,account,user,alloccpus,cputimeraw,state,workdir%60" -X |awk 'BEGIN{total=0}{total+=$6}END{print total}'" yields the accurate values, which are around 10 minutes for dj1. Attachment are the details.

1 0

Association limit problem
by Gestió Servidors 17 Apr '24

17 Apr '24

Hello, I'm doing some test with "associations" with "sacctmgr". I have created three users (user_1, user_2 and user_3). For each of these users, I have created an association: [root@myserver log]# sacctmgr show user user_1 --associations User Def Acct Admin Cluster Account Partition Share Priority MaxJobs MaxNodes MaxCPUs MaxSubmit MaxWall MaxCPUMins QOS Def QOS ---------- ---------- --------- ---------- ---------- ---------- --------- ---------- ------- -------- -------- --------- ----------- ----------- -------------------- --------- user_1 test None q50004 test aolin.q 1 4 2 10 normal user_1 test None q50004 test cuda-staf+ 1 4 2 10 normal [root@myserver log]# sacctmgr show user user_2 --associations User Def Acct Admin Cluster Account Partition Share Priority MaxJobs MaxNodes MaxCPUs MaxSubmit MaxWall MaxCPUMins QOS Def QOS ---------- ---------- --------- ---------- ---------- ---------- --------- ---------- ------- -------- -------- --------- ----------- ----------- -------------------- --------- user_2 test None q50004 test cuda-int.q 1 4 normal [root@myserver log]# sacctmgr show user user_3 --associations User Def Acct Admin Cluster Account Partition Share Priority MaxJobs MaxNodes MaxCPUs MaxSubmit MaxWall MaxCPUMins QOS Def QOS ---------- ---------- --------- ---------- ---------- ---------- --------- ---------- ------- -------- -------- --------- ----------- ----------- -------------------- --------- user_3 test None q50004 test research.q 1 2 1 normal user_3 test None q50004 test xeon.q 1 2 1 normal All users belong to "Test" account: [root@myserver log]# sacctmgr show account test --association Account Descr Org Cluster ParentName User Share Priority GrpJobs GrpNodes GrpCPUs GrpMem GrpSubmit GrpWall GrpCPUMins MaxJobs MaxNodes MaxCPUs MaxSubmit MaxWall MaxCPUMins QOS Def QOS ---------- -------------------- -------------------- ---------- ---------- ---------- --------- ---------- ------- -------- -------- ------- --------- ----------- ----------- ------- -------- -------- --------- ----------- ----------- -------------------- --------- test test test q50004 root 1 normal test test test q50004 user_1 1 4 2 10 normal test test test q50004 user_1 1 4 2 10 normal test test test q50004 user_2 1 4 normal test test test q50004 user_3 1 2 1 normal test test test q50004 user_3 1 2 1 normal When I submit with "user_1", all tests are running fine: some jobs are queued and executed and some jobs are rejected because of the limits. However, with users "user_2" and "user_3" I can't submit any job. All jobs are rejected with these messages: 11168 research. test user_3 PENDING 0:00 2024-04-17T12:53:21 N/A 1 1 OK N/A (AssocMaxCpuPerJo (null) 11173 research. test user_3 PENDING 0:00 2024-04-17T13:06:02 N/A 1 1 OK N/A (AssocMaxCpuPerJo (null) 11174 research. test user_3 PENDING 0:00 2024-04-17T13:06:16 N/A 1 1 OK N/A (AssocMaxCpuPerJo (null) 11176 research. test user_3 PENDING 0:00 2024-04-17T13:07:23 N/A 1 1 OK N/A (AssocMaxCpuPerJo (null) 11180 research. test user_3 PENDING 0:00 2024-04-17T13:08:45 N/A 1 1 OK N/A (AssocMaxCpuPerJo (null) For example, user "user_3" are trying to submit in this way (test.sh script only is a simple "sleep 50": sbatch -p aolin.q -N 2 ./test.sh --> sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified sbatch -p aolin.q -N 1 ./test.sh --> sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified sbatch -p research.q -N 1 ./test.sh --> submitted but not running --> nodelist(reason)= (AssocMaxCpuPerJobLimit) -> WHY??? sbatch -p research.q -N 1 -n 1 ./test.sh --> submitted but not running --> nodelist(reason)= (AssocMaxCpuPerJobLimit) --> WHY??? sbatch -p xeon.q -N 1 -n 1 ./test.sh --> submitted and running!! [root@myserver log]# squeue JOBID PARTITION NAME USER STATE TIME SUBMIT_TIME START_TIME NODE CPUS OVER_S TRES_PER_NODE NODELIST(REASON) DEPENDENCY REQ_NODES NODELIST 11202 research. test user_3 PENDING 0:00 2024-04-17T13:33:31 N/A 1 1 OK N/A (AssocMaxCpuPerJo (null) 11200 research. test user_3 PENDING 0:00 2024-04-17T13:33:17 N/A 1 1 OK N/A (AssocMaxCpuPerJo (null) 11212 xeon.q test user_3 RUNNING 0:18 2024-04-17T13:36:10 2024-04-17T13:36:10 1 1 OK N/A aolin-cpu-1 (null) aolin-cpu-1 Why? What am I doing wrong? Where is the limit that I am not seeing? Thanks a lot!

1 0

Redirect jobs submitted to old partition to new
by wdennis＠nec-labs.com 16 Apr '24

16 Apr '24

Hi all, I have a single-partition Slurm cluster (the single partition name being "default_queue") that I now want to implement multiple different queues on to subdivide the resources. Say the new default queue is "queue1"; Should I set the "default_queue" to `State=INACTIVE` and then use `Alternate=queue1` on it to redirect jobs sent to the old "default_queue" to end up on "queue1"? Thinking it would be nice to have an AltPartitionName= construct to handle this... (must be a reason this doesn't exist)

2 1

Slurm version 23.11.6 is now available
by Marshall Garey 16 Apr '24

16 Apr '24

We are pleased to announce the availability of Slurm version 23.11.6. The 23.11.6 release includes two different problems with the priority/multifactor plugin: a crash and a miscalculation of AssocGrpCPURunMinutes after a slurmctld reconfiguration/restart. The wsrep_on errors that sites running MySQL or older MariaDB should happen much less frequently and has a clarifying statement when it is an innocuous error. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Marshall > * Changes in Slurm 23.11.6 > ========================== > -- Avoid limiting sockets per node to one when using gres enforce-binding. > -- slurmrestd - Avoid permission denied errors when attempting to listen on > the same port multiple times. > -- Fix GRES reservations where the GRES has no topology > (no cores= in gres.conf). > -- Ensure that thread_id_rpc is gone before priority_g_fini(). > -- Fix scontrol reboot timeout removing drain state from nodes. > -- squeue - Print header on empty reponse to `--only-job-state`. > -- Fix slurmrestd not ending job properly when xauth is not present and a x11 > job is sent. > -- Add experimental job state caching with > SchedulerParameters=enable_job_state_cache to speed up querying job states > with squeue --only-job-state. > -- slurmrestd - Correct dumping of invalid ArrayJobIds returned from > 'GET /slurm/v0.0.40/jobs/state'. > -- squeue - Correct dumping of invalid ArrayJobIds returned from > `squeue --only-job-state --{json|yaml}`. > -- If scancel --ctld is not used with --interactive, --sibling, or specific > step ids, then this option issues a single request to the slurmctld to > signal all jobs matching the specified filters. This greatly improves > the performance of slurmctld and scancel. The updated --ctld option also > fixes issues with the --partition or --reservation scancel options for jobs > that requested multiple partitions or reservations. > -- slurmrestd - Give EINVAL error when failing to parse signal name to numeric > signal. > -- slurmrestd - Allow ContentBody for all methods per RFC7230 even if ignored. > -- slurmrestd - Add 'DELETE /slurm/v0.0.40/jobs' endpoint to allow bulk job > signaling via slurmctld. > -- Fix combination of --nodelist and --exclude not always respecting the > excluded node list. > -- Fix jobs incorrectly allocating nodes exclusively when started on a > partition that doesn't enforce it. This could happen if a multi-partition > job doesn't specify --exclusive and is evaluated first on a partition > configured with OverSubscribe=EXCLUSIVE but ends up starting in a partition > configured with OverSubscribe!=EXCLUSIVE evaluated afterwards. > -- Setting GLOB_SILENCE flag no longer exposes old bugged behavior. > -- Fix associations AssocGrpCPURunMinutes being incorrectly computed for > running jobs after a controller reconfiguration/restart. > -- Fix scheduling jobs that request --gpus and nodes have different node > weights and different numbers of gpus. > -- slurmrestd - Add "NO_CRON_JOBS" as possible flag value to the following: > 'DELETE /slurm/v0.0.40/jobs' flags field. > 'DELETE /slurm/v0.0.40/job/{job_id}?flags=' flags query parameter. > -- Fix scontrol segfault/assert failure if the TRESPerNode parameter is used > when creating reservations. > -- Avoid checking for wsrep_on when restoring streaming replication settings. > -- Clarify in the logs that error "1193 Unknown system variable 'wsrep_on'" is > innocuous. > -- accounting_storage/mysql - Fix problem when loading reservations from an > archive dump. > -- slurmdbd - Fix minor race condition when sending updates to a shutdown > slurmctld. > -- slurmctld - Fix invalid refusal of a reservation update. > -- openapi - Fix memory leak of /meta/slurm/cluster response field. > -- Fix memory leak when using auth/slurm and AuthInfo=use_client_ids. -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Fwd: sreport cluster UserUtilizationByaccount Used result versus sreport job SizesByAccount or sacct: inconsistencies
by KK 15 Apr '24

15 Apr '24

---------- Forwarded message --------- 发件人： KK <daijiangkuicgo(a)gmail.com> Date: 2024年4月15日周一 13:25 Subject: sreport cluster UserUtilizationByaccount Used result versus sreport job SizesByAccount or sacct: inconsistencies To: <slurm-users(a)schedmd.com> I wish to ascertain the CPU core hours utilized by user dj1 and dj. I have tested with sreport cluster UserUtilizationByAccount, sreport job SizesByAccount, and sacct. It appears that sreport cluster UserUtilizationByAccount displays the total core hours used by the entire account, rather than the individual user's cpu time. Here are the specifics: Users dj and dj1 are both under the account mehpc. In 2024-04-12 ~ 2024-04-15, dj1 used approximately 10 minutes of core time, while dj used about 4 minutes. However, "*sreport Cluster UserUtilizationByAccount user=dj1 start=2024-04-12 end=2024-04-15*" shows 14 minutes of usage. Similarly, "*sreport job SizesByAccount Users=dj start=2024-04-12 end=2024-04-15*" hows about 14 minutes. Using "*sreport job SizesByAccount Users=dj1 start=2024-04-12 end=2024-04-15*" or "*sacct -u dj1 -S 2024-04-12 -E 2024-04-15 -o "jobid,partition,account,user,alloccpus,cputimeraw,state,workdir%60" -X |awk 'BEGIN{total=0}{total+=$6}END{print total}'*" yields the accurate values, which are around 10 minutes for dj1. Here are the details: [root@ood-master ~]# sacctmgr list assoc format=cluster,user,account,qos Cluster User Account QOS ---------- ---------- ---------- -------------------- mehpc root normal mehpc root root normal mehpc mehpc normal mehpc dj mehpc normal mehpc dj1 mehpc normal [root@ood-master ~]# sacct -X -u dj1 -S 2024-04-12 -E 2024-04-15 -o jobid,ncpus,elapsedraw,cputimeraw JobID NCPUS ElapsedRaw CPUTimeRAW ------------ ---------- ---------- ---------- 4 1 60 60 5 2 120 240 6 1 61 61 8 2 120 240 9 0 0 0 [root@ood-master ~]# sacct -X -u dj -S 2024-04-12 -E 2024-04-15 -o jobid,ncpus,elapsedraw,cputimeraw JobID NCPUS ElapsedRaw CPUTimeRAW ------------ ---------- ---------- ---------- 7 2 120 240 [root@ood-master ~]# sreport job SizesByAccount Users=dj1 start=2024-04-12 end=2024-04-15 -------------------------------------------------------------------------------- Job Sizes 2024-04-12T00:00:00 - 2024-04-14T23:59:59 (259200 secs) Time reported in Minutes -------------------------------------------------------------------------------- Cluster Account 0-49 CPUs 50-249 CPUs 250-499 CPUs 500-999 CPUs >= 1000 CPUs % of cluster --------- --------- ------------- ------------- ------------- ------------- ------------- ------------ mehpc root 10 0 0 0 0 100.00% [root@ood-master ~]# sreport job SizesByAccount Users=dj start=2024-04-12 end=2024-04-15 -------------------------------------------------------------------------------- Job Sizes 2024-04-12T00:00:00 - 2024-04-14T23:59:59 (259200 secs) Time reported in Minutes -------------------------------------------------------------------------------- Cluster Account 0-49 CPUs 50-249 CPUs 250-499 CPUs 500-999 CPUs >= 1000 CPUs % of cluster --------- --------- ------------- ------------- ------------- ------------- ------------- ------------ mehpc root 4 0 0 0 0 100.00% [root@ood-master ~]# sreport Cluster UserUtilizationByAccount user=dj1 start=2024-04-12 end=2024-04-15 -------------------------------------------------------------------------------- Cluster/User/Account Utilization 2024-04-12T00:00:00 - 2024-04-14T23:59:59 (259200 secs) Usage reported in CPU Minutes -------------------------------------------------------------------------------- Cluster Login Proper Name Account Used Energy --------- --------- --------------- --------------- -------- -------- mehpc dj1 dj1 dj1 mehpc 14 0 [root@ood-master ~]# sreport Cluster UserUtilizationByAccount user=dj start=2024-04-12 end=2024-04-15 -------------------------------------------------------------------------------- Cluster/User/Account Utilization 2024-04-12T00:00:00 - 2024-04-14T23:59:59 (259200 secs) Usage reported in CPU Minutes -------------------------------------------------------------------------------- Cluster Login Proper Name Account Used Energy --------- --------- --------------- --------------- -------- -------- mehpc dj dj dj mehpc 14 0 [root@ood-master ~]# sacct -u dj1 -S 2024-04-12 -E 2024-04-15 -o "jobid,partition,account,user,alloccpus,cputimeraw,state,workdir%60" -X |awk 'BEGIN{total=0}{total+=$6}END{print total}' 601 [root@ood-master ~]# sacct -u dj -S 2024-04-12 -E 2024-04-15 -o "jobid,partition,account,user,alloccpus,cputimeraw,state,workdir%60" -X |awk 'BEGIN{total=0}{total+=$6}END{print total}' 240 [root@ood-master ~]# sreport cluster AccountUtilizationByUser accounts=mehpc start=2024-01-01 end=2024-04-15 -------------------------------------------------------------------------------- Cluster/Account/User Utilization 2024-01-01T00:00:00 - 2024-04-14T23:59:59 (9072000 secs) Usage reported in CPU Minutes -------------------------------------------------------------------------------- Cluster Account Login Proper Name Used Energy --------- --------------- --------- --------------- -------- -------- mehpc mehpc 14 0 mehpc mehpc dj dj dj 14 0 mehpc mehpc dj1 dj1 dj1 14 0

1 0

Slurm.conf and workers
by Xaver Stiensmeier 15 Apr '24

15 Apr '24

Dear slurm-user list, as far as I understood it, the slurm.conf needs to be present on the master and on the workers at slurm.conf (if no other path is set via SLURM_CONF). However, I noticed that when adding a partition only in the master's slurm.conf, all workers were able to "correctly" show the added partition when calling sinfo on them. Is the stored slurm.conf on every instance just a fallback for when connection is down or what is the purpose? The documentation only says: "This file should be consistent across all nodes in the cluster." Best regards, Xaver

2 1

Interfaces of topology/tree and Topology Awareness
by nico.derl＠tutanota.com 15 Apr '24

15 Apr '24

Hello everyone, I'm trying to improve topology awareness in a local Slurm-managed HPC system. It's using the default hierarchical 3-level topology with the tree-plugin. It however does not always confine jobs to the most tightly packed group of nodes, seems to over-provision switches for smaller jobs, and gets slow or overwhelmed with jobs that have a high node count. I'd like to implement something more literally aligned with best-fit, but I'm having trouble understanding the relevant interfaces to hook into the topology model of Slurm. I would like a high-level explanation of how the tree- and common topology components work, how they integrate into the higher scheduling logic and what the internal topology model looks like. Or some pointers to relevant docs discussing this. I have read the topology guide and its dev-doc, which does note some of the caveats I mentioned. It however only talks about providing a set of weights to the upper logic levels in the form of a node ranking. I can't see how this ranking resembles the topology and how it's being used. From looking at the signatures and C-code I can tell this much: topology-tree consumes the topology.conf and generates a ranking of some kind that is passed to topology-common. topology-common consumes a ranking and uses its own gres-sched to figure out what nodes can fit a job (possibly pulling info from the gres-select-plugin to determine node capabilities). It's then supposed to apply a best-fit algorithm to efficiently fill up vacant cluster-capacity, but I can't manage to follow this part in the code as everything crumbles into separate files that I can't link correctly in my head. Thanks in advance. referenced docs: <https://slurm.schedmd.com/topology.html> <https://hpc.rz.rptu.de/documentation/topology_plugin.html> <https://github.com/SchedMD/slurm/tree/master/src/plugins/topology/common>

2 1

Re: slurmrestd connect to 192.168.87.113:6819 Connection refused
by shaobo liu 14 Apr '24

14 Apr '24

Thanks, The reason was found. It was caused by the expiration of the rest api token. <nico.derl(a)tutanota.com> 于2024年4月12日周五 22:56写道： > If you say DBd isn't using 6819, in the sense that you selected a > different port, make sure the dbdport directive reflects that in both > slurmdbd.conf and AccountingStoragePort in slurm.conf. > It must be getting the 6819 from somewhere. > > > 12. Apr. 2024, 16:05 von dspam.liu(a)gmail.com: > > slurmctrld and rest are on the same machine, No firewall. secondary > slurmdbd is background mode, slurmdbd does not listen on port 6819. > > OS: ubuntu 20.04 > SLURM: 23.11.0 > > <nico.derl(a)tutanota.com> 于2024年4月12日周五 20:18写道： > > Hey, > Are slurmctrld and restd on separate machines? Can you manually reach > them? Could there be a firewall/closed port in the way? > > > 12. Apr. 2024, 11:36 von slurm-users(a)lists.schedmd.com: > > hi，slurm configured primary and secondary，The error when requesting > slurmrest api is as follows, may I ask what is the reason? > > # scontrol ping > Slurmctld(primary) at node003 is UP > Slurmctld(backup) at node113 is UP > > > # systemctl status slurmrestd.service > ● slurmrestd.service - Slurm REST daemon > Loaded: loaded (/lib/systemd/system/slurmrestd.service; enabled; > vendor preset: enabled) > Active: active (running) since Fri 2024-04-12 17:07:08 CST; 21min ago > Main PID: 705425 (slurmrestd) > Tasks: 21 (limit: 629145) > Memory: 20.3M > CGroup: /system.slice/slurmrestd.service > └─705425 /usr/sbin/slurmrestd -f /etc/slurm/slurm.conf > unix:/var/spool/slurm/slurmrestd.socket 0.0.0.0:6820 -vvv > > Apr 12 17:08:46 node003 slurmrestd[705425]: debug2: _slurm_connect: failed > to connect to 192.168.87.113:6819: Connection refused > Apr 12 17:08:46 node003 slurmrestd[705425]: debug2: Error connecting slurm > stream socket at 192.168.87.113:6819: Connection refused > Apr 12 17:08:46 node003 slurmrestd[705425]: slurmrestd: error: > slurm_persist_conn_open_without_init: failed to open persistent connection > to host:node113:6819: Connection refused > Apr 12 17:08:46 node003 slurmrestd[705425]: slurmrestd: error: Sending > PersistInit msg: Connection refused > Apr 12 17:08:46 node003 slurmrestd[705425]: slurmrestd: error: > slurm_rest_auth_p_get_db_conn: unable to connect to slurmdbd: Connection > refused > Apr 12 17:08:46 node003 slurmrestd[705425]: slurmrestd: error: > init_connection[v0.0.39]:[[2.0.1.191]:50652] rc[7000]=Unable to connect to > database -> openapi_get_db_conn() failed to open slurmdb connecti> > Apr 12 17:08:46 node003 slurmrestd[705425]: error: > slurm_persist_conn_open_without_init: failed to open persistent connection > to host:node113:6819: Connection refused > Apr 12 17:08:46 node003 slurmrestd[705425]: error: Sending PersistInit > msg: Connection refused > Apr 12 17:08:46 node003 slurmrestd[705425]: error: > slurm_rest_auth_p_get_db_conn: unable to connect to slurmdbd: Connection > refused > Apr 12 17:08:46 node003 slurmrestd[705425]: error: > init_connection[v0.0.39]:[[2.0.1.191]:50652] rc[7000]=Unable to connect to > database -> openapi_get_db_conn() failed to open slurmdb connection > > > >

1 0

slurmrestd connect to 192.168.87.113:6819 Connection refused
by shaobo liu 12 Apr '24

12 Apr '24

hi，slurm configured primary and secondary，The error when requesting slurmrest api is as follows, may I ask what is the reason? # scontrol ping Slurmctld(primary) at node003 is UP Slurmctld(backup) at node113 is UP # systemctl status slurmrestd.service ● slurmrestd.service - Slurm REST daemon Loaded: loaded (/lib/systemd/system/slurmrestd.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2024-04-12 17:07:08 CST; 21min ago Main PID: 705425 (slurmrestd) Tasks: 21 (limit: 629145) Memory: 20.3M CGroup: /system.slice/slurmrestd.service └─705425 /usr/sbin/slurmrestd -f /etc/slurm/slurm.conf unix:/var/spool/slurm/slurmrestd.socket 0.0.0.0:6820 -vvv Apr 12 17:08:46 node003 slurmrestd[705425]: debug2: _slurm_connect: failed to connect to 192.168.87.113:6819: Connection refused Apr 12 17:08:46 node003 slurmrestd[705425]: debug2: Error connecting slurm stream socket at 192.168.87.113:6819: Connection refused Apr 12 17:08:46 node003 slurmrestd[705425]: slurmrestd: error: slurm_persist_conn_open_without_init: failed to open persistent connection to host:node113:6819: Connection refused Apr 12 17:08:46 node003 slurmrestd[705425]: slurmrestd: error: Sending PersistInit msg: Connection refused Apr 12 17:08:46 node003 slurmrestd[705425]: slurmrestd: error: slurm_rest_auth_p_get_db_conn: unable to connect to slurmdbd: Connection refused Apr 12 17:08:46 node003 slurmrestd[705425]: slurmrestd: error: init_connection[v0.0.39]:[[2.0.1.191]:50652] rc[7000]=Unable to connect to database -> openapi_get_db_conn() failed to open slurmdb connecti> Apr 12 17:08:46 node003 slurmrestd[705425]: error: slurm_persist_conn_open_without_init: failed to open persistent connection to host:node113:6819: Connection refused Apr 12 17:08:46 node003 slurmrestd[705425]: error: Sending PersistInit msg: Connection refused Apr 12 17:08:46 node003 slurmrestd[705425]: error: slurm_rest_auth_p_get_db_conn: unable to connect to slurmdbd: Connection refused Apr 12 17:08:46 node003 slurmrestd[705425]: error: init_connection[v0.0.39]:[[2.0.1.191]:50652] rc[7000]=Unable to connect to database -> openapi_get_db_conn() failed to open slurmdb connection

2 1