I am confused by the reported amount of Down and PLND Down by sreport. According to it, our cluster would have had a significant amount of downtime, which I know didn't happen (or, according to the documentation "time that slurmctld was not responding", see https://slurm.schedmd.com/sreport.html)

Could it be my purge settings causing this problem? How can I check (maybe in some logs, maybe in the future) if actually slurmctld was not responding? The expected long-term numbers should be less than the ones reported for last month when we had an issue with a few nodes.... 

Thanks!


[davide@login ~]$ grep Purge /opt/slurm/slurmdbd.conf
#JobPurge=12
#StepPurge=1
PurgeEventAfter=1month
PurgeJobAfter=12month
PurgeResvAfter=1month
PurgeStepAfter=1month
PurgeSuspendAfter=1month

[davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=2/1/22
--------------------------------------------------------------------------------
Cluster Utilization 2022-02-01T00:00:00 - 2024-08-21T23:59:59
Usage reported in Percentage of Total
--------------------------------------------------------------------------------
  Cluster      TRES Name       Allocated            Down      PLND Down             Idle   Planned         Reported
--------- -------------- --------------- --------------- -------------- ---------------- --------- ----------------
  cluster            cpu          19.50%          12.07%          3.92%           64.36%     0.15%          100.03%
  cluster            mem          16.13%          13.17%          4.56%           66.13%     0.00%           99.99%

[davide@login ~]$    sreport -t percent -T cpu,mem cluster utilization start=2/1/23
--------------------------------------------------------------------------------
Cluster Utilization 2023-02-01T00:00:00 - 2024-08-21T23:59:59
Usage reported in Percentage of Total
--------------------------------------------------------------------------------
  Cluster      TRES Name       Allocated            Down      PLND Down            Idle   Planned         Reported
--------- -------------- --------------- --------------- -------------- --------------- --------- ----------------
  cluster            cpu          28.74%          18.80%          6.44%          45.77%     0.24%          100.02%
  cluster            mem          22.52%          20.54%          7.38%          49.55%     0.00%           99.98%

[davide@login ~]$  sreport -t percent -T cpu,mem cluster utilization start=2/1/24
--------------------------------------------------------------------------------
Cluster Utilization 2024-02-01T00:00:00 - 2024-08-21T23:59:59
Usage reported in Percentage of Total
--------------------------------------------------------------------------------
  Cluster      TRES Name      Allocated            Down      PLND Down            Idle  Planned        Reported
--------- -------------- -------------- --------------- -------------- --------------- -------- ---------------
  cluster            cpu         29.92%          24.88%         17.73%          27.45%    0.02%         100.00%
  cluster            mem         20.07%          28.60%         19.57%          31.76%    0.00%         100.00%

[davide@login ~]$  sreport -t percent -T cpu,mem cluster utilization start=8/8/24
--------------------------------------------------------------------------------
Cluster Utilization 2024-08-08T00:00:00 - 2024-08-21T23:59:59
Usage reported in Percentage of Total
--------------------------------------------------------------------------------
  Cluster      TRES Name     Allocated         Down PLND Dow           Idle  Planned       Reported
--------- -------------- ------------- ------------ -------- -------------- -------- --------------
  cluster            cpu        15.96%        2.53%    0.00%         81.51%    0.00%        100.00%
  cluster            mem         9.18%        2.22%    0.00%         88.60%    0.00%        100.00%

[davide@login ~]$  sreport -t percent -T cpu,mem cluster utilization start=7/7/24
--------------------------------------------------------------------------------
Cluster Utilization 2024-07-07T00:00:00 - 2024-08-21T23:59:59
Usage reported in Percentage of Total
--------------------------------------------------------------------------------
  Cluster      TRES Name      Allocated          Down PLND Dow           Idle  Planned       Reported
--------- -------------- -------------- ------------- -------- -------------- -------- --------------
  cluster            cpu         27.07%         2.57%    0.00%         70.34%    0.02%        100.00%
  cluster            mem         17.35%         2.26%    0.00%         80.40%    0.00%        100.00%