I am confused by the reported amount of Down and PLND Down by sreport. According to it, our cluster would have had a significant amount of downtime, which I know didn't happen (or, according to the documentation "time that slurmctld was not responding", see https://slurm.schedmd.com/sreport.html)
Could it be my purge settings causing this problem? How can I check (maybe in some logs, maybe in the future) if actually slurmctld was not responding? The expected long-term numbers should be less than the ones reported for last month when we had an issue with a few nodes....
Thanks!
[davide@login ~]$ grep Purge /opt/slurm/slurmdbd.conf #JobPurge=12 #StepPurge=1 PurgeEventAfter=1month PurgeJobAfter=12month PurgeResvAfter=1month PurgeStepAfter=1month PurgeSuspendAfter=1month
[davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=2/1/22 -------------------------------------------------------------------------------- Cluster Utilization 2022-02-01T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Down Idle Planned Reported --------- -------------- --------------- --------------- -------------- ---------------- --------- ---------------- cluster cpu 19.50% 12.07% 3.92% 64.36% 0.15% 100.03% cluster mem 16.13% 13.17% 4.56% 66.13% 0.00% 99.99%
[davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=2/1/23 -------------------------------------------------------------------------------- Cluster Utilization 2023-02-01T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Down Idle Planned Reported --------- -------------- --------------- --------------- -------------- --------------- --------- ---------------- cluster cpu 28.74% 18.80% 6.44% 45.77% 0.24% 100.02% cluster mem 22.52% 20.54% 7.38% 49.55% 0.00% 99.98%
[davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=2/1/24 -------------------------------------------------------------------------------- Cluster Utilization 2024-02-01T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Down Idle Planned Reported --------- -------------- -------------- --------------- -------------- --------------- -------- --------------- cluster cpu 29.92% 24.88% 17.73% 27.45% 0.02% 100.00% cluster mem 20.07% 28.60% 19.57% 31.76% 0.00% 100.00%
[davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=8/8/24 -------------------------------------------------------------------------------- Cluster Utilization 2024-08-08T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Dow Idle Planned Reported --------- -------------- ------------- ------------ -------- -------------- -------- -------------- cluster cpu 15.96% 2.53% 0.00% 81.51% 0.00% 100.00% cluster mem 9.18% 2.22% 0.00% 88.60% 0.00% 100.00%
[davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=7/7/24 -------------------------------------------------------------------------------- Cluster Utilization 2024-07-07T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total -------------------------------------------------------------------------------- Cluster TRES Name Allocated Down PLND Dow Idle Planned Reported --------- -------------- -------------- ------------- -------- -------------- -------- -------------- cluster cpu 27.07% 2.57% 0.00% 70.34% 0.02% 100.00% cluster mem 17.35% 2.26% 0.00% 80.40% 0.00% 100.00%