The times were correct via chrony but the timezones were UTC and NZDT which was the issue. Oddly nodes 1 and 2 didnt care about that, only no3 ***shrug***
regards
Steven
From: Chris Samuel via slurm-users <slurm-users@lists.schedmd.com> Sent: Tuesday, 10 December 2024 4:19 pm To: slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com> Subject: [slurm-users] Re: node3 not working - down
On 9/12/24 5:44 pm, Steven Jones via slurm-users wrote:
> [2024-12-09T23:38:56.645] error: Munge decode failed: Rewound credential
> [2024-12-09T23:38:56.645] auth/munge: _print_cred: ENCODED: Tue Dec 10
> 23:38:30 2024
> [2024-12-09T23:38:56.645] auth/munge: _print_cred: DECODED: Mon Dec 09
> 23:38:56 2024
> [2024-12-09T23:38:56.645] error: Check for out of sync clocks
One system is 24 hours behind/ahead of the other.
You should make sure NTP is set up and working on all these nodes.