Thanks,
The times were correct via chrony but the timezones were UTC and NZDT which was the issue. Oddly nodes 1 and 2 didnt care about that, only no3 ***shrug***
regards
Steven
________________________________ From: Chris Samuel via slurm-users slurm-users@lists.schedmd.com Sent: Tuesday, 10 December 2024 4:19 pm To: slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com Subject: [slurm-users] Re: node3 not working - down
On 9/12/24 5:44 pm, Steven Jones via slurm-users wrote:
[2024-12-09T23:38:56.645] error: Munge decode failed: Rewound credential [2024-12-09T23:38:56.645] auth/munge: _print_cred: ENCODED: Tue Dec 10 23:38:30 2024 [2024-12-09T23:38:56.645] auth/munge: _print_cred: DECODED: Mon Dec 09 23:38:56 2024 [2024-12-09T23:38:56.645] error: Check for out of sync clocks
One system is 24 hours behind/ahead of the other.
You should make sure NTP is set up and working on all these nodes.
-- Chris Samuel : https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.csamuel...http://www.csamuel.org/ : Berkeley, CA, USA
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com