Native Lustre encryption with Slurm: loading the key at job time without storing it on the remote network
Hi all, I’m new to this list, so hello to everyone. :-) Goal We want to use native lustre client-side encryption (fscrypt) to protect data on a remote HPC cluster, and run Slurm jobs that can work with that data. Current (not so good) concept: The key would temporarily live on the remote network: Key on NFS: /home/username/enc.key Encrypted data on Lustre: /lustre/project The workflow would be: 1. A Slurm prolog loads the key into the kernel keyring on the compute nodes. 2. The same script deletes enc.key from the remote network. 3. The job runs with access to the decrypted data. 4. A Slurm epilog script removes the key from the node’s keyring. The obvious problem: Until the job is scheduled, the key and the encrypted data are on the same remote network, which defeats much of the purpose. Accepted risks We are aware of other attack vectors: remote admins can read enc.key on NFS, and the key can be read from RAM while the job runs. Strict ACLs are essential, and we run the computes --exclusive We accept these risks for now. Question Does anyone know a way to avoid storing enc.key on the remote network before the job starts? Can the key be 'injected' at job start from our side (pushed or pulled) Does slurm have any native mechanism to deal with this? Is there a better pattern for this? Have you implemented something like this on your site, and how? Note: we have an existing site-to-site ipsec tunnel that we could use. Any tips or experiences are welcome. Thanks!
I would use a proper secret store for this. Hashicorp Vault or something similar. That's what we use for secret management across most of our HPC and HPC-adjacent services. For those more in bed with AWS you could use AWS Secrets Manager Tim From: sacawulu via slurm-users <slurm-users@lists.schedmd.com> Date: Saturday, 31 October 2026 at 10:26 To: slurm-users@schedmd.com <slurm-users@schedmd.com> Subject: [slurm-users] Native Lustre encryption with Slurm: loading the key at job time without storing it on the remote network This Message Is From an External Sender This message came from outside your organization. Report if suspicious or ignore. Hi all, I’m new to this list, so hello to everyone. :-) Goal We want to use native lustre client-side encryption (fscrypt) to protect data on a remote HPC cluster, and run Slurm jobs that can work with that data. Current (not so good) concept: The key would temporarily live on the remote network: Key on NFS: /home/username/enc.key Encrypted data on Lustre: /lustre/project The workflow would be: 1. A Slurm prolog loads the key into the kernel keyring on the compute nodes. 2. The same script deletes enc.key from the remote network. 3. The job runs with access to the decrypted data. 4. A Slurm epilog script removes the key from the node’s keyring. The obvious problem: Until the job is scheduled, the key and the encrypted data are on the same remote network, which defeats much of the purpose. Accepted risks We are aware of other attack vectors: remote admins can read enc.key on NFS, and the key can be read from RAM while the job runs. Strict ACLs are essential, and we run the computes --exclusive We accept these risks for now. Question Does anyone know a way to avoid storing enc.key on the remote network before the job starts? Can the key be 'injected' at job start from our side (pushed or pulled) Does slurm have any native mechanism to deal with this? Is there a better pattern for this? Have you implemented something like this on your site, and how? Note: we have an existing site-to-site ipsec tunnel that we could use. Any tips or experiences are welcome. Thanks! -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com ________________________________ AstraZeneca UK Limited is a company incorporated in England and Wales with registered number:03674842 and its registered office at 1 Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge, CB2 0AA. This e-mail and its attachments are intended for the above named recipient only and may contain confidential and privileged information. If they have come to you in error, you must not copy or show them to anyone; instead, please reply to this e-mail, highlighting the error to the sender and then immediately delete the message. For information about how AstraZeneca UK Limited and its affiliates may process information, personal data and monitor communications, please see our privacy notice at www.astrazeneca.com<https://www.astrazeneca.com>
participants (2)
-
Cutts, Tim -
sacawulu