I don't have anything I can share with you because we don't do precisely what you're describing (and I wouldn't be allowed to share our git repo even if we did)

But generally we used Hashicorp Vault for storing all secrets.  We use Vault in two ways; firstly to dynamically get secrets at runtime, the way you're invisaging for your prolog script.  More commonly we use Vault to store secrets which are then fetched during infrastructure deployment and included/updated in the configuration files, typically using Ansible.  We don't want secrets stored in the Ansible code, obviously, so Ansible fetches those secrets from Vault when the playbook runs.  Generally speaking, the GitHub action running Ansible to deploy the code runs as a service account which has the right permission to fetch the secrets from Vault.

This is of course somewhat different from the approach you need at a per-job level, but the tooling is similar.

Tim

From: sacawulu <cyusedfzfb@gmail.com>
Date: Wednesday, 4 November 2026 at 11:00
To: Cutts, Tim <tim.cutts@astrazeneca.com>; slurm-users@schedmd.com <slurm-users@schedmd.com>
Subject: Re: [slurm-users] Native Lustre encryption with Slurm: loading the key at job time without storing it on the remote network

This Message Is From an Untrusted Sender
You have not previously corresponded with this sender.
Report if suspicious or ignore.
 
Hi Tim, Thanks for your answer, I waited a bit, to see if there were other ideas as well. Seems not. :-) Do you perhaps have a github or something, that I could check as an inspiration to see how you connected slurm & lustre & Hashicorp Vault to each other? Thanks! On 10/31/26 11:47, Cutts, Tim wrote: > I would use a proper secret store for this.  Hashicorp Vault or > something similar.  That's what we use for secret management across most > of our HPC and HPC-adjacent services. > > For those more in bed with AWS you could use AWS Secrets Manager > > Tim > > *From: *sacawulu via slurm-users <slurm-users@lists.schedmd.com> > *Date: *Saturday, 31 October 2026 at 10:26 > *To: *slurm-users@schedmd.com <slurm-users@schedmd.com> > *Subject: *[slurm-users] Native Lustre encryption with Slurm: loading > the key at job time without storing it on the remote network > > This Message Is From an External Sender > This message came from outside your organization. > Report if suspicious or ignore. > > Hi all, I’m new to this list, so hello to everyone. :-) Goal We want to > use native lustre client-side encryption (fscrypt) to protect data on a > remote HPC cluster, and run Slurm jobs that can work with that data. > Current (not so good) concept: The key would temporarily live on the > remote network: Key on NFS: /home/username/enc.key Encrypted data on > Lustre: /lustre/project The workflow would be: 1. A Slurm prolog loads > the key into the kernel keyring on the compute nodes. 2. The same script > deletes enc.key from the remote network. 3. The job runs with access to > the decrypted data. 4. A Slurm epilog script removes the key from the > node’s keyring. The obvious problem: Until the job is scheduled, the key > and the encrypted data are on the same remote network, which defeats > much of the purpose. Accepted risks We are aware of other attack > vectors: remote admins can read enc.key on NFS, and the key can be read > from RAM while the job runs. Strict ACLs are essential, and we run the > computes --exclusive We accept these risks for now. Question Does anyone > know a way to avoid storing enc.key on the remote network before the job > starts? Can the key be 'injected' at job start from our side (pushed or > pulled) Does slurm have any native mechanism to deal with this? Is there > a better pattern for this? Have you implemented something like this on > your site, and how? Note: we have an existing site-to-site ipsec tunnel > that we could use. Any tips or experiences are welcome. Thanks! -- > slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe > send an email to slurm-users-leave@lists.schedmd.com > > ------------------------------------------------------------------------ > > AstraZeneca UK Limited is a company incorporated in England and Wales > with registered number:03674842 and its registered office at 1 Francis > Crick Avenue, Cambridge Biomedical Campus, Cambridge, CB2 0AA. > > This e-mail and its attachments are intended for the above named > recipient only and may contain confidential and privileged information. > If they have come to you in error, you must not copy or show them to > anyone; instead, please reply to this e-mail, highlighting the error to > the sender and then immediately delete the message. For information > about how AstraZeneca UK Limited and its affiliates may process > information, personal data and monitor communications, please see our > privacy notice at http://www.astrazeneca.com <https://www.astrazeneca.com > >

AstraZeneca UK Limited is a company incorporated in England and Wales with registered number:03674842 and its registered office at 1 Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge, CB2 0AA.

This e-mail and its attachments are intended for the above named recipient only and may contain confidential and privileged information. If they have come to you in error, you must not copy or show them to anyone; instead, please reply to this e-mail, highlighting the error to the sender and then immediately delete the message. For information about how AstraZeneca UK Limited and its affiliates may process information, personal data and monitor communications, please see our privacy notice at www.astrazeneca.com