[slurm-users] Custom Gres for SSD

[Matthias Loose]

On 2023-07-24 09:50, Matthias Loose wrote:

Hi Shunran,

just read your question again. If you dont want users to share the SSD, 
like at all even if both have requested it you can basically skip the 
quota part of my awnser.

If you really only want one user per SSD per node you should set the 
gres variable in the node configuration to 1 just like you did and then 
implement the prolog/epilog solution (without quotas). If the mounted 
SSD can only be written to by root no one else can use it and the job 
that requested it get a folder created by the prolog.

What we also do ist export the folder name in the user/task prolog to 
the environment so he can easely use it.

Out task prolog:

   #!/bin/bash
   #PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

   local_dir="/local"

   SLURM_TMPDIR="${local_dir}/job_${SLURM_JOBID}"

   # check for /local job dir
   if [[ -d ${SLURM_TMPDIR} ]]; then
     # set tempdir env vars
     echo "export SLURM_TMPDIR=${SLURM_TMPDIR}"
     echo "export TMPDIR=${SLURM_TMPDIR}"
     echo "export JAVA_TOOL_OPTIONS=\"-Djava.io.tmpdir=${SLURM_TMPDIR}\""
   fi

Kind regards, Matt

> Hi Shunran,
> 
> we do something very similar. I have nodes with 2 SSDs in a Raid1
> mounted on /local. We defined a gres ressource just like you and
> called it local. We define the ressource in the gres.conf like this:
> 
>   # LOCAL
>   NodeName=hpc-node[01-10] Name=local
> 
> and add the ressource in counts of GB to the slurm.nodes.conf:
> 
>   NodeName=hpc-node01  CPUs=256 RealMemory=... Gres=local:3370
> 
> So in this case the node01 has 3370 counts or GB of the gres "local"
> available for reservation. Now slurm tracks that resource for you and
> users can reserve counts of /local space. But there is still one big
> problem, SLURM hast no idea what local is and as u correctly noted,
> others can just use it. I solved this the following way:
> 
> - /local ist owned by root, so no user can just write to it
> - the node prolog creates a folder in /local in this form:
> /local/job_<SLURM_JOB_ID> and makes the job owner of it
> - the node epilog deletes that folder
> 
> This way you have already solved the problem of people/jobs not having
> reserved any local using it. But there ist still no enforcement of
> limits. For that I use quotas.
> My /local is XFS formatted and XFS has a nifty feature called project
> quotas, where you can set the quota for a folder.
> 
> This is my node prolog script for this purpose:
> 
>   #!/bin/bash
>   PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> 
>   local_dir="/local"
>   local_job=0
> 
>   ## DETERMINE GRES:LOCAL
>   # get job gres
>   JOB_TRES=$(scontrol show JobID=${SLURM_JOBID} | grep "TresPerNode="
> | cut -d '=' -f 2 | tr ',' ' ')
> 
>   # parse for local
>   for gres in ${JOB_TRES}; do
>     key=$(echo ${gres} | cut -d ':' -f 2 | tr '[:upper:]' '[:lower:]')
>     if [[ ${key} == "local" ]]; then
>       local_job=$(echo ${gres} | cut -d ':' -f 3)
>       break
>     fi
>   done
> 
>   # make job local-dir if requested
>   if [[ ${local_job} -ne 0 ]]; then
>     # make local-dir for job
>     SLURM_TMPDIR="${local_dir}/job_${SLURM_JOBID}"
>     mkdir ${SLURM_TMPDIR}
> 
>     # conversion
>     local_job=$((local_job * 1024 * 1024))
> 
>     # set hard limit to requested size + 5%
>     hard_limit=$((local_job * 105 / 100))
> 
>     # create project quota and set limits
>     xfs_quota -x -c "project -s -p ${SLURM_TMPDIR} ${SLURM_JOBID}" 
> ${local_dir}
>     xfs_quota -x -c "limit -p bsoft=${local_job}k bhard=${hard_limit}k
> ${SLURM_JOBID}" ${local_dir}
> 
>     chown ${SLURM_JOB_USER}:0 ${SLURM_TMPDIR}
>     chmod 750 ${SLURM_TMPDIR}
>   fi
> 
>   exit 0
> 
> This is my epilog:
> 
>   #!/bin/bash
>   PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> 
>   local_dir="/local"
>   SLURM_TMPDIR="${local_dir}/job_${SLURM_JOBID}"
> 
>   # remove the quota
>   xfs_quota -x -c "limit -p bsoft=0m bhard=0m ${SLURM_JOBID}" 
> ${local_dir}
> 
>   # remove the folder
>   if [[ -d ${SLURM_TMPDIR} ]]; then
>     rm -rf --one-file-system ${SLURM_TMPDIR}
>   fi
> 
>   exit 0
> 
> In order to use project quota you would need to activate it by using
> this mount flag: pquota in the fstab.
> I give the user 5% more than he requested. You just have to make sure
> that you configure available space - 5% in the nodes.conf.
> 
> This is what we do and it works great.
> 
> Kind regards, Matt
> 
> 
> On 2023-07-24 05:48, Shunran Zhang wrote:
>> Hi all,
>> 
>> I am attempting to setup a gres to manage jobs that need a
>> scratch space, but only a few of our computational nodes are
>> equipped with SSD for such scratch space. Originally I setup a new
>> partition for those IO-bound jobs, but it ended up that those jobs
>> might be allocated to the same node thus fighting each other for
>> IO.
>> 
>> With a look over other settings it appears that the gres setting
>> looks promising. However I was having some difficulties figuring
>> out how to limit access to such space to those who requested
>> --gres=ssd:1.
>> 
>> For now I am using Flags=CountOnly to trust users who uses SSD
>> request for it, but apparently any job submitted to a node with
>> SSD can just use such space. Our scratch space implementation is 2
>> disks (sda and sdb) formatted to btrfs and RAID 0. What should I
>> do to enforce such limit on which job can use such space?
>> 
>> Related configurations for ref:
>> gres.conf: NodeName=scratch-1 Name=ssd Flags=CountOnly cgroup.conf:
>> ConstrainDevices=yes slurm.conf: GresTypes=gpu,ssd
>> NodeName=scratch-1 CPUs=88 Sockets=2 CoresPerSocket=22
>> ThreadsPerCore=2  RealMemory=180000 Gres=ssd:1 State=UNKNOWN
>> Sincerely,
>> S. Zhang