[slurm-users] slurm array with non-numeric index values

[Renfro, Michael]

If the 500 parameters happened to be filenames, you could do adapt like (appropriated from somewhere else, but I can’t find the reference quickly:

=====

#!/bin/bash
 # get count of files in this directory
NUMFILES=$(ls -1 *.inp | wc -l)
# subtract 1 as we have to use zero-based indexing (first element is 0)
ZBNUMFILES=$(($NUMFILES - 1))
# submit array of jobs to SLURM
if [ $ZBNUMFILES -ge 0 ]; then
  sbatch --array=0-$ZBNUMFILES array_job.sh
else
  echo "No jobs to submit, since no input files in this directory.”
fi

=====

with:

=====

#!/bin/bash
#SBATCH --nodes=1  --ntasks-per-node=1 --cpus-per-task=1
#SBATCH --time=00:01:00
#SBATCH --job-name array_demo_2
 
echo "All jobs in this array have:"
echo "- SLURM_ARRAY_JOB_ID=${SLURM_ARRAY_JOB_ID}"
echo "- SLURM_ARRAY_TASK_COUNT=${SLURM_ARRAY_TASK_COUNT}"
echo "- SLURM_ARRAY_TASK_MIN=${SLURM_ARRAY_TASK_MIN}"
echo "- SLURM_ARRAY_TASK_MAX=${SLURM_ARRAY_TASK_MAX}"
 
echo "This job in the array has:"
echo "- SLURM_JOB_ID=${SLURM_JOB_ID}"
echo "- SLURM_ARRAY_TASK_ID=${SLURM_ARRAY_TASK_ID}"

# grab our filename from a directory listing
FILES=($(ls -1 *.inp))
FILENAME=${FILES[$SLURM_ARRAY_TASK_ID]}
echo "My input file is ${FILENAME}”

# make new directory, change into it, and run
mkdir ${FILENAME}_out
cd ${FILENAME}_out
echo "First 10 lines of ../${FILENAME} are:" > ${FILENAME}_results.out
head ../${FILENAME} >> ${FILENAME}_results.out

=====

If the 500 parameters were lines in a file, the same logic would apply:

- subtract 1 from the number of lines in the file to determine the array limit
- add 1 to ${SLURM_ARRAY_TASK_ID} to get a line number for a specific parameter
- something like "sed -n ‘${TASK_ID_PLUS_ONE}p’ filename” to retrieve that parameter
- run the Python script with that value

> On Jul 15, 2020, at 3:13 PM, c b <breedthoughts.www at gmail.com> wrote:
> 
> External Email Warning
> This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.
> I'm trying to run an embarrassingly parallel experiment, with 500+ tasks that all differ in one parameter.  e.g.:
> 
> job 1 - script.py foo
> job 2 - script.py bar
> job 3 - script.py baz
> and so on.
> 
> This seems like a case where having a slurm array hold all of these jobs would help, so I could just submit one job to my cluster instead of 500 individual jobs.  It seems like sarray is only set up for varying an integer index parameter.  How would i do this for non-numeric values (say, if the parameter I'm varying is a string in a given list) ?
> 
>