[slurm-users] How to use a pyhon virtualenv with srun?

[Williams, Gareth (IM&T, Black Mountain)]

Hi Yann,

The remaining problem may be that the ray processes are not waited on. I'm not sure, but hope this gets you looking in the right place. You may need to sleep indefinitely in the scripts that run the worker ray processes then when the master is finished making them work, cancel the workers then exit the main script.  If you just exit the main script computecanada will probably clean up for you automatically - but it is polite to clean up after yourself.

Gareth 

-----Original Message-----
From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of Yann Bouteiller
Sent: Monday, 18 November 2019 1:49 PM
To: Slurm User Community List <slurm-users at lists.schedmd.com>
Subject: Re: [slurm-users] How to use a pyhon virtualenv with srun?

Hello Brian, thank you for your answer.

Actually, you are not allowed to install things in your home on computecanada, this is why you need to install everything in a virtualenv with pip install. Also, you have to install each virtualenv in $SLURM_TMDIR which is the local drive of the node, because everything else is slow, so I think I cannot share homes.

Actually I succeeded at installing different virtualenvs on different nodes using a script for each worker that creates a local virtualenv, installs ray on it, and connects to the ray server running in the virtualenv of the head node (I mean the primary node, yes). I just call these scripts with srun. However, for some reason, the workers seem to connect fine to the server but are detected as dead after a
while: https://groups.google.com/forum/#!topic/ray-dev/INB_zVS5PWY

Yann



Brian Andrus <toomuchit at gmail.com> a écrit :

> I suspect when you say "head node" you mean the primary node from the 
> nodes your were allocated.
>
> Normally, when you use pip as a user, it installs in your home 
> directory. Are you certain all your nodes share the same homes?
> If they are merely synched, that would not be the same. Not actually 
> sharing homes could be the cause.
>
> Brian Andrus
>
>
> On 11/17/2019 11:24 AM, Yann Bouteiller wrote:
>>
>> Hello,
>>
>> I am trying to do this on computecanada, which is managed by slurm:  
>> https://ray.readthedocs.io/en/latest/deploying-on-slurm.html
>>
>> However, on computecanada, you cannot install things on nodes before 
>> the job has started, and you can only install things in a python 
>> virtualenv once the job has started.
>>
>> I can do:
>>
>> ```
>> module load python/3.7.4
>> source venv/bin/activate
>> pip install ray
>> ```
>>
>> in the bash script before calling everything else, but apparently 
>> this will only create-activate the virtualenv and install ray on the 
>> head node, but not on the remote nodes, so calling
>>
>> ```
>> srun --nodes=1 --ntasks=1 -w $node1 ray start --block --head
>> --redis-port=6379 --redis-password=$redis_password & # Starting the 
>> head ```
>>
>> will succeed, but later calling
>>
>> ```
>> for ((  i=1; i<=$worker_num; i++ ))
>> do
>>   node2=${nodes_array[$i]}
>>   srun --export=ALL --nodes=1 --ntasks=1 -w $node2 ray start --block 
>> --address=$ip_head --redis-pass$
>>   sleep 5
>> done
>>
>> ```
>>
>> will produce the following error:
>>
>> ```
>> slurmstepd: error: execve(): ray: No such file or directory
>> srun: error: cdr768: task 0: Exited with exit code 2
>> srun: Terminating job step 31218604.3 [2]+  Exit 2                  
>> srun --export=ALL --nodes=1
>> --ntasks=1 -w $node2 ray start --block --address=$ip_head 
>> --redis-password=$redis_password ```
>>
>> How can I tackle this issue, please? I am a beginner with slurm so I 
>> am not sure what is the problem here. Here is my whole sbatch
>> script:
>>
>> ```
>> #!/bin/bash
>>
>> #SBATCH --job-name=test
>> #SBATCH --cpus-per-task=5
>> #SBATCH --mem-per-cpu=1000M
>> #SBATCH --nodes=3
>> #SBATCH --tasks-per-node 1
>>
>> worker_num=2 # Must be one less that the total number of nodes 
>> nodes=$(scontrol show hostnames $SLURM_JOB_NODELIST) # Getting the 
>> node names nodes_array=( $nodes )
>>
>> module load python/3.7.4
>> source venv/bin/activate
>> pip install ray
>>
>> node1=${nodes_array[0]}
>> ip_prefix=$(srun --nodes=1 --ntasks=1 -w $node1 hostname
>> --ip-address) # Making address
>> suffix=':6379'
>> ip_head=$ip_prefix$suffix
>> redis_password=$(uuidgen)
>> export ip_head # Exporting for latter access by trainer.py
>>
>> srun --nodes=1 --ntasks=1 -w $node1 ray start --block --head
>> --redis-port=6379 --redis-password=$redis_password & # Starting the 
>> head sleep 5
>>
>> for ((  i=1; i<=$worker_num; i++ ))
>> do
>>   node2=${nodes_array[$i]}
>>   srun --export=ALL --nodes=1 --ntasks=1 -w $node2 ray start --block 
>> --address=$ip_head --redis-password=$redis_password & # Starting the 
>> workers
>>   sleep 5
>> done
>>
>> python -u trainer.py $redis_password 15 # Pass the total number of 
>> allocated CPUs
>>
>> ```
>>
>> ---
>> Regards,
>> Yann
>>
>>