<div dir="ltr">Hi Lisa,<div><br>Below is my jupyterhub slurm config. It uses the profiles, which allows you to spawn different sized jobs.  I found the most useful thing for debugging is to make sure that the --output option is being honoured; any jupyter python errors will end up there, and to to explicitly set the python environment at the start of the script. (The example below uses conda, replace with whatever makes sense in your environment).</div><div><br></div><div>Hope that helps,</div><div><br>Guy</div><div><br></div><div><br></div><div>#Extend timeouts to deal with slow job launch</div><div><font face="monospace">c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'<br>c.Spawner.start_timeout=120<br>c.Spawner.term_timeout=20<br>c.Spawner.http_timeout = 120<br><br># Set up the various sizes of job<br>c.ProfilesSpawner.profiles = [<br>("Local server: (Run on local machine)", "local", "jupyterhub.spawner.LocalProcessSpawner", {'ip':'0.0.0.0'}),<br>("Single CPU: (1 CPU, 8GB, 48 hrs)", "cpu1", "batchspawner.SlurmSpawner",<br> dict(req_options=" -n 1 -t 48:00:00 -p normal --mem=8G ")),<br>("Single GPU: (1 CPU, 1 GPU, 8GB, 48 hrs)", "gpu1", "batchspawner.SlurmSpawner",<br> dict(req_options=" -n 1 -t 48:00:00 -p normal --mem=8G --gres=gpu:k40:1")),<br>("Whole Node: (32 CPUs, 128 GB, 48 hrs)", "node1", "batchspawner.SlurmSpawner",<br> dict(req_options=" -n 32 -N 1  -t 48:00:00 -p normal --mem=127000M")),<br>("Whole GPU Node: (32 CPUs, 2 GPUs, 128GB, 48 hrs)", "gnode1", "batchspawner.SlurmSpawner",<br> dict(req_options=" -n 32 -N 1  -t 48:00:00 -p normal --mem=127000M --gres=gpu:k40:2")),<br>]<br><br>#Configure the batch job. Make sure --output is set and explicitly set up</font></div><div><font face="monospace">#the jupyterhub python environment<br>c.SlurmSpawner.batch_script = """#!/bin/bash<br>#SBATCH --output={homedir}/jupyterhub_slurmspawner_%j.log<br>#SBATCH --job-name=spawner-jupyterhub<br>#SBATCH --chdir={homedir}<br>#SBATCH --export={keepvars}<br>#SBATCH --get-user-env=L<br>#SBATCH {options}<br>trap 'echo SIGTERM received' TERM<br> . /usr/local/jupyterhub/miniconda3/etc/profile.d/conda.sh<br>conda activate /usr/local/jupyterhub/jupyterhub<br>which jupyterhub-singleuser<br>{cmd}<br>echo "jupyterhub-singleuser ended gracefully"<br>"""</font><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 5 May 2020 at 01:27, Lisa Kay Weihl <<a href="mailto:lweihl@bgsu.edu">lweihl@bgsu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">




<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I have a single server with 2 cpu, 384gb memory and 4 gpu (GeForce RTX 2080 Ti). </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Use is to be for GPU ML computing and python based data science.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
One faculty wants jupyter notebooks, other faculty member is used to using CUDA for GPU but has only done it on a workstation in his lab with a GUI.  New faculty member coming in has used nvidia-docker container for GPU (I think on a large cluster, we are just
 getting started)</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I'm charged with making all this work and hopefully all at once. Right now I'll take one thing working.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
So I managed to get Slurm-20.02.1 installed with CUDA-10.2 on CentOS 7 (SE Linux enabled). I posted once before about having trouble getting that combination correct and I finally worked that out. Most of the tests in the test suite seem to run okay. I'm trying
 to start with very basic Slurm configuration so I haven't enabled accounting. </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b>For reference here is my slurm.conf</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"># slurm.conf file generated by configurator easy.html.</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"># Put this file on all nodes of your cluster.</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"># See the slurm.conf man page for more information.</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">SlurmctldHost=cs-host</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255);min-height:13px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#authentication</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">AuthType=auth/munge</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">CacheGroups = 0</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">CryptoType=crypto/munge</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255);min-height:13px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#Add GPU support</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">GresTypes=gpu</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255);min-height:13px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#MailProg=/bin/mail</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">MpiDefault=none</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#MpiParams=ports=#-#</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255);min-height:13px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#service</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">ProctrackType=proctrack/cgroup</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">ReturnToService=1</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">SlurmctldPidFile=/var/run/slurmctld.pid</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#SlurmctldPort=6817</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">SlurmdPidFile=/var/run/slurmd.pid</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#SlurmdPort=6818</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">SlurmdSpoolDir=/var/spool/slurmd</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">SlurmUser=slurm</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#SlurmdUser=root</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">StateSaveLocation=/var/spool/slurmctld</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">SwitchType=switch/none</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">TaskPlugin=task/affinity</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"># TIMERS</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#KillWait=30</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#MinJobAge=300</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#SlurmctldTimeout=120</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">SlurmdTimeout=1800</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"># SCHEDULING</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">SchedulerType=sched/backfill</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">SelectType=select/cons_tres</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">SelectTypeParameters=CR_Core_Memory</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">PriorityType=priority/multifactor</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">PriorityDecayHalfLife=3-0</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">PriorityMaxAge=7-0</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">PriorityFavorSmall=YES</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">PriorityWeightAge=1000</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">PriorityWeightFairshare=0</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">PriorityWeightJobSize=125</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">PriorityWeightPartition=1000</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">PriorityWeightQOS=0</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"># LOGGING AND ACCOUNTING</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">AccountingStorageType=accounting_storage/none</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">ClusterName=cs-host</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#JobAcctGatherFrequency=30</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">JobAcctGatherType=jobacct_gather/none</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">SlurmctldDebug=info</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">SlurmctldLogFile=/var/log/slurmctld.log</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#SlurmdDebug=info</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">SlurmdLogFile=/var/log/slurmd.log</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"># COMPUTE NODES</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">NodeName=cs-host CPUs=24 RealMemory=385405 Sockets=2 CoresPerSocket=6 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:4</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255);min-height:13px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#PARTITIONS</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">PartitionName=DEFAULT Nodes=cs-host Shared=FORCE:1 Default=YES MaxTime=INFINITE State=UP</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">PartitionName=faculty<span> 
</span>Priority=10 Default=YES</span></p>
<br>
</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I have jupyterhub running as part of RedHat SCL. It works fine with no integration with Slurm. Now I'm trying to use batchspawner to start a server for the user.  Right now I'm just trying one configuration from within the jupyterhub_config.py and trying to
 keep it simple (see below).</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b>When I connect I get this error:</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt">500: Internal Server Error</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Error in Authenticator.pre_spawn_start: RuntimeError The Jupyter batch job has disappeared while pending in the queue or died immediately after starting.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b>In the jupyterhub.log:</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[I 2020-05-04 19:47:58.604 JupyterHub base:707] User logged in: csadmin</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[I 2020-05-04 19:47:58.606 JupyterHub log:174] 302 POST /hub/login?next= -> /hub/spawn (<a href="mailto:csadmin@127.0.0.1" target="_blank">csadmin@127.0.0.1</a>) 227.13ms</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[I 2020-05-04 19:47:58.748 JupyterHub batchspawner:248] Spawner submitting job using sudo -E -u csadmin sbatch --parsable</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[I 2020-05-04 19:47:58.749 JupyterHub batchspawner:249] Spawner submitted script:</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>   
</span>#!/bin/bash</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>   
</span>#SBATCH --partition=faculty</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>   
</span>#SBATCH --time=8:00:00</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>   
</span>#SBATCH --output=/home/csadmin/jupyterhub_slurmspawner_%j.log</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>   
</span>#SBATCH --job-name=jupyterhub-spawner</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>   
</span>#SBATCH --cpus-per-task=1</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>   
</span>#SBATCH --chdir=/home/csadmin</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>   
</span>#SBATCH --uid=csadmin</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255);min-height:13px">
<span style="font-variant-ligatures:no-common-ligatures"><span>    </span></span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>   
</span>env</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>   
</span>which jupyterhub-singleuser</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>   
</span>batchspawner-singleuser jupyterhub-singleuser --ip=0.0.0.0</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255);min-height:13px">
<span style="font-variant-ligatures:no-common-ligatures"><span>    </span></span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[I 2020-05-04 19:47:58.831 JupyterHub batchspawner:252] Job submitted. cmd: sudo -E -u csadmin sbatch --parsable output: 7117</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[W 2020-05-04 19:47:59.481 JupyterHub batchspawner:377] Job<span> 
</span>neither pending nor running.</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255);min-height:13px">
<span style="font-variant-ligatures:no-common-ligatures"><span>    </span></span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[E 2020-05-04 19:47:59.482 JupyterHub user:640] Unhandled error starting csadmin's server: The Jupyter batch job has disappeared while pending in the queue
 or died immediately after starting.</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[W 2020-05-04 19:47:59.518 JupyterHub web:1782] 500 GET /hub/spawn (127.0.0.1): Error in Authenticator.pre_spawn_start: RuntimeError The Jupyter batch job
 has disappeared while pending in the queue or died immediately after starting.</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[E 2020-05-04 19:47:59.521 JupyterHub log:166] {</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>     
</span>"X-Forwarded-Host": "localhost:8000",</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>     
</span>"X-Forwarded-Proto": "http",</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>     
</span>"X-Forwarded-Port": "8000",</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>     
</span>"X-Forwarded-For": "127.0.0.1",</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>     
</span>"Cookie": "jupyterhub-hub-login=[secret]; _xsrf=[secret]; jupyterhub-session-id=[secret]",</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>     
</span>"Accept-Language": "en-US,en;q=0.9",</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>     
</span>"Accept-Encoding": "gzip, deflate, br",</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>     
</span>"Referer": "<a href="http://localhost:8000/hub/login" target="_blank">http://localhost:8000/hub/login</a>",</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>     
</span>"Sec-Fetch-Dest": "document",</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>     
</span>"Sec-Fetch-User": "?1",</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>     
</span>"Sec-Fetch-Mode": "navigate",</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>     
</span>"Sec-Fetch-Site": "same-origin",</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>     
</span>"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>     
</span>"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36",</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>     
</span>"Upgrade-Insecure-Requests": "1",</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>     
</span>"Cache-Control": "max-age=0",</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>     
</span>"Connection": "close",</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>     
</span>"Host": "localhost:8000"</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><span>   
</span>}</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[E 2020-05-04 19:47:59.522 JupyterHub log:174] 500 GET /hub/spawn (<a href="mailto:csadmin@127.0.0.1" target="_blank">csadmin@127.0.0.1</a>) 842.87ms</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[I 2020-05-04 19:49:05.294 JupyterHub proxy:320] Checking routes</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[I 2020-05-04 19:54:05.292 JupyterHub proxy:320] Checking routes</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<br>
</p>
<div style="font-style:normal;font-variant-caps:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;margin:0px;background-color:rgb(255,255,255)">
<b style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;background-color:rgb(255,255,255)">In the slurmd.log (which I don't see as helpful):</b><br>
</div>
<div style="font-style:normal;font-variant-caps:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;margin:0px;background-color:rgb(255,255,255)">
<b style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;background-color:rgb(255,255,255)"><br>
</b></div>
<div style="font-style:normal;font-variant-caps:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;margin:0px;background-color:rgb(255,255,255)">
<b style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;background-color:rgb(255,255,255)"><br>
</b></div>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[2020-05-04T19:47:58.931] task_p_slurmd_batch_request: 7117</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[2020-05-04T19:47:58.931] task/affinity: job 7117 CPU input mask for node: 0x000003</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[2020-05-04T19:47:58.931] task/affinity: job 7117 CPU final HW mask for node: 0x001001</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[2020-05-04T19:47:58.932] _run_prolog: run job script took usec=473</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[2020-05-04T19:47:58.932] _run_prolog: prolog with lock for job 7117 ran for 0 seconds</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[2020-05-04T19:47:58.932] Launching batch job 7117 for UID 1001</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[2020-05-04T19:47:58.967] [7117.batch] task_p_pre_launch: Using sched_affinity for tasks</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[2020-05-04T19:47:58.978] [7117.batch] sending REQUEST_COMPLETE_BATCH_SCRIPT, error:0 status:32512</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">[2020-05-04T19:47:58.982] [7117.batch] done with job</span></p>
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b>In the jupyterhub_config.py (just the part for batchspawner):</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b><br>
</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">c = get_config()</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">c.JupyterHub.spawner_class = 'batchspawner.SlurmSpawner'</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures"># Even though not used, needed to register batchspawner interface</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">import batchspawner
<span>   </span></span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255);min-height:13px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">c.Spawner.http_timeout = 120</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255);min-height:13px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">c.SlurmSpawner.req_nprocs = '1'</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">c.SlurmSpawner.req_runtime = '8:00:00'</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">c.SlurmSpawner.req_partition = 'faculty'</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">c.SlurmSpawner.req_memory = '128gb'</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255);min-height:13px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">c.SlurmSpawner.start_timeout = 240</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255);min-height:13px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">c.SlurmSpawner.batch_script = '''#!/bin/bash</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#SBATCH --partition={partition}</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#SBATCH --time={runtime}</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#SBATCH --output={homedir}/jupyterhub_slurmspawner_%j.log</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#SBATCH --job-name=jupyterhub-spawner</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#SBATCH --cpus-per-task={nprocs}</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#SBATCH --chdir=/home/{username}</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">#SBATCH --uid={username}</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255);min-height:13px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">env</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">which jupyterhub-singleuser</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">{cmd}</span></p>
<p style="font:11px Menlo;margin:0px;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">'''</span></p>
<br>
</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I will admit that I don't understand all of this completely as I haven't written a lot of bash scripts. I'm getting that some of the things in {} are environment variables and others come from within this file and it seems they must be specifically defined
 in the batchspawner software somewhere.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Is the last piece trying to find the path of jupyterhub-singleuser and then launch it with {cmd}</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Feel free to tell me to go read the docs but be gentle <span id="gmail-m_5338306197073148082🙂">🙂</span> Because of the request to make ALL of this work ASAP I've been skimming and trying to pick up as much as I can and then going off examples trying to make this work.  I have a feeling
 that this command: <span style="color:rgb(0,0,0);font-family:"Andale Mono";font-size:12px;font-variant-ligatures:no-common-ligatures;background-color:rgb(255,255,255);display:inline">sudo -E -u csadmin sbatch --parsable output: 7117 </span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
is what is incorrect and causing the problems. Clearly something isn't starting that should be.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
If you can shed any light on anything or any info online that might help me I'd much appreciate it. I'm really beating my head over this one and I know inexperience isn't helping.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
When I figure out this simple config then I want to have the profile where I can setup several settings and have the user select.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
One other basic question. I'm assuming in Slurm language my server is considered to have 24 CPU with the cores and threads so that any of the Slurm settings that refer to things like CPU per task I could specify up to 24 if a user wanted. Also, in this case
 the node will always be 1 since we only have 1 server.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks!</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="gmail-m_5338306197073148082Signature">
<div id="gmail-m_5338306197073148082divtagdefaultwrapper" style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255);font-family:Calibri,Arial,Helvetica,sans-serif">
<div><font size="2">
<div>***************************************************************<br>
<p style="margin-top:0px;margin-bottom:0px">
<span>Lisa Weihl </span><em>Systems Administrator</em></p>
<p style="margin-top:0px;margin-bottom:0px">
<em>Computer Science, Bowling Green State University<br>
</em>Tel: (419) 372-0116   |    Fax: (419) 372-8061<br>
<a href="mailto:lweihl@bgsu.edu" target="_blank">lweihl@bgsu.edu</a><br>
<a href="http://www.bgsu.edu" target="_blank">www.bgsu.edu</a>​<br>
</p>
</div>
</font></div>
</div>
</div>
</div>
</div>

</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Dr. Guy Coates<br>+44(0)7801 710224<div><br></div></div></div>