[slurm-users] GRES GPU issues
Tina Friedrich
tina.friedrich at it.ox.ac.uk
Thu Dec 6 02:51:02 MST 2018
The available features / constraints aren't necessary; their purpose is
to offer a slightly more flexible way to request resources (esp. GPU).
As in, quite often people don't specifically need a P100 or V100, but
they can't run on a Kepler card; with the '--gres=gpu:p100:X' syntax
they can (I believe?) only ask for a specific model, which is a bit
limiting, and with the constraints, you can do something like '-C
gpu_gen:Pascal|gpu_gen:Volta'. That's why we have those.
Need to rush but I'll have a look at the config you sent later to see if
I can figure out what's going on.
Tina
On 05/12/2018 16:12, Lou Nicotra wrote:
> OK, after looking at your configs, I noticed that I was missing a
> "Gres=gpu" entry on my Nodename definition. Added and distributed...
> NodeName=tiger11 NodeAddr=X.X.X.X Sockets=2 CoresPerSocket=12
> ThreadsPerCore=2 Gres=gpu:1080gtx:0,gpu:k20:1 Feature=HyperThread
> State=UNKNOWN
>
> Assuming that 0 and 1 refer to device address as shown using nvidia-smi
> and it is not the number of GPUs in server... I have a multi GPU server
> with 8 GTXs, so I want to make sure I understand this correctly.
> scontrol shows...
> root at panther02 x86_64# scontrol show node=tiger11
> NodeName=tiger11 Arch=x86_64 CoresPerSocket=12
> CPUAlloc=0 CPUTot=48 CPULoad=19.96
> AvailableFeatures=HyperThread
> ActiveFeatures=HyperThread
> Gres=gpu:1080gtx:0,gpu:k20:1
> NodeAddr=X.X.X.X NodeHostName=tiger11 Version=18.08
> OS=Linux 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015
> RealMemory=1 AllocMem=0 FreeMem=268460 Sockets=2 Boards=1
> State=IDLE+DRAIN ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A
> MCS_label=N/A
> Partitions=tiger_1,compute_1
> BootTime=2018-04-02T13:30:12 SlurmdStartTime=2018-12-05T10:44:58
>
> CfgTRES=cpu=48,mem=1M,billing=48,gres/gpu=2,gres/gpu:1080gtx=1,gres/gpu:k20=1
> AllocTRES=
> CapWatts=n/a
> CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
> Reason=gres/gpu:1080gtx count too low (0 < 1)
> [slurm at 2018-12-05T10:36:28]
>
> What does the last line mean? nvidia-smi shows no jobs running on the
> 1080gtx...
>
>
> Also changed the gres.conf file to reflect my nodes that have 2
> different types of GPUs
> NodeName=tiger[02-04,06-09,11-14,16-19,21-22] Name=gpu Type=1080gtx
> File=/dev/nvidia0 Cores=0
> NodeName=tiger[02-04,06-09,11-14,16-19,21-22] Name=gpu Type=k20
> File=/dev/nvidia1 Cores=1
>
> This has allowed me to submit a GPU test job on tiger11. It is failing
> due to tensorflow environment parameters, but that is easy to fix...
>
> Another question, I see Tina has a number of available features listed
> for each node (cpu_gen, sku, cpu_mem, etc)... Is that necessary or is
> that just a sanity check?
>
> Once again, I like to thank all contributors to this thread... It has
> helped me get my cluster going!
>
> Thanks.
> Lou
>
>
>
> On Wed, Dec 5, 2018 at 9:41 AM Tina Friedrich
> <tina.friedrich at it.ox.ac.uk <mailto:tina.friedrich at it.ox.ac.uk>> wrote:
>
> Hello,
>
> don't mind sharing the config at all. Not sure it helps though, it's
> pretty basic.
>
> Picking an example node, I have
>
> [ ~]$ scontrol show node arcus-htc-gpu011
> NodeName=arcus-htc-gpu011 Arch=x86_64 CoresPerSocket=8
> CPUAlloc=16 CPUTot=16 CPULoad=20.43
>
> AvailableFeatures=cpu_gen:Haswell,cpu_sku:E5-2640v3,cpu_frq:2.60GHz,cpu_mem:64GB,gpu,gpu_mem:12GB,gpu_gen:Kepler,gpu_sku:K40,gpu_cc:3.5,
>
> ActiveFeatures=cpu_gen:Haswell,cpu_sku:E5-2640v3,cpu_frq:2.60GHz,cpu_mem:64GB,gpu,gpu_mem:12GB,gpu_gen:Kepler,gpu_sku:K40,gpu_cc:3.5,
> Gres=gpu:k40m:2
> NodeAddr=arcus-htc-gpu011 NodeHostName=arcus-htc-gpu011
> OS=Linux 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11
> UTC 2018
> RealMemory=63000 AllocMem=0 FreeMem=56295 Sockets=2 Boards=1
> State=ALLOCATED ThreadsPerCore=1 TmpDisk=0 Weight=96 Owner=N/A
> MCS_label=N/A
> Partitions=htc
> BootTime=2018-11-28T15:12:29 SlurmdStartTime=2018-11-28T17:58:55
> CfgTRES=cpu=16,mem=63000M,billing=16
> AllocTRES=cpu=16
> CapWatts=n/a
> CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
>
>
> gres.conf on arcus-htc-gpu011 is
>
> [ ~]$ cat /etc/slurm/gres.conf
> Name=gpu Type=k40m File=/dev/nvidia0
> Name=gpu Type=k40m File=/dev/nvidia1
>
> Relevant bits of slurm.conf are, I believe
>
> GresTypes=hbm,gpu
> (DebugFlags=Priority,Backfill,NodeFeatures,Gres,Protocol,TraceJobs)
>
> NodeName=arcus-htc-gpu009,arcus-htc-gpu[011-018] Weight=96 Sockets=2
> CoresPerSocket=8 ThreadsPerCore=1 RealMemory=63000 Gres=gpu:k40m:2
> Feature=cpu_gen:Haswell,cpu_sku:E5-2640v3,cpu_frq:2.60GHz,cpu_mem:64GB,gpu,gpu_mem:12GB,gpu_gen:Kepler,gpu_sku:K40,gpu_cc:3.5,
>
> Don't think I did anything else.
>
> I have other types of nodes - couple of P100s, couple of V100s, couple
> of K80s and one or two odd things (M40, P4).
>
> Used to run with a gres.conf that simply had 'Name=gpu
> File=/dev/nvidia[0-2]' (or [0-4], depending) and that also worked; I
> introduced the type when I gained a node that has two different nvidia
> cards, so what was on what port became important, not because the
> 'range' configuration caused problems.
>
> This wasn't a fresh install of 18.x - it was a 17.x installation that I
> upgraded to 18.x. Not sure if that makes a difference. I made no
> changes
> to anything (slurm.conf, gres.conf) with the update though. I just
> installed the new rpms.
>
> Tina
>
> On 05/12/2018 13:20, Lou Nicotra wrote:
> > Tina, thanks for confirming that GPU GRES resources work with
> 18.08... I
> > might just upgrade to 18.08.03 as I am running 18.08.0
> >
> > The nvidia devices exists on all servers and persistence is set.
> They
> > have been in there for a number of years and our users make use
> of them
> > daily. I can actually see that slurmd knows about them while
> restarting
> > the daemon:
> > [2018-12-05T08:03:35.989] Slurmd shutdown completing
> > [2018-12-05T08:03:36.015] Message aggregation disabled
> > [2018-12-05T08:03:36.016] gpu device number 0(/dev/nvidia0):c
> 195:0 rwm
> > [2018-12-05T08:03:36.017] gpu device number 1(/dev/nvidia1):c
> 195:1 rwm
> > [2018-12-05T08:03:36.059] slurmd version 18.08.0 started
> > [2018-12-05T08:03:36.059] slurmd started on Wed, 05 Dec 2018
> 08:03:36 -0500
> > [2018-12-05T08:03:36.059] CPUs=48 Boards=1 Sockets=2 Cores=12
> Threads=2
> > Memory=386757 TmpDisk=4758 Uptime=21324804 CPUSpecList=(null)
> > FeaturesAvail=(null) FeaturesActive=(null)
> >
> > Would you mind sharing the portions of the slurm.conf and
> corresponding
> > GRES definitions that you are using?. You have individual GRES
> files for
> > each server based on GPU type? I tried both, none of them work.
> >
> > My slurm.conf file has entries for GPUs as follows:
> > GresTypes=gpu
> > #AccountingStorageTRES=gres/gpu,gres/gpu:k20,gres/gpu:1080gtx
> > (currently commented out)
> >
> > gres.conf is as follows (had tried different configs, no change with
> > either one...)
> > # GPU Definitions
> > NodeName=tiger[01,05,10,15,20] Name=gpu Type=1080gtx
> File=/dev/nvidia0
> > Cores=0
> > NodeName=tiger[01,05,10,15,20] Name=gpu Type=1080gtx
> File=/dev/nvidia1
> > Cores=1
> > #NodeName=tiger[01,05,10,15,20] Name=gpu Type=1080gtx
> > File=/dev/nvidia[0-1] Cores=0,1
> >
> > NodeName=tiger[02-04,06-09,11-14,16-19,21-22] Name=gpu Type=k20
> > File=/dev/nvidia0 Cores=0
> > NodeName=tiger[02-04,06-09,11-14,16-19,21-22] Name=gpu Type=k20
> > File=/dev/nvidia1 Cores=1
> > #NodeName=tiger[02-04,06-09,11-14,16-19,21-22] Name=gpu Type=k20
> > File=/dev/nvidia[0-1] Cores=0,1
> >
> > What am I missing?
> >
> > Thanks...
> >
> >
> >
> >
> > On Wed, Dec 5, 2018 at 4:59 AM Tina Friedrich
> > <tina.friedrich at it.ox.ac.uk <mailto:tina.friedrich at it.ox.ac.uk>
> <mailto:tina.friedrich at it.ox.ac.uk
> <mailto:tina.friedrich at it.ox.ac.uk>>> wrote:
> >
> > I'm running 18.08.3, and I have a fair number of GPU GRES
> resources -
> > recently upgraded to 18.08.03 from a 17.x release. It's
> definitely not
> > as if they don't work in an 18.x release. (I do not
> distribute the same
> > gres.conf file everywhere though, never tried that.)
> >
> > Just a really stupid question - the /dev/nvidiaX devices do
> exist, I
> > assume? You are running nvidia-persistenced (or something
> similar) to
> > ensure the cards are up & the device files initialised etc?
> >
> > Tina
> >
> > On 04/12/2018 23:36, Brian W. Johanson wrote:
> > > Only thing to suggest once again is increasing the logging
> of both
> > > slurmctl and slurmd.
> > > As for downgrading, I wouldn't suggest running a 17.x slurmdbd
> > against a
> > > db built with 18.x. I imagine there are enough changes
> there to
> > cause
> > > trouble.
> > > I don't imagine downgrading will fix your issue, if you
> are running
> > > 18.08.0, the most recent release is 18.08.3. NEWS packed
> in the
> > > tarballs gives the fixes in the versions. I don't see any
> that
> > would
> > > fit you case.
> > >
> > >
> > > On 12/04/2018 02:11 PM, Lou Nicotra wrote:
> > >> Brian, I used a single gres.conf file and distributed to all
> > nodes...
> > >> Restarted all daemons, unfortunately scontrol still does not
> > show any
> > >> Gres resources for GPU nodes...
> > >>
> > >> Will try to roll back to 17.X release. Is it basically a
> matter of
> > >> removing 18.x rpms and installing 17's? Does the DB need
> to be
> > >> downgraded also?
> > >>
> > >> Thanks...
> > >> Lou
> > >>
> > >> On Tue, Dec 4, 2018 at 10:25 AM Brian W. Johanson
> > <bjohanso at psc.edu <mailto:bjohanso at psc.edu>
> <mailto:bjohanso at psc.edu <mailto:bjohanso at psc.edu>>
> > >> <mailto:bjohanso at psc.edu <mailto:bjohanso at psc.edu>
> <mailto:bjohanso at psc.edu <mailto:bjohanso at psc.edu>>>> wrote:
> > >>
> > >>
> > >> Do one more pass through making sure
> > >> s/1080GTX/1080gtx and s/K20/k20
> > >>
> > >> shutdown all slurmd, slurmctld, start slurmctl, start
> slurmd
> > >>
> > >>
> > >> I find it less confusing to have a global gres.conf
> file. I
> > >> haven't used a list (nvidia[0-1), mainly because I
> want to
> > specify
> > >> thethe cores to use for each gpu.
> > >>
> > >> gres.conf would look something like...
> > >>
> > >> NodeName=tiger[02-04,06-09,11-14,16-19,21-22]
> Name=gpu Type=k80
> > >> File=/dev/nvidia0 Cores=0
> > >> NodeName=tiger[02-04,06-09,11-14,16-19,21-22]
> Name=gpu Type=k80
> > >> File=/dev/nvidia1 Cores=1
> > >> NodeName=tiger[01,05,10,15,20] Name=gpu Type=1080gtx
> > >> File=/dev/nvidia0 Cores=0
> > >> NodeName=tiger[01,05,10,15,20] Name=gpu Type=1080gtx
> > >> File=/dev/nvidia1 Cores=1
> > >>
> > >> which can be distributed to all nodes.
> > >>
> > >> -b
> > >>
> > >>
> > >> On 12/04/2018 09:55 AM, Lou Nicotra wrote:
> > >>> Brian, the specific node does not show any gres...
> > >>> root at panther02 slurm# scontrol show partition=tiger_1
> > >>> PartitionName=tiger_1
> > >>> AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
> > >>> AllocNodes=ALL Default=YES QoS=N/A
> > >>> DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO
> > >>> GraceTime=0 Hidden=NO
> > >>> MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0
> LLN=NO
> > >>> MaxCPUsPerNode=UNLIMITED
> > >>> Nodes=tiger[01-22]
> > >>> PriorityJobFactor=1 PriorityTier=1 RootOnly=NO
> ReqResv=NO
> > >>> OverSubscribe=NO
> > >>> OverTimeLimit=NONE PreemptMode=OFF
> > >>> State=UP TotalCPUs=1056 TotalNodes=22
> > SelectTypeParameters=NONE
> > >>> JobDefaults=(null)
> > >>> DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
> > >>>
> > >>> root at panther02 slurm# scontrol show node=tiger11
> > >>> NodeName=tiger11 Arch=x86_64 CoresPerSocket=12
> > >>> CPUAlloc=0 CPUTot=48 CPULoad=11.50
> > >>> AvailableFeatures=HyperThread
> > >>> ActiveFeatures=HyperThread
> > >>> Gres=(null)
> > >>> NodeAddr=X.X.X.X NodeHostName=tiger11 Version=18.08
> > >>> OS=Linux 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19
> > 22:10:57 UTC 2015
> > >>> RealMemory=1 AllocMem=0 FreeMem=269695 Sockets=2
> Boards=1
> > >>> State=IDLE ThreadsPerCore=2 TmpDisk=0 Weight=1
> Owner=N/A
> > >>> MCS_label=N/A
> > >>> Partitions=tiger_1,compute_1
> > >>> BootTime=2018-04-02T13:30:12
> > SlurmdStartTime=2018-12-03T16:13:22
> > >>> CfgTRES=cpu=48,mem=1M,billing=48
> > >>> AllocTRES=
> > >>> CapWatts=n/a
> > >>> CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
> > >>> ExtSensorsJoules=n/s ExtSensorsWatts=0
> ExtSensorsTemp=n/s
> > >>>
> > >>> So, something is not setup correctly... Could it be
> a 18.X bug?
> > >>>
> > >>> Thanks.
> > >>>
> > >>>
> > >>> On Tue, Dec 4, 2018 at 9:31 AM Lou Nicotra
> > >>> <lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>
> > <mailto:lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>>
> <mailto:lnicotra at interactions.com <mailto:lnicotra at interactions.com>
> > <mailto:lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>>>> wrote:
> > >>>
> > >>> Thanks Michael. I will try 17.x as I also could
> not see
> > >>> anything wrong with my settings... Will report back
> > >>> afterwards...
> > >>>
> > >>> Lou
> > >>>
> > >>> On Tue, Dec 4, 2018 at 9:11 AM Michael Di Domenico
> > >>> <mdidomenico4 at gmail.com
> <mailto:mdidomenico4 at gmail.com> <mailto:mdidomenico4 at gmail.com
> <mailto:mdidomenico4 at gmail.com>>
> > <mailto:mdidomenico4 at gmail.com
> <mailto:mdidomenico4 at gmail.com> <mailto:mdidomenico4 at gmail.com
> <mailto:mdidomenico4 at gmail.com>>>> wrote:
> > >>>
> > >>> unfortunately, someone smarter then me will
> have to
> > help
> > >>> further. I'm
> > >>> not sure i see anything specifically wrong.
> The one
> > >>> thing i might try
> > >>> is backing the software down to a 17.x release
> > series. I
> > >>> recently
> > >>> tried 18.x and had some issues. I can't say
> whether
> > >>> it'll be any
> > >>> different, but you might be exposing an
> undiagnosed bug
> > >>> in the 18.x
> > >>> branch
> > >>> On Mon, Dec 3, 2018 at 4:17 PM Lou Nicotra
> > >>> <lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>
> > <mailto:lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>>
> > >>> <mailto:lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>
> > <mailto:lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>>>> wrote:
> > >>> >
> > >>> > Made the change in the gres.conf on local
> server file
> > >>> and restarted slurmd and slurmctld on master....
> > >>> Unfortunately same error...
> > >>> >
> > >>> > Distributed corrected gres.conf to all k20
> servers,
> > >>> restarted slurmd and slurmdctl... Still
> has same
> > error...
> > >>> >
> > >>> > On Mon, Dec 3, 2018 at 4:04 PM Brian W.
> Johanson
> > >>> <bjohanso at psc.edu <mailto:bjohanso at psc.edu>
> <mailto:bjohanso at psc.edu <mailto:bjohanso at psc.edu>>
> > <mailto:bjohanso at psc.edu <mailto:bjohanso at psc.edu>
> <mailto:bjohanso at psc.edu <mailto:bjohanso at psc.edu>>>> wrote:
> > >>> >>
> > >>> >> Is that a lowercase k in k20 specified in
> the batch
> > >>> script and nodename and a uppercase K
> specified in
> > gres.conf?
> > >>> >>
> > >>> >> On 12/03/2018 09:13 AM, Lou Nicotra wrote:
> > >>> >>
> > >>> >> Hi All, I have recently set up a slurm
> cluster
> > with my
> > >>> servers and I'm running into an issue while
> submitting
> > >>> GPU jobs. It has something to to with gres
> > >>> configurations, but I just can't seem to
> figure out
> > what
> > >>> is wrong. Non GPU jobs run fine.
> > >>> >>
> > >>> >> The error is as follows:
> > >>> >> sbatch: error: Batch job submission
> failed: Invalid
> > >>> Trackable RESource (TRES) specification after
> > submitting
> > >>> a batch job.
> > >>> >>
> > >>> >> My batch job is as follows:
> > >>> >> #!/bin/bash
> > >>> >> #SBATCH --partition=tiger_1 # partition
> name
> > >>> >> #SBATCH --gres=gpu:k20:1
> > >>> >> #SBATCH --gres-flags=enforce-binding
> > >>> >> #SBATCH --time=0:20:00 # wall clock limit
> > >>> >> #SBATCH --output=gpu-%J.txt
> > >>> >> #SBATCH --account=lnicotra
> > >>> >> module load cuda
> > >>> >> python gpu1
> > >>> >>
> > >>> >> Where gpu1 is a GPU test script that runs
> correctly
> > >>> while invoked via python. Tiger_1 partition
> has servers
> > >>> with GPUs, with a mix of 1080GTX and K20 as
> > specified in
> > >>> slurm.conf
> > >>> >>
> > >>> >> I have defined GRES resources in the
> slurm.conf
> > file:
> > >>> >> # GPU GRES
> > >>> >> GresTypes=gpu
> > >>> >> NodeName=tiger[01,05,10,15,20]
> Gres=gpu:1080gtx:2
> > >>> >> NodeName=tiger[02-04,06-09,11-14,16-19,21-22]
> > >>> Gres=gpu:k20:2
> > >>> >>
> > >>> >> And have a local gres.conf on the servers
> containing
> > >>> GPUs...
> > >>> >> lnicotra at tiger11 ~# cat /etc/slurm/gres.conf
> > >>> >> # GPU Definitions
> > >>> >> #
> NodeName=tiger[02-04,06-09,11-14,16-19,21-22]
> > >>> Name=gpu Type=K20 File=/dev/nvidia[0-1]
> > >>> >> Name=gpu Type=K20 File=/dev/nvidia[0-1]
> Cores=0,1
> > >>> >>
> > >>> >> and a similar one for the 1080GTX
> > >>> >> # GPU Definitions
> > >>> >> # NodeName=tiger[01,05,10,15,20] Name=gpu
> > Type=1080GTX
> > >>> File=/dev/nvidia[0-1]
> > >>> >> Name=gpu Type=1080GTX File=/dev/nvidia[0-1]
> > Cores=0,1
> > >>> >>
> > >>> >> The account manager seems to know about
> the GPUs...
> > >>> >> lnicotra at tiger11 ~# sacctmgr show tres
> > >>> >> Type Name ID
> > >>> >> -------- --------------- ------
> > >>> >> cpu 1
> > >>> >> mem 2
> > >>> >> energy 3
> > >>> >> node 4
> > >>> >> billing 5
> > >>> >> fs disk 6
> > >>> >> vmem 7
> > >>> >> pages 8
> > >>> >> gres gpu 1001
> > >>> >> gres gpu:k20 1002
> > >>> >> gres gpu:1080gtx 1003
> > >>> >>
> > >>> >> Can anyone point out what am I missing?
> > >>> >>
> > >>> >> Thanks!
> > >>> >> Lou
> > >>> >>
> > >>> >>
> > >>> >> --
> > >>> >>
> > >>> >> Lou Nicotra
> > >>> >>
> > >>> >> IT Systems Engineer - SLT
> > >>> >>
> > >>> >> Interactions LLC
> > >>> >>
> > >>> >> o: 908-673-1833
> > >>> >>
> > >>> >> m: 908-451-6983
> > >>> >>
> > >>> >> lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>
> > <mailto:lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>>
> > >>> <mailto:lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>
> > <mailto:lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>>>
> > >>> >>
> > >>> >> www.interactions.com
> <http://www.interactions.com>
> > <http://www.interactions.com> <http://www.interactions.com>
> > >>> >>
> > >>> >>
> > >>>
> >
> *******************************************************************************
> > >>> >>
> > >>> >> This e-mail and any of its attachments
> may contain
> > >>> Interactions LLC proprietary information,
> which is
> > >>> privileged, confidential, or subject to
> copyright
> > >>> belonging to the Interactions LLC. This
> e-mail is
> > >>> intended solely for the use of the
> individual or entity
> > >>> to which it is addressed. If you are not the
> intended
> > >>> recipient of this e-mail, you are hereby
> notified that
> > >>> any dissemination, distribution, copying, or
> action
> > taken
> > >>> in relation to the contents of and
> attachments to this
> > >>> e-mail is strictly prohibited and may be
> unlawful.
> > If you
> > >>> have received this e-mail in error, please
> notify the
> > >>> sender immediately and permanently delete
> the original
> > >>> and any copy of this e-mail and any
> printout. Thank
> > You.
> > >>> >>
> > >>> >>
> > >>>
> >
> *******************************************************************************
> > >>> >>
> > >>> >>
> > >>> >
> > >>> >
> > >>> > --
> > >>> >
> > >>> > Lou Nicotra
> > >>> >
> > >>> > IT Systems Engineer - SLT
> > >>> >
> > >>> > Interactions LLC
> > >>> >
> > >>> > o: 908-673-1833
> > >>> >
> > >>> > m: 908-451-6983
> > >>> >
> > >>> > lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>
> > <mailto:lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>>
> > >>> <mailto:lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>
> > <mailto:lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>>>
> > >>> >
> > >>> > www.interactions.com
> <http://www.interactions.com>
> > <http://www.interactions.com> <http://www.interactions.com>
> > >>> >
> > >>> >
> > >>>
> >
> *******************************************************************************
> > >>> >
> > >>> > This e-mail and any of its attachments may
> contain
> > >>> Interactions LLC proprietary information,
> which is
> > >>> privileged, confidential, or subject to
> copyright
> > >>> belonging to the Interactions LLC. This
> e-mail is
> > >>> intended solely for the use of the
> individual or entity
> > >>> to which it is addressed. If you are not the
> intended
> > >>> recipient of this e-mail, you are hereby
> notified that
> > >>> any dissemination, distribution, copying, or
> action
> > taken
> > >>> in relation to the contents of and
> attachments to this
> > >>> e-mail is strictly prohibited and may be
> unlawful.
> > If you
> > >>> have received this e-mail in error, please
> notify the
> > >>> sender immediately and permanently delete
> the original
> > >>> and any copy of this e-mail and any
> printout. Thank
> > You.
> > >>> >
> > >>> >
> > >>>
> >
> *******************************************************************************
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>>
> > >>> *Lou Nicotra*
> > >>>
> > >>> IT Systems Engineer - SLT
> > >>>
> > >>> Interactions LLC
> > >>>
> > >>> o: 908-673-1833 <tel:781-405-5114>
> > >>>
> > >>> m: 908-451-6983 <tel:781-405-5114>
> > >>>
> > >>> _lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>
> > <mailto:lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>>
> <mailto:lnicotra at interactions.com <mailto:lnicotra at interactions.com>
> > <mailto:lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>>>_
> > >>>
> > >>> www.interactions.com <http://www.interactions.com>
> <http://www.interactions.com>
> > <http://www.interactions.com/>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>>
> > >>> *Lou Nicotra*
> > >>>
> > >>> IT Systems Engineer - SLT
> > >>>
> > >>> Interactions LLC
> > >>>
> > >>> o: 908-673-1833 <tel:781-405-5114>
> > >>>
> > >>> m: 908-451-6983 <tel:781-405-5114>
> > >>>
> > >>> _lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>
> > <mailto:lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>>
> <mailto:lnicotra at interactions.com <mailto:lnicotra at interactions.com>
> > <mailto:lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>>>_
> > >>>
> > >>> www.interactions.com <http://www.interactions.com>
> <http://www.interactions.com>
> > <http://www.interactions.com/>
> > >>>
> > >>>
> >
> *******************************************************************************
> > >>>
> > >>> This e-mail and any of its attachments may contain
> Interactions
> > >>> LLC proprietary information, which is privileged,
> confidential,
> > >>> or subject to copyright belonging to the
> Interactions LLC. This
> > >>> e-mail is intended solely for the use of the
> individual or
> > entity
> > >>> to which it is addressed. If you are not the
> intended recipient
> > >>> of this e-mail, you are hereby notified that any
> dissemination,
> > >>> distribution, copying, or action taken in relation
> to the
> > >>> contents of and attachments to this e-mail is strictly
> > prohibited
> > >>> and may be unlawful. If you have received this
> e-mail in error,
> > >>> please notify the sender immediately and permanently
> delete the
> > >>> original and any copy of this e-mail and any printout.
> > Thank You.
> > >>>
> > >>>
> >
> *******************************************************************************
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >>
> > >> *Lou Nicotra*
> > >>
> > >> IT Systems Engineer - SLT
> > >>
> > >> Interactions LLC
> > >>
> > >> o: 908-673-1833 <tel:781-405-5114>
> > >>
> > >> m: 908-451-6983 <tel:781-405-5114>
> > >>
> > >> _lnicotra at interactions.com
> <mailto:lnicotra at interactions.com> <mailto:lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>>
> > <mailto:lnicotra at interactions.com
> <mailto:lnicotra at interactions.com> <mailto:lnicotra at interactions.com
> <mailto:lnicotra at interactions.com>>>_
> > >>
> > >> www.interactions.com <http://www.interactions.com>
> <http://www.interactions.com>
> > <http://www.interactions.com/>
> > >>
> > >>
> >
> *******************************************************************************
> > >>
> > >> This e-mail and any of its attachments may contain
> Interactions LLC
> > >> proprietary information, which is privileged,
> confidential, or
> > subject
> > >> to copyright belonging to the Interactions LLC. This
> e-mail is
> > >> intended solely for the use of the individual or entity
> to which
> > it is
> > >> addressed. If you are not the intended recipient of this
> e-mail,
> > you
> > >> are hereby notified that any dissemination, distribution,
> > copying, or
> > >> action taken in relation to the contents of and
> attachments to this
> > >> e-mail is strictly prohibited and may be unlawful. If you
> have
> > >> received this e-mail in error, please notify the sender
> immediately
> > >> and permanently delete the original and any copy of this
> e-mail and
> > >> any printout. Thank You.
> > >>
> > >>
> >
> *******************************************************************************
> > >>
> > >
> >
> >
> >
> > --
> >
> > *Lou Nicotra*
> >
> > IT Systems Engineer - SLT
> >
> > Interactions LLC
> >
> > o: 908-673-1833 <tel:781-405-5114>
> >
> > m: 908-451-6983 <tel:781-405-5114>
> >
> > _lnicotra at interactions.com <mailto:lnicotra at interactions.com>
> <mailto:lnicotra at interactions.com <mailto:lnicotra at interactions.com>>_
> >
> > www.interactions.com <http://www.interactions.com>
> <http://www.interactions.com/>
> >
> >
> *******************************************************************************
> >
> > This e-mail and any of its attachments may contain Interactions LLC
> > proprietary information, which is privileged, confidential, or
> subject
> > to copyright belonging to the Interactions LLC. This e-mail is
> intended
> > solely for the use of the individual or entity to which it is
> addressed.
> > If you are not the intended recipient of this e-mail, you are hereby
> > notified that any dissemination, distribution, copying, or action
> taken
> > in relation to the contents of and attachments to this e-mail is
> > strictly prohibited and may be unlawful. If you have received this
> > e-mail in error, please notify the sender immediately and
> permanently
> > delete the original and any copy of this e-mail and any printout.
> Thank
> > You.
> >
> >
> *******************************************************************************
> >
>
>
>
> --
>
> *Lou Nicotra*
>
> IT Systems Engineer - SLT
>
> Interactions LLC
>
> o: 908-673-1833 <tel:781-405-5114>
>
> m: 908-451-6983 <tel:781-405-5114>
>
> _lnicotra at interactions.com <mailto:lnicotra at interactions.com>_
>
> www.interactions.com <http://www.interactions.com/>
>
> *******************************************************************************
>
> This e-mail and any of its attachments may contain Interactions LLC
> proprietary information, which is privileged, confidential, or subject
> to copyright belonging to the Interactions LLC. This e-mail is intended
> solely for the use of the individual or entity to which it is addressed.
> If you are not the intended recipient of this e-mail, you are hereby
> notified that any dissemination, distribution, copying, or action taken
> in relation to the contents of and attachments to this e-mail is
> strictly prohibited and may be unlawful. If you have received this
> e-mail in error, please notify the sender immediately and permanently
> delete the original and any copy of this e-mail and any printout. Thank
> You.
>
> *******************************************************************************
>
More information about the slurm-users
mailing list