[slurm-users] Usage of particular GPU out of 4 GPUs while submitting

[Ravi Konila]

Hi Daniel Letai

Thanks for the quick response and guidance.

I have done the changes as mentioned in gres.conf and slurm.conf and now I 
am able to submit the jobs to a particular GPU.

Regarding MIG, it was just a thought came in m mind, in case studentA wants 
to submit jobs to both GPU partition (20G and 5G). But anyhow, referred 
Nvidia MIG user guide and your suggestion as mentioned above, I am clear 
now.

Thanks a lot for the support.


With Warm Regards
Ravi Konila

-----Original Message----- 
From: slurm-users-request at lists.schedmd.com
Sent: Monday, November 20, 2023 5:30 PM
To: slurm-users at lists.schedmd.com
Subject: slurm-users Digest, Vol 73, Issue 31

Send slurm-users mailing list submissions to
slurm-users at lists.schedmd.com

To subscribe or unsubscribe via the World Wide Web, visit
https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users
or, via email, send a message with subject or body 'help' to
slurm-users-request at lists.schedmd.com

You can reach the person managing the list at
slurm-users-owner at lists.schedmd.com

When replying, please edit your Subject line so it is more specific
than "Re: Contents of slurm-users digest..."


Today's Topics:

   1. Re: SLURM new user query, does SLURM has GUI /Web based
      management version also (Joseph John)
   2. Usage of particular GPU out of 4 GPUs while submitting jobs
      to DGX Server (Ravi Konila)
   3. Re: Usage of particular GPU out of 4 GPUs while submitting
      jobs to DGX Server (Daniel Letai)


----------------------------------------------------------------------

Message: 1
Date: Mon, 20 Nov 2023 03:44:48 +0000
From: Joseph John <jjk_saji at yahoo.com>
To: "Ole.H.Nielsen at fysik.dtu.dk" <Ole.H.Nielsen at fysik.dtu.dk>, Slurm
User Community List <slurm-users at lists.schedmd.com>
Subject: Re: [slurm-users] SLURM new user query, does SLURM has GUI
/Web based management version also
Message-ID:
<DU0PR10MB5775509BC25B2465601A4952FBB4A at DU0PR10MB5775.EURPRD10.PROD.OUTLOOK.COM>

Content-Type: text/plain; charset="us-ascii"

Thanks Ole
I was able to setup the SLURM for 4 nodes and tried out some  python code 
using srun and trying to understand and practice more of  SLURM commands
Thanks for the reply
Joseph John


From: slurm-users <slurm-users-bounces at lists.schedmd.com> on behalf of Ole 
Holm Nielsen <Ole.H.Nielsen at fysik.dtu.dk>
Date: Sunday, 19 November 2023 at 2:35 PM
To: slurm-users at lists.schedmd.com <slurm-users at lists.schedmd.com>
Subject: Re: [slurm-users] SLURM new user query, does SLURM has GUI /Web 
based management version also
On 19-11-2023 09:11, Joseph John wrote:
> I am new user, trying out SLURM
>
> Like to check if the SLURM has a GUI/web based management tool also

Did you read the Quick Start Administrator Guide at
https://slurm.schedmd.com/quickstart_admin.html ?

I don't believe there are any Slurm management tools as a web GUI, and
that would probably be a security nightmare anyway because privileged
system access is required.

There are a number of monitoring tools for viewing the status of Slurm jobs.

/Ole
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 
<http://lists.schedmd.com/pipermail/slurm-users/attachments/20231120/10e9cbc3/attachment-0001.htm>

------------------------------

Message: 2
Date: Mon, 20 Nov 2023 10:06:42 +0530
From: "Ravi Konila" <ravibhatk at gmail.com>
To: <slurm-users at lists.schedmd.com>
Subject: [slurm-users] Usage of particular GPU out of 4 GPUs while
submitting jobs to DGX Server
Message-ID: <8ED1EDA8185C4F1CAA1D0AB2D216B4B8 at RAVIKONILAPC>
Content-Type: text/plain; charset="iso-8859-1"

Hello Everyone

I am just beginner of slurm and started to use the same on our DGX Server 
which has 4 numbers of A100, 80GB GPUs.
Everything works fine, jobs goes to random GPUs (free available).
My question is related to submission of jobs to those GPUs. How do a student 
submit the job to a particular GPU out of 4 GPUs? For example, studentA 
should submit the job to GPU ID 1 instead of GPU ID 0.

Also we are planning for MIG in the server and we would like few students to 
submit the jobs to 20G partition and non critical jobs to 5G partition.
How should be the slurm.conf and gres.conf in this case.

Currently our configuration is as below:

gres.conf
Name=gpu    type=A100    file=/dev/nvidia[0-2,4]

------------
slurm.conf
.
.
.
GresTypes=gpu
NodeName=rl-dgxs-r21-l2 Gres=gpu:A100:4 CPUs=128 RealMemory=500000 
State=UNKNOWN
PartitionName=LocalGPUQ Nodes=ALL Default=YES MaxTime=INFINITE State=UP

-------------

Any suggestions or help in this regard is highly appreciated.

With Warm Regards
Ravi Konila
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 
<http://lists.schedmd.com/pipermail/slurm-users/attachments/20231120/018a699b/attachment-0001.htm>

------------------------------

Message: 3
Date: Mon, 20 Nov 2023 10:09:48 +0200
From: Daniel Letai <dani at letai.org.il>
To: slurm-users at lists.schedmd.com
Subject: Re: [slurm-users] Usage of particular GPU out of 4 GPUs while
submitting jobs to DGX Server
Message-ID: <31022502-46de-4a89-a092-7c32745c5ab3 at letai.org.il>
Content-Type: text/plain; charset="us-ascii"

An HTML attachment was scrubbed...
URL: 
<http://lists.schedmd.com/pipermail/slurm-users/attachments/20231120/cbdd1ef0/attachment-0001.htm>

End of slurm-users Digest, Vol 73, Issue 31
*******************************************