[slurm-users] REST-based CLI tools out there somewhere?

[Chip Seraphine]

We actually have a bunch of people doing that.  It’s fine for “I want to just run squeue and see how busy the cluster is without having to  head over there and look”, but it starts to break down with “my GUI framework runs squeue and sacct to check job status every few seconds, all day long, per user”.  ☹


From: slurm-users <slurm-users-bounces at lists.schedmd.com> on behalf of Jared Baker <jbaker at ucar.edu>
Reply-To: Slurm User Community List <slurm-users at lists.schedmd.com>
Date: Friday, November 10, 2023 at 1:03 PM
To: Slurm User Community List <slurm-users at lists.schedmd.com>
Subject: [ext] Re: [slurm-users] REST-based CLI tools out there somewhere?

At the risk of going a bit off the rails and alternative to the REST method (maybe), but not too far as we've been thinking of alternative ways for similar things (not slurm). Anyway, SSH certificates with a forced command entry and wrapper

At the risk of going a bit off the rails and alternative to the REST method (maybe), but not too far as we've been thinking of alternative ways for similar things (not slurm). Anyway, SSH certificates with a forced command entry and wrapper for slurm commands on submit hosts along with small wrappers on their workstation end could be viable... I'm sure there are many caveats here, but very doable too.

On Fri, Nov 10, 2023 at 11:36 AM Chip Seraphine <cseraphine at drwholdings.com<mailto:cseraphine at drwholdings.com>> wrote:
> what are the security concerns?

The cluster is shared between some business units that do not want to share data, so if we install the munge key on a machine that users have administrative or physical access to it could become compromised.  This could allow them to run jobs on the cluster as another user and retrieve data from shared filesystems.

Jupyterhub is heavily used, but suffers from the same problem- it needs to be running on a submit node.  And, as you mentioned, an interactive solution has drawbacks with many types of workloads.

On 11/10/23, 1:31 AM, "slurm-users on behalf of Hagdorn, Magnus Karl Moritz" <slurm-users-bounces at lists.schedmd.com<mailto:slurm-users-bounces at lists.schedmd.com> <mailto:slurm-users-bounces at lists.schedmd.com<mailto:slurm-users-bounces at lists.schedmd.com>> on behalf of magnus.hagdorn at charite.de<mailto:magnus.hagdorn at charite.de> <mailto:magnus.hagdorn at charite.de<mailto:magnus.hagdorn at charite.de>>> wrote:


Hi Chip,
what are the security concerns? Being able to control jobs is one
thing. Your users still need to setup the jobs which presumably
involves moving data. We have provided some launchers written in python
for specific use cases (that don't involve moving data). We also have a
jupyterhub that launches notebooks on the cluster. This works quite
well, but you do end up with interactive jobs which are most likely
less efficient than batch jobs. Again, this really depends on your use
cases.
Regards
magnus






On Thu, 2023-11-09 at 23:14 +0000, Chip Seraphine wrote:
> Hello,
>
> Our users submit their jobs from shared submit hosts, and have
> expressed an understandable preference for being able to submit
> directly from their own workstations. The obvious solution
> (installing the slurm client on their workstations, or providing a
> container that does something similar) are not available to us
> because of security concerns. This leaves REST as the best
> option. We’re hoping to provide a REST-based toolset that users
> familiar with the command line tools can make immediate use of (so,
> provides basic, stripped-down functionality of srun, squeue, sacct,
> and sinfo). Basically, we want to create a subset of the s* commands
> that can be run from some arbitrary machine if the user has the
> appropriate token.
>
> It’d be surprising if we were the first people to go down this path,
> but searching has turned up nothing. Is there a project anyone
> knows about out there for providing command-line SLURM commands that
> use REST to talk to the daemons? Or am I missing some obvious
> solution here?
>
> --
>
> Chip Seraphine
> Grid Operations
> For support please use help-grid in email or slack.
> This e-mail and any attachments may contain information that is
> confidential and proprietary and otherwise protected from disclosure.
> If you are not the intended recipient of this e-mail, do not read,
> duplicate or redistribute it by any means. Please immediately delete
> it and any attachments and notify the sender that you have received
> it by mistake. Unintended recipients are prohibited from taking
> action on the basis of information in this e-mail or any attachments.
> The DRW Companies make no representations that this e-mail or any
> attachments are free of computer viruses or other defects.


--
Magnus Hagdorn
Charité – Universitätsmedizin Berlin
Geschäftsbereich IT | Scientific Computing


Campus Charité Mitte
BALTIC - Invalidenstraße 120/121
10115 Berlin


magnus.hagdorn at charite.de<mailto:magnus.hagdorn at charite.de> <mailto:magnus.hagdorn at charite.de<mailto:magnus.hagdorn at charite.de>>
https://www.charite.de<https://urldefense.com/v3/__https:/www.charite.de__;!!EvhwMw!SIr6vqft4lrIWWkAUNnXlV7SdanIAvPQWp2DWWb1IuFEzlYVtFSi790uWXd1mepoliQwP9I6MqYMOxhxvUMy$> <https://www.charite.de<https://urldefense.com/v3/__https:/www.charite.de__;!!EvhwMw!SIr6vqft4lrIWWkAUNnXlV7SdanIAvPQWp2DWWb1IuFEzlYVtFSi790uWXd1mepoliQwP9I6MqYMOxhxvUMy$>>
HPC Helpdesk: sc-hpc-helpdesk at charite.de<mailto:sc-hpc-helpdesk at charite.de> <mailto:sc-hpc-helpdesk at charite.de<mailto:sc-hpc-helpdesk at charite.de>>