[slurm-users] Question: How to see all srun/sbatch commands in one place?

[mercan]

Hi;

You can use a job submit plugin to logging. We use lua job_submit 
plugin. The slurm.log_info() function writes a string to slurmctl log 
file. But we use a seperate file as a user activity log file. The 
logging lua code something as below:

         dt = os.date()
         jaccount = job_desc.account
         file = io.open("/log/sbatch.log", "a")
file:write("LUA\t",dt,"\t",job_desc.user_name,"\t",jaccount,"\t",job_desc.work_dir,"\n")
         file:close()

As you can see this code does not contain a host name info.  I've tried 
os.getenv("HOSTNAME") and some other things, but  I could not log the 
host name.

Regards,

Ahmet M.



On 24.01.2019 11:52, AiO wrote:
> Hi guys!
>
> Is there a way to 'tail -f' or equivalent some file (with some 
> slurmctld configuration) tå see what shell commands are deployed to 
> all the slurm hosts? I would like to centralize logging interleaved 
> with other security-related logging to get chronological chain of 
> events perfromed.
>
> I have played around with the LogLevels a bit but with no greater 
> understanding on the topic.
>
> Scenario:
>
>   host1: srun do_something_cool
>   host2: srun be_a_bit_awesome
>
>   slurm-controller: tail -f /var/some/file
>   ...... host1 invoked 'do_something_cool' on computenodeX .....
>   ...... host2 invoked 'be_a_bit_awesome' on computenodeY .....
>
> Obviously the log-format is totally bogus :) but somethings similar 
> would be nice to have.
>
> The ida is to let rsyslog poll such file for forwarding to a graylog 
> or something like that.
>
> Kind regards,
> AiO