You can create (fake or real) licenses for the applications you want to track and then use sreport to track their historical data.
https://slurm.schedmd.com/licenses.html
Pros:
- easy for you, it will take you few minutes to implement
- you may need/have that already in place if the software is licensed (if not, you can choose a number of licenses larger than the number of cores you have on your cluster and avoid artificially limiting concurrent jobs)
Cons:
- gotta convince users to use it, see my past messages about it on this list where I wanted to "hide" licensed software unless license was requested (spoiler alert, I found that too complicated and gave up)
- unless successful hiding binaries based on license request, users can still use it wrongly (not requesting them when needed), and in both case they can request "license" when not needed -- messing up your stats
Alternatively, you can make the real executables hidden and write wrapper scripts like this (and you may need an epilog to report "stopping" in case the job is cancelled, unless you do that separately from the JOBID data)
#!/bin/bash
LOG=make your own logic for how you want to store this (e.g. by user, date, etc)
date >> $LOG # you may want to use rfc or iso format for date
echo $USER is starting fluent in job $SLURM_JOBID >> $LOG
echo other info you may want to store >> $LOG
/full/hidden/path/to/fluent $@
echo $USER stopped fluent in job $SLURM_JOBID >> $LOG