<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">You could reconfigure the partition node lists on the fly using scontrol:</div><div class=""><br class=""></div><div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class="">$ scontrol update PartitionName=regular_part1 Nodes=<node list minus r00n00></div><div class=""> :</div>$ scontrol update PartitionName=regular_partN Nodes=<node list minus r00n00><div class="">$ scontrol update PartitionName=maint Nodes=r00n00</div></blockquote><div class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Should be easy enough to write a script that find the partitions containing node X, remove it, then add to partition "maint." The problem is restoring the node back to service, since you can't simply disable/down one particular node-in-a-partition.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class="">On Oct 5, 2018, at 9:06 AM, Michael Di Domenico <<a href="mailto:mdidomenico4@gmail.com" class="">mdidomenico4@gmail.com</a>> wrote:<br class=""><br class="">Is anyone on the list using maintenance partitions for broken nodes?<br class="">If so, how are you moving nodes between partitions?<br class=""><br class="">The situation with my machines at the moment, is that we have a steady<br class="">stream of new jobs coming into the queues, but broken nodes as well.<br class="">I'd like to fix those broken nodes and re-add them to a separate<br class="">non-production pool so that user jobs don't match, but allow me to run<br class="">maintenance jobs on the nodes to prove things are working before<br class="">giving them back to the users<br class=""><br class="">if i simply mark nodes with downnodes= or scontrol update state=drain,<br class="">slurm will prevent users from new jobs, but wont allow me to run jobs<br class="">on the nodes<br class=""><br class="">Ideally, i'd like to have a prod partition and a maint partition,<br class="">where the maint partition is set to exclusiveuser and i can set the<br class="">status of a node in the prod partition to drain without affecting the<br class="">node status in the maint partition. I don't believe I can do this<br class="">though. I believe i have to change the slurm.conf and reconfigure to<br class="">add/remove nodes from one partition or the other<br class=""><br class="">if anyone has a better solution, i'd like to hear it.<br class=""><br class=""></blockquote><br class=""><div class=""><br class="">::::::::::::::::::::::::::::::::::::::::::::::::::::::<br class="">Jeffrey T. Frey, Ph.D.<br class="">Systems Programmer V / HPC Management<br class="">Network & Systems Services / College of Engineering<br class="">University of Delaware, Newark DE 19716<br class="">Office: (302) 831-6034 Mobile: (302) 419-4976<br class="">::::::::::::::::::::::::::::::::::::::::::::::::::::::<br class=""><br class=""><br class=""><br class=""></div><br class=""></div></div></body></html>