- slurm-users - lists.schedmd.com

Re: Jobs not getting assigned the requested GPU card
by Dj Merrill 16 Oct '26

16 Oct '26

Hi Brian, Thank you for the input! I find this weird and interesting. For the past several years we have gone by the output of nvidia-smi -L to assign the GPUs in the gres.conf, and that's always worked. We also tried removing the fixed settings and tried both AutoDetect=nvml and AutoDetect=nvidia in the gres.conf and neither of those worked properly either. As you surmised, the minor does not match the ID: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 4 ID: 1 Minor Number : 3 ID: 2 Minor Number : 2 ID: 3 Minor Number : 0 ID: 4 Minor Number : 1 Interestingly, every time I reboot, I get a different result for the minor number, but the output of "nvidia-smi -L" always stays the same: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 1 ID: 1 Minor Number : 2 ID: 2 Minor Number : 3 ID: 3 Minor Number : 0 ID: 4 Minor Number : 4 # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 2 ID: 1 Minor Number : 4 ID: 2 Minor Number : 3 ID: 3 Minor Number : 1 ID: 4 Minor Number : 0 # nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3) # nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 GPU4 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PIX PIX NODE NODE NODE 0-7 0 N/A GPU1 PIX X PIX NODE NODE NODE 0-7 0 N/A GPU2 PIX PIX X NODE NODE NODE 0-7 0 N/A GPU3 NODE NODE NODE X PIX PIX 0-7 0 N/A GPU4 NODE NODE NODE PIX X PIX 0-7 0 N/A NIC0 NODE NODE NODE PIX PIX X I'm wondering if something changed in the recent nvidia driver / CUDA that was installed a few days ago. -Dj On 10/15/26 19:14, Brian Haymore wrote: > The device number that 'nvidia-smi' shows for the GPUs, 0, 1, 2, 3, > and 4, are not certain to map to /dev/nvidia$X. > > Could you run ' for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i > $i | grep "Minor Number"'; done > > If each matches then what I'm asking about is not the issue. If they > do not match then you will need to adjust the gres.conf file. > > Another command that would be good to see is 'nvidia-smi topo -m'. > > -- > Brian D. Haymore > University of Utah > Center for High Performance Computing > 155 South 1452 East RM 405 > Salt Lake City, Ut 84112 > Phone: 801-558-1150 > http://bit.ly/1HO1N2C <http://bit.ly/1HO1N2C> > ------------------------------------------------------------------------ > *From:* Dj Merrill via slurm-users <slurm-users(a)lists.schedmd.com> > *Sent:* Thursday, October 15, 2026 1:42 PM > *To:* slurm-users(a)lists.schedmd.com <slurm-users(a)lists.schedmd.com> > *Subject:* [slurm-users] Jobs not getting assigned the requested GPU card > We have a Slurm Cluster that has been working fine for many years. A > few months ago it was upgraded to Slurm version 25.11.65, and has been > working properly running that version. The systems are running Rocky > Linux 9.8 and are patched/updated on a regular basis. They are currently > running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from > Nvidia's repo. > > Today it was reported that jobs are not getting assigned the GPU card > they requested, so for example they may request a 5090 card but are > assigned an A100 card. This was working fine, so we are struggling to > figure out what has changed to cause this behaviour. > > We have a node with 5 GPU cards, of three types: > > % nvidia-smi -L > > GPU 0: NVIDIA GeForce RTX 5090 (UUID: > GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6) > GPU 1: NVIDIA GeForce RTX 3080 (UUID: > GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94) > GPU 2: NVIDIA GeForce RTX 3080 (UUID: > GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae) > GPU 3: NVIDIA A100 80GB PCIe (UUID: > GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) > GPU 4: NVIDIA GeForce RTX 3080 (UUID: > GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094) > > /etc/slurm/gres.conf matches the above output (and has been this way for > years): > > % cat /etc/slurm/gres.conf > Name=gpu Type=rtx5090 File=/dev/nvidia0 > Name=gpu Type=rtx3080 File=/dev/nvidia1 > Name=gpu Type=rtx3080 File=/dev/nvidia2 > Name=gpu Type=a100 File=/dev/nvidia3 > Name=gpu Type=rtx3080 File=/dev/nvidia4 > > > The /etc/slurm/slurm.conf entry for this node is the same as it has been > for years: > > NodeName=moose[63] CPUs=16 RealMemory=380000 > Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700 > > > When submitting a job such as: > > $ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash > srun: job 71048 queued and waiting for resources > srun: job 71048 has been allocated resources > > $ nvidia-smi -L > GPU 0: NVIDIA A100 80GB PCIe (UUID: > GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) > > Any suggestions as to where to look? > > Thank you, > > -Dj > > > > > -- > slurm-users mailing list -- slurm-users(a)lists.schedmd.com > To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com

2 6

5G Chipset Market Industry Trends and Global Forecasts to 2040
by nidhithankur987＠gmail.com 16 Oct '26

16 Oct '26

Market Size and Growth Forecast 2026 Valuation: USD 71 billion 2040 Forecast Valuation: USD 455 billion Compound Annual Growth Rate (CAGR): ~14.25% from 2026 to 2040 Key Market Segmentations By Type of Chipset: Application-Specific Integrated Circuits (ASICs): Custom-designed for specific tasks (such as Layer-1 scheduling in 5G radio units), providing significantly better performance-per-watt to manage severe thermal constraints in base stations and smartphones. Other configurations include standard modems, RFICs, and mixed-signal components. By Deployment / Application Type: Smartphones & Tablets: Currently capture the highest share of the 5G chipset market, supported by billions of shipped units globally. Affordability Impact: Driven by the decreasing price gap between 5G and 4G chipsets, making 5G devices increasingly accessible. High-Frequency Advantages: Utilization of millimeter-wave (mmWave) frequencies (ranging from 24 GHz to 100 GHz) delivers ultra-high data transmission speeds and optimizes Internet of Things (IoT) connected device experiences. By End-Users: Consumer Electronics Automotive Industry: Propelled by the rollout of connected cars and electric vehicles featuring embedded internet connectivity. Automation Industry: Expected to experience a notably high growth rate driven by the rising demand for industrial process automation. Healthcare and other sectors. Geographical Distribution: Asia-Pacific: Currently holds the majority share of the global 5G chipset market, supported by a highly developed telecommunication infrastructure. North America: Projected to witness a relatively higher CAGR during the forecast period (2026–2040). Core Market Drivers & Highlights Next-Generation Connectivity: Designed to deliver significantly higher data throughput and lower latency compared to legacy 4G chipsets. Emerging Technological Use Cases: Integration across high-speed telecommunications, smart manufacturing, C-V2X (Cellular Vehicle-to-Everything) automotive frameworks, and large-scale IoT ecosystems. https://www.rootsanalysis.com/5g-chipset-market

1 0

Jobs not getting assigned the requested GPU card
by Dj Merrill 15 Oct '26

15 Oct '26

We have a Slurm Cluster that has been working fine for many years. A few months ago it was upgraded to Slurm version 25.11.65, and has been working properly running that version. The systems are running Rocky Linux 9.8 and are patched/updated on a regular basis. They are currently running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from Nvidia's repo. Today it was reported that jobs are not getting assigned the GPU card they requested, so for example they may request a 5090 card but are assigned an A100 card. This was working fine, so we are struggling to figure out what has changed to cause this behaviour. We have a node with 5 GPU cards, of three types: % nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094) /etc/slurm/gres.conf matches the above output (and has been this way for years): % cat /etc/slurm/gres.conf Name=gpu Type=rtx5090 File=/dev/nvidia0 Name=gpu Type=rtx3080 File=/dev/nvidia1 Name=gpu Type=rtx3080 File=/dev/nvidia2 Name=gpu Type=a100 File=/dev/nvidia3 Name=gpu Type=rtx3080 File=/dev/nvidia4 The /etc/slurm/slurm.conf entry for this node is the same as it has been for years: NodeName=moose[63] CPUs=16 RealMemory=380000 Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700 When submitting a job such as: $ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash srun: job 71048 queued and waiting for resources srun: job 71048 has been allocated resources $ nvidia-smi -L GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) Any suggestions as to where to look? Thank you, -Dj

1 0

Making easy and simple security updates of Slurm compute nodes
by Ole Holm Nielsen 15 Oct '26

15 Oct '26

Dear Slurm users, Keeping Slurm compute nodes up-to-date with security updates is an important task, and since security updates are now released with a high frequency, it is desirable to make the updating process as simple and automatic as possible. The tools documented here in [1] offer a possible solution. It's enlightening to browse OS errata pages such as [2] or [3]. The design used in the present approach [1] is: * Configure all compute nodes to execute a crontab job script at every reboot. This script will usually do nothing, except in case the node has a Slurm "State=down" where it will perform the configured updates. * Making updates simply consists of rebooting compute node lists with the command "scontrol reboot asap nextstate=down <node-list>". Here we have employed (abused?) the Slurm "State=down" as a *trigger* telling the update script to perform some actions. Nodes with a "Reason=Node unexpectedly rebooted" are assumed to be unrelated to the present update system and will therefore be skipped. The page [1] gives details of configuration and the optional handling of mixed-state nodes. We have tested these tools on our 700-node cluster running RockyLinux 8 and 9 for some time now, and they work reliably in our environment when updating several times per week. I would welcome any comments or bug reports regarding these tools. Best regards, Ole [1] https://github.com/OleHolmNielsen/Slurm_tools/tree/master/nodes#compute-nod… [2] RockyLinux errata: https://errata.build.resf.org/ [3] AlmaLinux errata: https://errata.almalinux.org/ -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

1 0

Global AI in Drug Discovery Market Trends and Future Forecast Report
by abhaybhardwaj282＠gmail.com 15 Oct '26

15 Oct '26

The latest report published by Roots Analysis, titled Global AI in Drug Discovery Market offers a detailed study of the industry size, growth scenarios, and fundamental parameters of the market analysis. In this research report, the author provided an in-depth analysis of market dynamics, including key drivers, restraints, opportunities, and major threats that may impact growth during the upcoming years. Our market analysis report provides a brief discussion on leading market players, competition landscape, regional dominance, supply-demand ratio, business strategies, and future investment opportunities in the overall industry. This report provides a complete prototype of the estimated growth rate, market value, product types, applications, and type of end users active in this industry. What are the Report Highlights? The AI in Drug Discovery industry analysis report offers a comprehensive overview of the Market and its major growth prospects such as drivers, restraints, challenges, and upcoming opportunities. It also throws light on the current industry trends, major developments, and other interesting insights useful for this market. It further discusses the nature of the market competition, list of players and their key strategies to stay put in the competition. It also describes the table of segmentation and the names of the leading segment with market figures. The market report is provided on the company website. Major Players in AI in Drug Discovery market: Aiforia Technologies, Atomwise, BioSyntagma, Chemalive, Collaborations Pharmaceuticals, Cyclica, DeepMatter, Exscientia, InveniAI, MAbSilico, Optibrium, Recursion Pharmaceuticals, Sensyne Health and Valo Health. Leading market players and manufacturers are studied to help give a brief idea about them in the AI in Drug Discovery market report. The challenges faced by them and the reasons they are on that position is explained to help make a well informed decision. The competitive landscape of AI in Drug Discovery market is given presenting detailed insights into the company profiles, developments, mergers, acquisitions, economic status and best SWOT analysis. AI in Drug Discovery Market Segmentation: AI in Drug Discovery Market (2nd Edition): Industry Trends and Global Forecasts, till 2035 - Distribution by Drug Discovery Steps, Therapeutic Area and Key Geographies (North America, Europe, Asia-Pacific, Latin America, MENA, and RoW) Browse Complete Report @ https://www.rootsanalysis.com/reports/ai-based-drug-discovery-market.html

1 0

Fermenters and Bioreactors Market
by nidhithankur987＠gmail.com 15 Oct '26

15 Oct '26

Market Size & Growth Projections Market Valuation: The global industrial robotics market is projected to grow from USD 22 billion in 2026 to USD 35 billion by 2030, and reach USD 101 billion by 2040. Compound Annual Growth Rate (CAGR): The market is anticipated to expand at a robust 11.45% CAGR from 2026 through 2040, driven by the acceleration of Industry 4.0, smart factory automation, and rising global demand for consumer goods and automobiles. Core Market Segments & Share Insights By Type of Robot: Traditional Industrial Robots: Dominate the market with approximately 85.77% revenue share in 2026, valued for their high payload capacity, unmatched precision, and speed in heavy-duty sectors like automotive and metal fabrication. Collaborative Robots (Cobots): Represent the fastest-growing strategic segment, projected to expand at a 14.85% CAGR through 2040, fueled by labor shortages and adoption in SMEs, healthcare, logistics, and electronics. By Payload Capacity: Low Payloads (<20 Kgs): Lead the market with 54.20% revenue share in 2026, heavily deployed in electronics, pharmaceuticals, cosmetics, and semiconductor manufacturing. Medium Payloads (20 to 150 Kgs): Projected to register a fast 11.2% CAGR, ideal for mid-range manufacturing, welding, painting, and flexible assembly. By Mobility: Stationary Robots: Capture around 65% of the market share in 2026, essential for fixed, high-volume tasks such as welding and assembly. Mobile Robots: Represent the highest-growth mobility segment, expected to grow at a 16.5% CAGR, driven by warehouse automation, e-commerce fulfillment, and intra-factory logistics. By Application Area: Handling Operations: Currently lead the application sector, accounting for 42.0% of the market in 2026, supported by widespread compatibility and e-commerce logistics demands. Dispensing: Expected to see accelerated growth at a 12% CAGR, driven by precision material application needs across food, beverage, and agricultural industries. Key End-User Industries: Automotive Industry, Electronics Industry, Metal and Machinery Industry, Plastics Industry, and Food & Beverages Industry. Key Growth Drivers & Industry Trends Productivity & Safety: Automated labor environments report over a 5% increase in productivity, while industrial robot end-effectors and grippers safely execute hazardous or strenuous tasks (such as high-pressure vacuum or radiation chambers). Robot-as-a-Service (RaaS): Growing preference for flexible subscription and service-based deployment models is lowering upfront capital barriers for small and medium-sized enterprises (SMEs). AI & IoT Convergence: Next-generation industrial robots are increasingly integrated with artificial intelligence and IoT capabilities, enabling adaptive handling of complex production tasks and autonomous mobile navigation.

1 0

Slurm 26.05.4: idle~ nodes go to unknown after reconfigure
by Stefan Staeglich 14 Oct '26

14 Oct '26

Hi, since we updated our site from Slurm 24.11 to Slurm 26.05.4 the idle~ (powersaved) nodes goes to state unk (unknown) whenever running scontrol reconfigure. This state is not cleaned by Slurm itself. Is this a known issue of Slurm 26.05? Best, Stefan

4 8

Slurm version 26.05.5 is now available
by Marshall Garey 13 Oct '26

13 Oct '26

We are pleased to announce the availability of Slurm version 26.05.5. These releases fix some mild to moderate severity issues and potential crashes. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ Marshall Garey Senior System Software Engineer NVIDIA

1 0

Need configuration help
by Steffen Grunewald 13 Oct '26

13 Oct '26

Good afternoon all, after having run a Slurm cluster for a long time with exclusive node usage (the DEFAULT partition set to Shared=EXCLUSIVE, for historical reasons), I have now received a request to provide individual CPU cores as well, so "shared mode" would be it. I set up a new partition with OverSubscribe=NO, changed the DEFAULT stanza to OverSubscribe=EXCLUSIVE, and things seemed to work (after I changed SelectTypeParameters to CR_Core_Memory from CR_Core - apparently nobody ever hit a memory limit before...). But - now there are jobs that don't run anymore. After staring at them for hours, I found that they would request CPUs=32 (they always did, I suspect). Our compute nodes are 2s/16c/2t, "slurmd -C" on them reports CPUs=64 Boards=1 SocketsPerBoard=2 CoresPerSocket=16 ThreadsPerCore=2 Moving those jobs (that claim "never to be runnable") to the new partition now appears to work, and they claim enough memory not to share their node with other jobs, but I'd like to keep them in their original partition. Where's my mistake? Am I trying to mix things that can't be mixed? Thanks, S -- Steffen Grunewald, Cluster Administrator Max Planck Institute for Gravitational Physics (Albert Einstein Institute) Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany ~~~ Fon: +49-331-567 7274 Mail: steffen.grunewald(at)aei.mpg.de ~~~

3 2

SlurmDB Duplicate
by Paul Edmon 28 Sep '26

28 Sep '26

Currently we have a slurmdbd that is our main production, and thus under heavy traffic with query restrictions (1 week at a time currently). We also archive data that is older than 6 months so that the database stays lean and mean. This serves the purpose well for general day to day but limits us in terms of doing larger queries for the sake of summary statistics and billing. What I would like to stand up is a second slurmdbd that has no query restrictions and has all my historical data, while also being kept abreast of the live information coming off the cluster. Straight replication (slurmdbd or mysql) won't suffice as when I do my monthly purges, it would then propagate that purge. Likewise reimporting the archived data into an external database would be missing the current 6 months. My question is is there a clever way to run a duplicate slurmdbd that gets all the cluster traffic so it stays up to date, while also not purging data? Also in such as way that I could say have the duplicate slurmdbd offsite and not drag down cluster performance (as really the cluster should lean on the active database that is more proximate)? Looking at the documentation I can see an obvious method for HA, and AccountingStorageExternalHost which looks like it might work but may drag down performance due to needing to wait for DB operations to complete there. I'm curious if other sites have solved this. Summary programs like XDMod are useful but having the raw data available to cross check and process in new ways is what we are really looking for. Thanks in advance. -Paul Edmon-

4 4