- slurm-users - lists.schedmd.com

SURF slurm tools repo
by Bas van der Vlies 05 Nov '26

05 Nov '26

Hi, Maybe also useful to others we have put our SURF slurm tools on codeberg: https://codeberg.org/SURF/slurm_tools smon & snodes Command line tools for Slurm administrators, using the Slurm REST API (slurmrestd): * smon: rack overview of all nodes with their state or CPU load, a summary per state, or a list of busy nodes with jobs, load and memory. * snodes: list unavailable nodes grouped by reason, and drain or resume nodes with a note and an optional ticket number. Regards Bas -- -- Bas van der Vlies | High Performance Computing & Visualization | SURF| Science Park 140 | 1098 XG Amsterdam | T +31 (0) 20 800 1300 | bas.vandervlies(a)surf.nl | www.surf.nl

1 1

Native Lustre encryption with Slurm: loading the key at job time without storing it on the remote network
by sacawulu 04 Nov '26

04 Nov '26

Hi all, I’m new to this list, so hello to everyone. :-) Goal We want to use native lustre client-side encryption (fscrypt) to protect data on a remote HPC cluster, and run Slurm jobs that can work with that data. Current (not so good) concept: The key would temporarily live on the remote network: Key on NFS: /home/username/enc.key Encrypted data on Lustre: /lustre/project The workflow would be: 1. A Slurm prolog loads the key into the kernel keyring on the compute nodes. 2. The same script deletes enc.key from the remote network. 3. The job runs with access to the decrypted data. 4. A Slurm epilog script removes the key from the node’s keyring. The obvious problem: Until the job is scheduled, the key and the encrypted data are on the same remote network, which defeats much of the purpose. Accepted risks We are aware of other attack vectors: remote admins can read enc.key on NFS, and the key can be read from RAM while the job runs. Strict ACLs are essential, and we run the computes --exclusive We accept these risks for now. Question Does anyone know a way to avoid storing enc.key on the remote network before the job starts? Can the key be 'injected' at job start from our side (pushed or pulled) Does slurm have any native mechanism to deal with this? Is there a better pattern for this? Have you implemented something like this on your site, and how? Note: we have an existing site-to-site ipsec tunnel that we could use. Any tips or experiences are welcome. Thanks!

2 3

Making easy and simple security updates of Slurm compute nodes
by Ole Holm Nielsen 26 Oct '26

26 Oct '26

Dear Slurm users, Keeping Slurm compute nodes up-to-date with security updates is an important task, and since security updates are now released with a high frequency, it is desirable to make the updating process as simple and automatic as possible. The tools documented here in [1] offer a possible solution. It's enlightening to browse OS errata pages such as [2] or [3]. The design used in the present approach [1] is: * Configure all compute nodes to execute a crontab job script at every reboot. This script will usually do nothing, except in case the node has a Slurm "State=down" where it will perform the configured updates. * Making updates simply consists of rebooting compute node lists with the command "scontrol reboot asap nextstate=down <node-list>". Here we have employed (abused?) the Slurm "State=down" as a *trigger* telling the update script to perform some actions. Nodes with a "Reason=Node unexpectedly rebooted" are assumed to be unrelated to the present update system and will therefore be skipped. The page [1] gives details of configuration and the optional handling of mixed-state nodes. We have tested these tools on our 700-node cluster running RockyLinux 8 and 9 for some time now, and they work reliably in our environment when updating several times per week. I would welcome any comments or bug reports regarding these tools. Best regards, Ole [1] https://github.com/OleHolmNielsen/Slurm_tools/tree/master/nodes#compute-nod… [2] RockyLinux errata: https://errata.build.resf.org/ [3] AlmaLinux errata: https://errata.almalinux.org/ -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

2 4

Luxury Wines and Spirits: Exploring Craftsmanship, Heritage, and Changing Tastes
by wadekar.sagar087＠gmail.com 21 Oct '26

21 Oct '26

Luxury Wines and Spirits represent a distinctive part of the beverage world, shaped by tradition, craftsmanship, regional identity, and evolving consumer preferences. These beverages are often associated with special occasions, fine dining, cultural celebrations, and collecting. Their appeal extends beyond taste, reflecting the history, production methods, and expertise involved in creating distinctive products. Luxury wines are generally produced using carefully selected grape varieties and specialized winemaking techniques. Factors such as climate, soil composition, vineyard location, grape maturity, and aging conditions influence their aroma, flavor, and texture. Certain wine-producing regions have developed international recognition through generations of expertise and established production traditions. Vintage variations and limited production can also contribute to differences in quality, character, and perceived value. The luxury spirits category includes products such as whisky, cognac, rum, tequila, and premium gin. Their characteristics depend on ingredients, distillation processes, maturation periods, and blending techniques. For example, barrel aging can introduce complex aromas and flavors, while the origin of ingredients and the methods used by distillers help distinguish one spirit from another. Packaging, heritage, and production history may also influence how consumers perceive these beverages. Consumer interest in luxury wines and spirits is influenced by several factors, including culinary experiences, cultural traditions, gifting practices, and interest in premium craftsmanship. Some consumers explore these beverages to understand regional varieties and production techniques, while others value their connection to social occasions and personal collections. At the same time, changing lifestyle preferences have encouraged greater attention to responsible consumption, product transparency, and sustainability. Digital platforms have also changed how people discover and learn about luxury beverages. Online resources provide information about grape varieties, distilleries, tasting characteristics, and regional traditions, helping consumers make more informed choices. Overall, luxury wines and spirits combine agricultural expertise, technical knowledge, and cultural heritage. Their continued relevance reflects the relationship between traditional production practices and changing consumer expectations, making the category an interesting subject within the broader global beverage industry. https://www.marketresearchfuture.com/reports/luxury-wines-spirits-market-72…

1 0

Jobs not getting assigned the requested GPU card
by Dj Merrill 19 Oct '26

19 Oct '26

We have a Slurm Cluster that has been working fine for many years. A few months ago it was upgraded to Slurm version 25.11.65, and has been working properly running that version. The systems are running Rocky Linux 9.8 and are patched/updated on a regular basis. They are currently running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from Nvidia's repo. Today it was reported that jobs are not getting assigned the GPU card they requested, so for example they may request a 5090 card but are assigned an A100 card. This was working fine, so we are struggling to figure out what has changed to cause this behaviour. We have a node with 5 GPU cards, of three types: % nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094) /etc/slurm/gres.conf matches the above output (and has been this way for years): % cat /etc/slurm/gres.conf Name=gpu Type=rtx5090 File=/dev/nvidia0 Name=gpu Type=rtx3080 File=/dev/nvidia1 Name=gpu Type=rtx3080 File=/dev/nvidia2 Name=gpu Type=a100 File=/dev/nvidia3 Name=gpu Type=rtx3080 File=/dev/nvidia4 The /etc/slurm/slurm.conf entry for this node is the same as it has been for years: NodeName=moose[63] CPUs=16 RealMemory=380000 Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700 When submitting a job such as: $ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash srun: job 71048 queued and waiting for resources srun: job 71048 has been allocated resources $ nvidia-smi -L GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) Any suggestions as to where to look? Thank you, -Dj

2 1

Re: Jobs not getting assigned the requested GPU card
by Dj Merrill 16 Oct '26

16 Oct '26

Hi Brian, Thank you for the input! I find this weird and interesting. For the past several years we have gone by the output of nvidia-smi -L to assign the GPUs in the gres.conf, and that's always worked. We also tried removing the fixed settings and tried both AutoDetect=nvml and AutoDetect=nvidia in the gres.conf and neither of those worked properly either. As you surmised, the minor does not match the ID: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 4 ID: 1 Minor Number : 3 ID: 2 Minor Number : 2 ID: 3 Minor Number : 0 ID: 4 Minor Number : 1 Interestingly, every time I reboot, I get a different result for the minor number, but the output of "nvidia-smi -L" always stays the same: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 1 ID: 1 Minor Number : 2 ID: 2 Minor Number : 3 ID: 3 Minor Number : 0 ID: 4 Minor Number : 4 # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 2 ID: 1 Minor Number : 4 ID: 2 Minor Number : 3 ID: 3 Minor Number : 1 ID: 4 Minor Number : 0 # nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3) # nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 GPU4 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PIX PIX NODE NODE NODE 0-7 0 N/A GPU1 PIX X PIX NODE NODE NODE 0-7 0 N/A GPU2 PIX PIX X NODE NODE NODE 0-7 0 N/A GPU3 NODE NODE NODE X PIX PIX 0-7 0 N/A GPU4 NODE NODE NODE PIX X PIX 0-7 0 N/A NIC0 NODE NODE NODE PIX PIX X I'm wondering if something changed in the recent nvidia driver / CUDA that was installed a few days ago. -Dj On 10/15/26 19:14, Brian Haymore wrote: > The device number that 'nvidia-smi' shows for the GPUs, 0, 1, 2, 3, > and 4, are not certain to map to /dev/nvidia$X. > > Could you run ' for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i > $i | grep "Minor Number"'; done > > If each matches then what I'm asking about is not the issue. If they > do not match then you will need to adjust the gres.conf file. > > Another command that would be good to see is 'nvidia-smi topo -m'. > > -- > Brian D. Haymore > University of Utah > Center for High Performance Computing > 155 South 1452 East RM 405 > Salt Lake City, Ut 84112 > Phone: 801-558-1150 > http://bit.ly/1HO1N2C <http://bit.ly/1HO1N2C> > ------------------------------------------------------------------------ > *From:* Dj Merrill via slurm-users <slurm-users(a)lists.schedmd.com> > *Sent:* Thursday, October 15, 2026 1:42 PM > *To:* slurm-users(a)lists.schedmd.com <slurm-users(a)lists.schedmd.com> > *Subject:* [slurm-users] Jobs not getting assigned the requested GPU card > We have a Slurm Cluster that has been working fine for many years. A > few months ago it was upgraded to Slurm version 25.11.65, and has been > working properly running that version. The systems are running Rocky > Linux 9.8 and are patched/updated on a regular basis. They are currently > running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from > Nvidia's repo. > > Today it was reported that jobs are not getting assigned the GPU card > they requested, so for example they may request a 5090 card but are > assigned an A100 card. This was working fine, so we are struggling to > figure out what has changed to cause this behaviour. > > We have a node with 5 GPU cards, of three types: > > % nvidia-smi -L > > GPU 0: NVIDIA GeForce RTX 5090 (UUID: > GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6) > GPU 1: NVIDIA GeForce RTX 3080 (UUID: > GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94) > GPU 2: NVIDIA GeForce RTX 3080 (UUID: > GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae) > GPU 3: NVIDIA A100 80GB PCIe (UUID: > GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) > GPU 4: NVIDIA GeForce RTX 3080 (UUID: > GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094) > > /etc/slurm/gres.conf matches the above output (and has been this way for > years): > > % cat /etc/slurm/gres.conf > Name=gpu Type=rtx5090 File=/dev/nvidia0 > Name=gpu Type=rtx3080 File=/dev/nvidia1 > Name=gpu Type=rtx3080 File=/dev/nvidia2 > Name=gpu Type=a100 File=/dev/nvidia3 > Name=gpu Type=rtx3080 File=/dev/nvidia4 > > > The /etc/slurm/slurm.conf entry for this node is the same as it has been > for years: > > NodeName=moose[63] CPUs=16 RealMemory=380000 > Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700 > > > When submitting a job such as: > > $ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash > srun: job 71048 queued and waiting for resources > srun: job 71048 has been allocated resources > > $ nvidia-smi -L > GPU 0: NVIDIA A100 80GB PCIe (UUID: > GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) > > Any suggestions as to where to look? > > Thank you, > > -Dj > > > > > -- > slurm-users mailing list -- slurm-users(a)lists.schedmd.com > To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com

5 16

5G Chipset Market Industry Trends and Global Forecasts to 2040
by nidhithankur987＠gmail.com 16 Oct '26

16 Oct '26

Market Size and Growth Forecast 2026 Valuation: USD 71 billion 2040 Forecast Valuation: USD 455 billion Compound Annual Growth Rate (CAGR): ~14.25% from 2026 to 2040 Key Market Segmentations By Type of Chipset: Application-Specific Integrated Circuits (ASICs): Custom-designed for specific tasks (such as Layer-1 scheduling in 5G radio units), providing significantly better performance-per-watt to manage severe thermal constraints in base stations and smartphones. Other configurations include standard modems, RFICs, and mixed-signal components. By Deployment / Application Type: Smartphones & Tablets: Currently capture the highest share of the 5G chipset market, supported by billions of shipped units globally. Affordability Impact: Driven by the decreasing price gap between 5G and 4G chipsets, making 5G devices increasingly accessible. High-Frequency Advantages: Utilization of millimeter-wave (mmWave) frequencies (ranging from 24 GHz to 100 GHz) delivers ultra-high data transmission speeds and optimizes Internet of Things (IoT) connected device experiences. By End-Users: Consumer Electronics Automotive Industry: Propelled by the rollout of connected cars and electric vehicles featuring embedded internet connectivity. Automation Industry: Expected to experience a notably high growth rate driven by the rising demand for industrial process automation. Healthcare and other sectors. Geographical Distribution: Asia-Pacific: Currently holds the majority share of the global 5G chipset market, supported by a highly developed telecommunication infrastructure. North America: Projected to witness a relatively higher CAGR during the forecast period (2026–2040). Core Market Drivers & Highlights Next-Generation Connectivity: Designed to deliver significantly higher data throughput and lower latency compared to legacy 4G chipsets. Emerging Technological Use Cases: Integration across high-speed telecommunications, smart manufacturing, C-V2X (Cellular Vehicle-to-Everything) automotive frameworks, and large-scale IoT ecosystems. https://www.rootsanalysis.com/5g-chipset-market

1 0

Global AI in Drug Discovery Market Trends and Future Forecast Report
by abhaybhardwaj282＠gmail.com 15 Oct '26

15 Oct '26

The latest report published by Roots Analysis, titled Global AI in Drug Discovery Market offers a detailed study of the industry size, growth scenarios, and fundamental parameters of the market analysis. In this research report, the author provided an in-depth analysis of market dynamics, including key drivers, restraints, opportunities, and major threats that may impact growth during the upcoming years. Our market analysis report provides a brief discussion on leading market players, competition landscape, regional dominance, supply-demand ratio, business strategies, and future investment opportunities in the overall industry. This report provides a complete prototype of the estimated growth rate, market value, product types, applications, and type of end users active in this industry. What are the Report Highlights? The AI in Drug Discovery industry analysis report offers a comprehensive overview of the Market and its major growth prospects such as drivers, restraints, challenges, and upcoming opportunities. It also throws light on the current industry trends, major developments, and other interesting insights useful for this market. It further discusses the nature of the market competition, list of players and their key strategies to stay put in the competition. It also describes the table of segmentation and the names of the leading segment with market figures. The market report is provided on the company website. Major Players in AI in Drug Discovery market: Aiforia Technologies, Atomwise, BioSyntagma, Chemalive, Collaborations Pharmaceuticals, Cyclica, DeepMatter, Exscientia, InveniAI, MAbSilico, Optibrium, Recursion Pharmaceuticals, Sensyne Health and Valo Health. Leading market players and manufacturers are studied to help give a brief idea about them in the AI in Drug Discovery market report. The challenges faced by them and the reasons they are on that position is explained to help make a well informed decision. The competitive landscape of AI in Drug Discovery market is given presenting detailed insights into the company profiles, developments, mergers, acquisitions, economic status and best SWOT analysis. AI in Drug Discovery Market Segmentation: AI in Drug Discovery Market (2nd Edition): Industry Trends and Global Forecasts, till 2035 - Distribution by Drug Discovery Steps, Therapeutic Area and Key Geographies (North America, Europe, Asia-Pacific, Latin America, MENA, and RoW) Browse Complete Report @ https://www.rootsanalysis.com/reports/ai-based-drug-discovery-market.html

1 0

Fermenters and Bioreactors Market
by nidhithankur987＠gmail.com 15 Oct '26

15 Oct '26

Market Size & Growth Projections Market Valuation: The global industrial robotics market is projected to grow from USD 22 billion in 2026 to USD 35 billion by 2030, and reach USD 101 billion by 2040. Compound Annual Growth Rate (CAGR): The market is anticipated to expand at a robust 11.45% CAGR from 2026 through 2040, driven by the acceleration of Industry 4.0, smart factory automation, and rising global demand for consumer goods and automobiles. Core Market Segments & Share Insights By Type of Robot: Traditional Industrial Robots: Dominate the market with approximately 85.77% revenue share in 2026, valued for their high payload capacity, unmatched precision, and speed in heavy-duty sectors like automotive and metal fabrication. Collaborative Robots (Cobots): Represent the fastest-growing strategic segment, projected to expand at a 14.85% CAGR through 2040, fueled by labor shortages and adoption in SMEs, healthcare, logistics, and electronics. By Payload Capacity: Low Payloads (<20 Kgs): Lead the market with 54.20% revenue share in 2026, heavily deployed in electronics, pharmaceuticals, cosmetics, and semiconductor manufacturing. Medium Payloads (20 to 150 Kgs): Projected to register a fast 11.2% CAGR, ideal for mid-range manufacturing, welding, painting, and flexible assembly. By Mobility: Stationary Robots: Capture around 65% of the market share in 2026, essential for fixed, high-volume tasks such as welding and assembly. Mobile Robots: Represent the highest-growth mobility segment, expected to grow at a 16.5% CAGR, driven by warehouse automation, e-commerce fulfillment, and intra-factory logistics. By Application Area: Handling Operations: Currently lead the application sector, accounting for 42.0% of the market in 2026, supported by widespread compatibility and e-commerce logistics demands. Dispensing: Expected to see accelerated growth at a 12% CAGR, driven by precision material application needs across food, beverage, and agricultural industries. Key End-User Industries: Automotive Industry, Electronics Industry, Metal and Machinery Industry, Plastics Industry, and Food & Beverages Industry. Key Growth Drivers & Industry Trends Productivity & Safety: Automated labor environments report over a 5% increase in productivity, while industrial robot end-effectors and grippers safely execute hazardous or strenuous tasks (such as high-pressure vacuum or radiation chambers). Robot-as-a-Service (RaaS): Growing preference for flexible subscription and service-based deployment models is lowering upfront capital barriers for small and medium-sized enterprises (SMEs). AI & IoT Convergence: Next-generation industrial robots are increasingly integrated with artificial intelligence and IoT capabilities, enabling adaptive handling of complex production tasks and autonomous mobile navigation.

1 0

Slurm 26.05.4: idle~ nodes go to unknown after reconfigure
by Stefan Staeglich 14 Oct '26

14 Oct '26

Hi, since we updated our site from Slurm 24.11 to Slurm 26.05.4 the idle~ (powersaved) nodes goes to state unk (unknown) whenever running scontrol reconfigure. This state is not cleaned by Slurm itself. Is this a known issue of Slurm 26.05? Best, Stefan

4 8