[slurm-users] Kinda Off-Topic: data management for Slurm clusters
wdennis at nec-labs.com
Fri Feb 22 18:20:32 UTC 2019
On Friday, February 22, 2019 1:03 PM, Alex Chekholko said:
>If your bottleneck is now your network, you may want to upgrade the network. Then the disks will become your bottleneck :)
Via network bandwidth analysis, it's not really network that's the problem... It’s the NFS/disk I/O...
>For GPU training-type jobs that load the same set of data over and over again, local node SSD is a good solution. Especially with the dropping SSD prices.
Good to hear :)
>For an example architecture, take a look at the DDN "AI" or IBM "AI" solutions. I think they generally take a storage box with lots of flash storage and connect it via 2 or 4 100Gb links to something like an nvidia DGX (compute node with 8 GPU). Presumably they are doing mostly small file reads.
>In my case, I have whitebox compute nodes with GPUs and SSDs and whitebox ZFS servers connected at 40GbE. A fraction of the performance at a fraction of the price.
Same here, but connected at only 10G... Again, no budget (as of yet, anyhow) to do 25/40/50/100G network or all-flash storage :(
More information about the slurm-users