System overview

System overview

The SNOW Linux Cluster consists of 15 compute nodes, 11 of which have four 16-core processors per node, and the remaining 4 nodes have two quad-core processor each, for a total of 736 cores. It is configured with 3144Gb of total memory and 13TB of NFS disk space, although more storage can be transparently added. Its theoretical computer performance peak, according to the manufacturer specifications, has the capacity of 8490 Gflops.

System configuration

Calculation nodes

All SNOW nodes run Linux Debian 7.9 x64 OS and support batch services through Open Grid Engine 6.2. User data space is supported by a NetApp FAS 3140 appliance, giving NFS service to SNOW over a 10Gbps link.

There are:

- 11 x 4-U traditional servers, each one housing 4 AMD Opteron Abu Dhabi 6378 processors, 256Gb of RAM and a 240Gb SSD solid state disk. Each of this regular nodes are Supermicro Servers, manufactured by SIE under the SIE LADON brand. Beside these 11 server, are other 4 1-U servers, each one housing 2 Intel Xeon  E5504 CPU, 82GB of RAM and 1 Tb of local disk space.

- 4 x 1-U DL180 G6, each one housing 2 Intel Xeon E5504 2.00GHz, 64GB or 80GB of RAM

- 3 x 4-U ML350 G6, each one housing 2 Intel Xeon E5520 2.27GHz, 64GB or 80GB of RAM

- 1 x 1-U Dell R430, each one housing 2 Intel Xeon  E5-2600 v3 2.40GHz, 64GB of RAM

- 9 x GeForce GTX TITAN X Maxwell, each one have 3.072 CUDA Cores and 12 GB of GDDR5

- 5 x GeForce GTX TITAN X Pascal, each one have 3.584 CUDA Cores and 12 GB of GDDR5

- 1 x GeForce Tesla K40c, each one have 2.880 CUDA Cores and 12 GB of GDDR5

 

Fig. 1: Computing nodes.

 AMD PROCESSOR: SUPERMICRO SIE LADON                                                      INTEL PROCESSOR: HP PROLIANT DL160G6    

 

 INTEL PROCESSOR: HP ML350 G6                                                                        INTEL PROCESSOR: DELL R430    

 

 

GPU PROCESSOR

 

Table 1. Technical specifications for computing nodes:

Component Technology   Performance/Size
Nodes  AMD                    Four sixteen-cored AMD 6378 processors        11 Nodes / 704 Cores
Nodes INTEL Two quad-cored INTEL Xeon E5504 processors 7 Nodes / 80 Cores
Nodes INTEL 2 INTEL Xeon E5-2600 v3 1 Node / 32 Cores
Memory Distributed 3,3TB (Aggregate)
Shared Disk NFS  15Tb (resizable)
Local Disk 240Gb SSD 2,4Tb (aggregate)
Local Disk INTEL
HP 1T 3.5" SAS 7200 rpm
4Tb (aggregate)
Interconnect Gigabit ethernet *
GPU Nvidia GPU Titan X Maxwell 2 PCIe / 6144 Cores
GPU Nvidia GPU Tesla K40c 2PCIe / 4291 Cores
GPU Nvidia GPU Titan X Pascal 2PCIe / 10157 Cores

Head node

Additionally, SNOW is geared with a head node to support user interactive sessions, job managing and back-end VMs.

The SNOW head node is a single, 2-U server running Linux Debian 7, with 2 AMD Opteron 6328, 32Gb RAM and 4 10krpm, RAID5, SATA drives.

Fig. 2: Head node

Table 2. Technical specifications for the head node:

Component Technology   Performance/Size
Node                       Two twelve-cored AMD 6378 processors        1 Nodes / 24 Cores
Memory Single 32 Gb
Shared Disk NFS  6Tb (resizable)
Local Disk 5 disks 500Gb 10krpm, RAID5 1,8Tb
Interconnect Gigabit ethernet *
 

Network configuration

Several separate networks have been setup for the cluster different services: NFS, MPI, internal data moving, ssh.

Table 3. SNOW cluster networks

Network name Technology Performance
NFS                       TenGigabit  ethernet / Gigabit 10Gb uplink / 1Gbps
MPI Gigabit 1Gbps
IMPI Gigabit 1Gbps
DATA Paravirtualized Xen 1Gpbs
PUBLIc Gigabit 1Gbps

The AMD Abu Dhabi 6378 processor

The new Abu Dhabi 6378 is the main workhorse processor on the SNOW cluster. It includes this features:

  • On-chip integrated memory controllers
  • Big L3 cache: 16M
  • 48K L1 cache
  • Piledriver cores
  • Each FPU is shared between two cores.

Here are the full specs:

Table 4. Abu Dhabi processor full specs

Processor AMD Opteron™ 6300 Series Processor
Model 6378
AMD Virtualization Technology Yes
AMD-P (Power Management) Yes
OPN Tray OS6378WKTGGHK
OPN PIB n/a
Revision C0
Core Count 16
Base Speed (MHz) 2400
System Bus Speed (MT/s) 6400
Voltages  
Max Temps (C) 69'C
Wattage 115 W
L1 Cache Size (KB) 48
L1 Cache Count 16
L2 Cache Size (KB) 1000
L2 Cache Count 16
L2 Cache Speed (MHz) 2400
L3 Cache Size (KB) 16384
CMOS 32nm
Integrated Memory Controller Speed (MHz) 2000
Floating Point Unit 256-bit FPU
Socket G34

 

The INTEL XEON E5504 processor

It is the second processor model used in the cluster.  Its reduced number makes it to be considered the secondary processor.

Table5. Intel Xeon precessor full specs.

Processor Intel® Xeon® Processor
Model E5504
Intel Virtualization Technology Yes
Memory Types DDR3 800
Maximum Memory Size (GB) 144
Maximum Memory Channels    3
Core Count 4
Instuction Set  64 bits
Base Speed (MHz) 2000
System Bus Speed (MT/s) 4800
Voltages 0.75V - 1.35V
Max Temps (C) 76°C
Wattage 80 W
Cache Size (KB) 4000
CMOS 45nm
Integrated Memory Controller Speed (MHz) 2000
Socket FCLGA1366

 

The INTEL XEON E5520 processor

Table6. Intel Xeon processor full specs.

Processor Intel® Xeon® Processor
Model E5520
Intel Virtualization Technology Yes
Memory Types DDR3 800/1066
Maximum Memory Size (GB) 144
Maximum Memory Channels    3
Core Count 4
Instuction Set  64 bits
Base Speed (MHz) 2260
System Bus Speed (MT/s) 4800
Voltages 0.75V - 1.35V
Max Temps (C) 76°C
Wattage 80 W
Cache Size (KB) 8000
CMOS 45nm
Integrated Memory Controller Speed (MHz) 2000
Socket FCLGA1366
 
 

The INTEL XEON E5-2600 v3 processor

Table7. Intel Xeon processor E5 v3 Family full specs.

Processor Intel® Xeon® Processor
Model E5-2630 v3
Intel Virtualization Technology Yes
Memory Types DDR4 1600/1866
Maximum Memory Size (GB) 768
Maximum Memory Channels   4
Core Count 8
Instuction Set  64 bits
Base Speed (MHz) 2400
System Bus Speed (MT/s) 4800
Voltages 0.75V - 1.35V
Max Temps (C) 72.1°C
Wattage 85 W
Cache Size (KB) 8000
CMOS 22nm
Socket FCLGA2011-3
 

The Maxwell GPU processor

Table8. GTX Titan X Specs.

CUDA Cores 3.072
Base Clock (MHz) 1000
Boost Clock (MHz) 1075
Texture Fill Rate (GigaTexels/sec) 192
Memory Clock 7.0 Gbps
Standard Memory Config 12GB
Memory Interface GDDR5
Memory Interface Width 384-bit
Memory Bandwidth (GB/sec) 336.5
Maximum Digital Resolution 5120x3200
Maximum VGA Resolution 2048x1536
Multi Monitor  4 displays
Maximum GPU Temperature (in C) 91
Graphics Card Power (W) 250

The Tesla GPU processor

Table9. Tesla K40c Specs.

CUDA Cores 2.880
Base Clock (MHz) 745
Boost Clock (MHz) 875
Memory Clock 3.0 Gbps
Standard Memory Config 12GB
Memory Interface GDDR5
Memory Interface Width 384-bit
Memory Bandwidth (GB/sec) 288
Maximum GPU Temperature (in C) 70
Graphics Card Power (W) 245

The Pascal GPU processor

Table10. GTX Titan X Pascal Specs.

CUDA Cores 3.584
Base Clock (MHz) 1417
Boost Clock (MHz) 1531
Texture Fill Rate (GigaTexels/sec) 136
Memory Clock 10 Gbps
Standard Memory Config 12GB
Memory Interface GDDR5
Memory Interface Width 384-bit
Memory Bandwidth (GB/sec) 480
Maximum Digital Resolution 5120x3200
Maximum VGA Resolution 2048x1536
Multi Monitor  4 displays
Maximum GPU Temperature (in C) 91
Graphics Card Power (W) 250

Filesystems

SNOW includes several different filesystems, each one intended for different purposes. The main filesystems are:

  • $HOME: User-space filesystem
  • Local: Local filesystem for each node
  • SOFT: Storage area used for software deployment. Available to all nodes
  • SGE: Storage used for the scheduler for internal purposes. Available to all nodes.

Table 5. SNOW cluster filesystems

Storage Class Size Architecture Features
$HOME 6Tb NFS-accessed, NetApp WAFL back-end Mounted on /homedtic, /homedtcl, /homedcom (depending on  the user's department adscription).
Local 240Gb ext4 over a single SSD drive  Locally available to each node. Main feature is /scratch to speed up user I/O while calculating.
SOFT 6Tb NFS-accessed, NetApp WAFL back-end Mounted on /soft. 
/SGE 6Tb NFS-accessed, NetApp WAFL back-end Mounted on /sge.