EN Bereichsnavigation EN

Cray XE6 System - Rosa

The upgraded Monte Rosa comes with AMD Interlagos 2 x 16-core 64-bit CPUs, 32 GB per compute node and high performance networking with Gemini 3D torus interconnect. It features a total of 1496 nodes, that is 47872 cores, with a theoretical peak performance of 402 TFlops.


Production projects should target the Cray XE6, if they use highly scalable codes of excellent performance and make use of MPI or hybrid OpenMP/MPI parallelization schemes.

This page describes Rosa in the following sections

A Short Description of the Machine

Monte Rosa is a 16 cabinet Cray XE6 system with 8 login nodes, 1496 compute nodes and 21 nodes to provide I/O support on Lustre and other external file systems.
Each of the compute nodes consists of 2 16-core AMD Opteron 6272 2.1 GHz Interlagos processors, giving 32 cores in total per node with 32 GBytes of memory.
In total there are 47872 compute cores and over 46 Terabytes of memory available on the compute nodes.

Ho to Access Monte Rosa

Every user who has a valid approved proposal on Monte Rosa can have access to the system.

Since April 10th, 2013 Monte Rosa has three new very powerful external login nodes with the same processes as the compute nodes. This helps greatly when building applications. 

To access CSCS you need to ssh ela.cscs.ch

To access Monte Rosa's external login nodes you simply need to ssh rosa.cscs.ch

If you want to to access a specific node you have to explicitly ask for it.

Also note that /scratch, /project and /store will be mounted directly on the  nodes for high performance file transfer.

Programming Environment and Supported Software

The software environment on Rosa is controlled using the modules framework which gives an easy and flexible mechanism to access to all of the CSCS provided compilers, tools and applications.

    Compilers for the Rosa system are described on the Rosa compilers page. Rosa has a very rich array of compilers available with PGI, Pathscale and GNU as were already provided on the Cray XT3 systems under catamount, but with the new addition of a Cray compiler for XT systems running the Cray Linux Environment.

    Information about supported performance tools and debugging can be accessed from the Debugging and Performance Analysis page. In addition to the Cray-provided Craypat and Apprentice2 tools, the IPM (Integrated Performance Monitor) tool is available on the XE6 systems, and additional tools are being tested. Parallel debugging support on Rosa is provided by Totalview as with the other Cray XT systems at CSCS.

    More information of the programming environment and supported software available on Rosa is provided in the following pages:

     

     

    Batch Jobs and Interactive Jobs

    Monte Rosa uses the SLURM batch system for the submission of jobs. Jobs can run for a maximum time of 24 hours: the commands sinfo and scontrol show partition give information on the queues and partitions available on the system. The "normal" queue is meant for production runs up to 24 hours, while the "dev" queue is reserved to development projects and it allows runs up to 2 hours. If your username belongs as well to a development project, you will be able to choose the queue where to run your job by issuing the "--partition" directive in your batch script:
    #SBATCH --partition=<partition_name>

    Please refer to the man pages and the official SLURM documentation for the details.

    Interactive batch jobs should only be requested for small period of time and should be requested on as small a numbers of processors as is required.  Since an interactive batch job is allocated through the standard batch scheduling algorithms, you should only request this type of access if there are already sufficient free resources on the machine so that the interactive session can begin immediately. Note that your project will be charged for all of the time that is taken by the interactive batch job, and not just the time that you spend running applications on the compute nodes.

    Details of batch submission and how to set up a batch job are available on the following page:

    How to Run a Batch Job

    For a list of the most useful SLURM commands, please have a look at the corresponding FAQ section under the User Forum.

    Detailed Machine Description

    Monte Rosa is a Cray XE6 machine based around AMD Opteron Interlagos 16-core processors.

    The machine consists of 384 Cray XE blades arranged in 16 cabinets of 24 blades each. There are 10 service blades which provide essential services and allow external access, whereas the bulk of the machine is populated with 374 compute blades for running user applications.

    Each service blade has 4 nodes, each of which consists of a 6-core AMD Opteron processor and one PCI-Express card to give access to external services such as I/O and to allow a login facility for the machine. The 10 service blades therefore provide 40 service nodes, and these are configured as follows

    • 2 nodes are given over to essential system functionality such as booting the system and running the batch system server and scheduler
    • 8 nodes provide login functionality for user access and compilation
    • 5 nodes are used as part of the job launch mechanism of SLURM
    • 13 nodes are used for the server capabilities of a Lustre scratch file system
    • 5 nodes act as servers to allow access on the compute nodes to external file systems such as home and /project
    • 4 LNET router nodes to export the scratch filesystem to external clusters
    • 3 nodes are held in reserve for expansion in any areas where bottlenecks might be detected in the configuration

    Each of the compute blades has 4 compute nodes, and in turn each of the compute nodes consists of 2 16-core AMD Opteron Interlagos processors running at 2.1 GHz giving 32 cores in total per node with 32 GBytes of memory.

    Access to the high-speed network is through Gemini communications processor which is able to provide 5 GBytes/s of injection bandwidth in each direction per node, for the through-flow of packets out on the network.

    In total there are 47872 compute cores from AMD Opteron 2.1 GHz 16-core Interlagos processors and 46.75 Terabytes of DDR3-1600 memory available on the compute nodes. The peak performance of Rosa is 402 Teraflop/s.

    Rosa runs the Cray Linux Environment (CLE) operating system version 4.0.30, which means that a lightweight version of Linux is running on the compute nodes, which allows the use of mixed-mode MPI-OpenMP hybrid programming. The MPI-OpenMP hybrid approach implements  a distributed memory model using MPI communications between the nodes, and within a node it utilises a shared-memory threading paradigm.