Providing resources for scientists to expand their research in atmospheric and geospatial sciences

About the customer

High-performance computing (HPC) systems at the National Center for Atmospheric Research (NCAR) power the science behind predicting and researching global weather and climate. In early 2021, NCAR announced that their Cheyenne supercomputer, located at the NCAR-Wyoming Supercomputing Center (NWSC), will be replaced by a new HPE Cray EX system with 3.5 times more processing power, up to 19.87 petaflops with a combination of central processing unit (CPU) and graphical processing unit (GPU) nodes. The new system will be funded by the National Science Foundation (NSF) and will include efficient workload management and scheduling by HPCWorks™ PBS Professional™ software and HPCWorks Runtime Plus software, both part of the Siemens Xcelerator business platform of software, hardware and services.

The challenge

The global challenge of understanding, simulating and predicting the behavior of weather systems, climate patterns, and other atmospheric phenomena is critical for Earth and for everyone on it. Keeping up with climate change and understanding the actions we must take to preserve our planet and its diverse life requires powerful, sophisticated computing resources. To predict what happens next, NCAR needs the latest, most powerful and fully featured HPC solutions to enable weather and climate research.

Image ©University Corporation for Atmospheric Research

Our solution

NCAR already relies on HPCWorks PBS Professional for workload orchestration in Cheyenne. The team was running Slurm on their “Casper” data analysis and machine learning (ML) system, but they switched to HPCWorks PBS Professional to take advantage of advanced scheduling features and Siemens’ expert support. The Siemens solution will keep jobs running smoothly on the new HPE Cray system and will introduce features such as cloud bursting for near-infinite scalability; a requirement when it’s critical to quickly understand harsh weather conditions in places like Antarctica. With a limited window of opportunity to fly goods, equipment and people onto and off the frozen continent, knowing exactly what to expect is paramount to safety and efficiency.

Using HPCWorks PBS Professional also enables the NCAR team to use live fairshare data in job sort formulas and allows HPC administrators to make maintenance reservations. HPCWorks PBS Professional reservations are robust and can replace down nodes during the life of a reservation. Additional features on the new NCAR system will include high-throughput hierarchical scheduling with HPCWorks Runtime Plus for greater throughput, better license and resource utilization, and more flexible scheduler usage models. This will help NCAR develop and test the Weather Research and Forecasting (WRF) model for atmospheric research and operational forecasting applications.

HPC at NCAR is critical for studying phenomena from terrestrial weather patterns to solar storms. Image by Matthias Rempel, ©University Corporation for Atmospheric Research

Results

The new system, dubbed “Derecho” – a powerful type of storm – in a competition among Wyoming K-12 students, is slated for delivery in late 2021 and production in early 2022. It will replace NCAR’s Cheyenne and Laramie systems and is expected to rank among the top 25 supercomputers on the Top500 Cray list. “With its extra storage and a 3.5-fold capability improvement over the current NCAR supercomputer, Cheyenne, the new supercomputer will provide the necessary resources for our scientists to continue expanding their research in the atmospheric and geospatial sciences,” says Irfan Elahi, project director for the new system and director of the NCAR High-Performance Computing Division. “To provide this capability, the new supercomputer is designed for highly energy-efficient operations so it can exploit the workload manager and job scheduler features like green provisioning and energy-aware scheduling.”

“This new system is a major step forward in supercomputing power, providing the scientific community with the most cutting-edge technology to better understand the Earth system,” says Anke Kamrath, director, computational and information systems laboratory, NCAR.