Published on December 15, 2021 by Aaron Dubrow

Magnetospheric interaction of two neutron stars in their final orbits [Credit: E.R. Most and A. Philippov]

Peak performance on supercomputers may be rising unchecked, but getting real improvements in scientific productivity is an altogether different challenge.

Since 2019, the Texas Advanced Computing Center has been working with the National Science Foundation on an effort to create a Leadership-Class Computing Facility (LCCF) at The University of Texas at Austin. A key component of the facility will be a new instrument for science that can achieve an order of magnitude, or 10x, improvement in scientific efficiency over Frontera, the top academic supercomputer in the world today, when it comes online.

TACC plans to achieve these improvements, in part, through software, algorithmic, and environmental optimizations.

Speaking at the inaugural Frontera User Meeting in 2021, TACC executive director Dan Stanzione asked, "Is it realistic to expect us to make significant improvements in software over the next few years, given that some of our leading codes are decades old?"

His answer: Absolutely. To illustrate his point, he referenced NAMD (Nanoscale Molecular Dynamics), a popular biophysics code first introduced in 1995 that saw an 80% performance improvement when it was optimized for AVX 512, the latest x86 vector instruction set, in 2020.

 

Accretion dynamics around supermassive black hole mergers using the PatchworkMHD code. The view is on the equatorial plane. [Credit: Mark Avara, RIT]

 

Stanzione, in his talk, noted four ways researchers, through careful consideration of their computing problem, have been able to achieve massive improvements in code performance.

 

1. Change Algorithms

Manuela Campanelli, Distinguished Professor, Rochester Institute of Technology

 

Campanelli, principal investigator of two Frontera allocations, is among a team of astrophysicists who developed a ‘Patchwork' framework, where the global simulation is comprised of an arbitrary number of moving local meshes, or patches, which are free to employ their own resolution, coordinate system, physics equations, reference frame, and even, numerical methods.

The framework — primarily developed by postdoctoral researchers, Mark Avara (Rochester Institute of Technology) and Hotaka Shiokawa (John Hopkins University) — permits efficient computation of heterogeneous systems involving multiple kinds of physics, multiple length-scales, and multiple reference frames.

At the user meeting, Campanelli described the first successful Patchwork magneto-hydrodynamics (PWMHD) simulation of accreting binary supermassive black holes. Her team found that the long-term simulation, covering the full domain, was 30 times more efficient with PWMHD than with their previous methods.

"Algorithmic improvements like this multipatch scheme take years to develop and are a big lift and a big investment," Stanzione said. "But if we focus on them, and measure them, we can make codes better over time."

 

2. Optimize Codes for Processors

Elias Most, Associate Research Scholar, Princeton University, and Member, Institute for Advanced Study

 

The collision of two neutron stars might not only be accompanied by gravitational wave emissions and electromagnetic afterglows. The interaction of the stars' strong magnetic fields prior to the collision could give rise to as-yet-unobserved radio and X-ray transients.

Using Frontera, Most and Alexander Philippov (Flatiron Institute) have been investigating flares in relativistic magnetospheres of compact objects using global force-free simulations. In addition to their science findings, Most reported a 30% speed up of his simulations, mainly by vectorizing 99.1.% of the code.

He found that the code had been memory limited, but after optimizing it for Frontera's Intel processors, he was able to achieve up to 20% of peak performance on the code and good scaling up to 25,000 cores.

"We leverage all of the architectural advances of the Cascade Lake processors to the full extent to speed up our performance," Most said.

 

3. System Software Improvements

Hari Subramoni, Research Scientist, Ohio State University

 

MVAPICH is a leading implementation of MPI, the message-passing standard that is widely used on parallel supercomputing systems. Led by the Network-Based Computing Laboratory (NBCL) at The Ohio State University, MVAPICH has been used by the HPC community for more than 20 years and downloaded more than one million times.

The MVAPICH team is led by D.K. Panda, who also serves a co-principal investigator on Frontera. His team uses the system to test new features of their software at the most extreme scales.

"What are the requirements coming from applications and how can we best match those by providing newer solutions for the modern networking and computing hardware coming from vendors?" said Hari Subramoni, lead designer of the MVAPICH2/MVAPICH2-X software stack.

Subramoni showed how SHARP (Sharable Hierarchical Aggregation) can improve the performance of critical collective operations like all-reduce, reduce, and broadcast by five to nine times at full system scale, by offloading them to the switch.

"You'd be hard-pressed to find a code that doesn't have an all-reduce or broadcast operation," Subramoni said.

This massive improvement to a few functions could translate into a 5% or 10% performance improvement in a full code.

"By taking advantage of the right underlying primitives for point-to-point or collective operations, we can get significant benefits at the micro-benchmark and application levels," he said.

"We can do system software and firmware improvements that make the hardware run better over time and also invest in system libraries that can also improve performance," Stanzione said.

 

4. Tuning for the Specific System

Daniel Bodony, Professor of Aerospace Engineering, University of Illinois Urbana-Champaign

 

Bodony studies hypersonic flows using direct numerical simulations, or DNS. The problems he investigates are multi-scale in space and time and involve orchestrating between a number of different domain-specific codes, written primarily in C++ and Fortran, using MPI and OpenMP.

 

Frontera-computed Mach 6 turbulent flow past a deformable hypersonic vehicle control surface. Inset: Frontera-computed simulation of reactive, shock-laden flow past an entire vehicle [Credit: Daniel Bodony, University of Illinois at Urbana-Champaign]

 

Bodony's initial performance studies for an MPI-OpenMP DNS code on Frontera suggested that four MPI ranks per node would be optimal, with the rest of the hardware threads taken up with OpenMP tasks. However, after he included the I/O overhead, using parallel HDF5, he cut the number of MPI ranks in half to achieve fastest time-to-solution. Bodony found that Frontera I/O throughput was increased when one MPI task per socket was engaged in writing the data to parallel storage.

The change from four to two MPI tasks per node, while keeping the total number of MPI tasks x OpenMP threads constant, improved time-to-solution by 15%, with I/O.

"We're getting linear scaling on problems of production size, and have run on the full machine," Bodony said. His typical production size uses curvilinear structured meshes to support 30+ billion degrees of freedom to resolve the turbulence, shocks, and their interactions, and utilize between 1K and 2K Frontera nodes.

"Bodony's work shows that there's a lot of specific hardware tuning one can do to boost performance over time," Stanzione said.

 

COLLABORATING WITH CODE DEVELOPERS

 

In any substantially-sized user meeting, TACC hears examples of performance improvements related to each of these four categories. "However, they don't happen without a lot of effort, and a lot of that effort is largely unfunded," Stanzione said. "We want to address that to the best of our ability."

This is where TACC's new Characteristic Science Applications program comes in. Earlier this year, the program solicited application codes and ‘grand challenge'-class science problems from the community of current and emerging large-scale scientific computing users. Selected partners will receive funding over multiple years to refine their chosen applications to run on the future LCCF architecture.

In September 2021, NSF awarded TACC $7 million to select, study, and transform a set of Characteristic Science Applications to enable next-generation science for the LCCF. In 2022, TACC will name the approximately 20 applications selected for support.

"We want to work with some of these large application teams and co-evolve with them," Stanzione concluded. "This is how we advance science."