Communications of the ACM
Home/Magazine Archive/June 2022 (Vol. 65, No. 6)/Learning New Things and Avoiding Obstacles/Full Text
Last byte
Learning New Things and Avoiding Obstacles
Credit: Alexander Berg
ACM A.M. Turing Award recipient Jack Dongarra never intended to work with computers. Initially, the Distinguished Professor at the University of Tennessee and founder of the Innovative Computing Laboratory (ICL) thought he would be a high school science teacher. A chance internship at the Argonne National Laboratory kindled a lifelong interest in numerical methods and software—and, in particular, in linear algebra, which powered the development of Dongarra's groundbreaking techniques for optimizing operations on increasingly complex computer architectures.
Your career in computing began serendipitously, with a semester-long internship at Argonne National Laboratory.
As an undergraduate, I worked on EISPACK, a software package designed to solve eigenvalue problems. My role was helping to develop test problems and making sure things were working correctly. It was a wonderful environment. Then, I completed my master's degree, and they offered me a job.
"With my colleagues in Germany, Hans Meuer and Eric Strohmaier, we put together the first Top500 in 1993."
You began working on LINPACK around that time.
LINPACK was a National Science Foundation-funded project in the late 1970s that involved researchers at Argonne, University of New Mexico, University of California San Diego, and the University of Maryland. The goal was to design a software package for solving systems of linear equations that was based on state-of-the-art algorithms—and was portable, reliable, enhanced productivity, and provided efficient solutions to the scientific computer architectures in use at that time. As a way to measure efficiency, I constructed a benchmark to measure the performance of a computer when running the LINPACK software, which became the LINPACK benchmark. It appeared in a table in the LINPACK Users' Guide.
Figure. "The software packages that I have been involved in designing have found their way into the fabric of how problems in computational science are solved," observes 2021 ACM A.M. Turing Award recipient Jack Dongarra.
The table was the genesis of the TOP500 list of the most powerful computers.
With my colleagues in Germany, Hans Meuer and Eric Strohmaier, we put together the first TOP500 in 1993. Before then, Hans had a list of the fastest computers and I had a benchmark that rated those computers; Hans approached me about putting it together and calling it the TOP500.
You have been designing software packages over the last 50 years. Why?
The software packages that I have been involved in designing have found their way into the fabric of how problems in computational science are solved. As computer architectures evolve over time, from scaler to vector to multicore to distributed memory to hybrid architectures, the software packages are among the first to adapt to the changes. They must be rewritten to embrace the architecture.
You can see this in the evolution and development of the packages. EISPACK was designed for scalar computers and LINPACK for vector architectures; LAPACK and the BLAS for use on cache-based and shared memory computers; ScaLAPACK and MPI were intended for distributed memory architectures, and PLASMA and MAGMA developed as the need for multicore and hardware accelerators (GPUs) entered the computer landscape. And today we are working on SLATE, which addresses the challenges of exascale-based computing. Along the way, there is the need for performance evaluation, and that is where the benchmarks fit into the picture.
The high performance conjugate gradients (HPCG) benchmark solves matrix problems using an iterative approach that manipulates sparse matrices.
Over the years, Top500 rankings have evolved to include not just the LINPACK benchmark, but other ways to test how well computers can handle the sorts of tasks people need them to do.
The LINPACK benchmark has been in continuous use since the 1970s. It was born out of necessity, because it could quickly test the performance of vector subroutines, which served as a good approximation of performance for the rest of the LINPACK library. Thanks to the nature of the implementation, the LINPACK benchmark also served as a first-order approximation of other codes. That is partially due to the well-balanced hardware of the time, which offered plentiful bandwidth for every floating-point operation. Over the years, Moore's Law eroded the compute-to-bandwidth balance, resulting in a memory wall.
Figure. The authors of LINPACK—from left, Jack Dongarra, G.W. "Pete" Stewart, Jim Bunch, and Cleve Molar—photographed around Dongarra's car in 1978 in Downer's Grove, IL, near Argonne National Laboratory. Photo used with permission from Cleve Molar.
To reassess application needs in this new and different hardware regime, it is worthwhile to look at computational simulations. Many computational simulations involve heat diffusion, electromagnetics, and fluid dynamics. Unlike LINPACK, which tests raw floating-point performance, these real-world applications rely on partial differential equations (PDEs) that govern the continuous representations of physical quantities like particle speed, momentum, etc. These PDEs involve sparse (not dense) matrices that represent the 3D embedding of the discretization mesh. While the size of the sparse data fills the available memory to accommodate the simulation models of interest, most of the optimization techniques that help achieve close to peak performance in dense matrix calculations are only marginally beneficial in sparse matrix computations originating from PDEs.
Why is that?
The TOP500 LINPACK benchmark can be characterized as a dense matrix doing dense operations. Machines that do floating-point operations efficiently are going to look good on this benchmark, even though most real-world problems don't actually require it.
So we developed a benchmark called the high-performance conjugate gradients, or HPCG. HPCG solves matrix problems not with a direct approach—not based on matrix multiplication where, if you have two matrixes of roughly order n, the number of operations is n3 but the amount of data moved is only n2. Instead, HPCG uses an iterative approach that manipulates sparse matrices, which shows the characteristics of the hardware better in terms of real applications.
Just to put that into perspective, if I were to run the TOP500 benchmark on a computer, I would expect to see performance reach 75% of the theoretical peak. The HPCG benchmark shows performance of 3% of the theoretical peak. That's our "dirty little secret": most applications are far from having reached the theoretical peak of these high-performance computers. Still, it is a good way to expose the performance issues and look at how we might resolve and improve the situation.
Let's talk about what's new in the high-performance computing space. By the time we go to print, so-called exascale machines—which are capable of executing 1018, or a billion-billion, floating-point calculations per second—may finally be on top of the latest TOP500 list.
Oak Ridge National Laboratory is installing a computer called Frontier, which will be the first exascale machine in the U.S. They're running the benchmark on it now, and we expect that will be completed by the time the next TOP500 list comes out.
What about the economics of high-performance computers?
The exascale machines are very expensive. I think the price tag for the machine at Oak Ridge is between $500 million and $600 million, and the Department of Energy (DOE) is engaged in a program that's developing three of these exascale machines—in addition to their investment in developing the applications and software that will run on these systems. The total price tag for that program over seven years is going to be around $3.6 billion.
These supercomputers are powerful, sophisticated scientific instruments, like the James Webb telescope. They enable simulation and provide the opportunity to push back the frontiers of science. Today, we use numerical computations to understand and predict the behavior of scientifically or technologically important phenomena—and therefore accelerate the pace of innovation.
It is important to note that every time computing power increases by large factors, new benefits open before us. The benefits of exascale computing—which range from creating novel, more efficient combustion engines and new energy solutions to advances in healthcare, biology, and storm prediction—could potentially impact every person. The benefits of exascale computing will flow from classical simulations but also from large-scale data analysis, deep machine learning, and often the integration of the three approaches.
"It is important to note that every time computing power increases by large factors, new benefits open before us."
Your work has been fundamental to a huge number of applications, from cloud computing to large-scale physics experiments. Are there things you've found surprising or exciting, beyond the work itself?
There's always something new to learn and use in solving the current problems. I have been fortunate to work with an incredibly talented international community of people over the years to develop algorithms, software, and standards that have helped shape the computational science area. That work could not have happened without those people.
Having students is instrumental in that respect, because it helps us push forward and explore multiple fronts. We try to do research, not just development, meaning we need to experiment and we need to fail, because that's an important part of the learning process. Sometimes, our students come to me in search of a research problem to work on, and when I give them one, they immediately tell me that they don't know how to do it. And I say, "That's perfect. That's exactly the point. I wouldn't give you a problem if you already knew how to do it." That's where the excitement is, learning new things and overcoming obstacles.
©2022 ACM 0001-0782/22/6
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and full citation on the first page. Copyright for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or fee. Request permission to publish from [email protected] or fax (212) 869-0481.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2022 ACM, Inc.
Comments
Edward Anderson
From left to right, the authors of LINPACK gathered around Jack's car are Jack Dongarra, Cleve Moler, Pete Stewart, and Jim Bunch.
Displaying 1 comment