Leadership Systems and Leadership Science

By | 7/2/13

In 2002 the Japanese government announced the delivery of the ‘Earth Simulator,’ a 640-node vector parallel system capable of a peak performance of 40.96 teraflops. The high-performance-computing world was stunned: the climate-focused computer was more than three times as fast as the next-most powerful computer in the world and its theoretical peak performance was greater than the world’s next fastest half-dozen computers, combined. The high-speed-computing arms race had begun.

Within two years the Department of Energy High-End Computing Revitalization Act of 2004 was enacted and the phrase “leadership system” was coined. More than just a sobriquet, leadership systems represented for the US Department of Energy a new paradigm: a high-end computing system that is among the most advanced in the world in terms of performance in solving scientific and engineering problems. Now, more than a decade later, this vision to combine high-end computing with high-impact science is realized at the Leadership Computing Facility centers at Oak Ridge and Argonne National Laboratories, which field, respectively, the second and fifth most powerful computers in the world and jointly manage an allocation program annually awarding to researchers around the globe nearly five billion core-hours for science and engineering simulations.

Sixty percent of the available time on the 27-petaflops Cray XK7 “Titan” and 10-petaflops IBM Blue Gene/Q “Mira” are allocated through the Innovative and Novel Computational Impact on Theory and Experiment program, known as INCITE, which I manage on behalf of the two centers. We take the broadest possible interpretation of “science and engineering” and welcome applications from science communities ranging from biomedical research to astrophysics. Biologists, chemists, materials scientists, climate researchers, physicists and so on have found a receptive environment in INCITE, if they can demonstrate a true need for and ability to effectively use large portions of the leadership systems.

INCITE issues an annual call for proposals to which US- and non-US-based researchers can apply. Since the INCITE program awards time without reference to funding source (federal agency, state, private, etc) the review process is more extensive: we assess the potential impact of the project to the broader scientific and technical community. In fact, the INCITE review process is twofold. A panel of international reviewers in a particular domain assesses the impact of the work, and experts at the Leadership Computing Facility centers determine the ability of proposed INCITE projects to effectively use the resources being requested. Only those proposals determined by the peer-review panels to be of the greatest potential impact are considered for awards. From this list we then select those that successfully show that they can run well on the leadership systems. By “run well” I mean they can use a large fraction of the machine (we typically say twenty percent or more) for single-production or ensemble runs with a high degree of efficiency.

Sixty-one INCITE projects were awarded time for calendar year 2013, with per-project awards ranging from tens to hundreds of millions of core-hours. These projects are carrying out fundamental and applied research that is considered to be nonproprietary and no user fees are charged. When taking into consideration the cost to operate the center and to field these petascale systems, the effective value of any single INCITE award is on the order of one to several million dollars. The value to the center, the sponsors, and ultimately the researchers, is scientific discovery on a scale previously unimagined.

As far reaching as the scientific and engineering domains welcomed into the INCITE program, are their accomplishments. In 2007, modeling the basis of Parkinson’s disease was named the #1 computational accomplishment in DOE’s Breakthroughs. In 2008 and 2009, Nature and Science featured an INCITE project yielding the largest simulation of a galaxy’s worth of dark matter, which showed for the first time the fractal-like appearance of dark matter substructure. The world’s first continuous simulation of 21,000 years of Earth’s climate history in Science, 2009, was followed in 2010 by an unprecedented simulation of a magnitude-8 earthquake over 125 square miles. More recently, physicists examining the basic building blocks of atoms have calculated the number of bound nuclei in nature, as featured in Nature, 2012. Also last year, the US National Institute of Standards and Technology used the results of INCITE simulations on the flow of concrete, a multimillion dollar industry, to propose new standard reference materials.

From ten years ago to now, leadership systems and leadership science have come to the fore. Our vision is to couple high-end computing with high-impact science to enable scientific discovery by teams of researchers around the world. To learn more about the INCITE program, see Although the call for proposals awarding time for 2014 is closed, we encourage you to contact us and learn more about the leadership systems and access to prepare for the next call for proposals, which opens in April 2014.

About the Author:

Julia C. White, Ph.D. is the Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program manager. INCITE is a peer-review allocation program to award time on the US Department of Energy’s leadership-class supercomputers at Argonne and Oak Ridge National Laboratories. INCITE enables researchers around the country to carry out unprecedented scientific and engineering simulations. Annual total awards of time are roughly the equivalent of $70 million. White provides leadership and oversight of INCITE from the call for proposals through peer-review and final awards. She previously held management roles at Oak Ridge and Pacific Northwest National Laboratories and at Physical Review B, an international journal specializing in condensed-matter phenomena and materials physics. White holds a Ph.D. in chemistry from the Indiana University–Bloomington and an MBA from the University of Tennessee-Knoxville.


About The Blog

The HPC Blog is a dynamic online resource presenting the latest news and analysis on High-Performance Computing (HPC). By testing a new concept or product in virtual space, HPC modeling and simulation can dramatically reduce the time and physical effort necessary to bring a product to market. It shortens the development window, giving American companies an edge in an increasingly competitive global marketplace.