Enabling Technologies for Peta(FL)OPS Computing Thomas Sterling Universities Space Research Association Paul Messina California Institute of Technology Paul H. Smith National Aeronautics and Space Administration HQ i Abstract The Workshop on Enabling Technologies for Peta(FL)OPS Computing was held on February 22 through 24, 1994 at the DoubleTree Hotel in Pasadena, California. More than 60 experts in all aspects of high- performance computing technology met to establish the basis for con- sidering future research initiatives that will lead to the development, production, and application of PetaFLOPS scaled computing systems. Theobjectives oftheworkshopwere to: 1)Identifyapplicationsthatre- quire PetaFLOPS performance and determine their resource demands, 2) Determine the scope of the technical challenge to achieving e(cid:11)ective PetaFLOPS computing, 3) Identify critical enabling technologies that lead to PetaFLOPS computing capability, 4) Establish key research is- sues, and 5) Recommendelements of a near-term research agenda. The workshop focused on four major and inter-related topic areas: ApplicationsandAlgorithms,Device Technology,Architecture andSys- tems, and Software Technology. The workshop participants engaged in focused sessions of small groups and plenary sessions for cross-cutting discussions. The (cid:12)ndings produced re(cid:13)ect the potential opportunities and the daunting challenges that confront designers and users of future PetaFLOPS computing systems. A PetaFLOPS computing system will be feasible in two decades and will be important, perhaps even criti- cal, to key applications at that time. This prediction is based, in part, on the key assumptionthat there willbe acontinuationthroughout the twenty-yearperiodofthecurrentsemiconductorindustryadvancesboth in speed enhancement and in cost reduction through improved fabrica- tion processes. While no paradigmshift is required in systems architec- ture, active latency managementwill be essential requiring a very high degree of (cid:12)ne-grain parallelism and the mechanisms to exploit it. A mix of technologies will be required including semi-conductor for main memory, optics for inter-processor (and perhaps inter-chip) communi- cations and secondary storage, and possibly cryogenics (e.g., Josephson Junction) for very high clock rate and very low power processor logic. E(cid:11)ectiveness and applicabilitywill rest on dramatic per device cost re- ductionandinnovativeapproachestosystemsoftwareandprogramming methodologies. Near-term studies are required to re(cid:12)ne these (cid:12)ndings through more detailed examinationof system requirements and technology extrapola- ii tion. Thisreportdocumentstheissuesand(cid:12)ndingsofthe1994Pasadena PetaFLOPS workshop and makes speci(cid:12)c recommendations for near- term research initiatives. iii Acknowledgments The editors of this publicationwish to thank allthose who participated intheworkshopformakingitanhistoricaleventintheevolutionofhigh- performance computing. In addition, the editors wish to acknowledge theimportantcontributionsmadebyseveralassociateswhowererespon- sible for the excellent workshop arrangements and the high professional qualityofthis publication. MichaelMacDonaldprovidedtechnical edit- ing,reviewingallaspectsofthisreportandcontributingsubstantivelyto anumberofitssections. TerriCanzianprovidedexhaustiveanddetailed editing of the entire text and is responsible for the document’s profes- sionalformatandtypesetting. TinaPauna’spainstakingeditingweeded out countless awkward phrases and glitches. Tim Brice is credited for the success of the local arrangements and excellent logistical support throughout the workshop. Michele O’Connell provided important as- sistance to the workshop organizers prior to, during, and following the workshop and was responsible for coordination between the organizing committee, program committee, and local arrangements. Mary Goro(cid:11), ErlaSolomon,andChipChapmanassisted withregistration,computers and copying equipment, and in handling the many details that arise in the course of a dynamicworkshop. Executive Summary A PetaFLOPS is a measure of computer performance equal to a mil- lion billion operations (or (cid:13)oating point operations) per second. It is comparable to more than ten times all the networked computing ca- pability in America and is ten thousand times faster than the world’s most powerful massively parallel computer. A PetaFLOPS computer is so far beyond anything within contemporary experience that its ar- chitecture, technology, and programmingmethods may require entirely new paradigmsinorder toachievee(cid:11)ective use ofcomputingsystemsat this scale. For the U.S. to retain leadership in high-performance com- puting development and application in the future, planning and even early research into PetaFLOPS system design and methodologies may be essential now. To start these processes a number of Federal agencies combined to sponsor the (cid:12)rst majorconference in this emergingarea. The Workshop on Enabling Technologies for Peta(FL)OPS Comput- ingwashostedbytheJetPropulsionLaboratoryinPasadena,California iv fromFebruary22through24,1994andincludedover60invitedcontrib- utors fromindustry, academia,and government. They met to establish the basisforconsideringfuture research initiativesthatwillleadtoU.S. preeminence in developing, producing, and applying PetaFLOPS-scale computingsystems. The broad goal of the Workshop on Enabling Technologies for Peta(FL)OPS Computingwastoconduct andproduce the (cid:12)rst compre- hensive assessment of the (cid:12)eld of PetaFLOPS computing systems and to establish a baseline of understanding of its opportunities, challenges, and critical elements with the intent of setting near-term research di- rections to reduce uncertainty and enhance our knowledge of this (cid:12)eld. The major objectives of the workshop were to Identify Applications ofeconomic,scienti(cid:12)c,andsocietalimportance requiring PetaFLOPS scale computing. Determine Challenge in terms of technical barriers to achieving ef- fective PetaFLOPS computing systems. Reveal Enabling Technologies that may be critical to the imple- mentationofPetaFLOPScomputersanddeterminetheirrespectiveroles in contributing to this objective. Derive Research Issues that de(cid:12)ne the boundary between today’s state-of-the-art understanding and the critical advanced concepts toto- morrow’sPetaFLOPS computingsystems. Set Research Agenda for initialnear-term work focused on immedi- ate questions contributing to the uncertainty of our understanding and imposingthe greatest risk to launching a major long-term research ini- tiative. Theworkshopwas sponsored jointlybytheNationalAeronautics and Space Agency, the Department of Energy, the National Science Foun- dation, the Advanced Research Projects Agency, the National Security Agency, and the Ballistic Missile Defense Organization. Invited partic- ipants were selected to ensure the highest quality and coverage of the driving technical areas as well as representation from all elements of the high-performance computing community. The direction and nature of the workshop were set by opening talks presented by Seymour Cray and Konstantin Likharev. The workshop was organized into four work- ing groups re(cid:13)ecting the pace-setting disciplines that both enable and v limitprogress toward practical PetaFLOPS computing systems. These working groups were Applications and Algorithms (cid:15) Device Technology (cid:15) Parallel Architectures and System Structures (cid:15) System Software and Tools. (cid:15) The Applications Working Group considered the classes of applica- tions and algorithms that were both important to national needs and capableofexploitingthisscaleofprocessing. Throughthese discussions, some understanding of the resource requirements for such applications wasderived. TheDevice TechnologyWorkingGroupexplored the three most likely technologies to contribute to achieving PetaFLOPS perfor- mance: semiconductor, optics, and cryogenic superconducting. This groupestablishedprojectionsofthecapabilitiesforeachtechnologyfam- ilyanddistinguishedthemintermsoftheirstrengths andweaknesses in supporting PetaFLOPS computing. The Architecture Working Group examined three alternative structures comprising processor, communi- cation, and memorysubunits enabled by future technologies and scaled to PetaFLOPS performance. They investigated the most likely organi- zationsandmixesoffunctionalelementsatdi(cid:11)erent levelsoftechnology capabilityto reveal aspectrum of possible systems. The SoftwareTech- nology Working Group took on the challenging task of delineating the principal obstacles imposed by current software environments to e(cid:11)ec- tive application of future PetaFLOPS computing systems. They also examinedtheimplicationsofalternativeenvironmentsandfunctionality that mightsubstantively contribute to enhanced usefulness. This (cid:12)rst comprehensive review of the emerging (cid:12)eld of PetaFLOPS computing systems produced a number of important (cid:12)ndings that broadly de(cid:12)ne the challenge, opportunities, and approach to realizing this ambitions goal. These were as much derived from interactions amongthe workinggroups as comingfromdeliberations withinanysin- gle group. The following re(cid:13)ect the major (cid:12)ndings of the workshop combiningkey contributions fromall four of the workinggroups: 1. Construction of an e(cid:11)ective PetaFLOPS computing system will be fea- sible in approximately20 years, based on current technology trend pro- vi jections. 2. Thereareandwillbeawiderangeofapplicationsinscience,engineering, economics,andsocietalinformationinfrastructureandmanagementthat willdemand PetaFLOPS capability in the near future. 3. Cost, more than any other single aspect of a PetaFLOPS initiative, will dominate the ultimate viability and the time frame in which such systems will come into practical use. 4. Reliability of PetaFLOPS computer systems will be manageable but only because cost considerations will preclude systems having a much greaternumberofcomponentsthancurrentmassivelyparallelprocessing systems. 5. No fundamental paradigm shift in system architecture is required to achieve PetaFLOPS capable systems. Advanced variations on the NUMA MIMD (and possibly SIMD) architecture model should su(cid:14)ce, although speci(cid:12)c details mayvary signi(cid:12)cantly from today’s implemen- tations. 6. It is likely that a PetaFLOPS computer will exhibit a wide diameter, i.e., the propagation delay across a system measured in system clock cycles. Latency management techniques and very high concurrency on the order of a million-foldwill be key facets of systems of this scale. 7. The PetaFLOPS computer will be dominated by its memory. But, at leastforscienceandengineeringapplications,memorycapacitywillscale less than linearly with performance. A system capable of PetaFLOPS performance willrequire on the order of 30 terabytes of mainmemory. 8. To achieve PetaFLOPS performance, such computers will comprise a mixoftechnologyprovidingbetter performancetocost thanpossibleby anysingletechnology. Semiconductortechnologywilldominatememory withsomelogic,andprogresstowardthisgoalwillbetiedtoadvancesin the semiconductor industry. Optics will provide high bandwidth, inter- module communication at all levels and mass storage but little or no logic. Superconducting Josephson Junction technology may yield very high-performance logic and exceptionally low power consumption. 9. Majoradvancesinsoftwaremethodologiesforprogrammingandresource managementwillbenecessaryifsuchsystemsaretobepracticalforend- user applications. vii During the course of deliberations amongthe workshop participants, manyissues were brought to light, clarifyingthe space of opportunities and obstacles but leaving many questions unanswered. For example, assumptions about semiconductor technology in 20 years were derived fromSIAprojectionstotheyear2007andrequiredextrapolationbeyond thatpoint. Theeconomicsofspecialtyhardwarewasquestioned,leaving unresolved the degree towhichanyfuture PetaFLOPS computerdesign must rely on commodity parts developed for more general commercial application. The nature oftheuser base forPetaFLOPS computerswas highlycontested. Thepossibilitiesincludedclassicalscience/engineering problems, total immersion virtual reality human interfacing, and mas- sive information management and retrieval. The di(cid:14)culty of program- ming even today’s massively parallel processing systems left open the possibility that signi(cid:12)cant resources would be committed to achieving ease-of-use at the cost of sustained performance. But how such systems would ultimately be programmed is uncertain. The narrow scope of architectures examined was still very broad with respect to the tech- nology issues they posed. Although for each of the three architectures latency is seen as an issue driving system architecture decisions, the space of alternatives was too wide to permit a speci(cid:12)c approach to be recommended over all others. And, beyond the approaches explicitly examined, there remains the possibility of completely untried architec- tures that mightaccelerate greatly the pace to PetaFLOPS computing. These and other issues, while revealed as important at this workshop, remained unresolved at its close. Finally,the workshop concluded with key recommendationsfor near- term initiatives to reduce uncertainty and advance U.S. capability to- ward the achievement of PetaFLOPS computing. In the area of de- vice technology,itwas considered imperativethatbetter projections for semiconductor evolution be developed, and that the true potential of superconducting technology be better understood. With regards to ap- plications,speci(cid:12)c examplesidenti(cid:12)edascandidatesforPetaFLOPS ex- ecutionshouldbestudiedindepthtodeterminethebalanceofresources required atthatscale inorder tovalidatethe appropriateness ofthe pri- mary candidate architectures. Such a study should include at least one exampleofanapplicationforwhich there islittlecurrent use but which is potentially important to the future. The architecture working group covered manyfacets of PetaFLOPS architecture and produced a mean- viii ingfuloverview of a tenable PetaFLOPS computer structure, but many details had to be left unspeci(cid:12)ed. It is recommended that a near-term study be initiated to (cid:12)ll in the gaps, determining the requirements of the constituent elements of such a future machine. These speci(cid:12)cations are essential for validatingthe approach and determining requirements for allof the technologies used in its implementation. In conclusion, the Workshop on Enabling Technologies for Peta(FL)OPSComputingwasanhistoricmeetingthatbroughttogether a remarkable set of experts in the (cid:12)eld of high-performance computing and focused their talents on a question of great future importance to ourNation’sstrength inscience andengineering,as wellas itseconomic leadership in the world of the next century. Ideas, both conservative and controversial,were explored and the workshop resulted inan initial set of (cid:12)ndings that will set the course toward the ultimateachievement of a PetaFLOPS computer. But, an important immediate consequence ofthisworkshopbeyond the greater understanding ofPetaFLOPS com- puting systems achieved was the extraordinary synergism and cross fer- tilizationofideasthatoccurred amongsomeofthisNation’smajorcon- tributors to computer science. Contents Abstract i Acknowledgments iii Executive Summary iii PART I x 1 Introduction 1 1.1 Overview 1 1.2 Objectives 3 1.2.1Identify the Applications 3 1.2.2Determine the Challenge 4 1.2.3Reveal the Enabling Technologies 4 1.2.4Derive the Research Issues 4 1.2.5Set the Research Agenda 4 1.3 Approach 4 1.4 Background 7 1.5 Issues 10 1.6 Report Organization 12 2 PetaFLOPS from Two Perspectives 15 PART II 27 3 Summary of Working Group Reports 29 3.1 Applications 29 3.2 Device Technology 32 3.2.1Semiconductor Technologies 32 3.2.2Optical Technologies 34 3.2.3Superconducting Technologies 36 3.3 Architecture 37