The Route Less Taken: The Homegrown Los Alamos Integrated Computer Network

. Between the 1970s and 1990s, Los Alamos National Laboratory built and utilized a largely custom computer network for the Lab’s supercomputers. Designed to support the unusual performance, storage, and security requirements of an American weapons lab, the Los Alamos Integrated Computer Network, as the focus of historical study, complicates and enriches the history of computer networking development, exploring the approaches and contributions to computer networking of an institution outside the better-known worlds of industry, academia, and the military. For example, the Lab's reticence to adopt TCP/IP due to performance and security concerns further complicates the narrative of the ARPANET/Internet protocol suite's adoption among advanced networking sites in the 1980s and 90s.


Introduction
Since the Second World War, the Laboratory at Los Alamos has engaged in advanced scientific computing as part of its primary mission as a nuclear weapons lab, and increasingly for other fields of computing-intensive scientific research, such as climate modelling. Because of its unusual computational demands, Los Alamos, during and after the Cold War, often sought levels of computing performance, and hardware and software capabilities, before high-tech vendors were capable or interested in meeting those requirements. As a consequence, Los Alamos had a history of developing custom solutions to computing-related problems, particularly where new technologies were not yet available on the market, or existing systems lacked key features or capabilities Lab supercomputer users desired. Homegrown solutions were costlier in terms of development and maintenance, but were typically well-tailored to the Lab's technical and security requirements. The custom approach also provided in-house knowledge and expertise for technologies important to the

Early Lab Computing
Founded in 1943 as the primary R&D site of the Manhattan Project, the Laboratory at Los Alamos emerged from World War II as a major center of scientific computing in the United States. Using, building, and buying some of the earliest digital computers, 1 For an overview of Los Alamos and its computing history, see Donald MacKenzie, "The Influence of the Los Alamos and Livermore National Laboratories on the Development of Supercomputing," Annals of the History of Computing 13, No. 2 (1991). 2  Los Alamos played a role in the maturation of computer hardware and software in the first decades of the Cold War, going back to its use of ENIAC on that computer's very first calculation. As part of its collaboration with IBM on the design of the IBM 7030 "Stretch," Los Alamos helped to create and define the concept of supercomputing. Research and development on operating systems, programming languages, and compilers also accompanied the Lab's work on hardware, creating the software tools which vendors like IBM, Control Data Corporation, and Cray Research either could not, or would not, provide for the small (but growing) niche of highperformance scientific computing. 3 Beginning in the early 1950s, when digital computing was still in its infancy, Los Alamos actively engaged with the broader, though still small, scientific computing community, exchanging correspondence and software with sites like the RAND Corporation, and with other researchers, taking part in conferences and user groups, like IBM SHARE, of which Los Alamos was a founding member. The purpose was to share in the community's pool of knowledge and experience concerning a new technological field. These exchanges continued and became more elaborate and important over time, with Los Alamos directly funding R&D at universities and among vendors, including IBM, RCA, and Burroughs, on computing technologies, such as high-speed storage and memory, which weren't feasible or cost-effective to research at the Lab. Participating in the scientific computing community and encouraging R&D activities outside the Lab were considered to be investments that would reduce the need for costlier in-house development. However, in-house development was a common practice in Los Alamos computing, particularly along new technological branches, like high-performance and high-security networking in the 1970s. 4 4

C Division and Custom Lab Networking
In 1968, Los Alamos formed its first dedicated Computing or "C" Division, its personnel drawn from the three computing-related groups within the Theoretical Physics or "T" Division. T Division had led the original Los Alamos hydrogen-bomb project, and was still the most computing-intensive division in the 1960s, occupying 40% of the Lab's available computing time. Transferring oversight of the Lab's supercomputers to a dedicated computing division was intended to unify the Lab's computing efforts, as multiple science divisions were purchasing their own machines, and were pursuing differing technical paths from one another. A dedicated computing division was also intended to mitigate the potential conflict of interest found with the most computer-dependent division allocating access to all other divisions. 5 Between the mid-1960s and the formation of the new C Division, T Division faced a rapidly growing demand from users on Lab computing resources. The actual computing power at the Lab's Central Computing Facility (CCF) quadrupled every two years between 1966 and 1974, but this increase still did not meet the user demand. With the mission-critical nature of Lab computing resources, particularly amid the escalating demands from users, T Division was reluctant to adopt new technologies that had the potential to disrupt vital services or security, at least until those technologies had undergone extensive testing before entering widespread use at the Lab. Computer users from other divisions who were dissatisfied with T Division's conservative approach did not find significant change with the formation of C  6 The increasing user demand upon computer resources at Los Alamos between the mid-1960s and mid-70s was due to a variety of factors: First, the Partial Test-Ban Treaty of 1963 moved all nuclear testing underground, which made weapons tests more expensive, and increased the reliance upon computer simulation. Second, after having diversified in their research, the US weapons labs experienced a reducing budget year-over-year beginning in 1960, spurring Los Alamos to transfer large numbers of personnel back toward its core mission, which increased the number of heavy computer users, despite the Lab population remaining stable. Third, the weapons themselves became more sophisticated and design tolerances became tighter starting in the mid-1960s, as US nuclear strategic policy shifted toward a counter-force (targeting military assets) strategy. A mix of more complex weapons For example, Lazarus resisted the development or adoption of time-sharing operating systems at Los Alamos, believing that time sharing, in its early stages in the late 1960s and early 1970s, posed security risks, and wasted clock cycles at a time when the Lab still could not keep up with the growing demand for computing power. Computer users from heavily computer-dependent divisions, such as P (Physics), MP (Material Physics), and W (Weapons), found this approach to time sharing to be too cautious, and successfully petitioned for Lazarus to be replaced in 1973. His successor, Frank McGirt, quickly entered into collaboration with the Lawrence Livermore Lab to utilize its LTSS (Livermore Time-Sharing System) operating system on the CDC (Control Data Corporation) 7600 supercomputers at Los Alamos. Livermore's Computation Division, under Sidney Fernbach from 1955 to 1982, typically approached new technical solutions more readily than did Los Alamos' C Division under Lazarus. Curiously, Fernbach was less open than Los Alamos to the potential of smaller computer systems in the 1970s, as he remained focused on large machines. He usually delegated responsibility for smaller systems to others in the Computation Division. 7 Cautious toward the new arena of computer networking, Roger Lazarus oversaw the construction of an experimental remote-access system initiated under T Division. Called "MUX," the Multiple User eXperiment was meant to determine the utility and viability of remote computer connections at Los Alamos. MUX provided up to 75 users the ability to run batch jobs on one of the Lab's CDC 6600 supercomputers via remote terminals at 300bits/s. Its specifications completed in 1967, and the system entering use in 1968, MUX consisted of about $50,000 worth of hand-built hardware and custom application software, using 8,000 integrated circuits and two miles of twisted-pair cable. The custom MUX application software resided on the 6600 and designs and more complex codes used for development and simulation also markedly increased the processing and storage demands of the heaviest users of Lab computing resources between the 1960s and 70s. Part of T and then C Division's conservatism with new technologies was the need to prevent disruptions in the already overbooked CCF's production schedules. 7  Fernbach stepped down as the Livermore Computation Division leader in 1982, with Fernbach arguing that he had been removed. Fernbach at Livermore had heavily backed the development of the Control Data STAR-100 supercomputer in the early 1970s, but the poor performance of that system (of which Livermore purchased two, while Los Alamos backed out of its contract to purchase a single STAR) had consequences for Fernbach's position, and for the relative freedom of the two labs in their selection of new computing systems. one of its Peripheral Processing Units (PPU), which handled I/O operations for the CPU. The application placed remotely submitted jobs into the 6600's job queue, then returned completed jobs to users via the terminal interface built by the MUX team. MUX was not a time-sharing system, as users could not interact with their jobs as they ran on the 6600, but it offered a platform for introducing and testing timesharing-like capabilities in the Lab setting. Ron Christman, one of MUX's three original developers, recalled that the system was reliable to the point that, when it did fail on rare occasions, its maintenance staff had to "re-learn" how MUX worked in order to repair it. The experience with MUX, as a tentative first step into networking technology, characterized C Division's cautious approach to networking development between the 1960s and 70s. 8 MUX was enormously popular with users, and remained in use for over a decade. While quite limited in its capabilities, MUX indicated to Lazarus and C Division that a demand for computer networking existed at Los Alamos, and that the technology could be implemented securely and reliably in the Lab setting. While Lab supercomputer users were, indeed, enthusiastic over the potential of larger-scale networking at Los Alamos, MUX itself did not fully prepare C Division, despite its usual caution, for the technical challenges of building a computer network. Even with inside knowledge of a similar networking effort at another American weapons lab, Los Alamos wrestled with many of the same problems facing other early computer networking sites, but with the added complications introduced with the niche requirements of supercomputing at a secure facility. 9

Hydra
In 1970, with the success of MUX, Lazarus authorized development of the Lab's first computer network. The network came online in 1974, one year later than planned. Ron Christman, also a designer of the new network, indicated that the delay was largely due to an overly ambitious early design, and insufficient coordination within the development team for such a complex system. The relative simplicity and small scale of MUX did not prepare the C Division networking team for the challenges of interconnecting multiple supercomputers in the Central Computing Facility (CCF). The resulting network was, by necessity, greatly scaled-back from the original plan, with proposed features like graphical interface capabilities stripped from the final design. Called "Hydra", the network initially connected each of the Lab's four CDC 7600 supercomputers to a shared pool of short-term, long-term, and archival storage systems through a central front-end machine, itself a Control Data 6600 7 supercomputer. The front-end machine, doing no production work of its own, also managed remote terminal connections to the four 7600s. Although the original plans called for a new computer to serve as the front-machine, C Division opted to convert one of its production 6600s for use on Hydra due to an initial lack of funds. The one 6600, itself, represented only a small portion of the Lab's overall computing capacity. Christman argued at the time of Hydra's development that the benefits of more flexible, shared storage options would outweigh the small loss in processing power. Roger Lazarus agreed. 10 Lawrence Livermore, beginning development in the mid-1960s, constructed a similar network before Hydra, called "Octopus," which used two DEC PDP-6 computers as its front-end machines. Despite the historical rivalry between Los Alamos and Lawrence Livermore, the two nuclear weapons design labs of the AEC complex, their computing divisions maintained (and still maintain) collaborative relationships on the development, purchase, and usage of HPC systems, as both labs had similar computing requirements and faced similar technical problems. However, particularly between the 1950s and 1980s, the two often followed somewhat different technological paths, as in the case of networking development, due to differences in funding, lab-specific goals, particular demands from users, and styles of leadership. Livermore was founded in the 1950s largely to stimulate a competitive environment with Los Alamos, with either lab pursuing similar lines of R&D, but from slightly different philosophical angles. These tendencies were reflected in their approaches to computing. Los Alamos C-Division staff remained apprised of the problems with Livermore's Octopus through publications and inter-lab technical meetings, and attempted to avoid similar issues with Hydra. Hydra had fewer initial performance and reliability issues than did Octopus, partly due to the higher performance of the CDC 6600 as the Hydra front-end machine, and the lessons learned from Livermore's earlier foray into centralized-storage networking. However, both custom networks had similar drawbacks for maintenance, operability, and reliability. The total dependence on a central front-end machine was the single largest drawback for network reliability. C Division prepared one of the production 6600s in the Central Computing Facility (CCF) as a backup for the Hydra front-end machine, allowing the other CDC computer to take over if the primary 6600 failed, but that solution could not completely address the inherent deficiencies in Hydra's configuration, only minimize their impact. 11 The purpose of Hydra, a "star" configuration network, was to reduce the redundancy of data storage purchased for each of the Lab's supercomputers, and to allow users to access their files on any of the machines in the CCF, regardless of 10  8 whichever machine they had used previously. Aside from the Control Data channel couplers, the point-to-point interconnects which C Division had repurposed to link each of the supercomputers to the Hydra front-end machine at 12Mbits/s, the network was entirely custom to Los Alamos, including the Hydra OS, which replaced the CDC Scope operating system on the 6600 front-end machine. After the MUX development team found the Scope OS to be unreliable and difficult to adapt to the needs of the project, C Division opted to pursue its custom OS for Hydra, despite the greater development costs. Data storage on Hydra consisted of CDC disk units for short-and long-term storage, and the IBM 1360 "Photostore", a massive archival system that stored data on photographic film chips. The Photostore, one of only six IBM constructed, provided about one-terabit of storage in total. 12

The Homegrown Tradition
Retired Lab networking experts Don Tolmie and John Morrison recalled that C Division opted to develop its custom networking and storage system because the commercial market for supercomputer networking did not yet exist in the early 1970s. Los Alamos and Livermore were among the few civilian facilities at that time pursuing the kinds of data speeds, volumes, and security needed for interconnecting supercomputers for classified work. This was not an unusual situation at the Laboratory. In the early 1950s, the digital computing industry was still in its infancy, with very few computers in existence worldwide. In addition to its modified IBM accounting machinery and a Hand Calculation Group of human computers, Los Alamos had spent the immediate post-war years utilizing whatever available computer resources existed outside the Lab, such as the ENIAC in both Pennsylvania and Maryland, and IBM's electromechanical SSEC machine in New York. The most demanding of these calculations were usually in support of the hydrogen bomb project. However, traveling great distances for computer access, particularly to machines that were not typically well suited to the types of calculations Los Alamos performed in the early Cold War, was not a practical solution. 13 In the 1950s, Los Alamos cultivated a native capacity for developing and maintaining its own computing solutions. For example, the MANIACs I and II, from 1952 and 1957, respectively, were first-generation digital computers constructed at Los Alamos. These machines provided computing capacity for the Lab when vendor 12 "The Computer Science and Services Division Annual Report, January-December 1975," 5-6, 9; Christman, "A Review of the Hydra Project"; Christman, interview by Nicholas Lewis, 2014; John Morrison, interview by Nicholas Lewis, Los Alamos, NM, 7/1/2014. 13 Morrison, interview by Nicholas Lewis, 2014; Don Tolmie, interview by Nicholas Lewis, Los Alamos, NM, 7/14/2014; The first ever calculation on the ENIAC was a Los Alamos feasibility study of the hydrogen bomb. The calculation was less than ideal, because ENIAC's limited memory forced Los Alamos to simplify the variables of the problem to the point that the results were mostly inconclusive. However, Los Alamos would use ENIAC repeatedly in the 1940s, due to the scarcity of digital computers in the early Cold War years.
9 solutions were still relatively scarce. Just as important, the intimate knowledge of digital computing gleaned from developing the MANIACs also provided the Lab community with in-depth knowledge of a mission-critical technology, which continued forward as Los Alamos transitioned to the maturing array of vendor solutions that appeared from the mid-1950s through the early 60s. The Lab continued to customize vendor hardware as needed, and to create bespoke devices when no others were available. T and then C Divisions of the 1950s and 60s developed much of their own software in-house, due to the relative dearth of software for the niche scientific-computing market, and to maintain a local capacity for producing software tailored to the Lab's requirements. Much was the case in the 1970s with computer networking. C Division opted to pursue its own networking solutions to meet the Lab's unusual requirements while the supercomputer-networking industry was young and provided few ready-made options. Pursuing technically challenging custom solutions, according to the C Division leadership of the 1970s, also fostered local networking expertise, and aided the division in attracting new talent. 14

The Integrated Computer Network
C Division began to modify the centralized Hydra network soon after it first entered service, replacing the 6600 with a CDC Cyber 73 as the front-end machine in 1975, when the funding became available. C Division also took advantage of the growing range of minicomputers available on the market to augment network services. Minicomputers became the Swiss Army knives of Lab networking, with C Division customizing UNIX-powered DEC PDP-11s, and later DEC VAX, SEL, and Gould minicomputers, to serve in a variety of roles. For example, in order to connect remote users to Hydra, Los Alamos developed the KCC (Keyboard Communications Concentrator), a custom device based on a PDP-11 and DEC interface unit to provide a single connection to the Hydra front-end machine for up to 96 keyboard-based terminals. The CCC (Computer Communications Concentrator) provided a similar connection for up to 18 computer-based terminals, which were usually other DEC minicomputers. 15 In 1975, when the number of keyboard-based terminals in use exceeded the capacity of a single KCC, C Division developed the PDP-11-based Synchronous Concentrator (SYNC). The SYNC served as a concentrator and message switcher to interconnect the KCCs with the Hydra front-end machine. More importantly, the SYNC provided direct terminal connections with each of the networked supercomputers in the CCF. Providing direct terminal access to the "worker" supercomputers reduced the centralization of the Hydra system, which C Division increasingly recognized as an unreliable network model. The CDC channel couplers proved to be the single most unreliable component of Hydra, as they were never intended to connect more than two CDC computers together. Troubleshooting a connection error required maintenance staff to take both the originating machine and the Hydra front-end computer offline. 16 As frustrating as Hydra's shortcomings were to users and to C Division, including its problematic interconnect technologies, a front-end machine that CDC had not intended to be a storage or terminal-access controller, and an interface that was unfriendly to users, they offered a hands-on learning experience with the complexities of networking in the Lab environment, just as the Lab's home-built computers had with digital computing in the 1950s. Having a clearer understanding of what did and did not work well at Los Alamos, and in supercomputer networking in general, C Division's networking team developed a network plan for the remainder of the 1970s. The plan emphasized a reduced dependency on Hydra and the eventual phasing-out of the Hydra storage system completely. The plan also called for an increase in network speeds to support supercomputers with faster I/O capabilities, such as the new Cray-1, and growing the network to accommodate a variety of new services and users. The plan meant adding new transport, switching, and security systems, all of which would be custom to Los Alamos, primarily due to the relative infancy of the highperformance computer networking industry as the 1970s came to a close. 17 One of the first products of C Division's new networking plan was the Los Alamos File Transport machine, introduced in 1977. While the SYNCs provided complete terminal access to the supercomputers of the CCF independent of Hydra, Los Alamos lacked similar direct connectivity for file access. The File Transport (FT) machine used the same architecture and low-level protocols as the SYNC to provide high-speed data links of up to 40Mbit/s between the CCF supercomputers. The first FT linked together the Cray-1 and two CDC 7600s. Additional FT machines formed a File-Transport Network (FTN), creating a high-speed backbone for direct file transfers, which reduced the need for routing data through the Hydra front-end machine. The FTN was also important for serving the rapidly growing number of high-speed graphics terminals at the Lab. The new terminals, once connected to the FTN, allowed users to stream color graphical data at 150kbits/s from the supercomputers of the Central Computing Facility directly to their offices. 16  Conventional terminals operated at only about 9.6kbits/s. Improving support for a growing range of small computing systems would become an important theme in Los Alamos network development over the following decade. While the Lab network would grow far beyond its original size and complexity as the 1980s began, reorienting to accommodate small-scale computing, in addition to adapting to more powerful and data-intensive supercomputers in the CCF, would pose a significant challenge to C Division, as it often meant rethinking how and why computer networks were used at Los Alamos. Those challenges were only emerging during the ICN's first great expansion, taking place in the late 1970s. 18

ICN Security in the Late 1970s
Increasing the range of services and users on the new Integrated Computer Network in the late 1970s required a significant reconfiguration of network security. The original Hydra network had a simple approach to the complex issue of security at a weapons laboratory; only classified users were allowed on the network. In order to accommodate unclassified supercomputer users, and to facilitate administrative Lab functions across the network, C Division developed a custom, minicomputer-based Network Security Controller (NSC) to monitor and restrict file and resource access based upon user passwords. The NSCs separated the ICN into three partitions: Secure, Administrative, and Open. Several of the supercomputers in the CCF were moved to the Open partition for unclassified use, while the Administrative partition users had dedicated CDC Cyber 73 mainframes. Users of the Administrative or Open partitions could share files with users on the Secure partition, but not the other way around, in order to prevent a "write down" of classified data to a lower classification level. Each data packet ran through two NSC checks, providing redundancy to guard against error. The NSCs did not run any user code, and prevented the need to rely upon worker machines to enforce security rules. Using the NSC to monitor and control network security on the single physical network was far less costly than developing separate networks for each security level, allowing classified and unclassified users to share the same resources. 19

The Los Alamos High-Speed Parallel Interface
The ICN jumped from about 2,200 mostly local users in 1977 to over 6,000 by the mid-1980s, with an additional 2,000 users remote to Los Alamos. As networked resources became more widely used and more mission-critical to the Laboratory, the demands of connecting larger numbers of machines and services to the ICN grew more difficult and time consuming. The various supercomputers, minicomputers, 18  storage systems, printing services, and other devices employed at the Lab were never intended to function together on a single network. A variety of incompatible interconnect technologies and proprietary communication protocols meant that C Division had to develop expensive hardware and software conversion systems that could translate the native I/O of one machine into that of every other device with which it needed to connect on the network. This engineering task grew increasingly impractical as the network became larger and more heterogeneous as the 1970s drew to a close. C Division's solution was one of the first efforts toward network interface standardization in the supercomputing arena. 20 Beginning development in 1978, the Los Alamos High-Speed Parallel Interface (HSPI) converted the native I/O of a computer or another networked device into the standard ICN higher-level protocol suite, which was also developed at Los Alamos. The suite primarily consisted of SIMP (Simple Inter-Machine Protocol), PTP (process-to-process), and AFT (File Transport), which would remain in use for well over a decade, much like the HSPI. The custom Lab protocols were designed to accommodate large data blocks with low latency, high burst rates, extensive error detection and correction, and minimal message passing. C Division had to design and construct a HSPI, including the hardware and drivers, for each new device placed on the network, but each HSPI only had to convert the native I/O interface and protocols of the new device into the standard ICN protocols. The HSPI of each receiving device then converted the ICN protocols back into that device's native I/O. Once connected to its HSPI, a device had full access to the entire Los Alamos network, which significantly reduced the time required to integrate new equipment onto the ICN. 21 HSPIs provided full-duplex point-to-point transfers to the various machines of the CCF, with error correction integrated into the HSPI hardware. The HSPI supported data transfers of up to 60Mbits/s, which was intended to exceed the I/O capabilities of the fastest devices on the network, so that the network interfaces would not act as performance bottlenecks. In order to accommodate different data rates (for example, 3Mbits/s for the PDP-11, and 50Mbits/s for the Cray-1), a handshake between HSPIs determined the maximum data rates of the exchanging machines. The HSPI employed hardware error detection and correction while data was on the fly, so as not to slow the data transfer. When double-bit errors were detected, the HSPI hardware aborted the transfer. The first completed HSPI was for the ubiquitous PDP-11, with others following in short succession in the 1970s and 80s. 22

CFS and the ICN of the 1980s
The late 1970s' introduction of higher networking speeds, new security partitions, and standardized interfaces and protocols on the Integrated Computer Network coincided with the end of Hydra in 1979, completing the Lab's transition to a modular network configuration. Replacing Hydra as the networked storage system on the ICN was the Common File System (CFS), which represented a middle ground between vendorsourced networked storage systems and the entirely custom Hydra system. The networked-storage market had improved significantly in the years after Hydra's deployment, and C Division was able to use off-the-shelf IBM storage hardware and operating systems, rather than developing its own. IBM disc units replaced older Control Data storage devices, and the tape-cartridge-based IBM 3850 replaced the Photostore. While CFS still required a central storage controller, unlike the repurposed CDC front-end computers of Hydra, the CFS controller was designed and used specifically for storage. The CFS hardware offered a maximum data transfer rate of about 5Mbits/s, and stored about 384 gigabytes of combined online and offline data on the network as of 1980, with a growth rate of about 175 gigabytes per year. While the hardware and operating system were vendor-sourced, saving C Division considerable development time and cost, the division had to develop in-house the CFS application and interface software, as vendors in the HPC space still did not offer a complete package needed for tying together their storage devices with the complex, high-performance network environment of a national laboratory. 23 After the introduction of CFS, the C Division leader, Robert "Bo" Ewald, noted a drop in the number of tapes at the Central Computing Facility, as the storage system reduced the number of redundant, partially filled tapes in circulation. Much of the CFS development team originally worked on the Hydra project, and had feared a repeat of the design difficulties that had delayed the rollout of Hydra. As a consequence, the team leaders invested in a "structured" design process of allocating and reviewing programming work. Individual programmers on the project were assigned specific parts of the CFS coding effort, with their code being checked at least three times by other programmers for errors and efficiency before the code was added to the CFS application suite. The process itself was time-consuming, but team 14 members recalling the effort cite it as a key reason for the on-time, unusually smooth roll-out of CFS, and its long-term reliability. 24 To the astonishment and delight of the CFS development team, Los Alamos network users, who were never shy about voicing their dissatisfaction over the problems and perceived shortcomings of the Lab's network services, wrote letters of praise to CFS for its on-time delivery, reliability, and intuitive, platform-agnostic interface. The transition to CFS concluded with the Photostore, which IBM no longer supported, being placed into a read-only mode until 1980, so that users could move their remaining files before Hydra was finally shutdown. With its success at Los Alamos, General Atomics Corporation sold CFS commercially as DataTree. The CFS development team aided in creating or reconfiguring components of the CFS application suite for use in a variety of supercomputing environments. Over the next decade CFS, as DataTree, entered use at several dozen data-intensive computing sites worldwide. Los Alamos networking personnel often visited sites, including those overseas, that had purchased the DataTree system, and aid in its installation, optimization, and personnel training for that site's unique requirements. CFS, and its success at and beyond Los Alamos, during the 1980s represented a transitionary period in networked data storage, when the number of data-intensive computing sites had grown markedly since the development of Hydra in the early-to-mid 1970s. CFS also marked the important role that Los Alamos would play in networked storage development, with the Lab collaborating with partners in industry and academia on many other high-speed and high-capacity storage systems, such as HPSS and Panasas, from the 1990s to the present. 25 By 1981, the Los Alamos ICN consisted of several modules. The modules included the "worker" machines (the nine supercomputers in the CCF, which performed the bulk of the scientific computing work at the Lab), the Common File System (which allowed the supercomputers to store and share files via the network), the Terminal Network (connecting terminal users from the Secure, Administrative, and Open security partitions to the ICN), and PAGES (a service center that allowed users of the ICN to print to shared laser printers, plotters, or other graphical media). The various modules were interconnected through the File Transport Network, which provided the ICN's high-speed data transfer capability. Two additional modules catered to differing levels of production control on the network. FOCUS, a VAX-11based control interface, allowed operators to monitor and manage the Lab's Cray and CDC 7600 supercomputers via the ICN. The final module, XNET (eXtended Network Access System), marked the beginning of the Lab's incorporation of outside protocols onto portions of the ICN. Beginning in 1979, the XNET module brought "distributed processing" to Los Alamos, that is, support for user-operated minicomputers on the network, which a growing number of ICN users were requesting as the 1970s drew to a close. Text-based and graphical terminals performed no production work of their own, and simply connected remote users to the supercomputers of the CCF. VAX-11 minicomputer "distributed processors" allowed users to perform production work locally, or on a remote VAX machine, while retaining full access to the resources of the ICN. Most XNET users were local to Los Alamos, but some were located in other states. Smaller-scale computing complicated the structure and philosophy of computer networking at Los Alamos, as it did elsewhere, with small, but powerful systems performing an increasing amount of production work that normally would have been done on the Lab's larger computers. C Division invested significant time and research energies into supporting smaller computing systems in a variety of capacities on the ICN over the 1980s, while attempting to maintain the overall performance and security of the larger network. 26 The VAX distributed processors initially used the proprietary DECnet protocol to communicate with the XNET concentrators, forming a smaller network of minicomputers on a module of the larger ICN. The use of DECnet on the XNET module offered the Lab several advantages. For one, it allowed Los Alamos ICN users access to the latest tools available from vendors, such as smaller-scale computing and networking, without requiring the Lab to develop its own, costlier solutions. Los Alamos already followed a similar, but custom path with its terminal access, as it continually upgraded the terminal network module to coincide with improvements in vendor-supplied terminals over the 1970s. However, upgrading the custom system took longer, and offered no vendor support. The XNET module also allowed Los Alamos to experiment and gain experience with outside networking advancements, such as the proprietary DECnet system, while retaining its custom, mission-critical portions of the ICN. 27 In the early 1980s, Los Alamos became part of the broader networking community, establishing links in 1983 with multiple external networks, including a purchased link to GTE's Telenet, a commercial packet-switching network. Connecting with external networks brought resources to Los Alamos from beyond the Lab, and allowed easier collaboration and information access for remote users. The Telenet connection, for example, allowed Open partition users access to globally distributed computing and information resources. Telenet also offered remote dialup access to the ICN for a substantially lower cost than a dedicated phone line. 1983 also marked the Lab's first link with ARPANET, the Department of Defense's packet-switching network of universities, research institutions, and unclassified DoD traffic. C Division connected an ARPANET IMP (Interface Message Processor) to a custom VAX gateway, which translated ARPANET's TCP/IP traffic into the ICN's local protocols. For security purposes, traffic from external networks connected only to the Open partition of ICN. 28 Although TCP was being forcibly rolled-out onto ARPANET in 1983, and was spreading elsewhere due to ARPA and the Department of Defense incentivizing its adoption among outside networks and computing vendors, Los Alamos had little interest in TCP/IP in the early 1980s. The ARPANET suite had little to offer in terms of performance, security, or compatibility with systems in use on the Lab network. Indeed, maintaining a custom-networking environment offered the Lab distinct security advantages, as few outsiders were familiar with the Los Alamos network or its largely bespoke systems and protocols. However, as with changes in supercomputing technologies, the Computing Division actively studied new networking technologies that were emerging from the maturing computer networking community in the 1980s. 29 While Los Alamos remained committed to its locally developed protocol suite for mission-critical parts of the ICN, such as the worker machine and File Transport modules, C Division was open to expanding its use of external protocols where they offered greater compatibility with the growing range of networked devices entering the market. Such protocols and devices had the potential to reduce the high development and maintenance costs associated with custom solutions. In 1983, C Division began investigating the potential of the X.25 networking protocol for its distributed processor network, as DEC was slated to incorporate X.25 support into its VAX VMS operating system. As an international standard, X.25 was also promising as the Lab's preferred protocol for interfacing with commercial packet-switching networks, such as Telenet and Tymnet. By 1985, C Division had converted XNET to run the X.25 protocol over DECnet, with over 75 VAX distributed processors on XNET connecting to the ICN via three, and soon four, gateways. After extensive security testing, the Lab allowed distributed XNET processors access to the Secure network partition, bringing small-scale computing to all network users at Los Alamos. 30 The Lab's connections with the wider networked world changed rapidly in the mid-1980s. In 1984, the DoD separated MILNET from ARPANET. ARPANET retained the university and research traffic, while MILNET linked together government, Defense Department, and Department of Energy facilities, including Los Alamos. Los Alamos established direct links to three other MILNET nodes. Each node connected to MILNET via a gateway host computer. The Lab's host allowed minicomputers on DECnet to exchange e-mail, and enabled TCP/IP access to the open partition of the larger Department of Defense Network. In 1986, Los Alamos added a fourth security partition to the ICN, the National Security partition, so that Department of Defense users who lacked Department of Energy (DOE) clearances could have access to secure computing resources separate from secure DOE computing work. Los Alamos moved one of its Cray-1A supercomputers into the new security partition for the benefit of military users and Defense contractors, like the Defense Nuclear Agency (DNA), SDI, and DARPA. 31 As a site once selected for its geographical isolation, the Lab's experience with computer networking quickly brought Los Alamos into the center of collaborative information processing, distribution, and management for the Department of Energy in the 1980s. The Department of Energy's WBCN (Wide-Band Communications Network) and NWCNET (Nuclear Weapons Complex Network) projects exemplified the Lab's place in bridging remote government computing sites. Begun in 1985, the DOE initiated the two related networking projects to link together the computing resources of the national Nuclear Weapons Complex, which consisted of Los Alamos, Lawrence Livermore National Laboratory, Sandia National Laboratory, and eleven other research and development sites. WBCN was the hardware component that allowed for secure transmission of data among the sites of the nuclear weapons complex, and consisted of DEC VAX-11/750 minicomputers that served as gateway controllers at each location, along with accompanying encryption and communication hardware. 32 WBCN initially ran over terrestrial circuits, which were replaced with satellite circuits in the late 1980s. NWCNET comprised the basic tools used at each site for the storage and transmission of weapons-related data across WBCN. The first NWCNET tool was CFS, the Common File System developed at Los Alamos, which stored and managed the secure data at each location in the complex. Another tool was the software that interfaced CFS with the WCBN, while a third tool was the software that allowed the various NWCNET devices to communicate via Hyperchannel. A fourth tool allowed for secure e-mail capability between the different locations. The system first linked Los Alamos and the Rocky Flats weapons production facility in Colorado in 1986, with the other sites following soon thereafter. Many of the components and services driving the WBCN and NWCNET systems originated at Los Alamos. C Division produced a modified version of CFS that could be used at the other weapons complex locations, and created a network interface for the IBM MVS operating system (the OS platform for the CFS application suite), which allowed users and worker machines access to CFS across a Hyperchannel link. Because of its extensive networking experience, Los Alamos functioned as both the development site and the Tech Control for WBCN, testing the network's components on the Lab's ICN, and ensuring the smooth operation of the other WBCN nodes. 33 As Los Alamos became the lead development site for networking efforts inside the DOE complex, the Lab continued its onsite supercomputer-networking development, which brought Los Alamos into the world of network standards setting. By the mid-1980s, the Los Alamos HSPI, the custom network interface the Lab had employed since the late 1970s, was reaching the end of its lifespan, as its 60Mbits/s throughput could not accommodate the anticipated performance increases of the next decade. The rapid increase in traffic across the ICN, and the introduction of new supercomputers with much faster I/O capabilities, such as the Cray Y/MP, meant that Los Alamos would soon require interface speeds far beyond what the networking industry offered in the late 1980s. However, developing interfaces in-house for each device the Lab purchased was expensive, and meant the Lab had to provide its own support. C Division had little choice but to create custom interfaces when it developed the HSPI in the 1970s and early 1980s, but vendors and standards-setting bodies associated with networking had evolved considerably between that time and the late 1980s. C Division proposed a standard interface for the supercomputing industry, which would provide an alternative to proprietary vendor interconnects. 34

The "Lunatic Fringe" and the HIPPI Standard
In 1987, a Los Alamos delegation, led by C Division networking expert Don Tolmie, approached the American National Standards Institute (ANSI) Networking Task Group with a proposal for the High-Performance Parallel Interface (HIPPI), the first gigabit network interface. Although ANSI initially labelled the Los Alamos group as "the lunatic fringe," because of the extreme performance it was proposing, ANSI soon 33 Ibid. 34  formed a new task group (the first ever in the supercomputer industry) to pursue the Lab's interface. Developed at Los Alamos, the HIPPI became an ANSI standard in 1991, and an ISO (International Organization for Standardization) standard soon thereafter, marking the first standard networking interface for supercomputers and their peripherals. Much as the Lab's collaborative development of the first supercomputer with IBM signified the maturation of digital scientific computing, the standardization of the HIPPI represented a similar maturation of supercomputer networking in the 1980s. Since the 1970s, the number of institutions utilizing supercomputers, including industry and centers of scientific research, had increased significantly, as the computers themselves became more compact and reliable, and as the software tools for using them improved. Older, advanced scientific computing centers, such as Los Alamos, had endured the challenges of early computing, which included the need to write locally much of the software used on their machines. This barrier largely precluded the deployment of supercomputers where the users lacked the ability or willingness to write their own application software. 35 By the mid-1970s, institutions like NCAR, the National Center for Atmospheric Research, began to demand from vendors the inclusion of basic software packages with the supercomputers they bought, as was the case with NCAR's purchase of the first commercially available Cray-1 in 1977. Advancing supercomputing hardware and software, and the growing number and range of users, fed into a rapidly maturing scientific computing environment by the 1980s, including the maturation of ancillary technologies, such as high-speed networking. With the Los Alamos HIPPI standard in place, and with a growing range of customers to provide pressure on vendors to use the standard, supercomputers and related devices began to ship with the HIPPI interface already built-in, reducing the cost and difficulty of supercomputer networking. In the 1990s, the HIPPI, and the high-speed networking interconnects it spawned and inspired, provided part of the vital networking infrastructure that underpinned the development of cluster computing. 36 As Los Alamos participated in the development and evolution of networking beyond the Lab, its local ICN continued to evolve in the late 1980s, transitioning away from custom protocols and systems, and toward the new range of industry standard solutions entering the market. While distributed processors and external network connections provided impetus for the Lab to explore X.25, the arrival of even smaller computing platforms and an increasing reliance upon connections to external networks spurred changes to the Los Alamos ICN in the second-half of the 1980s, along with preparations for even more extensive changes in the coming decade. By 1985, Sun Microsystems scientific workstations and IBM-compatible personal computers were rapidly entering use throughout the Lab as lower-cost alternatives to 20 minicomputer-based distributed processors and computer-based terminals. C Division spent the latter-half of the 1980s exploring whether Ethernet-based LANs (Local Area Networks) of small computers could be utilized at Los Alamos and still maintain the integrity of the larger ICN. 37 Scientific workstations and PCs initially operated as stand-alone units, while C Division experimented with secure means of connecting them with the ICN's custom architecture. PCs soon entered use as terminals on the Terminal module of the ICN, but only in the unclassified partitions. Between 1985 and 1987, the Lab connected workstations and personal computers to the ICN by creating small LANs whose traffic went through the VAX gateways used to interface with ARPANET. Routing traffic through the ARPANET gateways was a temporary workaround solution, while the Lab investigated the routing and interface hardware needed to connect LANs more directly to the larger network. In 1989, a KCC team developed a more permanent solution, creating a new DEC MicroVAX-powered Keyboard Communications Concentrator that supported file traffic between workstations and the ICN. C Division likewise enabled TCP/IP compatibility on the XNET module for the distributed processor network, after DEC had added TCP/IP support to its VAX operating systems in response to DoD paid incentives. 38 Although Los Alamos had little interest in the DoD's TCP/IP protocol suite in the early 1980s, by the mid-to late-1980s, the suite had spread far beyond its original application on ARPANET. Los Alamos networking expert John Morrison later recalled the issues with TCP/IP, particularly its poor performance with large file transfers, its lack of built-in security features, and its error-detection deficiencies, which initially dissuaded the Lab from pursuing the protocol suite beyond developing gateways with ARPANET and MILNET. However, the snowball effect of TCP/IP's adoption among business, consumer, and government networks and computing systems expanded the market for TCP/IP-compatible systems and devices, and vendors followed the demand. As a consequence, in the late-1980s, C Division began to develop a strategy to convert selected modules of the ICN to TCP/IP, and migrate away from the custom ICN protocol suite, as the promise of practical vendor-supplied networking solutions outweighed the deficiencies of the DoD protocols. 39 In 1987, C Division began experimenting with building TCP/IP support into CTSS, the Livermore and Los Alamos co-developed operating system used on the Lab's Cray supercomputers. The Lab soon dropped the proposed modification of CTSS in favor of converting its supercomputers to the Cray-sourced version of Unix, 37 Yvonne Martinez, et al., "Computing Division Two-Year Operational Plan, FY 1984FY -1985 called UNICOS, which supported TCP/IP. C Division also tested an experimental Ethernet network, using TCP/IP to connect a DEC VAX to a Cray supercomputer through an IP router on the Secure partition. Ethernet's poor performance in the test gave further justification to pursue the HIPPI standard. The Lab eventually returned to Ethernet in the mid-1990s as a solution for message passing between the supercomputers and their storage systems, saving the faster HIPPI links for data transfer. 40

The ICN2
The Lab spent the late 1980s and early 1990s preparing for a gradual transition of the ICN away from custom protocols and devices as part of its "Network Modernization Project." Modernization was planned to occur in stages, proceeding from the least to the most critical modules and systems, to avoid unexpected performance or security issues. Once under way, the overall conversion was expected to take place over a decade or more, but DOE intervention necessitated a more abrupt break with the homegrown Los Alamos network. The resulting conversion to TCP/IP-compatible systems occurred as one part of a total restructuring of the ICN architecture. In 1991, the DOE Office of Safeguards and Security determined that the Lab's partitioning of the ICN into Open, Administrative, Secure/Unclassified, National Security, and Secure/Unclassified partitions via custom security controller hardware and software would not meet new DOE-wide security requirements. Safeguards and Security sought uniform network security policies and enforcement across the Department of Energy, and custom systems were difficult, if not impossible, to verify in accordance with standardized metrics. Faced with the new DOE requirement, Los Alamos planned and constructed the "ICN2" between 1991 and 1996, which involved creating a physical separation between the Secure and Open portions of the network. The Lab completed its transition to industry standard systems as part of the restructuring, replacing the custom gateways and other bespoke systems with vendor-supplied, TCP/IP-compatible components. Taking five years to complete, the transition occurred rapidly, considering the size and complexity of the Lab network, and the need to prevent disruptions of vital services. While not the gradual transition Los Alamos had anticipated, the ICN restructuring provided the compatibility with industry networking standards envisioned in the late 1980s.

New Custom Solutions
Although the Los Alamos network was no longer comprised of mostly in-house systems, custom networking solutions never completely disappeared. Awaiting the development of its replacements, CFS remained in use until 2002, using a custom gateway to translate the ICN's TCP/IP protocols for the CFS controller. As it had done before, the Lab returned to custom networking when vendor solutions were either inadequate or missing from the market altogether. Before the term "cyber security" was coined, in the late 1990s Los Alamos developed its own high-speed security monitoring hardware to sift through Web-based traffic passing through the ICN. A purely hardware-based solution that split data traffic into multiple streams made it possible to screen the data for malicious packets without slowing network traffic. Beyond security, Los Alamos broke a world speed record in 1994 when it completed a 792Mbits/s data transfer via a Lab-developed HIPPI/SONET gateway to the San Diego Supercomputer Center, a distance of over 1,200 miles. This experiment helped to push the boundaries of remote-computing performance, allowing distant centers and users to share supercomputer resources. These examples are characteristic of the Lab's continuing ability and drive to pursue custom approaches in order to explore and push boundaries, and to stimulate the research, development, and application of new capabilities at and beyond Los Alamos. 42

Conclusion
The homegrown Los Alamos Integrated Computer Network of the 1970s and 80s represented a unique path of networking development, where the technical demands of a nuclear weapons facility required the pursuit of custom solutions to problems that few other computing centers had yet to encounter. Although unique, Los Alamos was not isolated from the broader networking community. Just as it fostered intellectual and financial ties with computer vendors, researchers, and other users, the Lab played an active role in outside networking development, lending its expertise to create and disseminate technical innovations, and incorporating new technologies as they offered benefits to ICN users. The Los Alamos network was not a linear development, changing its technical and philosophical orientations in relation to varying internal and external pressures, and sometimes adopting technically inferior solutions, such as TCP/IP, for economic or bureaucratic expedience. Examining the Laboratory at Los Alamos highlights how often-overlooked institutions experienced, and contributed to, computer networking differently than the better-known cases from academia, 23 industry, and the US military, which serves to complicate and enrich the narrative of computer networking history.