Virginia Tech Magazine

Summer FEATURE 2010

Factor of X by STEVEN MACKAY

In October 2003, Virginia Tech crashed the supercomputer market with a machine that put Blacksburg on the map of technology powerhouses. Known as System X--pronounced "ten," the Roman numeral, reflecting the goal of achieving 10 teraflops, or 10 trillion operations per second--the supercomputer landed Virginia Tech among the most powerful computational research facilities in the world.

As a production supercomputer, X certainly changed how research is done at Virginia Tech. Researchers no longer have wait months for government supercomputers to complete research. With X, the wait time is nearly nil, and the payoff is huge.

Scientists in Blacksburg and from around the world have used X to crack riddles of the blood system that have befuddled researchers for decades; discover previously unknown attributes of DNA, the basic building block of all life; and helped find missing genes that could lead to improved medicines.

"System X is a unique and powerful resource that enables a very broad range of research," says Mark Paul, an associate professor of mechanical engineering at Virginia Tech who conducts research with X and serves on the committee that selects the supercomputer's projects. "It plays a pivotal role in our ability to explore open challenges in computational science and engineering."

Beyond engineering, X is serving faculty members across campus, such as geosciences and other departments in the College of Science. On average, $20 million of the research money that flows into Virginia Tech annually is tied to research supported by System X, says Kevin Shinpaugh, director of Virginia Tech's Research & Cluster Computing Center and one of X's original architects. During the 2009-10 academic years, related funding bumped up to roughly $25 million.

Wu Feng, an associate professor with the computer science and the electrical and computer engineering departments, recently used System X for a massive study to identify genes previously missed by scientists.

X solves problems dramatically faster than it would take humans to plow through on pen and paper or an isolated personal computer. It is the launching pad pushing Tech's scientific and technological leadership into the stratosphere.

It also is a brain magnet for Virginia Tech, attracting researchers who want to work for a U.S. university that has the means to assemble such a massive beast of technology. In turn, these researchers have brought more grants, prestige, and headlines to the university. "System X has been a draw for faculty hiring in many areas of the College of Engineering for several years," says Don Leo, associate dean of research and graduate studies in the college. "Computer science, of course, is a major benefactor, but X also has helped us recruit faculty in the departments of engineering science and mechanics, and mechanical engineering. Faculty whose research relies on modeling of large-scale systems need access to high-performance computing facilities."

Not bad for a supercomputer that is 7 years old and had a start-up price tag of just $5.2 million.


	GREAT EXPECTATIONS The hype surrounding System X was reaching a crescendo seven years ago when Virginia Tech's Srinidhi Varadarajan presented the keynote address at the O'Reilly Mac OS X Conference, a hotbed annual gathering in Santa Clara, Calif., of the world's leading information technology researchers. The room was packed. Everyone wanted to know about System X. Varadarajan, then-assistant professor of computer science in the College of Engineering, delivered. Back in Blacksburg, in an office building at the university's Corporate Research Center, was X, a vastly high-tech supercomputer just assembled by Virginia Tech faculty, staff, and students, comprising 1,100 Apple PowerMac G5 computers right off the assembly line. It was rumored to rank among the largest supercomputers in the world. The New York Times and scores of industry publications already were publishing stories, building up anticipation as if X were a summer blockbuster film. Varadarajan, now associate professor of computer science and director of Virginia Tech's Center for High-End Computing Systems, was not expecting this attention. The supercomputer was not even fully operational at the time of his talk, and it was unknown whether the system would work in its entirety. The System X team had ordered crates of G5s on an act of blind faith, well before the G5 models hit the market. The near month-long process of assembly and testing was painstaking. "We were running first-generation hardware. They had never been connected before like that," says Varadarajan.

Srinidhi Varadarajan, the architect of System X, has seen the supercomputer assist countless research projects.

X's first power output numbers were 800 gigaflops, roughly one-twentieth of the supercomputer's potential output. The numbers leapfrogged, crawled, and leapfrogged again. The day a New York Times reporter called, X was at 7.5 Teraflops. When the story saw print the next day, X was at 8.5. Then it jumped to 9.5 teraflops for a week while Varadarajan was at the OS X Conference. A little more tinkering, pushing and problem-solving pushed the supercomputer to a then-resounding 10.3 teraflops.

"We were racing against time to submit the benchmark timing results in time for the fall 2003 release of the TOP500 list," says Cal Ribbens, associate professor of computer science and one of many original masterminds behind X, along with Erv Blythe, vice president for information technology; Hassan Aref, then dean of the College of Engineering; and Glenda Scales, associate dean of information technology for the college; and others. "And we were dealing with leading-edge technology--both hardware and software. So the system was not very stable at first. There were a few hardware failures. There were many software glitches."

The fallout would have been great if benchmarks were not met and the system failed to coalesce. "It would have been disastrous," Varadarajan says.

But it wasn't. Varadarajan's presentation was answered with thunderous applause. "I'm not used to getting a standing ovation at the end of a talk," said Varadarajan. But he got it. Adds Ribbens: "He was treated like a rock star." A rock star in a polo shirt and Khakis.

The intensive work paid off. In fall 2003, X landed at No. 3 on the TOP500, an international tabulation that has benchmarked the world's top high-performance computing systems since 1993. The machine achieved a sustained rate of 10.28 teraflops on the standard TOP500 benchmark calculation, ranking it behind only supercomputers in Japan and at the Los Alamos National Laboratory, both of which cost more than 10 times as much as System X. With a 2004 update, System X began running at a sustained rate of 12.25 teraflops, with a peak speed of 20.24 teraflops. "We had the fastest supercomputer machine in academia," says Varadarajan.

FINDING GENES AND SOLVING UNKNOWN PUZZLES

Wu Feng, an associate professor with Virginia Tech's computer science and electrical and computer engineering departments, has used X on numerous occasions. A 2005 study, just prior to his arrival at Virginia Tech, involved Feng and several other researchers focusing on parallel genomic sequencing. The findings resulted in a best paper nomination at the 2006 International Conference on High-Performance Computing Networking, Storage and Analysis.

Feng continues research into that field, having recently participated in a massive study with the Virginia Bioinformatics Institute that used X and six smaller supercomputers to locate genes in the genomes of microbes that were previously missed by gene annotation programs created by scientists.




		Cal Ribbens, associate professor of computer science and one of the many original masterminds behind X

Using computational software developed by Feng called mpiBLAST, researchers were able to compact work that would take a sole PC 90 years to complete into half a day. The vast amount of work and newly found genes could be used to create cutting edge antibiotics and vaccines against evolving diseases such as HIV or even the flu, according to researchers.

"We gathered the sequences of 780 microbial genomes, searching these sequences against each other," Feng says of his work. "This part took roughly 12,000 processor cores about 10 to 11 hours to complete. If a DNA segment is similar to the sequence of known genes, then the segment is likely to be a coding gene with a similar function."

The work by Feng was akin to looking for a single page within a comic book series, but with an index that is missing page references to that particular page. The "missing" page is in the collection, but there is no way to readily locate it. "Instead we had to thumb through by hand and find the pages that we wanted. X, using this software program, found the genes that were known, but missing, hidden among so many like ones," Feng says.

Alexey Onufriev, associate professor in the departments of computer science in the College of Engineering and of physics in the College of Science, was one of X's earliest users and remains a strong advocate. He used X to crack a decades-old riddle that befuddled scientists: tracking the routes oxygen uses as it moves in and out of myoglobin, the oxygen-binding protein found in the muscle tissue that has been the guinea pig of molecular science for the past 50 years. It is the protein that gives fresh meat its nice red color in your local supermarket, and principles that govern its function often are applicable to other, more complex proteins.

In separate work, Onufriev used X to explore the full range of motions of a long DNA strand--long enough to form the fundamental unit of DNA packed in a single living cell, the nucleosome. The computer simulation, using only 10 percent of X's total heft, scuttled the long-held belief that DNA is a rigid building block of life, like a Lincoln Log. Instead, the computational experiment showed DNA is much more flexible than previously thought, Onufriev says. These findings may go a long way in helping scientists track the molecular makeup of human beings.






				G FORCE



				System G uses half the power of its brother, X, but packs twice the computing punch

				System G sits behind two large, blank wooden doors in a building on Virginia Tech's Corporate Research Center campus. It has no visible signage that brags of its name, mass computing power, technical creativity, or computer science ingenuity. But it should. G is comprised of 325 Mac Pro computers, each with two 4-core, 2.8-gigahertz Intel Xeon processors and eight gigabytes of random access memory, cranking out 22.8 teraflops (trillion operations per second) of computing power, compared to System X with its 1,100 Apple PowerMac G5s, boasting a sustained 12.25 teraflops. "It's twice as powerful, a third the size, and uses a third of the power," says Kirk W. Cameron, associate professor of computer science in the College of Engineering. Cameron helped build G with Srinidhi Varadarajan, associate professor of computer science and director of the Center for High-End Computing Systems. Unlike System X, System G was created to improve system design first and foremost and to be used as a production computer second. The $1.2 million machine was built to develop new, high-performance software tools and applications with paramount efficiency. When Cameron and Varadarajan, assisted by a team of fellow faculty members and scores of graduate students, set out to build G, they had an end-goal in mind. They wanted G to not only be fast but also environmentally friendly. Hence the "G" for green. When Cameron arrived at Virginia Tech in 2004--on the heels of X's opening, and attracted to Tech because of X--"green computing" wasn't even an industry catchphrase. But Cameron already was looking ahead at the need to conserve power. He previously worked for Los Alamos National Laboratory, handling performance duties, "squeezing as much power as possible" out of the facilities' supercomputers, he says. This was in 2002, well before the recent call to arms in green energy. G went on line in 2008. "Now everybody says they are doing it, but we were lucky enough to have done it first," Cameron says. The system is stocked with advanced power-management capabilities, such as power-aware central processing units (CPUs), disks, and memory. A network of advanced power and thermal sensors runs throughout the entire system, with Cameron and Varadarajan able to track how much power each CPU unit in each Apple tower is using. Using power caps, the system is able to run on lower steam when not in use or in partial use, as opposed to older supercomputer systems that run at full blast, needlessly using power, on a continuous basis. Temperature control is carefully monitored as to not overuse the massive air-conditioning units that keep the stacked, side-by-side computers from overheating. "We wanted to try and find the most efficient way to use computers," says Cameron.
				S.M.

Naren Ramakrishnan is a professor of computer science whose research has focused on data mining--the science of processing massive quantities of data to discover patterns and to produce new insights. X is vital to this work. In 2008, Ramakrishnan and a colleague from India used X to catalogue templates of possible "switches" within a living human cell. To illustrate, imagine a massively large dissembled puzzle.

"The 'pieces' are chemical reactions, and we used System X to try out different ways to put together these reactions to form a biochemical switch," says Ramakrishnan. "It's actually a little bit more complicated because some of the pieces can be used multiple times in completing the jigsaw, some pieces need not be used, and each piece has many orientations. So there are millions of possible combinations, which is why we need a supercomputer to sift through all of them."

Paul, the associate professor of mechanical engineering, was attracted to Virginia Tech because of System X. He also conducts research under the flag of the College of Science's physics department, where he has used the supercomputer to conduct large-scale computational simulations to better understand the chaotic dynamics of the atmosphere and oceans. "We are exploring fundamental aspects of the chaotic dynamics that result when large systems are driven far from equilibrium," he says. "Examples include the weather and climate, the convection of biological organisms in the oceans, the efficiency of combustion and chemical reactions in complex flow fields, and fluid turbulence."

THE FAST TRACK

With a committee of just three members--Ribbens; Paul; and Daniel Crawford, professor of chemistry in the College of Science--approval time for using System X for a project is one or two days; and once an allocation of System X time is made, researchers on a given project can submit jobs to the machine with virtually no waiting time at all. Requests to use supercomputer resources at other installations can take anywhere from three months to a year--"and three months is incredibly lucky," says Varadarajan--while jobs often sit in a queue waiting to run for hours or days. The committee reviews 25-30 projects a year, declining only a handful of projects that don't justify the supercomputer's resources, Ribbens says.

From there, researches are given a project account. They log in from wherever they are in the world at an appointed time, enter and then run their data codes, and wait for the results. The amount of computing power and the complexity of the work determines the return rate, so a project involving the ignition of gases within an engine could take only a couple hours. But creating algorithms tied to, say, the formation of wildly infinitely complex protein or cells can take months, if not longer.

Virginia Tech researchers, including Onufriev, appreciate the fast approval and work flow. "I could have applied for time at national supercomputer centers. Problem is, the applications take a while, and then you have to stand in a queue to run your jobs," he says. "With the myoglobin project, it was really important to quickly test out many factors, so the on-demand access to fairly large chunks of X was important. I would have given up on the project if I did not have access to X, simply because it would have taken me years to do this work with ordinary computing power."

Multiple experiments and research projects can be handled simultaneously on X, as it is rare for any one project to consume the full blast of X's regular 12.25-teraflop computing power. "Many things can run at the same time, in parallel, like cars on a highway," says Crawford, one of the early consultants on X as it was being planned and built. "As scientists, we knew we'd be getting a lot of use out of it."

X's FUTURE

Supercomputer years make dog years look easy. Built in 2003, System X is ancient for its species. Among typical PCs, every 18 months computer processors can run twice as fast for roughly the same cost. For supercomputers the process is more drastic: performance increases by 50 percent every six months.

"We are looking for a successor," says Varadarajan. But the process may be grueling. Including upgrades, X has cost $6.5 million. A supercomputer now, to be competitive, would cost upward of $30 million to build and would need to run 1,000 times faster at peak power. As of publication, the No. 1 computer on the TOP500 list is the "Jaguar," peaking at 2,331 teraflops.

X and other supercomputers, now and in the future, will be vital tools for researchers such as Feng, Onufriev and Ramakrishnan. But they still are tools. They need cutting-edge users to ask the important questions.

"A supercomputer is a fast tool. It won't lead to discoveries, but it can help scientists and researchers discover things faster," says Feng. "One important thing to remember is a supercomputer is only as smart as its users. It does what it is told. That's it."

X is by no means ready for retirement. In the hands of Virginia Tech's world-class researchers, the supercomputer will continue to pave the way for groundbreaking research that will change the world.

To read about the launch of System X, see the Winter 2004 Virginia Tech Magazine story at www.vtmagazine.vt.edu/winter04 .

STEVEN MACKAY is the communications coordinator for the College of Engineering. Lynn Nystrom, director of news and external relations for the college, contributed to this article.