Microsoft is starting to immerse its servers in fluid to improve performance and energy efficiency. A server stand is now used for production loads in what looks like a floating bath. This immersion process has been in the industry for a few years now, but Microsoft claims that it is “the first cloud supplier to run two-phase immersion cooling in a manufacturing environment.”
The cooling works by immersing the server racks completely in a specially designed non-conductive liquid. The fluorocarbon-based liquid works by removing heat when it hits components directly and the liquid reaches a lower boiling point (1
“It’s really a bathtub,” Christian Belady, vice president of Microsoft’s Advanced Development Data Center, explains in an interview with The Verge. The stand will settle inside the bath, and what you see is boiling just as you see boiling in your pot. The boiling in your pot is at 100 degrees Celsius, and in this case it is at 50 degrees Celsius. ”
This type of liquid cooling has in recent years been used by cryptocurrencies to extract bitcoin and other cryptocurrencies. This method inspired Microsoft to test its use in recent years, by using it to test against spearheads and intensive workloads for applications such as machine learning.
Most data centers are air-cooled right now, using outdoor air and cooling it by releasing it to temperatures below 35 degrees Celsius using evaporation. This is known as swamp cooling, but it uses a lot of water in the process. This new liquid bath technique is designed to reduce water consumption. “It will potentially eliminate the need for water consumption in data centers, so it is very important to us,” says Belady. “It’s really about driving less and less load no matter where we land.”
This server tub also allows Microsoft to pack hardware more tightly together, which should reduce the amount of space needed in the long run compared to traditional air cooling. Microsoft is first trying this with a small internal production load, with plans to use it more widely in the future. “It’s in a small data center, and we’re looking at a rack that’s worth it,” says Belady. “We have a whole phase approach, and our next phase is pretty soon with more racks.”
Microsoft will mainly study the reliability implications of this new cooling and the types of burst workloads it may even help with demand from cloud and AI. “We expect much better reliability. Our work with the Project Natick program a few years ago really showed the importance of eliminating moisture and oxygen from an environment, Belady explains.
Project Natick saw Microsoft sink an entire data center to the bottom of the Scottish Sea, throwing 864 servers and 27.6 petabytes of storage into the water. The experiment was a success, and Microsoft had only one-eighth of the error rate of a land-based data center. “What we expect with immersion is a similar trend, because the liquid displaces oxygen and moisture, and both of these create corrosion … and those are the things that create faults in our systems,” says Belady.
Part of this work is also related to Microsoft’s environmental commitment to tackle water scarcity. The company is committed to replenishing even more water than it uses for its global operations by 2030. This includes Microsoft, which uses an on-site rainwater collection system in offices and collects condensation from air conditioners to water systems. Nevertheless, Microsoft extracted almost 8 million cubic meters of water from municipal systems and other local sources in 2019, compared to just over 7 million in 2018.
Microsoft’s efforts to address water consumption will be extremely challenging given its trend towards more water use, but projects such as two-phase immersion will certainly help if rolled out more widely. “Our goal is to reach zero water use,” says Belady. “That’s our calculation, so that’s what we’re working towards.”