How Can DC Distribution Improve Power Efficiency in Server Farms?

[stedwards]

1 Internet server farms demand a massive amount of power.
2 The power grid can supply 3 phase.
3 To increasing demands for data throughput rate both transistor size and voltage levels have been ever shrinking.

The server farm market wants efficient and reliable power conversion from 3 phase mains to low voltage DC supply rails that are (at a guess) a couple of volts.

How would you design the winning power conversion product?

 

[anorlunda]

District heating. Build the farm in a dense city center in a cold region with long winters and no air conditioning in summer.

Use water cooling for the server farm. Use the waste heat water to heat all the buildings and homes. Warm water return from the buildings melts the ice on streets and sidewalk. Then dump the water in the harbor, keeping it ice free through the winter. Shut down the city and send everybody on vacation during the hottest month.

I once lived in Västerås Sweden, a city of 150,000 people. They did all those things by diverting 25% of the low pressure steam energy from the local power plant. When the power plant reached end of life, they went back to consuming electricity to do all that stuff.

Using low grade heat (warm water below boiling temperature) is very difficult. District heating is almost the only way to use that heat productively. So my server farm design uses all that waste energy productively to to things that would otherwise have to consume electricity anyhow.

 

[tech99]

1 Internet server farms demand a massive amount of power.
2 The power grid can supply 3 phase.
3 To increasing demands for data throughput rate both transistor size and voltage levels have been ever shrinking.

The server farm market wants efficient and reliable power conversion from 3 phase mains to low voltage DC supply rails that are (at a guess) a couple of volts.

How would you design the winning power conversion product?

How about transforming down to 48V AC, then a 3-phase rectifier to obtain fairly smooth 48V DC. At this point a battery back up can be connected. Then convert it to 48V 400 Hz AC by electronic commutation and distribute it to each rack, where there would be a small 400 Hz transformer with several 3V rectified outputs.

 

[stedwards]

How about transforming down to 48V AC, then a 3-phase rectifier to obtain fairly smooth 48V DC. At this point a battery back up can be connected. Then convert it to 48V 400 Hz AC by electronic commutation and distribute it to each rack, where there would be a small 400 Hz transformer with several 3V rectified outputs.

The nice thing about 3 phase power into a transformer, is that it can output 6 phase with greatly reduced ripple. Rectified, is the ripple (13%) low enough to power servers directly, without filtering?

 

[tech99]

The nice thing about 3 phase power into a transformer, is that it can output 6 phase with greatly reduced ripple. Rectified, is the ripple (13%) low enough to power servers directly, without filtering?

I think 13% would probably be OK at input side of a 3V regulator. In any case, it is quite easy to filter this at 3 volts DC. Of course, every unit or circuit board in the rack would require a regulator followed by a small amount of noise filtering.

 

[f95toli]

The nice thing about 3 phase power into a transformer, is that it can output 6 phase with greatly reduced ripple. Rectified, is the ripple (13%) low enough to power servers directly, without filtering?

Servers are not (usually) powered directly. They are powered from UPS systems(batteries combined with e.g. diesel generators), which in the case of a farm can be quite large facilitates..
There is, however, no reason why you can't charge the batteries using supply with relatively large ripple.

 

[Baluncore]

By using power factor correction circuits, all available AC phases can be switched into one high voltage DC capacitor and battery bank. That will look like a resistive load to the grid. The HV DC can then be distributed to wherever the low voltage high current is needed. It can then be reduced efficiently to a couple of volts by using buck converters with synchronous rectifiers. That will give an overall efficiency of better than 90%.

 

[Jeff Rosenbury]

By using power factor correction circuits, all available AC phases can be switched into one high voltage DC capacitor and battery bank. That will look like a resistive load to the grid. The HV DC can then be distributed to wherever the low voltage high current is needed. It can then be reduced efficiently to a couple of volts by using buck converters with synchronous rectifiers. That will give an overall efficiency of better than 90%.

I like this idea. But doesn't switching place harmonics on the power lines?

I'm not sure one large switching system is the answer. Perhaps several offset by a few microseconds? Or possibly timing switches so the low order harmonics cancel? (The higher order ones are more easily filtered.)

 

[Jeff Rosenbury]

So from what I'm reading, transform the voltage to a reasonable level. Reasonable would be defined as one where we could operated a high efficiency DC to DC switcher. (Remember the line voltage is likely coming in on a transmission line for a large farm, so some transformers will be needed anyway.) Then use power electronics to switch the current into a DC capacitor bank in such a way as to condition the line back to the power company (PC) to offset other PC loads. Attach the UPS here and distribute to various server racks. Have each rack provide a DC to DC switcher set and a backup. Distribute the DC lines to the boards which should each have their own linear supplies.

I like Anolunda's idea for using cooling water to heat buildings and stuff. I also agree that the servers would do well in cool climates with lots of hydropower. (Northern Canada comes to mind.) Of course finding an area with that and a technically competent workforce will always be an issue. Lots of competent people seem to prefer at least some sunshine.

 

[Baluncore]

But doesn't switching place harmonics on the power lines?

Yes, but they are the harmonics of the PF correction circuit which operates at above 20kHz. Those harmonics are easily spread in time and attenuated by LC noise filters.

There would be redundancy, with more than two AC to HV DC units per server rack.

The advantage of HV DC is that HV swamps the PN junction voltage, reduces current through MOSFET/IGBT switches and reduces HV ripple since E = ½ C V².

The buck topologies that convert HV DC to a couple of volts DC are limited in efficiency by output diode voltage drop. Synchronous diodes made from MOSFETs resolve that inefficiency. The HV side needs low current switch semiconductors while the LV side needs LV high current semiconductors.

This is not new technology. It has been in use for 10 years in some fields.

 

[Jeff Rosenbury]

Do you have a name for this sort of high voltage DC converter. I looked up HV DC converters and they seemed to only go from low to high. I need something to google please.

I'm curious about the circuitry.

 

[jim hardy]

I like Baluncore's approach. It makes use of a subtlety that's not immediately obvious -

To make 3 volts DC you rectify some higher voltage. The voltage drop across your rectifiers is an unavoidable loss.
Using the best rectifiers around might make that only 1/4 volt loss.
Mosfets can rectify with only a few millivolts.loss by turning them on and off at the right times, ie synchronizing them with the incoming line...

A search on "Synchronous Rectification" produces lots of references
http://scholar.lib.vt.edu/theses/available/etd-09152003-180228/unrestricted/Ch2.pdf

 

[Baluncore]

Power Factor Correction. Search 'smps pfc'
http://en.wikipedia.org/wiki/Switched-mode_power_supply#Power_factor

See; LT8312. Boost Controller with Power Factor Correction.
http://cds.linear.com/docs/en/datasheet/8312f.pdf “Synchronous Converters” employs synchronous rectification;
http://en.wikipedia.org/wiki/Active_rectification
http://en.wikipedia.org/wiki/Switched-mode_power_supply#Voltage_converter_and_output_rectifier

Also see;
The LTC4357 “Positive High Voltage Ideal Diode Controller” can be used to replace Schottky power diodes in redundant power supply selection.
http://cds.linear.com/docs/en/datasheet/4357fd.pdf

 

[Jeff Rosenbury]

I was thinking something more like this. The higher voltage would limit copper losses. The big DC bus could be unconditioned (in its primary, user side path anyway) to avoid conditioning losses and use IGBT switchers for rack power. IGBTs have a higher gate capacitance which will limit the switch frequency, but the skin effect and other effects limit switcher speed for high currents anyway. That means spending more on inductors (and likely capacitors since we are likely looking at a resonance mode switcher), but the efficiency savings would likely be worth it.

There would be safety issues as well. A high voltage, high current DC power bus is a dangerous combination. It would need to be armored. But the power industry has been dealing with these problems for decades.

 

[Baluncore]

The IXYS Sixpack IGBT Module, VWI 6-12P1, is a 3PH 'H'-bridge.
It looks like it was designed to be used to convert HV DC to three phase AC for variable speed drives.

 

[jim hardy]

Thanks Baluncore

you showed me a whole new approach. That LTC4537 is an eye opener.

Hanging out at PF one learns a LOT and that's good for the psyche. Thanks for sharing your expertise.old jim

 

[Baluncore]

The karma is reciprocated.

 

[The Electrician]

Thanks Baluncore

you showed me a whole new approach. That LTC4537 is an eye opener.

Hanging out at PF one learns a LOT and that's good for the psyche. Thanks for sharing your expertise.old jim

If you liked the LTC4357, you'll love this: http://www.linear.com/product/LT4320

 

[jim hardy]

Wow.

I try to learn something every day. Those ideal rectifiers are a week's worth !

What a natural step from switching regulators. Interesting possibilities with synchronous alterators...

Thanks, guys..

 

[anorlunda]

I expect that the very big server farm owners would assign some of their best engineers to address this problem. Does anyone here know some of the things they have adopted? rejected?

 

[Baluncore]

Does anyone here know some of the things they have adopted?

http://www.directpowertech.com/docs/LEONARDO ENERGY.pdf

 

[anorlunda]

http://www.directpowertech.com/docs/LEONARDO ENERGY.pdf

Thanks Baluncore. It was very informative. They report several projects showing 10-20% energy savings with DC power distribution. They also discuss something not mentioned in this thread - reliability. Improved reliability with DC (via fewer components) actually trumps energy consumption.

 

[Baluncore]

Another advantage of DC distribution is the lack of reactive power in the system. Reactive power results in increased losses in AC systems due to larger current magnitude for an equal amount of transferred power. Non-linear loads, such as AC-DC converters (without power factor correction (PFC)), require reactive power in an AC system. In view of the large number of converters in data centers, much is to be gained by migrating to a DC system.

The problem with DC distribution systems is the insulation and inductance of the cables.
The insulation of AC is less demanding than the insulation of DC. Insulation of AC does not usually become polarised or grow conductive trees and whiskers, as often happens with DC.
Disconnecting DC can be difficult. Unplugging or switching off can strike an arc that is difficult to extinguish. AC reverses phase twice each cycle and so can be switched safely at lower cost. Look at the DC and AC ratings of switches and relay contacts to see a real advantage of AC over DC. As an example, 10A contacts are rated at either 30VDC or 250VAC.

I would distribute power within the server farm as DC, but I would do it with an advantageous twist to cut the weight and cost of cables, switch gear and connectors. To distribute power to racks I would use three phases of square wave AC. At any point in time two phases would supply the current while the third would have 2 msec to change polarity without load. The converter would operate at 50Hz or 60Hz. The 3PH SQ AC would be distributed over normal industrial cables and switch gear, with normal 3PH connectors. It is after all, 3PH 400VAC.

There would be two or more AC feeders entering the server farm from from separate sub-stations. Within the server farm there would be AC motors driving DC generators that charge three UPS battery banks to 400 VDC. Diesel engines could be clutched to two of the generators. All that is current marine propulsion technology. There would be crossbar connections to allow redundancy during fault conditions and while servicing.

Two independent supplies of 3PH SQ 400VAC would be generated from separate UPS battery banks by 3PH H-bridges. Those would operate between 50Hz and 60Hz. That redundant power would be distributed throughout the building to all racks.

Each power module in the rack would receive two 3PH SQ 400VAC inputs. A three phase, six diode bridge on each power input would regenerate 400VDC in the rack without ripple. (That would also minimise big electrolytic capacitor requirement and temperature problems in the rack supplies).

The rectified 400VDC would then be reduced efficiently using multiple distributed synchronous buck converters to LV DC for logic circuits.