Power in the Data Center and Balancing 3-phase power - some questions and answers

First, clearing up some misconceptions I've heard and giving out some basic data:

• This conversation is for power typically used in the US.
• 120V power is one phase wire to neutral wire.
• 208V power, across 2 phase wires, is still considered single-phase.
• There is no "two phase" power in common use these days.  Trust me.
• It is really important to balance use of the phases of 3-phase power.

Most data centers use large UPS systems for the room (before distribution of power to power panels and racks).  Smaller server rooms sometimes use rack-based UPS systems (after distribution to racks).

The step-down transformer sometimes used taking 208V down to 110V wastes energy.  I ballpark it at 10%.  In most server rooms 208V power is used for equipment like servers and storage.  Here are a few examples of how that power is often distributed and how to balance power across the phases in each case:

• A 3-phase power panel with 208V circuit-breakers wired to outlets near each rack.  This method is the usual method in older data centers, and requires an electrician for every change, which can be expensive over time.
• Balance power by placing gear in racks using under-utilized phases.  For example - Phase A is under-utilized by B and C phases are over-used.  Use racks with AB and/or CA to boost use of A more than B and C.
• 3-Phase from Starline Busway or equivalent to 208V plugin modules which provide outlets.  This type of distribution is quite flexible and a bit more expensive up front.  No electrician is needed for minor power distribution changes over time.
• Balance power with a short outage and switch modules to change phases in use for the outlet.
• 3-Phase to the PDU/power strip in the rack.  The equipment to do this is typically more expensive than other power distribution equipment.
• Balance power by changing which outlets you use in the rack.  Power must be balanced in each rack.

The real question in balancing 3-phase power is this: how much out-of-balance is too much?  Our standard is that power must not be more than 10% out of balance.  It is "ok" if a very temporary migration takes power use 15% out-of-balance for less than a few hours.

I have seen power panels 25% or more out of balance.  This typically causes some problems:

• Heat - this condition is detectable as heat
• Circuit Breaker outages - circuit breakers can flip
• Waste of power - Power will be induced in the neutral wire and be wasted
• Loss of capacity - this wasted power is lost and cannot be used, lowering the maximum capacity available.

Radical Automation

There are a lot of tools out there to help out operationally.
They help with numerous areas, including but not at all limited to:

• config file management
• patch and package management
• server builds
• remote management (even for virtualized/remote console)
• monitoring/alerting
• data gathering and data graphing (can be separate from monitoring/alerting
• CMDB and inventory management
• capacity planning/management
• performance monitoring/management/planning
• security monitoring/prevention/policy enforcement
• correlation engines
• release tools
• lifecycle management (hardware, software, etc)
• control management (automated error correction like restarting a failing application instance)
• reporting tools

Each of these items is useful in its own right.

When you have enough of these systems, they can seem like a lot to look at, and a lot of overhead.  But without them you are blind.

The solution is radical automation.

I've worked in a radically automated shop.  This was in the days before so much open source software, in the 90's.  We wrote systems that handled most of these issues end-to-end, and were written flexibly to be extensible.  We had what I think of as code-perfectionists who wrote self-documented code in perl, no less, that anyone could modify.  I started helping do it, and then others who were better at it and more dedicated to this space took over.  But even years later I could go and modify our code and automation because it was written so well.  But I digress.

Having all of these systems very tightly integrated did many things for us:

• Automation means less administrator-caused downtime.  Also, less experienced folks could take on more complex tasks that were "wrapped" in automation that safeguarded against most stupid mistakes and "fat fingering" problems.
• We had the data we wanted now, and on an ongoing basis.  And seeing and using that data, sharing that data, was easily doable.  
• Weekly reporting that could be changed week to week but otherwise be consistent
• 
  
• We could tackle enormous tasks.  When building and deploying a server is entering a tiny bit of data in a database about it and its role, and plugging it in and waiting, it is not hard to deploy hundreds of servers in a day with a few people.
• Budgeting and planning were eventually largely based on rules from automated capacity planning.
• We knew where and what and how many of everything there was and why.
• We did a LOT more to focus on vendor management when we were not so busy staring at our navel.

There are a ton of frameworks for automation, but none replace rolling up your team's sleeves and getting to work on writing some automation.  You can start small and grow from there.  Pay your sysadmins to be programmers and not just sysadmins.

Flash storage in the data center

Flash storage in the data center is here for many of us and is coming to the rest like a freight train.  There is a dizzying array of options, including bypassing flash and using instead a pile of RAM to consider.

What roles is flash used for, generally:

• Primary storage - when you always need blazing fast read performance.  I think hadoop clusters and databases when I think flash primary storage.
• Caching - when you need a boost to traditional storage performance, especially when using SATA storage (although some use it against 10K RPM disk as well).  This is very common for virtualization uses and is also to improve general performance of centralized storage.

Flash is used in 2 form factors:

• SSD (solid state disk) - replaces traditional disk in the form factor of a disk.  This can be used locally or in a storage appliance separated from a a server.
• PCIe - bypassing SAS protocols and throughput limits, flash storage is accessible to your server's motherboard

There is one more dimension of flash storage use: where do you put it?

• Locally - put it right in a server
• Remotely - put it in a central storage appliance

Both form factors and roles can be involved with both locations of storage.

Flash is generally not so great at sequential writes.  Dell is selling SSD storage and has a fascinating write up useful for thinking about flash in general, showing flash is generally not worth the money for sequential writes.  As I think about databases and convergence, I think about database storage history.  First we used to separate all different database uses for storage into separate physical disk.  Then having a big pool of disk became more important.  Now we're on the verge of separating out those uses again!  But I digress...

Here are a few options I have been involved in looking at or have been hearing about:

• FusionIO - mostly used as caching PCIe local storage
• They did what every hardware startup wants to do - got past several hurdles to success!  FusionIO lined up a first MAJOR customer (rumored to be Facebook) that helped them get to volume pricing, and then lined up an OEM agreement with a major manufacturer of servers (Dell).  It clearly works quite well for many/most folks, but does NOT work well for some folks using it as primary storage for Oracle...
• Dell MD1220 with 8 SSD's - Local SSD primary storage
• Dell's write up mentioned above is interesting!  600GB usable storage with mirroring data protection.
• Tintri - A network appliance using non-PCIe storage as a cache
• this is via NFS, it is fast, low latency...
• Violin Memory - flash and RAM in an appliance hooked up via fiber channel or on the network via iSCSI
• Network Appliance - This appliance can use the PCIe-type approach as cache in Flashcache as well as the SSD approach.
• Pure Storage - Appliance using flash as primary storage over fiber channel

VMWare Vsphere 5 - a few thoughts

VMWare has announced the features, and pricing structure, for vSphere 5.

The first item being heavily discussed is the new cost per-used-RAM. We use Enterprise Plus, which gives 48GB used per CPU. If you have an 8-core CPU, then that is only 6GB/core for your license! That sounds bad to some folks. Here is the other side of the story, however. If you have a redundant configuration as many of us do, then the unused RAM is not counted.  We found that when we added up our used memory we needed to buy no new licenses.

Update: vmware moved to 96GB/CPU for "old" enterprise plus licenses after the uproar from customers...   We became unfazed as our calculations led us to conclude it did not affect us.

RAM prices at a low: It is a better time than ever to use server virtualization!

RAM: Over the years I have watched as server RAM prices have gone down.  However, when I recently priced out a virtualization server with 192GB RAM and the RAM was half the price from 9 months ago, I thought WOW!  My purchase power to implement virtualization has increased dramatically!  With 2-socket servers with 192GB RAM, there is more compute power than most companies really need for most tasks in total; virtualization of some kind is required for most enterprises to make use of the hardware on the market.

There is no better time than the present thus far to implement virtualization.

New ASHRAE standards, energy efficiency, and HPC

The new ASHRAE standards, when vendors support them, allow for servers and storage that can be run continuously at a maximum of 113 degrees Fahrenheit. Your support contracts and warrantees are not voided when this is supported.

Dell is at the forefront of supporting these standards for the public. They have made public statements supporting these standards. Most of us have heard about the military running servers in the desert without air conditioning. Milspec is not for most IT organizations, but is paving the way for energy efficiency for the rest of us. It means in many places in the US, ambient air is plenty cold 100% of the time. No more cooling server rooms except as needed for people doing work in server rooms.

I have heard jokes in the past about hanging winter coats in server rooms in warm climates as people get very cold working for long periods in chilled rooms. Now the new joke is shorts, flip-flops, and tanktops on a rack by the server room door as we might start running these rooms much hotter.

How does this affect HPC?

Air cooling at any temperature is only good for racks holding up to about 20KW of power-using dense equipment. Any more than that and you just cannot get the heat out fast enough without help from a more dense material than air or by moving air so quickly that it is not cost effective. If I can run data centers on ambient air cooling, then running air cooled racks is getting cheaper and cheaper. Highly dense racks requiring chilled water or other solutions are becoming more expensive compared to these air cooled racks.

It has been the trend to try to use smaller and smaller server rooms, and get more and more dense. The benefits seem to now peak to 20KW/rack and go down from there. If you do not really need extreme HPC density for performance reasons, extra density is not cost effective.

Methods to retro-fit ambient air cooling into existing data centers are critically needed. There is not nearly enough action in the market in this area.

The cloud: hype and reality

The cloud. Everyone in IT is in a buzz about it. It means many things to many people, and sometimes seems to mean anything anyone wants it to mean.

First let me say that, if it is not obvious already, this is not a one-size-fits-all world. As businesses tackle the next buzzword concept, it is very important to keep that in mind. One great was outsourcing, which is great for lots of situations, but definitely not all. Now the cloud.

I was asked to ignore all "business problems" and evaluate moving all servers my department runs in the cloud. It was going to cost twice what it currently costs us in terms of costs like equipment replacement and support contracts. Running servers in the cloud is great if you are small and it helps you avoid hiring your first systems administration team, and it is great if you are huge and need help not buying equipment for rare peaks of capacity utilization, especially if you also have to buy equipment for a DR site for those peaks as well.

As far as SaaS goes, it is a clear winner for many cases. If you do not need to run a local email system, sure, outsource it to a "cloud provider". Salesforce CRM and more can be great instead of running it yourself. Examining every service you run and seeing if it could be more effectively run by others is a sensible thing to do. Sometimes software makers move to a hosted model that is more effective.

Running your own "local cloud" can be effective too, assuming you have appropriate control of and planning for your infrastructure. Moving into server or desktop virtualization when you may not have the network bandwidth or central storage IO to handle it is a disaster.

As always, if you do what makes sense, things go well. Do things because "they are the hot thing to do" rarely goes well for anyone.