Tips for Choosing a Data Center Part 2: Facility, Location, and Power Systems

Are you picking a data center? Unsure of what you should be looking for? Can’t figure out what makes one data center provider better than another? Well, you came to the right place.

In the last video I talked about building your own vs. leasing colo space (spoiler don’t build your own), and we talked about what the different tier levels mean and we talked about redundancy factors as well as some frequently used terms. So, if you didn’t see part 1 of this series, check it out here, I’ll wait.

The Facility Location

Let’s talk about the facility itself. There are three key words, location, location, location, that is the real estate mantra, and it is true with data centers as well. There are several ways in which location is important. First, you want a data center in a location near your IT resources and staff. Frequent trips to a remote location can be problematic and expensive for regular maintenance. If you need to do emergency work, having your data center quickly accessible may be critical for you.

You should also consider the challenge of moving data. You may need a data center in close proximity to the source of your data generation. You should also consider how you manage DR and replication and whether you need a low latency connection to reduce your RTOs and RPOs or if you are running an active active environment that needs low latency to maintain consistency.

Perhaps one of the biggest factors these days is proximity to cloud resources. If you are thinking of pursuing a hybrid cloud strategy, having a data center that has direct connect facilities to your cloud provider of choice is critical. This can allow you to access the cloud through a simple cross connect. Many data centers have connectivity to cloud providers through partner networks, but the direct connect facilities are your best bet if you are going to do anything latency sensitive. There are not a lot of them, and for the most part, they tend to be premium data center facilities, so some of that vetting work has already been done for you bu the cloud providers.

Consider the facilities proximity to utilities like electrical grids and natural gas service if they have bi-fuel generators. Make sure these utilities have diverse paths and entrances to the building. Consider the proximity to telco carriers that are important to you. Make sure the facility you are moving into has access to circuits that are diverse to the building and access to regionally diverse paths. Here in Chicago, most telco runs through the NAP (network access point) at 350 Cermak, but there are options for circuits that bypass the NAP to give you a diverse route in the event the NAP goes down. 

Avoiding natural and man-made disasters

You should look for a building that is fully dedicated to being a data center, single use. Being in a building with other businesses can introduce risks. You don’t want your data center damaged by a kitchen fire or a flood from other businesses in the building that don’t have the same standards as a data center. 

Check to see if the facility is on a flood plane, check for seismic activity. Is it in an area prone to tornadoes, and if it is, is the facility hardened and protected from extreme weather. Be cautious about facilities near airports, train line or highways. A chemical spill from a truck on the highway could shut your facility down, hazardous material from a derailed train could put you out of commission. Often, businesses will have a requirement to maintain a certain distance from nuclear facilities or chemical plants. There can also be a requirement for a certain minimum distance between facilities. So you need to take all of those things into consideration when choosing your location.=

What shape is the building in

Ask questions about the building itself. What shape is the roof in? This may seem silly, but I can’t begin to tell you how big a problem this is for so many data centers. If the roof leaks you have major problems. If it needs to be replaced, that is very messy and disruptive construction. If the facility looks beat up and shabby, it probably is, and it probably has bigger problems that you can’t see. 

Construction

Building data centers is a very capital intensive business, so they are often built in phases. It is generally a good thing to have additional phases that will come online in the future as that gives you the option to move to new, more modern facilities down the road. However, construction can create problems, be aware of what construction is ongoing and what is planned and how will that be contained relative to your location in the facility. 

Construction can bring power outages, damage, fire, inconvenience, and usually tons of dust, the enemy of computer gear everywhere. If managed and planned well, new construction can be safe, and a sign of a healthy business. Justmake sure it won’t impact you. I worked on a data center build at the 111 8th facility in New York, and they were rapidly developing the building. It seemed like every day, the path to get to the data center changed as walls went up and moved around. Some days I would have to go up a floor and then down a floor and then up again to get around construction or a new data center that suddenly appeared in my path to the facility.

Company financials

This is probably a good time to tell you to check on the business’ financials. As I said, this is a very capital intensive business. Facilities can cost hundreds of millions of dollars, and they are often heavily leveraged. Try to be sure the company is not over its skis. They will always paint a rosy picture of their finances, so ask around to see what the word on the street is about the facility, check what people in the industry say about them.  Much of it will be FUD, but sometimes where there is smoke, there is fire. Look for signs of lack of investment in a facility, or kludgy repairs. Good brokers can often help steer you away from failing businesses.

Consider your needs

Do you just need a locked cabinet in a shared space, do you need a SCIF (secure compartmentalized Information Facility), or something in-between. Do you need some work space to have a desk and store some tools. Do you have regulatory concerns like ITAR where foreign nationals can not be in the building, or do you have to satisfy NIST SP 800-53 standards for government work. Will the facility support those efforts?

MEP and PUE

MEP is a term you will hear being used in the industry. MEP stands for Mechanical, Electrical, Plumbing. This covers electrical systems cooling systems air handling systems, etc. MEP is the lifeblood of the data center, see what their capabilities are for maintaining these systems. Do they have electrical engineers on staff for both low voltage and high voltage. Do they have HVAC engineers, etc. Having engineers on staff to handle these systems is a positive sign that they are making the investment in the people to run the facility.

PUE is another term you will hear referenced quite frequently. PUE stands for Power Usage Effectiveness. This number is a ratio that describes how efficiently a facility uses power. How much power is used by the computing equipment vs how much is used by the facility cooling and overhead. The lower the number the better the efficiency. This number is not an exact science, and quite frankly, it is often just guessed at or completely made up. Look to see if you can see evidence of these efficiencies.

Power

Make sure the power systems will meet your needs. Understand what types of circuits you are going to need, 3-phase, 220, 110, etc. Ask if they will support all of the socket types you will need. Here is a tip, I have seen so many organizations put in all 220, not thinking they might need some 110 at some point. You will invariably need it for some charger or power supply that only runs on 110. Find out how many kilowatts they can support in a cabinet if you have large power needs. Do you want power fed off a bus so you can reconfigure things easily, or or is it OK to have power coming off of whips, which are cables running back to a panel. Understand your options. Ask if they supply in-cabinet ATS units for equipment that has only one power supply. You may need this if the facility does A/B power outage testing.

Make sure there is sufficient power capacity to grow. The utility allocates a certain power reserve to an area, if the facility is running near capacity, see if they utility has capacity to provision more power and ask what their plans are to add more.

Power delivery path

Ask about the full power path. There should be the diverse entrances. From these entrances, the conduit paths should be completely divergent and not overlap at any point. One thing that people often ask about is whether the facility is receiving electrical service from multiple power grids. For most facilities, this isn’t practical, and I would argue it is generally unnecessary. The electrical infrastructure is in pretty dodgy shape here in the US and in many other countries it is worse. If a grid goes down there is often a high probability of a larger cascading failure, so I don’t put a lot of stake in the multiple grids thing. What is important is onsite generation capabilities.

Onsite power generation

Onsite generation is usually supplied by generators, typically diesel. There are some bi-fuel generators that can run on diesel and natural gas, and there are even some fuel cell options out there. Bi-fuel generators are nice because they can greatly increase runtimes. The natural gas supply is a fully diverse source to the electrical grid, and since it is a pressurized system, it can keep going when the power is out. The problem with bi-fuel is that these generators are typically smaller, in the 600kW range, and can’t provide enough power for many facilities. When you are in the multi-megawatt range diesel is king. 

Ask about their maintenance and testing procedures. Diesel generators need to be tested frequently. In cold weather, they need to be started on a regular basis so that you can be sure they will start when you need them. In very cold winters, we used to run our generators every day to make sure they were going to work in an emergency. Burning fuel also helps with keeping the fuel free from fungus and mold, diesel can be tricky to store and manage over a long period of time.

For diesel, check what the onsite storage capacity is. How many hours of runtime do they have with the fuel onsite. This will change over time as the facility power consumption increases or decreases, so ask what the system is engineered to deliver at maximum capacity. Ask about their supplier contracts, do they have rolling stock, or fuel in trucks. Consider that life safety services are going to get priority in a major catastrophe, so all those contracts might go out the window. 

I had a situation once where the region we had a facility in got hit by some tornados. Power was out for several days. It was our practice to top off the fuel tanks a couple of times during the day. After getting topped off, the driver said to us, that this was going to be his last delivery, which is not what you want to hear when the power is still out. He explained that the power was out at the fuel storage depot, and they didn’t have any backup generators to operate the pumps to fill the trucks. That was a failure mode that I hadn’t previously considered. In the end we had enough capacity to weather the storm, but I felt bad for those who had less capacity.

TVSS and ATS

So let’s cover the rest of the power path. I knew a guy at a facility who could talk for hours about how they designed the grounding systems, which is a really important thing for a facility and it usually doesn’t get enough attention. There are the TVSSs, Transient Voltage Surge Suppressors for the facility and ATSs or Automatic transfer switches, which rapidly transfer power sources in an outage situation. Engineers are often proud of their designs and will love to talk about this stuff, and you can get some good information talking to them about the systems. If they aren’t proud of the design, it is good to know that too.

UPSs

Check on the UPSs. Much of the infrastructure in a data center is depreciated over 10-15 years check on the age of the equipment. Some facilities will buy equipment used, so it may be older than you would suspect. Make sure they are on a regular maintenance schedule. Ask when they had the batteries replaced or the capacitors redone in the UPSs. This is all work that can be disruptive. Sometimes you encounter UPSs that don’t have bypasses built in, so they have to take a hard outage for any maintenance. Some facilities can have more exotic things like flywheels or UPSs with Lithium Ion batteries, but the bottom line is having enough time to get to generator power.

 Ask about their maintenance contracts and who does the maintenance. Ask about how much runtime the UPSs have. Typically they are only there to bridge the amount of time for the generators to start up, but a bigger buffer is better, because generators can be difficult to start, especially in the winter. Check that the electrical work is done correctly, are all of the cables run through conduits,  finished and dressed nicely? If they take the time to make it all look good, they probably did a good job of installing it.

Check on the PDUs, not the ones in the cabinet, but the big power distribution units on the floor and that the A and B sides are isolated from each other. It is also good to see that the phases are properly balanced to see if they manage their power well, but it is unlikely they will let you get to that level of detail. 

Is all of this equipment out on the floor, or is it contained in a galley. Having it in a galley means they can do maintenance activities away from your equipment without disturbing you. See if they can show you their MEP rooms, they can be really interesting.

You might want to inquire as to how green the power sources are if that matters to your organization. Is it coming from coal, nuclear, wind, hydro, or solar. Different markets offer different things, and different markets can have wildly different prices per kWh or even subsidized services.

One other thing that is rare to find, but is something you might want to consider is EMP shielding. EMP is a very real problem, from high altitude nuclear detonations, to solar flares, to gamma ray bursts from exploding stars, any of these events can cause significant damage to your equipment. Depending on the nature of your business, you might want to take a look at it. This is a very cool and complex topic that is too much to go into here, but I did a video on EMP and EMP shielding facilities that you can check out here.

I don’t want these videos to get too long, so I will pick up cooling systems in the next video followed by Network, Fire suppression and detection, Security, monitoring, add-on services and finally, tips for negotiating the contract. 

Please click like, subscribe and hit that bell icon for updates. If there is anything you want me to cover, let me know in the comments, and I will see you in the next video.