We’ve been using Linux for routing, firewalling, and internal services like NAT, VPN, etc now for years.  Vyatta has wrapped his up in a self contained system that provides a nicer frontend for configuring and mananging all of this.  You can download Vyatta and install it yourself, including the option to buy a commercial version with more features and support, or you can buy pre built appliances from Vyatta as well if you don’t want the hassle.

Regardless of which way you choose, the overwhelming feature that makes Vyatta nice is that pound for pound it’s much cheaper than Cisco or the other alternatives.  And while we have plenty of Cisco gear, and in certain cases it makes sense to go with one of the bigger vendors, for smaller routing projects where you aren’t using a lot of the features Vyatta makes much more sense.  That’s where we use it, and we can pass on that savings to our customers as a result.  More importantly, we usually can buy more than 1 of the units to either run in active/passive mode or to keep a spare on the shelf for failure – for less than what we’d pay for a more name brand router.

I’m excited about the 6.1 release as it provides support for more IPV6, which is a must anymore, as well as some layer 2 bridging which is probably a result of the need for spanning multiple racks at a layer 2 level for cloud related projects.

First, some background: our building is well protected from lightning strikes by what’s known as transient voltage surge suppresion (TVSS).  This is similar to the surge protection strips most of us have at home that we plug our valuable electronics into.  However, we have three fold protection within Data Cave:

1. Incoming protection at the electrical mains.  Our switchgear has built in TVSS on the incoming power lines, rated for 250kA.  This technology basically clamps the line voltage in case of a spike, due to lightning or some other disturbance, and keeps it from entering further into the building.
2. Electrical power that is being sent to the critical IT equipment goes through our flywheel UPS units.  In these units we convert the AC power to DC power, then back to AC, as part of the rectification process.  Any surges or sags get filtered out by these units.
3. Our data suite Power Distribution Units also have TVSS units on them, which is what connects directly to the IT loads, in case any surges were somehow able to make it to this point.

One of the TVSS Modules protecting Data Cave's incoming power

So overall, we have great surge suppression.  So imagine our surprise earlier this summer when a large thunderstorm came through, and a nearby lightning strike caused some damage to the hardware that controls the automated gates to the facility.  In addition, we had some power over ethernet switches inside of our building also go “flaky”, for lack of a better term. This prompted us to investigate what was really going on, and what we could do to better protect ourselves (and our customers).

Our Findings

The fence and gates that surrounding our data center were in the process of being installed, but not finished, when the internals to the data center itself were commissioned.  Something that was overlooked in this process was the protective grounding of these items from lightning strikes, which is a different beast than just grounding them from an electrical safety issue.

The fence posts themselves stick into the ground, and in general that means the fence is grounded.  But in our condition, it’s not “well grounded”.  The soil in this area is a bit sandy to a bit clay like, and thus to achieve a good earth ground we prefer to go 20 feet deep.  The fence posts are only about 3 feet deep, which means they aren’t achieving as good of ground as they could be.  To fortify this, we drove 20 foot copper ground rods in periodically along the perimeter of the fence, and bonded the post to the rod, to give better overall grounding.

Copper grounding of fence post

Grounding of the gate

Other outdoor circuits

In examining the fence and gate, another vulnerable spot we found was in the underground conduit that goes from inside of our data center to the gates.  Because the gates are electrically controlled, they need electrical power – which is provided from inside of our building via an underground conduit.  In addition, our gates are key carded to allow customers and staff in upon a swipe, so data cabling was also necessary.

However, these two electrical lines now represent two new pathways for lightning to enter our building.  While our incoming electrical lines to the facility are protected, if lighting was to strike the gate, it could go “backwards” through the wiring and back into our building in the opposite direction!

Circumventing this is easy.  First, we installed transient voltage surge suppression devices directly at the electrical service at the gate, so in case of a lightning strike it would suppress any large voltages that may generate.  Then we protected the data cabling by the use of inline ethernet surge suppression devices.  These devices cause any voltage spikes that may be generated to be suppressed on the data lines as well, limiting any kind of backfeed into the building.

Ethernet surge suppression device

Conclusion

With the addition of better lightning protection of things outside of the building, we feel better equipped at mitigating future lightning strikes in the area.

In the part 2, we will highlight some of the existing lightning protection features of Data Cave and how they work to keep lightning away in the first place.

Understanding Data Center Power

Each cabinet or rack installed in the data center needs power to operate.  Data centers generally charge for this power at a flat fee per circuit, based on the total capacity of the circuit.

When data centers charge for power in this fashion, their total charge must include 4 items:

1. Their cost for the incoming power used in the circuit
2. Their cost of the overhead for maintaining the power
3. Their cost for the cooling required, since the power will turn into heat
4. Markup

The above calculation is usually easy for the data center.  They know how much they are paying for power, they can factor in an overhead rate for the ancillary equipment (UPS systems, cabling, maintenance, surge suppression, etc).  #3, the cost for the cooling, is a bit tricker as it requires the data center to know how efficient their systems are to understand how much power is required to cool the IT systems.  Many data centers will estimate this, the most common estimation being the same cost as #1.  That is, there’s just as much cost in cooling the load as there is in heating it up.  While for well design data centers, that is not the case, it provides a rule of thumb that many follow.

Thus, the overall price a customer will pay for power is usually more than 2x the actual cost for just that power, simply due to the overhead.

Playing off inefficiencies

While the data center may provide a 120V/20A circuit, rarely are all 20A of that circuit used.  In fact, 20% of that circuit simply cannot be used, because of electrical code rules.  But furthermore, in many cases, a customer may only be utilizing a fraction of the total available power on a circuit.  This is common place in the data center, and it is also a big source of profit.

The profit stems from the fact that the data center is charging the customer for the worst case usage – that is, for 20A.  In reality, the customer may be only drawing 10A.  As such, the data center is profiting from the difference.  This profit comes from lower electrical usage, lower cooling usage, and lower total overhead.

While the overall cost and profit may be small at this scale, imagine a data center customer with multiple cabinets and multiple electrical circuits.  What if, instead of 1 circuit, the customer had 40.  And what if, on after, those circuits were only being 1/2 utilized.  The customer is overpaying for power that they are not utilizing.

Enter Metered Power

Data Cave’s electrical infrastructure was designed from the start for metering power at the individual circuit level.  With metered power, the end customer pays for power just like they were paying the electric company – based on actual consumption.  This billing arrangement can provide significant savings over a flat circuit fee, as the total consumption is aggregated across all customer circuits.

It also allows the customer better insight into how much power is actually being consumed.  It gives the customer feedback to know which systems draw more power, and which draw less.  Based on that information, a customer may be able to do a cost benefit analysis on when it’s time to replace systems with those that are more efficient – and reap the savings.

Unfortunately, at the same time, our current host decided to do maintenance on the webserver. We currently are using an offsite host for web and email for the thedatacave.com domain, due to the fact that the data center is under construction (trust me, we’ll host ourselves internally soon). And I suppose you have to live with the ups and downs of someone else controlling your infrastructure.

This particular host has really given us nothing but grief. The outages are one thing, but the lack of explanation beyond “we’re working on it” and “it’s now up” is very frustrating. I want details. I want to know what’s wrong, what’s being worked on, and what their thoughts are on having it fixed. If you tell me 1 hour, and it takes 2, that’s ok. I just want to know 1hour vs. 24 hours, vs never. When I ask questions, all I ever get back is “no”, with no explanation.

Once you lose a customer, it’s hard to get them back. Any business without a firm grasp on good customer services is just shooting themselves in the foot these days. Trust me when I say that as a company, Data Cave will always put the customer first. We strive for perfection, but at the times we aren’t perfect, at the very least we’ll be transparent about it.

It’s no fun to have an upset customer, but you’ll have an appreciative one if you simply keep them informed.

Caleb

All of the conduits run thus far were the services into the building, from the transformers and generators. These were the very deep conduits, as they have to go under the foundation of the building.

Once these were all in place, the process was to backfill. We backfilled to the 3 foot mark, and then left water to run overnight to harden the ground. The next day, the ground was so hard you coudn’t break it with a spade shovel. We felt pretty good about this level compaction.

The final step was to run the conduits necessary for the utility service – the primary high voltage lines that service the transformers. These are all outside, and are only at the 3 foot deep level. So these conduits were run next.

The generators also require some extra wiring for running pumps, and to control when they start and stop. These are run in smaller 1-2 inch conduits.

While this was happening, the site foundation received its stone base.

And by Friday, everything had returned to its normal level.

We’ve been working on our electrical services. The electrical service for the building is massive. Like, mondo. Like, so big you can’t even fathom until you see it.

We’re starting in just one section and with one service (actually, two services via two transformers, but that can be routed by the internal switchgear as we want), but ultimately we’ll have four sets of these services.

This requires a lot of piping to get into the building. And it has to go under the foundation.

Where the pipe sticks up in the picture above is inside the building. It then runs out underground to the transformers outside of the building.

First steps, dig a BIG hole:

Also, buy a lot of pipe. I think for this trench work we bought just over 10,000 feet of 4″ PVC (that’s almost two miles). We also have somewhere around 4000 4 foot sweeping elbows. This thing is mammoth.

Now, start putting it all in place.

Each electrical service requires 8 conduits, in groups of four.

Here’s one group of four:

And then the next day we had a few groups of 8s installed

After the pipe is laid, we began to backfill:

From the other angle:

Another angle, showing the massive scope of the project:

We will be posting more of the details surrounding the construction, design, and decision making of the building as we move forward here. Stay tuned.