Vyatta 6.1, we love it.
September 1, 2010 by caleb · Leave a Comment
Recently, Vyatta announced a 6.1 release of their router software platform. At the Cave, we absolutely love Vyatta and the software.
We’ve been using Linux for routing, firewalling, and internal services like NAT, VPN, etc now for years. Vyatta has wrapped his up in a self contained system that provides a nicer frontend for configuring and mananging all of this. You can download Vyatta and install it yourself, including the option to buy a commercial version with more features and support, or you can buy pre built appliances from Vyatta as well if you don’t want the hassle.
Regardless of which way you choose, the overwhelming feature that makes Vyatta nice is that pound for pound it’s much cheaper than Cisco or the other alternatives. And while we have plenty of Cisco gear, and in certain cases it makes sense to go with one of the bigger vendors, for smaller routing projects where you aren’t using a lot of the features Vyatta makes much more sense. That’s where we use it, and we can pass on that savings to our customers as a result. More importantly, we usually can buy more than 1 of the units to either run in active/passive mode or to keep a spare on the shelf for failure – for less than what we’d pay for a more name brand router.
I’m excited about the 6.1 release as it provides support for more IPV6, which is a must anymore, as well as some layer 2 bridging which is probably a result of the need for spanning multiple racks at a layer 2 level for cloud related projects.
Data Center Lightning Protection – Part 1
August 3, 2010 by caleb · Leave a Comment
One of the projects we focused on this summer was analysis and fortifying Data Cave’s lightning protection system. I wanted to share a little bit about what we’ve learned.
First, some background: our building is well protected from lightning strikes by what’s known as transient voltage surge suppresion (TVSS). This is similar to the surge protection strips most of us have at home that we plug our valuable electronics into. However, we have three fold protection within Data Cave:
- Incoming protection at the electrical mains. Our switchgear has built in TVSS on the incoming power lines, rated for 250kA. This technology basically clamps the line voltage in case of a spike, due to lightning or some other disturbance, and keeps it from entering further into the building.
- Electrical power that is being sent to the critical IT equipment goes through our flywheel UPS units. In these units we convert the AC power to DC power, then back to AC, as part of the rectification process. Any surges or sags get filtered out by these units.
- Our data suite Power Distribution Units also have TVSS units on them, which is what connects directly to the IT loads, in case any surges were somehow able to make it to this point.
One of the TVSS Modules protecting Data Cave's incoming power
So overall, we have great surge suppression. So imagine our surprise earlier this summer when a large thunderstorm came through, and a nearby lightning strike caused some damage to the hardware that controls the automated gates to the facility. In addition, we had some power over ethernet switches inside of our building also go “flaky”, for lack of a better term. This prompted us to investigate what was really going on, and what we could do to better protect ourselves (and our customers).
Our Findings
The fence and gates that surrounding our data center were in the process of being installed, but not finished, when the internals to the data center itself were commissioned. Something that was overlooked in this process was the protective grounding of these items from lightning strikes, which is a different beast than just grounding them from an electrical safety issue.
The fence posts themselves stick into the ground, and in general that means the fence is grounded. But in our condition, it’s not “well grounded”. The soil in this area is a bit sandy to a bit clay like, and thus to achieve a good earth ground we prefer to go 20 feet deep. The fence posts are only about 3 feet deep, which means they aren’t achieving as good of ground as they could be. To fortify this, we drove 20 foot copper ground rods in periodically along the perimeter of the fence, and bonded the post to the rod, to give better overall grounding.
Copper grounding of fence post
Grounding of the gate
Grounding of the gate
Other outdoor circuits
In examining the fence and gate, another vulnerable spot we found was in the underground conduit that goes from inside of our data center to the gates. Because the gates are electrically controlled, they need electrical power – which is provided from inside of our building via an underground conduit. In addition, our gates are key carded to allow customers and staff in upon a swipe, so data cabling was also necessary.
However, these two electrical lines now represent two new pathways for lightning to enter our building. While our incoming electrical lines to the facility are protected, if lighting was to strike the gate, it could go “backwards” through the wiring and back into our building in the opposite direction!
Circumventing this is easy. First, we installed transient voltage surge suppression devices directly at the electrical service at the gate, so in case of a lightning strike it would suppress any large voltages that may generate. Then we protected the data cabling by the use of inline ethernet surge suppression devices. These devices cause any voltage spikes that may be generated to be suppressed on the data lines as well, limiting any kind of backfeed into the building.
Ethernet surge suppression device
Conclusion
With the addition of better lightning protection of things outside of the building, we feel better equipped at mitigating future lightning strikes in the area.
In the part 2, we will highlight some of the existing lightning protection features of Data Cave and how they work to keep lightning away in the first place.
Metered Power
January 5, 2010 by caleb · Leave a Comment
Data Cave’s metered power offering can be a huge cost benefit to the end user. Read more
DCK Generator Post
December 31, 2008 by caleb · Leave a Comment
We had a nice acknowledgement of our generator installation video this morning on DCK.
Unfortunately, at the same time, our current host decided to do maintenance on the webserver. We currently are using an offsite host for web and email for the thedatacave.com domain, due to the fact that the data center is under construction (trust me, we’ll host ourselves internally soon). And I suppose you have to live with the ups and downs of someone else controlling your infrastructure.
This particular host has really given us nothing but grief. The outages are one thing, but the lack of explanation beyond “we’re working on it” and “it’s now up” is very frustrating. I want details. I want to know what’s wrong, what’s being worked on, and what their thoughts are on having it fixed. If you tell me 1 hour, and it takes 2, that’s ok. I just want to know 1hour vs. 24 hours, vs never. When I ask questions, all I ever get back is “no”, with no explanation.
Once you lose a customer, it’s hard to get them back. Any business without a firm grasp on good customer services is just shooting themselves in the foot these days. Trust me when I say that as a company, Data Cave will always put the customer first. We strive for perfection, but at the times we aren’t perfect, at the very least we’ll be transparent about it.
It’s no fun to have an upset customer, but you’ll have an appreciative one if you simply keep them informed.
Caleb
More Conduit
This past week, we finished up the conduit work.
All of the conduits run thus far were the services into the building, from the transformers and generators. These were the very deep conduits, as they have to go under the foundation of the building.
Once these were all in place, the process was to backfill. We backfilled to the 3 foot mark, and then left water to run overnight to harden the ground. The next day, the ground was so hard you coudn’t break it with a spade shovel. We felt pretty good about this level compaction.
The final step was to run the conduits necessary for the utility service – the primary high voltage lines that service the transformers. These are all outside, and are only at the 3 foot deep level. So these conduits were run next.
The generators also require some extra wiring for running pumps, and to control when they start and stop. These are run in smaller 1-2 inch conduits.
While this was happening, the site foundation received its stone base.
And by Friday, everything had returned to its normal level.
Electrical Conduit
The site has been very active lately!
We’ve been working on our electrical services. The electrical service for the building is massive. Like, mondo. Like, so big you can’t even fathom until you see it.
We’re starting in just one section and with one service (actually, two services via two transformers, but that can be routed by the internal switchgear as we want), but ultimately we’ll have four sets of these services.
This requires a lot of piping to get into the building. And it has to go under the foundation.
Where the pipe sticks up in the picture above is inside the building. It then runs out underground to the transformers outside of the building.
First steps, dig a BIG hole:
Also, buy a lot of pipe. I think for this trench work we bought just over 10,000 feet of 4″ PVC (that’s almost two miles). We also have somewhere around 4000 4 foot sweeping elbows. This thing is mammoth.
Now, start putting it all in place.
Each electrical service requires 8 conduits, in groups of four.
Here’s one group of four:
And then the next day we had a few groups of 8s installed
After the pipe is laid, we began to backfill:
From the other angle:
Another angle, showing the massive scope of the project:
Inaugural Blog
I’ve done a few posting so far over on my personal site, tarkblog, but we felt it was appropriate to differentiate Data Cave related goodies into its own blog – hence this site.
We will be posting more of the details surrounding the construction, design, and decision making of the building as we move forward here. Stay tuned.