The Recent Amazon Outage
July 10, 2012 by David Krider · Leave a Comment
Amazon’s Elastic Computing Cloud recently suffered an outage. Summarizing, Amazon’s explanation was that power to the facility failed, their generators failed to provide “stable” power, and energy within their UPS’s was exhausted before power could be restored. It only took about 20 minutes to fix the generators and get power back online, but as anyone who has supported large distributed systems knows, it’s not that easy. Getting services back online involves more than just restoring power, and made for serious interruptions for several popular web sites, such as Pinterest and Instagram. For instance, Netflix was down for three hours.
The Hacker News community discussed the event at length. There are a couple of takeaways that I would like to point out.
First, it’s not common for a data center — even a large data center — to have fully redundant power equipment. Data Cave does. We have two power feeds and two generators which all feed into electrical switch gear which can choose either utility power or generator power for either the “A” or “B” side. In addition, both “sides” are sized to run the entire demand alone, should the need arise, and the switchgear can feed both sides with any one of the 4 sources. Both sides then feed dedicated flywheel UPS systems, which, in turn, supply the A/B PDUs which then break down power by rack through breakers.
The main weakness in this setup is customer equipment. Most “enterprise”-level equipment has redundant power supplies, but they are not always connected. Also, it’s critical that the load gets split even across the A/B sides of power — and that no more than 50% of the breaker capacity is used on either side — so that if one fails, breakers aren’t tripped and power is lost when the load falls fully to the other side. At Data Cave, we help monitor this situation for our customers.
Second, it’s not common for data centers — even large ones — to have people on staff that really understand generators. Data Cave does. Our affiliate company, located right next door, is a world leader in high-end, large-displacement diesel engine testing. As such, we have, at our disposal, experts with both large diesel engines, and the large electrical motors they drive. If something goes wrong with either one, someone can fix it immediately. We don’t need to call a service company to come fix “their” equipment; these are not “black boxes” to us.
Data Cave was designed, from the ground up, to be a world-class data center, and the implementation of our systems would make a textbook of best practices. Our staff is filled with subject matter experts in everything relating to building and running a facility like this. If you’re in the market for a colocation or disaster recovery site, you owe it to yourself to come see the features of Data Cave for yourself.
The Cost of Being Unprepared
June 12, 2012 by Kara Manon · Leave a Comment
Halloween 2011, New England. Mother Nature provided more tricks than treats when she dumped a massive amount of snow and ice. 1.8 billion homes and businesses were without power for over a week. A Symantec Disaster Preparedness Survey estimated that downtime costs $12,500 a day for small to medium sized businesses. Potentially, businesses in New England could have lost at least $87,000, if they weren’t prepared.
Data is quickly becoming one of the most valuable assets a company has, but many companies don’t take the time, effort and money to be prepared and to protect their data. If you are a business owner or rely on data for smooth operation of your business, ask yourself the following questions:
- Do you have an emergency or disaster preparedness plan? 66% of businesses don’t. (Ad Council)
- Do you think it’s important to recover data after a disaster? 71% think it is important.
- Do you have any remote data backup? Even though 71% think it’s important, only 31% take action to back up their data.
Too many businesses are falling victim to myopic vision and the ever popular “That could never happen to me.” Company executives are focused on revenue, but forget about their backup plans. This limited thinking can (and will) cost them a lot of time and money.
Luckily, hindsight is 20/20. Past disasters, like last Halloween, have painted prime examples of other companies.
- Honda (and other Japanese car manufacturers) halted their supply chains and manufacturing processes for months after the tsunami in 2011.
- American Express suffered great data and equipment losses in the September 11th attack of the World Trade Centers.
In fact, in 2011 the U.S. broke its record for the cost of natural disasters at a staggering 1 billion dollars.
The risks are real, and your data and equipment is important. So what can you do to prevent loss and continue operating in the case of a disaster?
- Use off-site backup
- Use disaster recovery
- Have business continuity services available
It is crucial to move your data somewhere else. Diversify your data’s location to minimize your risk. By doing this, you will create your own insurance policy and ensure that your valuable data is protected.
At Data Cave, we’ve made it our business to be prepared for the worst-case scenario. We strategically chose an Indiana location with a low risk of natural disasters. Plus, we purpose built our data center to protect against the most common Midwest disasters – storms.We don’t only offer your data a safe home but we also have the option to provide you with an office, should yours become inoperable.
Nowadays, your information, data, and technology are incredibly valuable. Using Data Cave to protect these is your best insurance policy.
The Data Cave Advantage, Part 1
April 6, 2012 by Kara Manon · Leave a Comment
Here are just a few of the things that we think make Data Cave stand out in the crowd.
Purpose-built: If you’ve seen our facility, then you know, it was meant to be a data center. Many people who see the building say it looks like a prison, and they aren’t wrong. The building is nondescript and menacing, as it should be. We want to keep the general public out of the facility for the security of our customers. No one should be around the building that isn’t employed by or working with Data Cave. Our customers feel safe when they visit and know their equipment is safe. Not only that, but because we custom-built the facility, we purposely installed 12-inch thick block around the perimeter of the data center. The rebar that continues through the middle of each block is nearly an inch thick and the blocks are filled with concrete. The rebar continues down into the ground and up into the roof. Too many data centers out there were retrofitted into existing buildings and don’t provide the safety and privacy that we believe a data center should have in order to give customers maximum protection.
Privately Owned & Operated: This speaks for itself. We own the land and the building. There are no liens on the property and we don’t have anyone to answer to, including a board of directors. A lot of data centers lease space which could potentially cause future problems when lease terms end. We also have continuity here. The same people who built Data Cave are still around today. Our facility manager has been working at Data Cave since the beginning and helped actually build Data Cave.
The Roof: We’ve blogged about this before but it’s pretty impressive so we’ll say it again: Our roof weighs approximately 4.5 MILLION pounds. This is fairly jaw dropping. As you can see from the above photo, no equipment sits on the roof and we have no roof access. We built Data Cave this way to mitigate any potential leaks that could occur. The rebar mentioned earlier actually helps hold the roof down in the event of high winds or tornadoes. You can read more here.
Data Suites: Instead of the traditional data center with one big room where customers have to cage their equipment, Data Cave is split into individual 1,300-square-feet data suites. We chose that size to minimize hotspots and provide a higher level of privacy and security for customers. This layout allows one client to lease a suite which only they have access to, at any time. One of our data suite customers must comply with HIPAA and HITECH acts, which means everyone going in and out of the suite is monitored. We send the customer a report at the end of each month listing who has been in the area. That’s just one of the suites that our staff stays out of unless absolutely necessary. This setup really benefits companies needing to comply with certain regulations.
Come see for yourself the advantages of choosing Data Cave. Call us at 866.514.2283.