On July 12th a major power event occurred at the prominent carrier hotel at 56 Marietta Street responsible for network interconnections for a significant portion of the southeast including over 60 carriers and over 100 telecom providers. The incident occurred during planned power distribution upgrades and created quite a number of issues for both regional and national organizations.
Digital Realty, the owner of the building which it acquired from Telx a number of years ago, released the following statements direct to customers, but made no mention of the incident on Twitter where they typically post multiple times per day:
During scheduled maintenance, the engineering team at 56 Marietta reports that the generator failed resulting in a loss of critical IT load to equipment in the space. The site team is investigating the source of the issue and scope of the impact. Further updates will be provided as additional information becomes available.
As an update to the Generator failure at 56 Marietta, the engineering team reports that power has been restored and all power systems have been normalized. A RFO, that will outline the root cause as well as the scope of the event, will be published. Further updates will be provided as they become available.
Several customers on a popular outage list noted that both their A & B redundant power feeds went down at the same time for both A/C and D/C, proof that even some of the largest and most well funded organizations can suffer significant downtime without the proper protocols, procedures and failsafes in place.
Total Uptime systems detected over 100 client devices unreachable as a result of this event. Those subscribing to our Cloud Load Balancer or DNS Failover Service had their traffic automatically rerouted to alternate locations outside of the area based on customer configurations.
Ironically after writing a blog post earlier today commenting on how ISPs are not the best organization to handle DNS because it is not their core competency, AT&T suffered a significant DNS outage that affected many customers across the United States for many hours. Network World has a few more details as does ComputerWorld. AT&T did not release […]
In 1959 there was a fire in the Pentagon. It resulted in seven million dollars’ worth of damage, taking out three mainframe computers. In today’s currency that would be $58.1 million. The National Fire Protection Agency (NFPA) was then tasked with creating rules and regulations to manage risks in computer environments. They came up with […]
A service provider that offers software-as-a-service or another cloud-based solution should understand what customers are looking for and what compels those very customers to choose an off-premise, “cloud-based” solution vs. the more traditional on-premise, self-hosted solution. As a cloud service provider ourselves, we set out to understand how our customers went about choosing one service […]
Redundancy is indispensable in the world of information technology. Of course, redundancy is not welcome in every aspect of life. If your company doesn’t need you anymore and makes you “redundant”, you’ll have to look for another job. Poorly written text may be credited to the Department of Redundancy Department. The concept of redundancy is […]