If you heard or felt Internet outages today, it wasn’t just you. It seems many end-users were filing complaints on downdetector.com for a vast number of unrelated ISPs. From our global vantage point, we could see that the Internet of things was acting strange. There may have been a number of small unrelated events that contributed to the chaos, but what really seems to have triggered the vast amount of routing issues we saw was the fact that the IPv4 BGP routing table grew beyond 512k routes today as seen here.
At Total Uptime, our infrastructure sits in many datacenters around the world and traverses most networks, so we have a unique vantage point into Tier-1 and peering networks that many do not. We first noticed strange behavior when a Level3 link in Washington DC seemed to dead-end traffic shortly before 5:00 AM EST. This was non-impacting for us because we connect to many providers and can seamlessly reroute traffic, but this was followed by a lot of routing changes all morning and then in the early afternoon we saw similar behavior with level3 out of San Jose.
Overall during the day, it seems routes were changing significantly as various network providers worked to resolve this issue as quickly as possible. Steven Vaughan-Nicols at ZDNet seems to have written the best article we’ve seen thus far, and it is worth the read. Hopefully today’s events were significant enough to have flushed out the issue, but insignificant enough to have kept the pain to a minimum. Fortunately for the customers we protect with our Cloud Load Balancer, it was business as usual.
On July 12th a major power event occurred at the prominent carrier hotel at 56 Marietta Street responsible for network interconnections for a significant portion of the southeast including over 60 carriers and over 100 telecom providers. The incident occurred during planned power distribution upgrades and created quite a number of issues for both regional […]
We came across an article today that was written almost a year ago in Computer World that we all found interesting (and affirming of what we do). A new vendor-sponsored study of the 1,000 biggest websites quantifies how much better third-party managed service providers are doing at operating DNS services across the Internet than enterprises that run […]
In the early days of the Internet, humans and other computers located the few massive interconnected computers the same way: by their numeric Internet Protocol (IP) addresses. Soon, the impracticality of memorizing all these numbers became obvious and a rudimentary naming scheme was developed. A central repository of names and their associated IP addresses was […]
Ironically after writing a blog post earlier today commenting on how ISPs are not the best organization to handle DNS because it is not their core competency, AT&T suffered a significant DNS outage that affected many customers across the United States for many hours. Network World has a few more details as does ComputerWorld. AT&T did not release […]