[Wiretap]

Hey guys, I'm sure at least some people noticed the issues we were having this morning (I'm surprised that nobody said anything). It appears our hosting provider suffered a large DDoS attack which took down not just DOBC but thousands of other sites too. Below are the updates released by the provider with regards to the attack:

Quote

Major System-Wide Outage


We are investigating the cause and will update this post as soon as we know more.

Update 2:54pm PST: One of our network providers is having issues right now. Our network admins are working on moving all traffic to our two other service providers. We will keep updating this post as soon as we have more information. -Oscar

Update 3:10pm PST: We are having issues with one of our transit links to our LAX datacenter, our network administrators are investigating the issue and we hope to have it resolved as soon as humanly possible. This is causing problems with both external and internal networking – we’ll update this post as soon as we have more information or the issue is resolved. -John

Update 3:30pm PST: Our network engineers are on site at the source of the issue and working on resolving it. More information to follow as it becomes available. -John

Update 3:50pm PST: We are still working feverishly to resolve this issue – some portions of our network are responding better now but there’s more work to be done. -John

Update 4:10pm PST: We are seeing improvement but are still working on this issue. -John

Update 4:40pm PST: Our network engineers rebooted the core router at our LAX facilities but our providers are still flapping. It appears that the LAX facility may be stabilized but we’re experiencing high load on our core routers as a result of all the BGP flapping. We are still investigating the issue but current traffic patterns possibly indicate an attack on our network. -John

Update 5:00pm PST: We are still working to defeat this attack on our network. Nullifying certain source IPs has seemed to help matters, but that attack vectors and profiles are still being introduced and identified. -John

Update 5:25pm PST: Some attacks have been blocked that were inbound and we have identified some that were outbound and blocked those as well. We’re still working on the matter and the network responsiveness is improving but the attacks are still causing BGP flapping. -John

Update 5:50pm PST: We’ve pinned down the sources and nature of the attacks and are in the process of scanning for more attack software as well as filtering bad traffic at the core routers. The issue is not completely resolved yet but we’re making good progress now. More updates to follow. -John

Update 6:15pm PST: In a nutshell we suffered an extremely sophisticated attack. It took a while to get things under control enough to see what was going on and then start not only blocking attack vectors but track down and disable software being used to launch attacks from our network as well. Things are almost under control currently and once they are we’ll get a full and detailed report from our network engineers for you. -John

Update 6:35pm PST: Good news – all inbound attacks appear to now be screened and the CPU levels on our networking equipment is currently stable. We’ve strengthened our defenses at all levels of our network as well – borders are hardened, cores are protected, transit interfaces are stable. That said, we’re remaining vigilant as these attacks are much more advanced than what we’ve seen previously. -John

Update 6:50pm PST: All indications are that everything is under control so we are marking this matter as resolved. We will still provide details on what happened as soon as we’re able to do so. -John

Update 7:12pm PST: It’s back.. we’re on it again.. more updates soon. -Josh

Update 7:31pm PST: It seems that we’re seeing issues with our transit links to our Alchemy datacenter. Our network engineers are working on the issue and we will update this as soon as we have more information. – Oscar

Update: 8:02pm PST: We are seeing some more interruptions in networking, our admins and network engineers are still looking into the issue. – Justin

Update: 8:25pm PST: Our abuse team is getting involved as well at this point. We have resolved issues on most of our routers, and have found over 500 compromised sites so far, that we are working on fixing. Overall things should improve over time. -Justin

Update: 8:53pm PST: Things once again seem to be resolved.. we’re going to be cautiously optimistic and say nobody should be seeing any network issues at this time. We’re still going through squashing bad guys here and there, but the system is now sound. We’ll update this post if anything changes. -Josh

As I was posting this:

Quote

Update: 9:14pm PST: I spoke too soon again.. we’re still on it. -Josh

[Shale]

Why the fuck is some little punk doing a DDoS attack on the host? -______- wankers.

[boostin]

For the uninitiated... WTF is DDOS??

[Wiretap]

Distributed Denial of Service,

Essentially a 'botmaster' takes control of a whole bunch of compromised (read, virus/spyware infected) computers and directs them to send bad data by the truckload at a specified destination... If it doesn't nail the devices on the other end into falling over, it at the very least makes it difficult if not impossible for genuine users to access the service.

[boostin]

Ah, right you are! English is so much easier to understand!!

[Oscar]

I be not affected by the new fail. First time that Vodafone did something right.

[Shale]

Was just Xtra. Seems to have fixed itself. I have NFI WTF happened.

[chrisjunkie]

If I understand it correctly:

Basically the way that Telecom's internet data flows into the place that hosts this site was a different route to other people. This route had problems hence the Telecom users experiencing problems.

Think of it as finding two ways to a destination. One road is closed (Telecom) and the other road is running fine (other people)

Wiretap can correct me if I'm way out of line though I think that is basically right