9

One of the sites I work with has recently started to get DoS'd. It started out at 30k RPS and now it's at 50k/min. The IP's are pretty much all unique, not in the same subnet, and are in multiple countries. They only request the main page. Any tips on how to stop this?

The servers are running on Linux with the Apache as the web server.

Thanks

Improve this question
2
  • What kind of traffic is it? Have you identified what kind of DDoS it is? ie, is it consuming your bandwidth or is it consuming your system resources?
    – Josh Brower
    Commented Dec 29, 2009 at 18:29
  • This is a great question but apparently there is no real answer. Wow, I never knew that the DoS brick wall was so thick.
    – Xeoncross
    Commented Jun 13, 2010 at 20:51

4 Answers 4

Reset to default
4

You're not just trying to withstand a DoS, you're trying to withstand a DDoS, which is distributed and much more difficult to deal with.

Essentially, you're trying to identify illegitimate traffic and block them. Ideally, you want to null route this traffic (even better get your upstream providers to null route it.)

The first port of call is identification. You need to find some way to identify the traffic that is being sent to your host. Whether it's a common user agent, whether it's the fact that they're not actually using a proper browser (HINT: do they act like proper browsers - i.e. follow 301 redirects), whether all requests flood in at the exact same time or by how many requests each IP is hitting your server per hour.

You cannot block them without identifying them and you need to find some way of doing that.

Those DDoS mitigation tools essentially do the same thing, except in real time and cost a bomb. Half of the time there's false positives or the DDoS is so big it doesn't matter anyways, so be careful where you put your money here if you do decide to invest in one of them either now or in the future.

Remember: 1. IDENTIFY 2. BLOCK. 1 is the hard part.

Improve this answer
4
  • 1
    The problem isn't blocking, the problem is identifying. You can't block something if you can't identify it. So far we've seen no patterns at all. Real browsers, no patterns in request times, completely different countries, no referrer, they follow redirects, they accept cookies. They act just like normal users. This looks like it's almost impossible to tell. We're thinking of routing all traffic to Amazon, have Amazon handle all the requests for the homepage that will be cached, and all other pages handled by our web app for now. Thanks for the answer though.
    – William
    Commented Dec 29, 2009 at 17:43
  • Minor correction: they're probably not real browsers, keep that in mind when working on identification. Also, what's your user-base look like? If it's all US-centric, you might want to block offshore requests as a stopgap to buy you a little bit of breathing room...
    – pboin
    Commented Dec 29, 2009 at 17:48
  • They're not "real" browsers in a sense that they're using Firefox, Chrome, etc for their requests. One thing you'll notice is how I said these are unique IP's, running for hours, at that high of a RPS. This "person" has a HUGE botnet apparently, even our data center (ThePlanet) can't figure out a way to stop it either. It's not very easy to tell if it's a browser or not. If it follows redirects, stores cookies, etc how do you tell? Besides you need to remember something, each request is unique. So banning an IP means nothing. The requests need to be blocked before they hit our server.
    – William
    Commented Dec 29, 2009 at 18:05
  • Non-browsers, or text based browsers don't tend to run javascript? What user agent header are they supplying also?
    – Philip Reynolds
    Commented Dec 29, 2009 at 18:58
1

You're assuming that this is an intentional DDoS. The first thing to try is changing the IP address. If it's not in fact intentional, then it will stop.

Where would these requests be coming from if it's not intentional? It could be random, or it could be a mistaken target. Unlikely, but worth a try.

Are you sure you're not just getting loads of legitimate traffic? Maybe you've been slashdotted, or something. Try looking at the referrers in the logs.

Improve this answer
0

Does your front-end router/load-balancer not have DOS-attack management? Ours does and it makes a world of difference.

Improve this answer
2
  • The problem is, ALL ip's are unique, from different countries, etc. There is really no way to tell the attacker from a legitimate user. ALL of our bandwidth is being eaten up right now, we can'd do anything.
    – William
    Commented Dec 29, 2009 at 17:08
  • But DOS-managing routers and load-balancers don't care where traffic comes from, if they see a lot of a certain type of DOS-related traffic from certain IP's then they ignore it and carry on with their work regardless, allowing servers to server and customer traffic to be treated correctly. People like Cisco and Foundry make a lot of money from their work in this area and what you're seeing isn't out of the ordinary at all.
    – Chopper3
    Commented Dec 29, 2009 at 19:31
0

You can ask your upstream provider to ask their upstream to assist. Lets say for instance that you run a website with UK users only. Then you can check where the traffic in general originates from using some whois database. Lets say for instance that a significant amount of your unwanted traffic happens to originate from russia, china and/or korea. Then you can call up your upstream provider and have them call theirs to have them nullroute your IP addresses temporarily from these areas, assuming they have routers close to the sources.

THis isnt a long term solution but it helps if your userbase is clustered in a few geographical areas. In the past Ive helped customers like this, simply not announcing them to foregin peers, just national ones. THis did take some of their business away (users that found them unreachable because they werent available internationally anymore) but its alot better than just beeing out of service alltogeather.

But at the end of the day this is more of a desperate act. But its better to cut of a limb than loose the body.

If youre in luck your upstream providers provider has the equipment and are willing to help you filter most of the undesired traffic away.

Good luck :-)

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .