0

Hi All,

For the last 6 months our site has been under severe brute force, syn flood attack. They keep bombarding a single URL of the server and it is xml file. They are not attacking any other URL.

e.g. http://www.example.com/rss123/attackedfilename.xml

We have removed the xml page from our site but still they keep on sending requests, this is for the last 6 months non stop.

The IP has been changed just to see and they are sending several thousand requests per second. The requests come from different IPS and different ranges, so you can not even block the IP’s. They seem to be coming from a legitimate IP’s.

Due to this I have had to pay for an extremely expensive server which holds 8 GB of RAM and quad core processor etc, however, even with this the server server still reaches critical level, just because these requests are eating up my resources.

Our technical team has been working on all aspects of apache server security, external modules, firewall, hardware firewall from beginning but still we are not able to stop them.

We have installed following modules.

1) mod_security
2) mod_evasive
3) Firewall
4) SYS_Cookies enabled

We have worked with the hosting company and their technical team leader, he installed the best CISCO hardware firewall and tried to stop them, but in vain.

We have checked our server to see if anything from our site is causing the request, no extra file uploaded on to the server. For example if some file has been upload or some text has been added to the file (checked if we’ve been hacked). Even though we checked for any hacks, I am still wondering if there is something we do not know about. Can a hack lead to huge amounts of traffic?

We need some help to stop these attacks. We have searched a lot and have found that sites that get attacked like this have only one option is to shut down till it stops. I really hope that will not be the case for us. Please let us know if any one has any ideas to deal with this.

We are willing to try any small suggestion which might help from coding to scripting to modules to firewall. So please provide suggestion/solutions and way to get us out of this.

Also could it be our own part of php code which can do this? We are ready to check every php file to make sure it does not have any line of code which can be dangerous?

We worked with hardware firewall company to drop a request on the spot coming for the single URL but it is getting setup.

We have antivirus running on server however if any specific antivirus or antimalware is needed, we can try that.

Following are the details I have got from my linux admin. This will help you to trace the issue in better way.

Problem: Apache SYN_RECV

OS - RHEL5
kernels - 2.6.18-92.1.22.el5-x86_64
              2.6.18-92.el5-x86_64

rpms:-
kernel-devel-2.6.18-92.el5
kernel-headers-2.6.18-92.1.22.el5
kernel-devel-2.6.18-92.1.22.el5
kernel-2.6.18-92.1.22.el5
kernel-2.6.18-92.el5

OS Type:
cat /etc/issue
Red Hat Enterprise Linux Server release 5.2 (Tikanga)
> cat /proc/version
Linux version 2.6.18-92.1.22.el5 (mockbuild@hs20-bc2-5.build.redhat.com) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)) #1 SMP Fri Dec 5 09:28:22 EST 2008

We are providing 403 code for the URL request. Following is part of Access Log

94.70.118.139 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; FunWebProducts; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; CT2077543_4.5.188.7)"
89.216.230.148 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; sr; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10"
82.81.54.226 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB5; CT2077543_4.5.191.15)"
85.229.15.86 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6; .NET CLR 2.0.50727; CT2088752_4.5.188.7)"
92.237.189.17 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10"
84.106.127.218 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; eSobiSubscriber 2.0.4.16; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; InfoPath.2; .NET CLR 3.5.30729; .NET CLR 3.0.30618; CT2088433_4.5.188.7)"
87.93.30.98 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30618; CT2088347_4.5.188.7)"
93.86.61.247 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.0" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.2; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; CT2077543_4.5.188.7)"
91.152.228.27 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; Creative ZENcast v2.00.13; CT2088347_4.5.188.7)"
94.69.164.32 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; FunWebProducts; .NET CLR 1.1.4322; .NET CLR 2.0.50727; CT2088700_4.5.191.15)"
82.201.180.177 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB6; CT2077543_4.5.189.28)"
83.248.2.230 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SIMBAR={46BC3752-9118-483D-8E88-CD3E89FCD192}; GTB6; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; CT2088752_4.5.188.7)"
99.235.137.30 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6; .NET CLR 2.0.50727; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; CT2077543_4.5.191.15)"
216.155.136.84 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.0.04506; .NET CLR 1.1.4322; CT2077543_4.5.188.7)"
217.123.166.205 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB5; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.2; .NET CLR 3.5.30729; .NET CLR 3.0.30618; CT2088433_4.5.188.7)"
86.96.227.88 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; CT2077543_4.5.188.7)"
203.115.189.77 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; CT2077543_1.5.48.2; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10"
203.82.79.102 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; CT2088347_4.5.188.7)"
88.195.52.126 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET CLR 3.5.30729; .NET CLR 3.0.30618; CT2088347_4.5.189.24)"
77.81.114.171 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6; SIMBAR={B471FCBA-22ED-11DE-91A3-00196693641D}; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; CT2077543_4.5.191.15)"
92.84.250.65 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; CT2077543_4.5.191.15)"


netstat:

tcp        0      0 domain.com:http   85-156-91-20.elisa-mo:55168 SYN_RECV
tcp        0      0 domain.com:http   220.255.7.227:27183         SYN_RECV
tcp        0      0 domain.com:http   5e03cbc4.bb.sky.com:51086   SYN_RECV
tcp        0      0 domain.com:http   79.126.234.198:18139        SYN_RECV
tcp        0      0 domain.com:http   78.148.175.148:11115        SYN_RECV
tcp        0      0 domain.com:http   83-154-143-68.rev.lib:61479 SYN_RECV
tcp        0      0 domain.com:http   ABTS-North-Static-248:54775 SYN_RECV
tcp        0      0 domain.com:http   90-230-131-95-no130.tb:1134 SYN_RECV
tcp        0      0 domain.com:http   static-host119-73-6-2:49538 SYN_RECV
tcp        0      0 domain.com:http   222.127.130.238:gtp-control SYN_RECV
tcp        0      0 domain.com:http   acl1-1571bts.gw.smartbr:g5m SYN_RECV
tcp        0      0 domain.com:http   athedsl-282427.home.o:60002 SYN_RECV
tcp        0      0 domain.com:http   CPE-58-166-77-138.nsw:60067 SYN_RECV
tcp        0      0 domain.com:http   C-59-101-99-107.syd.c:51097 SYN_RECV
tcp        0      0 domain.com:http   ti0111a380-2667.bb.on:60993 SYN_RECV
tcp        0      0 domain.com:http   92.81.2.242:60451           SYN_RECV
tcp        0      0 domain.com:http   118.100.120.248:pserver     SYN_RECV
tcp        0      0 domain.com:http   triband-del-59.178.84:50140 SYN_RECV
tcp        0      0 domain.com:http   cpc4-leds5-0-0-cust82:obrpd SYN_RECV
tcp        0      0 domain.com:http   ALyon-153-1-8-78.w86-:59494 SYN_RECV
tcp        0      0 domain.com:http   120.28.199.183:3comnetman   SYN_RECV
tcp        0      0 domain.com:http   h248.4.16.98.dynamic.:60758 SYN_RECV
tcp        0      0 domain.com:http   89.211.205.59:64217         SYN_RECV
tcp        0      0 domain.com:http   CPE-124-187-26-30.qld:ff-sm SYN_RECV
tcp        0      0 domain.com:http   frw.Gloworld.com:59104      SYN_RECV
tcp        0      0 domain.com:http   220.255.7.182:winpoplanmess SYN_RECV
tcp        0      0 domain.com:http   srisaionline180.excell:1232 SYN_RECV
tcp        0      0 domain.com:http   CPE-60-230-16-150.vic:52611 SYN_RECV
tcp        0      0 domain.com:http   203.82.91.102:41318         SYN_RECV
tcp        0      0 domain.com:http   69.171.165.50:32454         SYN_RECV
tcp        0      0 domain.com:http   dsl-TN-static-195.:corbaloc SYN_RECV
tcp        0      0 domain.com:http   210.186.66.179:49330        SYN_RECV
tcp        0      0 domain.com:http   ABTS-North-D:xinuexpansion3 SYN_RECV
tcp        0      0 domain.com:http   c122-106-133-46.livrp:49273 SYN_RECV
tcp        0      0 domain.com:http   173.subnet125-1:nssalertmgr SYN_RECV
tcp        0      0 domain.com:http  121.246.52.30.dynamic:63977 SYN_RECV
tcp        0      0 domain.com:http   mobile-3G-dyn-BC-179-1:4464 SYN_RECV
tcp        0      0 domain.com:http   crd48.neoplus.adsl.t:aminet SYN_RECV


Following we have done till now is mentioned below for the configurations.

###############
 sysctl.conf 

##############
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536

# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
# Enable IP spoofing protection, turn on Source Address Verification
net.ipv4.conf.all.rp_filter = 1
# Enable TCP SYN Cookie Protection
net.ipv4.tcp_syncookies = 1

# 65536 seems to be the max it will take
net.ipv4.ip_conntrack_max = 1048576
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 87380 8388608
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_window_scaling = 1


#############
fwsnort, bfd burnintest  chkrootkit  ddos  faf lsm  nobody_check  sim apf 

#############
 modsecurity-apache

LoadModule evasive20_module   /usr/lib64/httpd/modules/mod_evasive20.so

<IfModule mod_evasive20.c>
    DOSHashTableSize    3097
    DOSPageCount        3
    DOSSiteCount        50
    DOSPageInterval     1
    DOSSiteInterval     1
    DOSBlockingPeriod   30
</IfModule>

LoadModule security_module    /usr/lib64/httpd/modules/mod_security.so

<IfModule mod_evasive20.c>
    DOSHashTableSize    3097
    DOSPageCount        3
    DOSSiteCount        50
    DOSPageInterval     1
    DOSSiteInterval     1
    DOSBlockingPeriod   30
</IfModule>

####################

Again,
Hope this helps you to see the issue in detail. I have also put the latest configurations to keep site going on.
Besides all these details, please let me know for any suggestion you think will be helpful or details you want.
Thank you for your help in advance! Help!

Regards,

Sam

Edited by Reverend Jim: Fixed formatting

2
Contributors
5
Replies
6
Views
8 Years
Discussion Span
Last Post by phpfreek2007
0

Hi,

Any idea regarding specific firewall which can stop the request for specific URL at entry point?

Sam

0

Hi,

Just finished checking full server again, no unintentional file or code on the server.

I will see for this option to check the server for spyware or virus and malware.

Also, anyone has idea regarding the Firewall which drops request at entry point for specific URL request? Currently we have tried are IP and pattern based only to slow down the attack, however, they are being smarter and keep generating new bunch of IP address.

Sam

0

Hi all,

Scanned the server with rootkit antispyware, no infection found. Regarding the firewall, put on BFD firewall over APF, still requests are not getting down.

Also IP table is getting full of new ips and it keeps network slow. Please advice.

IS there a way like we can set some intermediate setup which directs these request to some dummy server and we do not get actual request on our actual server?

Sam

0

Hi,

The issue is so serious and complex that I need to post it to several forums.

However, can you please help me in solving the issue? I have tried a lot and still not getting any way closer to the solution.

Thanks in advance.

Sam

This topic has been dead for over six months. Start a new discussion instead.
Have something to contribute to this discussion? Please be thoughtful, detailed and courteous, and be sure to adhere to our posting rules.