I just updated Ubuntu from 12.04 to 14.04 in a xenserver vm and ran into this error:
xsshotStarting VM
Using <class ‘grub.GrubConf.Grub2ConfigFile’> to parse /grub/grub.cfg – WARNING:root:Unknown directive load_video – WARNING:root:Unknown directive terminal_output – WARNING:root:Unknown directive else – WARNING:root:Unknown directive elif – WARNING:root:Unknown directive else – WARNING:root:Unknown directive else – WARNING:root:Unknown directive else – WARNING:root:Unknown directive else – WARNING:root:Unknown directive export – WARNING:root:Unknown image directive recordfail – WARNING:root:Unknown image directive gfxmode – WARNING:root:Unknown image directive recordfail – WARNING:root:Unknown directive submenu – WARNING:root:Unknown image directive recordfail – WARNING:root:Unknown image directive gfxmode – WARNING:root:Unknown image directive recordfail – Traceback (most recent call last): – File “/usr/bin/pygrub”, line 850, in ? – raise RuntimeError, “Unable to find partition containing kernel” – RuntimeError: Unable to find partition containing kernel

So what’s going on here?

This happens because after updating your kernel, grub adds a couple of lines. If this was a regular machine you would just see those added like in the image below.

sshot169

But now that the grub has these extra entries Xenserver sometimes fails here and results in the above grub error. You could fix this manually by mounting the bootloader (xe-edit-bootloader -n your_vm_name -p 1 ) and then comment out every thing under the “Previous Linux Versions” or after doing an upgrade running a purge on previous kernel versions with a fancy command like (dpkg -l ‘linux-*’ | sed ‘/^ii/!d;/'”$(uname -r | sed “s/\(.*\)-\([^0-9]\+\)/\1/”)”‘/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d’ | xargs sudo apt-get purge -y )

But that won’t fix anything in the future. Do it the proper way and fix your xenserver. With the following.

1. Backup your /usr/lib/python2.4/site-packages/grub/GrubConf.py

2 Replace it with this working version:

GrubConf.py

3. Blamo! Your machines may upgrade now and your broken ones will boot!

 

Banning China DDOS with HAProxy and fail2ban

I awoke this morning to alerts that the load on my application servers were skyrocketing. Checked the access logs and saw a bucket of GET requests with Bittorrent in the name. Here’s some of the logs I saw.

/var/logs/apache2/access.log
111.192.44.32 - - [23/Jan/2015:03:21:13 -0800] "GET /announce?info_hash=%05%06%2F%83%2F%8C%F6%0C%8C%26%17%C4%DF%8Fk%7E%17%06%C2%D0&peer_id=%2DSD0100%2D%3C%D0v%DF%04%0FR%88JE%85%EF&ip=172.16.93.62&port=10426&uploaded=405633701&downloaded=405633701&left=349640112&numwant=200&key=23928&compact=1 HTTP/1.0" 403 507 "-" "Bittorrent"
101.22.73.220 - - [23/Jan/2015:03:21:59 -0800] "GET /announce.php?info_hash=%EFyT%86%B5A%99%E5%5BH%F9U%A6%95%B7%F5r%27%CE%CE&peer_id=%2DSD0100%2D%22%CA%29%86%1D%82%E6%00%FC%7B%C9%F4&ip=101.22.73.220&port=27399&uploaded=58982400&downloaded=58982400&left=138369872&numwant=200&key=7620&compact=1 HTTP/1.0" 403 511 "-" "Bittorrent"
118.114.177.119 - - [23/Jan/2015:03:22:01 -0800] "GET /announce.php?info_hash=ROIFGKj%BE%DF%15%EA%BDG%E9%A508%11%BC%7E&peer_id=%2DSD0100%2D%90%29%02%13%01%E8P%AA%07%FB%CD%99&ip=192.168.1.102&port=8263&uploaded=84324757&downloaded=84324757&left=1008939937&numwant=200&key=26946&compact=1 HTTP/1.0" 403 512 "-" "Bittorrent"

Thank Zuul for HAPROXY keeping my site up. When one application server dropped out, another was still there and they just kept getting shot down. How I stopped it was the following.

1 . Enable local logging on HAProxy. /etc/haproxy/haproxy.cfg
In the Global section add:

log 127.0.0.1 local5 info

Place the below under the Default section:

option httplog clf

In the /etc/rsyslog.conf add this line to the end of the file.

local5.* /var/log/haproxy.log

and un-comment or place following in rsyslog.conf:

$ModLoad imudp
$UDPServerRun 514
$UDPServerAddress 127.0.0.1

2. apt-get install fail2ban. Add the following to the bottom of /etc/fail2ban/jail.conf

[dnspoison]
enabled = true
port = http,https
protocol = all
filter = bittorrent-block
banaction = iptables-allports
logpath = /var/log/haproxy.log
maxretry = 1
bantime = 604800

3. Create filter.d file: /etc/fail2ban/filter.d/bittorrent-block.conf

[Definition]
failregex = .*]: <HOST> .*GET \/announce.*hash.*
ignoreregex =

4. reload
service haproxy reload
service fail2ban restart
services rsyslog restart

5. watch the banning begin:
tail -f /var/log/fail2ban.log

ban_hammer

Banning China DDOS with just fail2ban

I awoke this morning to alerts that the load on my application servers were skyrocketing. Checked the access logs and saw a bucket of GET requests with Bittorrent in the name. Here’s some of the logs I saw.

/var/logs/apache2/access.log
111.192.44.32 - - [23/Jan/2015:03:21:13 -0800] "GET /announce?info_hash=%05%06%2F%83%2F%8C%F6%0C%8C%26%17%C4%DF%8Fk%7E%17%06%C2%D0&peer_id=%2DSD0100%2D%3C%D0v%DF%04%0FR%88JE%85%EF&ip=172.16.93.62&port=10426&uploaded=405633701&downloaded=405633701&left=349640112&numwant=200&key=23928&compact=1 HTTP/1.0" 403 507 "-" "Bittorrent"
101.22.73.220 - - [23/Jan/2015:03:21:59 -0800] "GET /announce.php?info_hash=%EFyT%86%B5A%99%E5%5BH%F9U%A6%95%B7%F5r%27%CE%CE&peer_id=%2DSD0100%2D%22%CA%29%86%1D%82%E6%00%FC%7B%C9%F4&ip=101.22.73.220&port=27399&uploaded=58982400&downloaded=58982400&left=138369872&numwant=200&key=7620&compact=1 HTTP/1.0" 403 511 "-" "Bittorrent"
118.114.177.119 - - [23/Jan/2015:03:22:01 -0800] "GET /announce.php?info_hash=ROIFGKj%BE%DF%15%EA%BDG%E9%A508%11%BC%7E&peer_id=%2DSD0100%2D%90%29%02%13%01%E8P%AA%07%FB%CD%99&ip=192.168.1.102&port=8263&uploaded=84324757&downloaded=84324757&left=1008939937&numwant=200&key=26946&compact=1 HTTP/1.0" 403 512 "-" "Bittorrent"

How I stopped it with just fail2ban was the following.

1. Install fail2ban.

sudo apt-get update
sudo apt-get install fail2ban

2. Add the following to the bottom of /etc/fail2ban/jail.conf

[bittorrent]
enabled = true
port = http,https
protocol = all
filter = bittorrent-block
banaction = iptables-allports
logpath = /var/log/apache2/access*.log
maxretry = 1
bantime = 604800

3. Create filter.d file: /etc/fail2ban/filter.d/bittorrent-block.conf

[Definition]
failregex = <HOST> .*GET \/announce.*hash.*
ignoreregex =

4. reload
service fail2ban restart

5. watch the banning begin:
tail -f /var/log/fail2ban.log

ban_hammer