i installed the akismet anti-comment-spam plugin for wordpress a few days ago. so far it is working GREAT. i was getting pretty sick of tagging every spam entry in my moderation queue (there were a LOT), all while keeping an eye on legitimate comments.

highly recommended!

i spent most of tuesday messing around with various freeBSD and lighttpd settings. even though netstat -m didn't report insufficient buffers, i was getting no buffer space errors in my log files. so, i upped kern.ipc.nmbclusters from 6000-ish to 16000 and rebooted.

the errors continued. however, netstat -an counted more connections before dying than before. hmmm…

netstat also revealed that many, many, many of those connections were hanging around in the TIME_WAIT state. i googled that and found some articles about MSL and socket connections… to be honest i didn't understand it all, but it seemed that it was possible to reduce the amount of time that sockets remained in the TIME_WAIT state by changing the value of net.inet.tcp.msl. on freeBSD 4.10 the default value is 30000 (30 seconds), so i reduced it to 7500 (7.5 seconds). i understand that lowering it too much can have other consequences, like having old data from an old connection getting mixed up in a current stream?? i admit i am a TCP/IP newbie.

anyway, i changed that value, and restarted the server. so far so good… the server seems to be handling the load quite nicely now, with no connection failed/buffer space errors. i don't know if increasing nmbclusters was necessary, but since things are working ok now i am just going to leave it there for now

lighttpd is great (serves a lot, quickly), but under very heavy load it seems to act rather erratically. i'm not sure where the fault lies, though: in lighttpd, in fast-CGI (which is how lighttpd runs PHP), or in freeBSD, or some combination.

the symptoms are error messages like this:

2005-12-19 10:11:52: (mod_fastcgi.c.1561) connect failed: 242 Connection refused 61 0 /tmp/php-fastcgi.socket-0
2005-12-19 10:11:52: (mod_fastcgi.c.2663) socket failed: No buffer space available 263 4086
2005-12-19 10:11:52: (mod_fastcgi.c.2663) socket failed: No buffer space available 263 4086
2005-12-19 10:11:52: (mod_fastcgi.c.2663) socket failed: No buffer space available 263 4086

"no buffer space available" is supposed to suggest that the system is running out of network buffers. running netstat -m should reveal that buffer memory was exhausted, but the results of netstat show:

158/1552/18176 mbufs in use (current/peak/max):
        127 mbufs allocated to data
        31 mbufs allocated to packet headers
74/496/4544 mbuf clusters in use (current/peak/max)
1380 Kbytes allocated to network (10% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

which says that there's plenty of buffer space…

then there's this ticket on the lighttpd bug tracker… the bug is old, since lighttpd no longer consumes 100% cpu, but it still becomes unresponsive and hangs under the same conditions.

all in all i probably just need to add another server. i need to compile some statistics on exactly how many pets etc i serve up, but i stopped running a stats package on the image/swf servers because the log files were too large to analyze with the amount of RAM i've got. i guess i can just grep -c something appropriate, or heck, even just count the # of lines with wc -l.

for the last week or so the main bunnyherolabs.com server had been experiencing connection issues–stylesheets and images would fail to load, ssh login attempts would time out or be refused outright, mysql connections would fail randomly, emails wouldn't send, etc. etc. it seemed to happen only under heavy load. when i inspected /var/log/messages, it was full of No buffer space available errors.

to google i went. i read some suggestions to run netstat -m to see how many buffers short i was running. however, the output of that command indicated that i wasn't actually running out of mbufs??

other people with this problem seemed to have solved it by replacing the network card. i may eventually ask my hosting provider to do that (layered technologies, who are awesome, by the way) if the problem crops up again.

i ended up resolving it (apparently) by sending more of my traffic to the second server i recently rented (petimage.bunnyherolabs.com). for example, most of my <img> tags are now pointing there. i wonder if i'm simply hitting the limits of the hardware… i guess we shall see?