i spent most of tuesday messing around with various freeBSD and lighttpd settings. even though netstat -m didn't report insufficient buffers, i was getting no buffer space errors in my log files. so, i upped kern.ipc.nmbclusters from 6000-ish to 16000 and rebooted.

the errors continued. however, netstat -an counted more connections before dying than before. hmmm…

netstat also revealed that many, many, many of those connections were hanging around in the TIME_WAIT state. i googled that and found some articles about MSL and socket connections… to be honest i didn't understand it all, but it seemed that it was possible to reduce the amount of time that sockets remained in the TIME_WAIT state by changing the value of net.inet.tcp.msl. on freeBSD 4.10 the default value is 30000 (30 seconds), so i reduced it to 7500 (7.5 seconds). i understand that lowering it too much can have other consequences, like having old data from an old connection getting mixed up in a current stream?? i admit i am a TCP/IP newbie.

anyway, i changed that value, and restarted the server. so far so good… the server seems to be handling the load quite nicely now, with no connection failed/buffer space errors. i don't know if increasing nmbclusters was necessary, but since things are working ok now i am just going to leave it there for now

lighttpd is great (serves a lot, quickly), but under very heavy load it seems to act rather erratically. i'm not sure where the fault lies, though: in lighttpd, in fast-CGI (which is how lighttpd runs PHP), or in freeBSD, or some combination.

the symptoms are error messages like this:

2005-12-19 10:11:52: (mod_fastcgi.c.1561) connect failed: 242 Connection refused 61 0 /tmp/php-fastcgi.socket-0
2005-12-19 10:11:52: (mod_fastcgi.c.2663) socket failed: No buffer space available 263 4086
2005-12-19 10:11:52: (mod_fastcgi.c.2663) socket failed: No buffer space available 263 4086
2005-12-19 10:11:52: (mod_fastcgi.c.2663) socket failed: No buffer space available 263 4086

"no buffer space available" is supposed to suggest that the system is running out of network buffers. running netstat -m should reveal that buffer memory was exhausted, but the results of netstat show:

158/1552/18176 mbufs in use (current/peak/max):
        127 mbufs allocated to data
        31 mbufs allocated to packet headers
74/496/4544 mbuf clusters in use (current/peak/max)
1380 Kbytes allocated to network (10% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

which says that there's plenty of buffer space…

then there's this ticket on the lighttpd bug tracker… the bug is old, since lighttpd no longer consumes 100% cpu, but it still becomes unresponsive and hangs under the same conditions.

all in all i probably just need to add another server. i need to compile some statistics on exactly how many pets etc i serve up, but i stopped running a stats package on the image/swf servers because the log files were too large to analyze with the amount of RAM i've got. i guess i can just grep -c something appropriate, or heck, even just count the # of lines with wc -l.

so far using coral CDN is working great! my bandwidth usage has dropped dramatically.

by the way, to send the appropriate header from lighttpd so as to trigger the fallback over-quota behaviour, just add this line to lighttpd.conf:
setenv.add-response-header = ( "X-Coral-Control" => "redirect-home" )

also, i am apparently now one of the major users of coral.

i'm trying out the coral content distribution network as a way of reducing the amount of bandwidth i use by hosting pets. we'll see how it works out… i just worry in general about relying on stuff outside of my immediate control–what if the response time is slow? or if i hit their bandwidth limits? on the other hand, their new over-quota system means that if i do use too much of their bandwidth, requests will just get redirected back to my server, so in theory the pets should never become unavailable.

anyway, it's certainly worth trying, and easily reversed if it doesn't work out.

EDIT: hmmm it looks like the new over-quota behaviour only kicks in if you serve an additional header in your HTTP responses. i'll have to see how to do this in lighttpd (which is what is serving up the pet's SWF files).

i wrote a simple awk script to tally the different versions of safari that appear in my apache log files. i did this because awstats doesn't differentiate between different versions of safari. awstats does track different versions of netscape, but i get more safari users than netscape users…

read more for the script itself.
(more…)

well the pets now allow a wider range of characters to be part of the name and adopter name. this is cool, but the downside is that the SWF files are now anywhere from 20-35% larger, due to the increased size of the embedded font. this has a direct impact on my bandwidth usage… and now that i've allowed those characters, it would be unfair to remove them, as it would break existing pets.

one way around this would be to use a device font instead of an embedded font. the downside of that is that unless the user had the century gothic font installed, it wouldn't look as nice.

hmmm now that i think about it… i should double check to see how lighttpd handles cache headers and conditional http GETs and stuff…

update: lighttpd returns HTTP 304 (not modified) just fine when appropriate. perhaps i should look at the individual swf files and see what cruft i can chuck out of them to slim down the file sizes.