Upgrading to Rails 0.13.1 pains
Dreamhost has upgraded to Rails 0.13.1, so I figured I should too. The actual application upgrade was simple enough, the main tricky part of figuring out which files I should just ditch and replace with new copies. ‘config/environment.rb’ was probably the only one.
I did the upgrade locally and it worked fine on WEBrick. Things didn’t go as smoothly when I upgraded it on the server, first there were stupid things like forgetting to make ‘production’ the default Rails environment again (overwriting environment.rb will do that). But the more serious problem of the site timing out. I came across this my Apache error log:
[Sat Aug 6 08:51:48 2005] [error] [client x.x.x.x] FastCGI: comm with (dynamic) server "/home/xxx/yyy/public/dispatch.fcgi" aborted: (first read) idle timeout (240 sec)
[Sat Aug 6 08:51:48 2005] [error] [client x.x.x.x] FastCGI: incomplete headers (0 bytes) received from server "/home/xxx/yyy/public/dispatch.fcgi"
Not a good sign, and not obvious what’s causing the problem. At this point I figured I better put the old version back (copied the backup over rather than any fancy CVS tricks). To avoid future downtime I setup another domain to act as a staging server. This means I can test upgrades on an identical setup as the live server. But of course DNS updates take time so I didn’t have access to it straight away.
Rather than wait, I figured I’d just install Apache/FastCGI locally. Should be fairly straightforward, but these things never are. I followed this guide:
http://wiki.rubyonrails.com/rails/show/HowtoInstallOnGentooWithApache
Straightforward enough but I kept getting 403 errors when I tried to access the site. I thought it was some weird Apache access error and spent forever looking at the Deny and Allow settings in the config files. After a lot of wasted time I realised while the app directory was set to 755, my home directory was set to 700, doh!
So once that was fixed I could access the site and get the same timeout problem as I did on the live copy. Googling the FastCGI error message turned up a lot of red herrings but I eventually came across a post saying it might be a permissions problem with the interprocess communication file (IPC) that FastCGI uses. I quick check in my commonapache2.conf file showed this wasn’t set anywhere, so I threw this in the end:
<ifmodule modfastcgi.c>
FastCgiIpcDir /tmp/fcgiipc
FastCgiServer /…/rails-app/public/dispatch.fcgi -initial-env RAILS_ENV=production -processes 15 -idle-timeout 60
</ifmodule>
Voila! It now loaded but crashed with a MySQL error:
HY000Host ‘x.x.x.x’ is not allowed to connect to this MySQL server
which was very confusing since ‘x.x.x.x’ was my external IP, not the IP of my laptop. I first thought MySQL must be setup to only accept connections from localhost. But digging around the config files didn’t turn up anything that looked like per-host access/deny. This is because connections from different hosts are setup on a per user basis, not a per server basis. This makes sense, but is different from other servers I configured, so it took a while to find this out. With the new fancy user setup it still failed with the same message. After a lot of head banging I realised it’s connecting to the production database not the development one! Hacking my database.yml proved this to be correct.
I finally had a working Apache/FastCGI setup, and confirmed a fix is setting up the FastCGI IPC file. It looks like the root of the problem is the switch to RailsFCGIHandler in dispatch.fcgi, which no longer works with the old Apache setup. But of course it’s set in the Apache config file, which I have no control over on DreamHost, time to raise a ticket!
