Chef Upgrade Issue

posted Jun 18, 2012, 6:10 AM by Tyler Akins   [ updated Jun 18, 2012, 6:13 AM ]
Once again, I had a problem with Opscode Chef, but for a very understandable reason.  First, while trying to spin up an instance, I see messages like this at the end.

ERROR: Server returned error for http://ec2-50-17-230-193.compute-1.amazonaws.com:4000/cookbooks/phpunit/0.9.1/files/1ac61a28fa057aeb34ca4e5071e9c96c, retrying 2/5 in 8s
ERROR: Server returned error for http://ec2-50-17-230-193.compute-1.amazonaws.com:4000/cookbooks/phpunit/0.9.1/files/1ac61a28fa057aeb34ca4e5071e9c96c, retrying 3/5 in 16s
ERROR: Server returned error for http://ec2-50-17-230-193.compute-1.amazonaws.com:4000/cookbooks/phpunit/0.9.1/files/1ac61a28fa057aeb34ca4e5071e9c96c, retrying 4/5 in 29s
ERROR: Server returned error for http://ec2-50-17-230-193.compute-1.amazonaws.com:4000/cookbooks/phpunit/0.9.1/files/1ac61a28fa057aeb34ca4e5071e9c96c, retrying 5/5 in 53s
ERROR: Running exception handlers
FATAL: Saving node information to /var/chef/cache/failed-run-data.json
ERROR: Exception handlers complete
FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
FATAL: Net::HTTPFatalError: 500 "Internal Server Error"

I then peeked into /var/log/chef/server.log and saw some messages. Not quite helpful, but perhaps a clue?

merb : chef-server (api) : worker (port 4000) ~ Params: {"cookbook_version"=>"1.0.0", "action"=>"show_file", "cookbook_name"=>"apache2", "checksum"=>"ab1792e9de7461ddf4861e461c0c8a24", "controller"=>"cookbooks"}
merb : chef-server (api) : worker (port 4000) ~ undefined method `file_location' for # - (NoMethodError)

The file_location was set correctly, so now I am stumped. I restarted chef server, rebuilt the database as I mentioned on a previous blog post and uploaded everything again to the server. No luck with any of those. The failure point wasn't always on the same package. It seemed to hop around to different packages at different times.

So, now I check versions of the packages that are installed.

19:32 utilities:/tmp$ sudo dpkg -l | grep chef
ii  chef                              0.10.8-2                                   A systems integration framework, built to br
ii  chef-expander                     0.10.4-1                                   A systems integration framework, built to br
ii  chef-server                       0.10.4-1                                   A meta-gem to install all server components
ii  chef-server-api                   0.10.4-1                                   A systems integration framework, built to br
ii  chef-server-webui                 0.10.4-1                                   A systems integration framework, built to br
ii  chef-solr                         0.10.4-1                                   Manages search indexes of Chef node attribut

You'll see that one package is at 0.10.8 and the rest are all 0.10.4. Could that be it? Reinstalling the chef package didn't force upgrades of the others, so I just manually used apt-get to upgrade the other chef packages and it started to work again.

Problem solved.
Comments