Lazy Testing your URLs

scottru — Fri, 23 Jul 2010 19:49:13 +0000

So I have a Rails app that’s broad but not very deep: there are a lot of different types of URLs and thousands of pages, generated dynamically or built through a CMS. No one page is that complicated, but there’s a fair amount of shared code and thus some tentacles throughout. Yes, I should have written behavior-level and unit-level tests from the beginning, but I didn’t, don’t judge me.

Anyway, after I broke the site yet again, I realized what I really wanted was a post-release test suite – I wanted to test a whole bunch of pages in production and make sure they’re still rendering.

Stealing from the Net::HTTP canonical examples, then, I wrote a standalone script:

require ‘net/http’

URIs = ['http://example.com/', 'http://example.com/page/2']

puts “Running URL Test Suite…”
URIs.each do |uri|
  url = URI.parse(uri)
  req = Net::HTTP::Get.new(url.path)
  res = Net::HTTP.start(url.host, url.port) { |http| http.request(req) }
  unless res.code == ’200′
    puts “ERROR: #{uri} #{res.code}”
  end
end
puts “Test Suite Complete!”

Then, since I deploy directly with git, I added a call to this script in my post-receive hook. As long as I watch the deployment (which I do anyway), I’ll see if there’s a problem and fix it immediately. (I can handle a few minutes of downtime or just roll back.)

There are obvious improvements to this (like not storing the URLs in the file), and this doesn’t handle redirect, workflow, or authentication scenarios, but it’s a perfectly acceptable 99% case for a content site.

Note that if you are doing Rails development and you’ve built a custom 404 page, you’ll want to make sure that you’re actually returning a 404 status code, or this won’t catch errors: you can test your status codes, and if you aren’t, you can add code in your application controller to do the right thing.

Vaporbase micro-CMS: Performance Improvement

scottru — Fri, 26 Sep 2008 15:02:05 +0000

(Warning: code talk ahead.)

When I started building the app I’m working on, I needed a quick CMS system that didn’t impose its own parsing language, and I was just learning RoR, so I implemented the Vaporbase micro-CMS based on this great tutorial. While you wouldn’t confuse this with a pro-grade CMS system, it works just fine for the most part, and doesn’t require you to learn a number of new things.

Since implementing it, I’d made a few minor cleanups, but nothing significant, until I had 60 pages in the CMS, and saw a problem: the list of pages was taking a very long time to load (21sec on avg on my dev machine).

Sure enough, there’s a problem: in the Edit and Show Tree Hierarchy section, here’s a line that says

<% unless index_item.children.nil? %>

Looks innocuous, right? Not exactly: for every single page in the tree, this does a separate SQL query (which is a full table scan, though likely on a tiny table) to find out if that item has any children. So creating the page is O(# pages in table), and while the queries itself aren’t that expensive, building the page just takes a while. If your tree is mostly flat – almost certainly the case for this kind of CMS – you’re wasting a lot of time.

Fortunately, there’s an easy solution: when you get the list of pages before creating the tree, figure out which pages have children, and then store that list to the side. It adds a bit of code, but not a lot.

# Original: from the index action in the controller
# This gets just the root nodes and then recurs down the tree
@pages = Page.find( :all, :conditions => ['parent_id IS NULL'], rder => :position)

# Replacement:
# build two arrays –
# @pages consists of all of the root nodes (since the view walks down the tree)
# @pageIDs_with_children consists of all pageIDs which are the parent for >=1 page

@pages = []
@pageIDs_with_children = []

@all_pages = Page.find(:all)
@all_pages.each do |page|
if (page.parent_id.nil?)
@pages << page
else
@pageIDs_with_children << page.parent_id
end

(Note that the list might have duplicates, but that doesn’t matter. If that makes you unhappy, check for existence in the else clause.)
Then replace the view line above with

<% if @pageIDs_with_children.index(index_item.id) %>

I saw a >80X performance improvement on this page with this change in development (from >20sec to ~0.25sec with 60 pages), and you’ve just replaced an expensive linear scaling step with a very inexpensive one.

ScottRu » Rails

Lazy Testing your URLs

Vaporbase micro-CMS: Performance Improvement