Archive for the 'Technology' Category


O’Reilly’s Velocity Conference & Spike Night

FYI, I just guest-posted on O’Reilly’s Radar Blog about an event I’m running at this year’s Velocity Web Operations & Performance conference. It’s called Spike Night, and it’s a real-world look at managing operational spikes in two interesting, high-scale businesses (MySpace and

It wasn’t the easiest thing to bring together – not least because I have hardly had time to think about extracurricular activities, but also because it seems scary to potential participants, and scary means lots of approvals, etc. – but I’m excited and it should be good fun. See you in San Jose!


Your iPhone flies (to you) for 46 cents

It also bends time and space

It also bends time and space

On Friday, June 19, the iPhone 3G S launches, and for the first time, Apple has made launch day home delivery available. This has made some things clear – for example, we know that phones are shipping from China to Alaska all on the same day. I was curious about the magnitude of this delivery – size, cost, etc. Here are some estimates:

Total Sales

Apple sold at least 1 million 3G’s in the first weekend. It’s reasonable to assume that the number will be about the same – ease of home delivery counterbalancing the extra few days. So we assume 1,000,000 phones shipping from China.

Getting to the US

We know that the phones are shipping from Hong Kong to Anchorage via UPS. Unless UPS has some undiscovered stealth supertechnology, it’s by plane. Let’s assume they’re all shipping at the same time (though that doesn’t matter, it’s more interesting).

I don’t know the whole UPS fleet, but they bought 27 Boeing 767-300ER’s in 2007, so we’ll use those for our size analysis.

How many planes?

The limiting factor could be volume or could be weight.


The 767-300ER holds 30 LD2s. Now, the phones could be shipping in of three forms:

  • raw phones in trays to be Apple-packaged and ship-boxed in Alaska or another port.
  • Already-assembled Apple boxes, still not packaged for ship.
  • Fully shippable boxes (for home delivery)

Obviously the tradeoff in each case is volume you can ship v. assembly time on landing. The first seems really unlikely – there’s no way accessory boxing is happening across the country in ports in such a tight timespan. I would assume the second case, because ship boxing is a partly-automatable process and because the volume savings are so significant.

We don’t know the size of the new 3G S box, but the 3G box was 2.25″ x 3.5″ x 5.75″, so let’s assume the same dimensions. Each LD2 is 61.5″ x 60.4″ x 64″ (in their more generous size).

As I haven’t yet solved the packing problem, let’s just assume 80% utilization of the full volume. (That’s probably generous with pallets and such.)

181683 in2(LD2 volume) / 45.25 in2 (3G volume) * .8 = 4200 boxes

So 30 DL2s = 126,000 boxes: thus it takes 8 fully-loaded cargo planes to bring 1,000,000 iPhones to the US.


Weight’s not a limiting factor. UPS says my package is 0.5kg, and the 767-300ER can carry 96,870 kgs – at 80% utilization that’s 155,000 boxes/plane.

Estimate, then, is 8 cargo planes to get the phones to the US. I like to think of them in Top Gun formation.


The 767-300ER uses 3.47gal/mi: the air distance is ~5100 miles, so each plane uses 17,700 gallons: the estimated price today for jet fuel is $1.80/gallon (nice price if you can get it – assumes bulk discounts), so that’s $32,000 for gas – add another 20% for overhead and you have ~$38,000/flight to Anchorage. Anchorage to Miami is another 4000 miles: assume an average of 2000 miles for in-US travel (and higher overhead) and you add another $20,000/flight’s worth of travel (obviously spread across many other planes, many with smaller cargo holds). That’s $464,000 in flight costs, or $0.46/phone. (I haven’t factored in packaging and local delivery costs – either from Shenzhen to Hong Kong [23 miles] or from the airport to your home or store – which aren’t insignificant.)

It’s also ~57,000 lbs of CO2, but, y’know, whatever.

Final Words

Only $500K to get the phones to local ports in the US? Really? I’m surprised it’s so low, but math is math (though my packing assumption could be flawed).

If you add the UPS packaging in China or HK, you probably add 3-4X the planes and the cost, so there’s no way that’s happening. Anchorage was probably quite busy today.


Software Management: Avoiding Boss Bugs

So you’re a manager of a team, and you’re building a cool customer experience. You’ve got developers, you’ve got testers, you’re running around balancing schedules and worrying about processes and getting the legal team on board and dealing with deployment issues and watching the bug count and haranguing the team to fix one more thing and then you say it’s done and you ship.

And then you get the mail, and you know what it’s going to say as soon as you see

From:  Boss, Your
Subject: Re: Project Rubik Launched Today!

Sure enough, you open the mail, and Ms. Boss (or Ms. Boss’s Boss) has found a bug. And it’s a stupid bug. It’s a spelling error on the second page, or it’s I clicked the link on the bottom and nothing happened, or it’s I tried in IE6 on my home machine and all the words are jumbled, or it’s I typed “mass” in the body and it was rejected for profanity. And you see this bug, and all you think is I should have caught this, Ms. Boss must think my team has no quality control, and damn it, that’s all anyone is going to remember. Congrats – you’ve snatched defeat from the jaws of victory.

Has this happened to you? It’s happened to me. It probably happened on half of the customer-facing features I launched at Amazon, where Boss was played by my actual boss, or my SVP, or Jeff Bezos, each one of whom was a passionate advocate for the customer’s experience and was going to click through the moment we shipped to take a real look, no matter how many times she had seen it during the planning and testing phases. The WhitePages CEO had the same skills and obsessions. Aaaaggghhh – we should have done better.

The good part is that you really can avoid Boss Bugs, and you’ll be happier if you do so. There are simple and complex things you can do, and I try now to do all of them, and it mostly seems to work.

  • Take a break, then be a user. The hard part is the precedent – take a break. This doesn’t mean just go get coffee – you need to find a way to clear your head of all of the things that you are thinking about related to the project. Do it first thing in the morning, or right after dinner, or after a bike ride. But make yourself do it – go through the whole process. Do it slowly, read the screens, take your time. Your users will not be as fast as you are.
  • Demo to someone you know. Show off your work to someone you trust, in the software business or not – really use the features and look at the pages while you’re doing it. Ask them to drive, too.Both of these first two techniques will also help you find things that you need to check “later” or have someone look at, so have a pen handy.
  • Do a real bug bash, and be part of it. Yes, bug bashes are a pain to schedule, but I’ve never been unhappy that I did it once it was done, and they’re always useful. Make someone else deal with coordination – you should put on your headphones and do your own bash. I try to “win” (find the most bugs) and encourage others to do the same. Great managers are great bug finders. And if you find a lot of bugs? You’re further away than you thought. (Good to know now.)
  • Set your expectations with your QA team clearly. It doesn’t matter if they work for you or for someone else – your QA team is the way you make sure your product is good enough to ship. So make sure they understand what your quality bar is, and that even if you have good specs, you need them to help make sure that the product is great, not just that it matches the specs. The worst product I almost shipped came about because I didn’t make sure that my QA team knew that unfinished sentences on web pages were ship-stoppers, and that there was nobody else who was going to find them. Quality isn’t just that the API or Selenium tests pass – it’s that the product looks and feels right, and if that’s subjective, there you go – if it was easy, we wouldn’t be paid for it.

On balance, having a boss who cares about the customer experience is like having a boss who’s technically competent – it’s great having someone who gets it in your chain. Pretend to be them and you will deliver better software, guaranteed.

If you have other tips to suggest, by all means, please add them to the comments.

(BTW, Boss Bugs don’t just apply to software – they also apply to processes and planning. I had a Boss Bug in my own planning process recently, and my Boss caught it – good for him and good for the project.)


Programming: write it down (again)

Last day of last month, I talked about the importance of writing down your questions when you’re programming. Last day of this month, similar post – now, it’s about writing down your algorithms when you’re programming.

Since college (when I would “program” by the light of a desk lamp at the top of a bunk bed – pre-laptop), whenever I have a reasonably complex piece of logic to work through, I always start with a piece of paper and pencil, and I sketch out the entire algorithm in pseudo-pseudocode – basic structures, algorithms, control flow. (It’s like being at the board for an interview, but nobody’s watching.)

Why do this? In order of importance:

  • You can focus on the problem. When you’re writing real code, you’re focusing on a lot of things – the logic, but also the syntax and markup, the object-method structures, the tabs, the comments, the format. Each of those things takes some thought, and each moment causes a mental task switch from solving the real problem. (Maybe you’re both so talented and so experienced in a particular language and IDE that none of those things actually require any thought – you’re as fluent as you are in your native tongue. But I’m not, and you’re probably not either.)
  • It’s much faster. You might type faster than you write when taking notes, but nothing’s faster than writing fake code on a piece of paper, drawing arrows and braces to mark sections, using carets to insert content, etc. You aren’t worrying about any of the things above: you will find the problems in your thinking much, much more quickly.
  • You can do it anywhere. On the bus, by the bed lamp, when you’re pretending to make art with your three-year-old. You can think about the algorithm in between things and record your notes, rather than waiting for the moment you’re sitting down. Paper & pen are still more accessible and portable than the MacBook Pro.

Highly recommended approach. Let me know if this or similar things work for you!


Vaporbase micro-CMS: Performance Improvement

(Warning: code talk ahead.)

When I started building the app I’m working on, I needed a quick CMS system that didn’t impose its own parsing language, and I was just learning RoR, so I implemented the Vaporbase micro-CMS based on this great tutorial. While you wouldn’t confuse this with a pro-grade CMS system, it works just fine for the most part, and doesn’t require you to learn a number of new things.

Since implementing it, I’d made a few minor cleanups, but nothing significant, until I had 60 pages in the CMS, and saw a problem: the list of pages was taking a very long time to load (21sec on avg on my dev machine).

Sure enough, there’s a problem: in the Edit and Show Tree Hierarchy section, here’s a line that says

<% unless index_item.children.nil? %>

Looks innocuous, right? Not exactly: for every single page in the tree, this does a separate SQL query (which is a full table scan, though likely on a tiny table) to find out if that item has any children. So creating the page is O(# pages in table), and while the queries itself aren’t that expensive, building the page just takes a while. If your tree is mostly flat – almost certainly the case for this kind of CMS – you’re wasting a lot of time.

Fortunately, there’s an easy solution: when you get the list of pages before creating the tree, figure out which pages have children, and then store that list to the side. It adds a bit of code, but not a lot.

# Original: from the index action in the controller
# This gets just the root nodes and then recurs down the tree
@pages = Page.find( :all, :conditions => [‘parent_id IS NULL’], :order => :position)
# Replacement:
# build two arrays –
# @pages consists of all of the root nodes (since the view walks down the tree)
# @pageIDs_with_children consists of all pageIDs which are the parent for >=1 page

@pages = []
@pageIDs_with_children = []

@all_pages = Page.find(:all)
@all_pages.each do |page|
if (page.parent_id.nil?)
@pages << page else @pageIDs_with_children << page.parent_id end

(Note that the list might have duplicates, but that doesn’t matter. If that makes you unhappy, check for existence in the else clause.)
Then replace the view line above with

<% if @pageIDs_with_children.index( %>

I saw a >80X performance improvement on this page with this change in development (from >20sec to ~0.25sec with 60 pages), and you’ve just replaced an expensive linear scaling step with a very inexpensive one.


Search Evernote, not Google

I’ve played with 100 note-taking/history-recording tools over the years, and none of them have ever really taken off for me. I started playing with Evernote soon after the beta began, and while I loved the interface and the story, my uses were very occasional.

Now, however, I’m constantly in Evernote – it’s third only to my browser and Textmate – as I’m coding. 

Why? I’m using it as a programming information repository. Every day that I program in Ruby on Rails, I learn a few new things or change the way I do things – today, for example, I learned that interpolation is faster than concatenation, and yesterday I finally found a clear explanation on how yields work. What I do now is every time I find an interesting piece of information, I clip the whole page to Evernote, add a few tags, and then keep on going. Occasionally I’ll paste screenshots from the PDFs of some books I own as well, which then (usually) get magically transcribed for me.

I’ve realized that I was using Google to search for the same information over and over again, and then trying to remember which link I liked the best. With Evernote, I’ve already recorded that memory, so I just have one source. Every day a greater percentage of my programming-related searching is happening in Evernote.

I’ve thought about making a separate notebook for my Rails/Ruby content and making it public – if I do so, I’ll post it here – though obviously it’s not a replacement for an actual text, it just has pointers to things I’ve cared about. However, if a dozen people did this, I’d love to have links to all of their notebooks.


On Apple, Amazon, Reviewing, and Large Companies

Two interesting stories going around in the last week, which I find to be similar (even if their impact is different): Apple’s being raked over the coals for rejecting an iPhone app that duplicates Apple functionality, and Amazon’s been dealing with the customer review attack on Spore, including purging and then restoring all of the reviews.

As I’ve mentioned before, I used to manage the Amazon customer reviews business, and so I know very well what the current team is going through. My assumption is that the Apple app store review business has some similar processes and problems. Here are some things I learned while dealing with this:

You start with some philosophical rules, and you try to make them stick. Providing guidelines is the only way to start. Example philosophies for Amazon (made-up, these aren’t real, don’t quote them anywhere else) could include “our customer is the Amazon buyer” (so no, Ms. Vendor, we won’t take down the negative reviews of your book, even though you spend a lot of money on advertising with us), “we eliminate reviews with demonstrably false information”, and “fairness is more important than justice” (so if you generally write good reviews and then get caught plagiarizing once, you can be given more chances). 

All sensible on face and all make sense to folks who think in these kinds of abstractions all day – there may still be debate but these are good places to start. 

There’s a clear chain of command for decisions. The escalation path from “customer service rep in her fourth week receives a review complaint in the mail queue” to “Jeff decides the review stays” should be very clear. (In my ~2 years dealing with customer reviews, btw, Jeff only engaged once on actual content, and the issue was much larger than just reviews (and he was getting hundreds of mails on this topic) – he generally trusted the heads of these teams to do the right thing as long as they could articulate the philosophy.)

All of this sounds good, of course, but then people get involved. And customer service reps are trying to interpret the philosophies (if they can find them among hundreds of pages of other rules), and some of them are judgment calls (what is “demonstrably false?” If I say “the defibrillator didn’t work and my dad died,” is someone going to check? are comments on voting records trustworthy? etc.) that different people will make, and of course you don’t want Jeff or Steve Jobs or anyone making every decision.

So it’s messy, and when it’s messy, strange things happen – reviews appear and disappear, apps go away and come back (like Netshare), etc. 

This is a long way of saying that it’s entirely likely that the banning of Podcaster is a problem of human judgment in a theoretically well-structured system – not least because the decision seems inconsistent – and that could easily come back, not because of a correction of a philosophy, but because of a correction of a human error.

Now, it’s Apple’s responsibility to make that correction, and then to treat the errant employee with respect and look at how the company can do a better job. 



Programming: When you’re stuck, write it down

Hey. Long time, no blog. Been busy.

Thought I’d share a tip that I’ve used for many years when programming (or learning almost any new thing) and I’ve been using recently: when you’re stuck, write it down.

Say you’re trying to figure out how to do something in [pick a framework], and you’ve Googled the heck out of the most-likely search terms, and nothing’s coming up.

Then write down your question as if you were going to ask a teacher/email it to a friend/post to a Google group/etc. Write down all the details: explain the thing you’re trying to do, the problem you have, and the number of things you’ve tried. Be as clear as you can, but don’t worry about being concise.

Literally every single time I’ve ever done this – and my rule-of-thumb is to do it after ~1.5 days worth of trying to figure it out myself – I find a number of new avenues to try, and almost always solve the problem on my own.

Writing it down forces you to take the jumbled thoughts in your head (they probably weren’t jumbled when you started, but you’ve changed paths so many times now) and turn them into a narrative. The process makes visible paths that your random walk opened up but that you didn’t see.

(This has been a procrastination for completing a writeup of my own.)


Not-at-all-fun things to do with Twitter

Every once in a while I hit another breathless article about all the cool things you can do with Twitter. Most of them involve either following things I don’t actually care about or submitting obscure numbers for reasons unknown.

So here’s the start of a list of things that might seem less fun.

  • Get your stuff stolen. When I get into burglary, I’ll start with Twitter. Hey, Ian’s at a play! That’s plenty of time to take his MacBook and mardi gras beads.
  • Get spammed. OK, The Onion stopped, but still, it’s like double opt-in to friendly spam.
  • Be fooled by a fake celebrity. Ira Glass? Fake Ira Glass, currently dealing in non-sequiturs. (Some were likely devastated.)
  • Keep up with dictators.
    what are you doing?

    (Sadly, I believe Twitter has started blocking the creation of other dictator’s names: Pol Pot & Robert Mugabe were “unavailable” but have no pages. Probably that’s a good thing.)
  • Wait for pages to load. You already knew that one, but I was Internetally obligated to include it. Sorry, I don’t make the law.

Anything else to add?


Starting a Seattle CTO Support Group

I’ve been thinking for a while about trying to put together a Seattle “CTO Support Group” and have mentioned this to a few folks, and I’m finally kicking this off. I posted basically this message to the Seattle Tech Startups mailing list this past Friday, but I’m not sure who that hits and misses. I’ve had ~20 people express interest so far.

Opening thoughts:

  • Attendees are CTOs (or whatever title denotes “head of technology” – software and ops) at local companies/govts/etc. Starting point is that this is one person in a company, feel free to tell me that this doesn’t make sense for your organization, but the goal isn’t a replacement for other tech gatherings with more open invites.
  • Figure the makeup of the group will determine the balance between small/medium/large companies, private v. public, funded v. bootstrapped, etc. Early folks have been primarily (but not entirely) small startups.
  • Would have regular get-togethers – breakfast? lunch? – hosted by some company, ideally in a regular location (prob. in downtown Seattle or Pioneer Square) or in a rotating location.
  • Get-togethers would be 70% mingling, 30% presentation (probably from someone in the group) on an interesting topic – could be technical, could be managerial, could be look-at-the-industry. Assumption is that the content and topics are meant for hands-on leaders and are real-world, not stuff we’d read in a Gartner report if we read them anymore. Obviously interactive.
  • Assumption is that everything is under FrieNDA.

If you fit the description and would be interested enough to attend regularly, let me know (and if you have a preference b/w breakfast or lunch, or another bright idea, let me know that as well). There’s an “Email Me” link in the sidebar of I’ll likely host the first one in the next N weeks at WhitePages HQ in downtown Seattle (and probably could host regularly, we have a good space depending on size).

Twitter Updates

[aktt_tweets count="5"]