Getting Mashed as a start-up

Here at Hashrocket Labs, we follow a philosophy that we think other startups should follow: build something with the bare minimum feature set and release it. Let it grow.

One way to "let it grow" on a budget is to put your application on a host like Slicehost (http://www.slicehost.com). Our slice was the bare minium at $20/month; throughout the course of the day, we repeatedly increased the specs of our slice, but it turns out that wasn't even necessary.

So, what caused our growing pain or "scaling issue"?

ReadWriteWeb had written an introductory article about our new product Tweetblocker that generated some good traffic. What happened after that, was both exciting and stressful.

We got "mashed".

Mashable, a popular online blog, wrote an article about our application that we had no idea about. It was a great thing. We saw enormous amounts of traffic. What came next was what every startup has nightmares about. Our site crashed hard. We did NOT know what was going on.

Was it a Rails scaling issue?

No. The "can rails scale" question has been the most overused question in our community to this day. Yesterday was no exception. The reason our site crashed had nothing to do with Ruby or Rails.

The Problem

One of the views being used was querying the largest table (500,000 records) and was taking around 400 seconds to complete. It was a basic SELECT. The problem code was below:

CREATE VIEW the_view_table AS

SELECT user_id, twitter_id FROM users_friends

UNION

SELECT user_id, twitter_id FROM user_followers

And we were using the following in our code:

has_and_belongs_to_many :our_followers_and_friends,

:join_table => 'the_view_table',

:class_name => 'TwitterUser'

The above union does NOT use the indexes. This wasn't a Rails issue. It was Mysql. Soon as we removed the union and did two separate SELECT's, our queries went from 400 seconds to under 1 second. Our site came back and handle the onslaught of traffic with ease.

The Conclusion

Now we have a web application that is running amazingly fast with the user traffic we're experiencing now. And Hashrocket is fully comfortable with the performance of our application thus far. The code base is clean and has a tiny footprint. The way Rails should be.