In Search of Everything All at Once

Charles Jolley, CEO of Sproutit and I have been talking quite a bit about data storage problems, particularly in the ineffectiveness and the eventual scaling block you run into with most SQL-based solutions.

Facebook's success story with MySQL aside, most top sites either don't use SQL or use such simple queries in their data that the searches are simplistic, and the heavy lifting is done by careful indexing. Complex joins are almost unheard of in these situations, and actually having to perform one of these searches while the user is waiting can be a painful experience.

Ruby on Rails actually makes constructing these complex and slow joins painfully easy. You get instant results just by adding water. Perform complex joins, multiple levels of deep associations, and find_by_anything till you're blue in the face and you just won't notice it, until your application is an overnight success.

It's not Rails fault, or even SQL's fault. Having to give serious thought to how you're going to scale your data storage has always been a necessity for big applications, it's just something that you can defer now with modern frameworks. I suppose that's a good thing, as long as you can make the scaling happen on the fly.

We're actually working on some techniques to combat this problem, using things like Memcached, S3, and custom data formats. If you're interested, stay tuned. I'll be posting more on the subject soon.