Partly Cloudy

We have a dream
A vision
An aspiration
To compute in the cloud
To pay as we go
To drink by the sip
To add cores at our whim
To write to disks with no end
To scale up with demand
And scale down when it ends
Elasticity
Scalability
Redundancy
Computing as a utility
This is our dream
Becoming reality

But…
There’s a hitch.
There’s a bump in the road
There’s a twist in the path
There’s a detour ahead on the way to achieving our goal

It’s the Database
Our old friend
He is set in his ways
He deals in transactions to keeps things consistent
He maintains the integrity of all his relations
He eats disks for breakfast
He hungers for RAM
He loves queries and joins, and gives each one a plan
He likes his schemas normal and strict
His changes are atomic
That is his schtick

He’s an old friend as I said
We all know him well
So it pains me to say that in this new-fangled cloud
He doesn’t quite fit

Don’t get me wrong, our friend can scale as high as you want
But there’s a price to be paid
That expands as you grow
The cost is complexity
It’s more things to maintain
More things that can go wrong
More ways to inflict pain
On the poor DBA who cares for our friend
The one who backs him up and, if he dies, restores him again

I love our old friend
I know you do too
But it is time for us all to own up to the fact
That putting him into the cloud
Taking him out of the rack
Just causes us both more pain and more woe
So…
It’s time to move on
Time to learn some new tricks
Time to explore a new world that is less ACIDic

It’s time to meet some new friends
Those who were born in the cloud
Who are still growing up
Still figuring things out
There’s Google’s BigTable
and Werner’s SimpleDB
There’s Hive and HBase and Mongo and Couch
There’s Cassandra and Drizzle
And not to be left out
There’s Vertica and Aster if you want to spend for support
There’s a Tokyo Cabinet and something called Redis I’m told

It’s a party, a playgroup of newborn DB’s
They scale and expand, they re-partition with ease
They are new and exciting
And still flawed to be sure
But they’ll learn and improve, grow and mature

They are our future
We developers should take heed
If our databases can change, then maybe
Just maybe
So can we

UPDATE: Someone shot a video of me reading the poem and posted it on YouTube.

UPDATE 2: Another video posted to YouTube, this one has the entire poem and a little better audio.

In 1995 or 1996 I got a prophetic email from Mason Hale, who had discovered Frontier and thought it would make a wonderful environment for CGI scripting. He was right, but I came to wish I had never gotten that email. Seriously.

While it is nice to be remembered, I wish it were a little more fondly. I hate to think Dave genuinely regrets our interaction and the cool things we built together or pushed each other to build back in the early years of the web. We did some crazy stuff. Some of which has faded to history, but I still remember well — and fondly. Like when we wired Frontier up to Netscape to run client-side scripts using a custom usrtlk: protocol. It was a security nightmare waiting to happen, but it was geeky-cool and a fun and still mostly uncharted territory to explore.

So it is a bit sad for me to read that Dave associates me and that time with a negative turning point in his life. I have not kept in touch with Dave over the years. We didn’t have a falling out or anything, we just started working on different things. I am thankful for the many lessons I learned from working with Dave Winer. I wish him well, and I am sorry to hear he’s had a rough stretch these last few years.

The biggest news is that I left my job as Chief Technologist at frog design in June 2006. I left to help start up a new company, LargeSmall Systems, with another friend, Matt Cohen. We are not quite ready to publicly disclose the nature of our business or product, but it has been a great ride so far. We’ve assembled a terrific team and together we are building some really cool technology. It’s been hard work, and a lot of fun at the same time. I’ll write more about this new venture in the future.

Leaving a comfortable position at frog for the relative uncertainty of a not-quite-yet-funded hatchling startup last year was a tough decision to make. Having gone through a boom and bust during my eight years at frog, the digital technology team I founded in 1998 was doing better than ever. We were doing great work and some strategic bets we had placed earlier were starting to pay off. Yet, I had the opportunity to do something new and exciting with this start-up. I was fortunate to be able to leave frog at the top of my game, so to speak, with my department growing and with strong leadership in place behind me to keep things rolling. For me, it was a textbook case of how to leave your job on good terms.

It’s been great to watch from sidelines, since I left, and see my old team continue to crank out great work and to establish themselves as leaders in some hot emerging technology areas. One example: a frog project I played a key role in early on, the Alltel Celltop, was recently awarded Best of Show at the CTIA Wireless industry conference. Another: the new Hawaiian Airlines website is gorgeous, highly-functional, and technologically sleek and elegant. I am bursting with pride.

While I would be enjoying a bit more glory about now had I stayed at frog, I don’t have any regrets. The last nine months have been intensive training for me in the starting, funding and growing of a company. For the first time in my life I am fully focused on creating a single product. It’s been an awesome experience and the future only looks brighter.

There have been some benefits to my personal life too. For one, my commute is now just under a mile. And while at frog I was a platinum-level frequent flyer, now I am closer to home and have been able to do things like coach my daughter’s soccer team — something I never would have been able to do with my previous travel schedule. I’m still working as hard as ever, yet I’ve also been able to spend more time with my family. And that has been priceless.

More to come…