How Stripe Got Me Thinking: Distributed Applications

A few weeks ago I competed in the ethical hacking competition, Stripe CTF 2.0: Web Edition.

If you read the shared blog posts of my team members, you’ll quickly realize just how bright these guys are. On a whim, they beat over 16,000 other participants in capturing the flag. Color me impressed. So needless to say, apart from the altruistic team player attitude one should adopt when on a team, I really wanted, and needed, these guys’ help.

In order for that, I would have to make it easy for us to collaborate. But, I would need to be able to set it up fast before they got too much further ahead (Yes, Dustin and Andre had gotten to the conference room a few minutes early and had already beaten levels 1 and 2).

Now, I had been experimenting with Gollum recently on my local machine, using it as a personal wiki to save myself from re-googling things all the time. It had become *raspy voice*…my precious…*end raspy*. Gollum is a “simple wiki system built on top of Git that powers GitHub Wikis.” More specifically it is a Ruby gem that parses markdown files and serves them up through Sinatra, leveraging Git for version control.

Since I already had it installed, I figured I’d just startup another instance in it’s own directory for us to use as an ad-hoc wiki. It was easy, quick, and done.

It worked out pretty well at first. We were able to share documentation links, code snippets, and encoded strings with the entire team but only had to explain it once. This allowed us to document in as much or little detail as we’d like. It was great!

That is, until we packed up for the night. You see, we’re all competitive and wanted to get ahead, so we continued to work independently before meeting up again to continue the next evening. And, since I was hosting the wiki on my local machine, only I had access to the files. The team members who were behind couldn’t reference any of the shared materials while the team members who were ahead couldn’t share their techniques through the community wiki.

So the next evening usage of the wiki dropped off dramatically. Even though it was less convenient, people had resorted back to their tried and true methods of email chains, IMs, and drop box. This was because the wiki content was out of date. Also, the wiki was unreliable since they didn’t know when they would have access to it again. They simply didn’t trust it. Ironic in a hacking competition, no?

I was kinda bummed out. I really liked using Gollum how we did the first night, but it’s drawbacks during our independent working time made it unusable. You could always put it on a server, but if the data is private (like in a competition, or actual business case) you want to control access, which brings a whole slew of other issues. Also, the web’s traditional client server architecture assumes internet connectivity. Although constant connectivity is quickly becoming the status quo in our society, this isn’t the norm in other societies. And it likely wouldn’t be the norm in extenuating circumstances such as natural disasters, or war scenarios.

This got me thinking… then I realized how dumb I was.

Our problem was already solved. Furthermore, the solution was already baked into the technology stack of our wiki. We were all using the web interface to contribute to the wiki. After all it was a wiki, that’s how Wikipedia works. But we were using Gollum which is really just a bunch of text files in a Git repository. The whole team could’ve cloned my entire repo!!!

We could’ve continued to work on it independently, and then synced back up once we met again.

Now this really got me thinking. Other team members could also clone personal copies of the wiki they wouldn’t have to share. Or, what if they shared certain pieces with certain members? But what if we applied them to our content? And what if web applications were really selectively distributed applications?

Well, a few others have had similar ideas. Checkout this new open distributed social networking project called Tent, and Chris Parson’s post on the open twitter clone called OStatus. Both of these got mentioned on Hacker News today, so I thought I’d throw in my two cents as well.

Interesting stuff…

So, I’ve since begun experimenting with some of the above concepts both in distributed web applications and in continuous deployment workflows for some projects. I’ll be posting examples and descriptions of both in the near future. Stay tuned!