As an engineering manager tackling a new problem, it’s a natural question to ask yourself. Your engineers kick ass, and given enough time, they’ll build a solution that fits your needs better than anything on the market. Of course, their time is also one of your company’s scarcest resources, and it’s your job to make sure it doesn’t get wasted — plus, you need this solution yesterday. So what do you do?
The most widely-cited rule of thumb is to build things that are “core” to your business, and buy (a term used loosely, in this age of SaaS and open source) the rest. This rule makes sense, and for some people it works well in practice. For me though, it leaves something to be desired, and I have a hard time applying it universally.
The reason is yak shaving. A large chunk of your team’s time is unavoidably spent doing things that actually aren’t core to your business, or each person’s role. For your developers, it might be maintaining integrations with buggy third-party APIs, managing dependencies, or refactoring old code. For marketing, it might be manually entering data, and creating and distributing reports. For sales, it might be pulling the same data on your customers over and over. And even though this type of work is unavoidable, minimizing it is the key to productivity.
The trouble with bought solutions is that, in many cases, they tend to attract yaks. There may be feature mismatch, which slows down the users of the solution and increases communication overhead. In the worst cases it can even prevent the solution from delivering on its promised value. Moreover, if there’s a technical integration, your engineering team often winds up shimming and debugging it, without the benefit of having written the code themselves. In comparison, an internal solution can save your team time by providing exactly the features you need, and technical modifications are easier since you wrote the code. It’s an upfront investment to be sure, but it can be worth it. Of course, this investment is obviously a giant hairy yak all by itself, and if it’s a hard problem, there’s a good chance you’ll actually do a worse job in the long-run than a dedicated outside team.
For these reasons, I believe build or buy is a tricky call, and there are plenty of war stories favoring both sides. Wikipedia articles, for example, are written in a special markup language, originally developed as a simpler alternative to HTML. Over time, though, the language has grown organically to the point that no one actually knows how it works. If you want to see how something will render on Wikipedia, you just have to feed it in and see what you get. Though the original design goal was sound, it’s clear that an off-the-shelf solution would have left the project in a better place today.
Tornado on the other hand, is a popular Python web framework that was developed internally by FriendFeed. Usually building your own framework or language is a jumbo-sized yak to be avoided at all costs (cf. Asana) — after all, there’s plenty of these tools that work fine for thousands of developers already. But, given FriendFeed’s unusual (in 2008 anyway) real-time demands, and the quality of their implementation, Tornado wound up being a major win for them, and is credited with supporting both their core features and their eventual acquisition by Facebook.
A New Rule of Thumb
So what should you do when faced with a build or buy dilemma? I propose a new rule of thumb: build a quick-and-dirty solution internally, then reevaluate. This gives you some of the payoff quickly and without a cash expense (compared to commercial solutions). But more importantly, by biasing toward building it ensures that you truly grok the problem space. As an engineer, you’re probably the kind of person who best understands something by trying to build it yourself (potentially after you took it apart). Once you understand the requirements and the implementation difficulties deeply, you’ll be much better prepared to evaluate the available third-party options. Likewise, you’ll have a good idea of what it would take to build the full solution yourself — just be careful not to get sucked down this path prematurely.