Sunday, December 8, 2013

My Take on Development Best Practices

I was recently asked to contribute to a VentureBeat feature on best practices for software engineering. I'll withhold my opinion on the piece; for the record, here's the full text I provided:

A true master developer knows how to maximize velocity while maintaining quality.

Here's my top ten:
  1. KISS - complexity is your #1 enemy. Everything should be as simple as possible. Invest in simplifying your architecture and codebase and it will pay off in velocity. When there's more than one way to go, choose the simpler way. Don't over-engineer things. Anything that's "unavoidably complex" is best avoided entirely.
  2. TDD is how you build defect-free code as quickly as humanly possible. Never, ever, say you'll write the tests later, or "it's done except for writing tests" - that's how they did it in the '90s. We know better than that now. Measure test coverage and keep it at 100%.
  3. Build your prototypes under the assumption that they'll evolve into your product whether you like it or not; you may intend to throw it away and start from scratch later but it usually doesn't work out that way. This means start writing tests when you start writing code.
  4. Develop your software securely from inception since it's extremely painful to "add security" later. If you don't know how to do this, get someone on your team who does.
  5. Keep your workspace clean - purge all unnecessary code, remove unnecessary dependencies, drop DB tables you no longer need (though you might need to retain backups), keep your team wiki up-to-date.
  6. Automate QA and ops as much as possible - development is inherently iterative so maximizing velocity means driving your iteration time down. Aim for at least push-button if not fully automatic deploys. Build on every commit to master.
  7. Make your tests as high-veracity as possible. It might seem like a lot of work, but it's an investment you'll be thankful for when you narrowly avoid shipping a subtle, nasty bug.
  8. Software engineering is a team sport; peer review all commits. A critique-seeking, low-ego culture fosters team-wide excellence.
  9. Deploy whenever you have something ready to release; otherwise you've paid for the work but you're not realizing the value of it.
  10. Only refactor when the problem and the solution are both staring you in the face - premature refactoring is a huge waste of time. Don't invest in perfecting anything that might conceivably be dropped in the next 6 months. And premature optimization is the root of all evil.
Success is when a single developer can fearlessly rewrite core modules in the morning and deploy to production in the afternoon with zero disruption for your customers.

Amplifying Expertise – and Accelerating Learning – through Apprenticeship

Organization is a means of multiplying the strength of an individual. - Peter Drucker, The Effective Executive
In software engineering, conventional wisdom holds that the time a senior engineer spends not being hands-on is time wasted, and this includes time spent discussing and reviewing the work of junior engineers. As one writer put it: "[Having] junior developers report to senior ones...reduces the time spent on development by the best technicians."

I contend the opposite is true: any time a lead engineer spends doing low-value work like writing accessors is an expensive waste, like using an atomic clock to time your eggs. I'd say the ideal use of a senior engineer's time is directing junior engineers in the execution of designs, investigations, and experiments. At architecture firms, partners don't make the blueprints.

In my ideal dev team, a master engineer designs the solution, then communicates it to midlevel engineers, who create skeleton classes and tests and then hand these off to junior engineers, who fill in the required functionality. Defects flow in the opposite direction - junior engineers do the initial investigation, asking for help if they need to. Once the bug has been found, the junior engineer can fix it or escalate it; either way, his solution is reviewed by a more experienced engineer.

In this way, the junior members of the team learn from the senior members, just as they do in law firms and hospitals - under close supervision. This is very different from just turning inexperienced engineers loose on the codebase, as is commonly done. I know I would have benefited enormously from closer supervision and assistance early in my career, in both far greater productivity, which would also have benefited my team, and accelerated professional development.

Monday, January 21, 2013

What does it mean to be a tech team lead?

In journalism, a reporter writes a story and an editor edits it - reading through the whole piece, catching errors or omissions in writing or reporting, asking questions, and making suggestions at all levels - from how to catch and hold the reader's interest to how to structure the story to how to smooth out an awkward paragraph or sentence. This is A Good Thing, because a) it's always good to have someone else go over your work and b) the editor is usually the more experienced writer. In fact, it's such a good thing that on the occasions when an editor writes a piece, it undergoes something called a "top-edit", where it's edited by another editor. This is because it would obviously be crazy to publish something without someone else having read and reviewed it. But in software engineering we do this all the time, and consider it a cornerstone of our hacker-ethos, meritocratic culture.

I think this is bonkers. I've noticed that the code of junior engineers is often flawed, sometimes at multiple levels, in ways that are easy to avoid and easy to teach them to avoid. And I've noticed that if I delegate a modest-sized project and then walk away until a code review a few weeks later, it will often be the case that the architecture was ill-conceived and the process won't scale, or has lots of nasty failure modes, and we'll have to either go with what's been built or go back and do it properly.

I've also noticed that junior engineers will silently bang their heads against things that I can often help them resolve if I just occasionally stop by and ask them how things are going. But instead of just giving them the solution I'll try to teach them how to ask the right questions and design and run experiments to figure out what's going on for themselves.

Eventually, I came to think of these editorial and mentoring activities as not ancillary to or a distraction from my job, but the core of my job, and my reasoning comes down to amplification. In journalism, you typically have several reporters funneling their work through one editor, and it seems obvious to me that the aim is to achieve the volume of output of many writers at the quality level of a very experienced writer, while maintaining a consistent voice across the entire organization. The editor isn't sidelined by not doing her own writing, she's amplified. This pattern is everywhere - from the fine arts to the sciences to law to most trades I can think of - but I've never seen it in software engineering, perhaps because senior engineers tend to enjoy doing their own coding and resist being pulled away from it.

So I now view my role as something like an editor, coordinating and overseeing a team of engineers to amplify my own skill and experience and help them develop theirs. Any time I'm off by myself churning out test cases is time I'm not solving hard problems or helping someone else learn how to solve a hard problem. But for any code I write, I implement top-edits by submitting pull requests for my own contributions - and sure enough,  junior engineers will catch me making stupid mistakes. Also, having them examine my code critically keeps them abreast of changes to the codebase and gives them a chance to learn new techniques and ask questions. It's similar to pair programming, but one of the pair is always me, and I'm always rotating among the team, offering help and feedback wherever I can. The result? The productivity of many engineers, at the quality level of a lead engineer, and a happy team, rapidly growing their skills.

Tuesday, December 11, 2012

How to add NAT to an existing EC2 instance in an AWS VPC

I'm in the process of moving infrastructure from the AWS public cloud into a VPC, and since my VPC has private subnets, I need NAT to allow machines in these subnets to open connections to S3, Ubuntu repos, etc. The VPC wizard can create a NAT EC2 instance for you, but I'm a penny pincher and didn't want to dedicate a whole instance just to NAT - e.g. I also wanted to use it as my VPN server. But I couldn't find specific instructions on exactly how to add NAT to an existing instance with a single NIC (most solutions require two NICs in a standard gateway configuration, one for the LAN IP and one for the WAN IP, which you can actually do with VPC, but why do it if you don't have to?). After a bit of research and some reverse engineering of the AWS NAT solution, here's what I came up with:

  1. In the EC2 console, disable source/dest checking by right clicking on the instance you want to use for NAT and choosing "Change Source / Dest Check".
  2. Create a security group having an inbound rule allowing ALL from and associate it with your NAT instance.
  3. On the NAT instance, create /etc/network/if-pre-up.d/nat-setup as:

    echo 1 > /proc/sys/net/ipv4/ip_forward
    iptables -t nat -A POSTROUTING -s -j MASQUERADE
  4. chmod +x the script, then run it. This script will automatically be run when the machine reboots, so your NAT will survive a restart.
  5. Make sure all your private subnets have a default route to use the NAT instance as a gateway (create a route for and associate it with your NAT instance in the route table(s) associated with your private subnets).
  6. test your NAT by pinging something from an EC2 instance in a private subnet
Note: the NAT instance needs to have an Elastic IP, of course.

Wednesday, November 7, 2012


I've always wondered how the relay got its name; I thought it was a strange moniker for an electromechanical switch. Reading The Information the other day, I finally found out. I had thought relays were originally invented to enable automatic circuit switching in telephone networks, but it turns out the telegraph was the necessity of their invention.

Telegraph signals weaken as they travel down a wire; beyond a certain distance the original signal can no longer power a receiver. So what do you do? You say, well, let's have an intermediate telegraph operator: he can be stationed where the signal is still intelligible, and simply re-transmit what he receives, using his own power supply to generate a full-strength signal.

And then someone clever realizes you can use an electromagnet to operate the re-transmitting key and the incoming line to power it. Yes, the first relay was a telegraph key with an electromagnet stuck on it.

Friday, October 19, 2012

A simple quantitative metric for comparing programming languages

Try this: pick up a copy of JavaSript: The Good Parts ($30; 176pp) in one hand and a copy of The C++ Programming Language ($75; 1040pp) in the other. Now: not knowing anything else about them, which language would you rather use? Which language would you suspect has more sharp edges that you'll have to arduously learn to avoid as you cut yourself on them, one by one, over the course of months or years?

The C++PL is a great book - comprehensive, well organized, clearly and enjoyably written - but surely the heft of the user's manual says something about a gadget. And before you argue that the JS book doesn't have any I/O complexity to deal with, I'll throw in the entire Node.js manual for free - and I'll bet the difference in word count is still an order of magnitude. In UI design, the goal is to not need a manual at all - shouldn't that hold true for programming languages as well? What would Tony Hoare say?

He would say this:

"Programmers are always surrounded by complexity; we cannot avoid it. Our applications are complex because we are ambitious to use our computers in ever more sophisticated ways. Programming is complex because of the large number of conflicting objectives for each of our programming projects. If our basic tool, the language in which we design and code our programs, is also complicated, the language itself becomes part of the problem rather than part of its solution."

Sunday, October 14, 2012

Venn Diagram Fail

Presumably, multiple individuals saw, interacted with, and approved this graphic for it to be in an advertisement, and none of them know how Venn diagrams work.