Find us on GitHub

Teaching basic lab skills
for research computing

Software Engineering: Introduction

Hello, and welcome to the first episode of the Software Carpentry lecture on software engineering. This episode will explain what software engineering is, how it's different from software carpentry, and give you some rules of thumb for figuring out how formal your software development process should be.

The term "software engineering" was first used at a NATO conference in 1968, and was chosen to get people thinking about how we ought to build large, complex software systems.

People have argued ever since about what the term actually means, and whether "engineering" is the right model to aspire to.

For example, should we be able to sue programmers for making flaky software the same way that we can sue engineers for designing faulty dams and bridges?

On the one hand, quality would probably improve.

On the other hand, despite fifty years of hard work, mathematical techniques for analyzing programs are still quite limited, and it's hardly fair to require someone to meet standards that aren't yet achievable.

Either way, most people's definition of "software engineering" includes project management, which is what this lecture will focus on.

For our purposes, the difference between software carpentry and software engineering is the difference between putting up some drywall and building a school.

A reasonably skilled amateur can do the first in a few hours or a couple of days with a little bit of planning, a couple of sketches, and a few hand tools.

The second is more involved: it requires a team of people (some of them specialists), bigger tools, and a lot more planning and monitoring.

The good news is, the survey we did in 2009 showed that almost all of the software that scientists and engineers write lives at the carpentry end of the scale.

The bad news is, many people don't recognize when the techniques that have served them well for years are no longer adequate, and that's one of the main reasons that many large software projects (in industry as well as in research) come in late or over budget, or wind up failing completely.

While there's no clear dividing line between software carpentry and software engineering, we believe very strongly that nobody should try to tackle the second until they've mastered the first.

We also think it's important for people to realize that there are differences: things that work well when a single person is writing code for their own use can only be scaled up so far.

So here's a quick rule of thumb: if a dozen people are involved in your project, or if the work is going to take more than a dozen weeks (i.e., one quarter), then you're doing engineering.

And like it or not, you're going to have to think about how you plan, scope, and monitor your work.

The only question is whether you're going to do this at the start of the project, or after everything has come crashing down around you.