Hello, and welcome to the third episode of the Software Carpentry lecture on Make. In this episode, we'll have a look at how to use patterns in filenames.
As we said in earlier episodes, we're exploring a tool that will manage tasks and dependencies for us.
Here are the dependencies for the paper the robot is working on: paper.pdf depends on paper.wdp, figure-1.svg, and figure-2.svg, while figure-1.svg depends on summary-1.dat, which in turn depends on all the files with names like data-1-1.dat, data-1-2.dat, and so on.
To re-create summary-1.dat from the raw data files, we run a Python script we wrote ourselves called stats.py. We're still updating that and fixing bugs as we find them. Each time we change it, we want to regenerate summary-1.dat, so stats.py is actually another dependency of summary-1.dat, as well as being the tool used to create it.
The problem we're going to explore in this episode is how to express the idea "all the files named data-1-something.dat". We don't know in advance how many of these there will be, and we don't want to have to rewrite our Makefile each time we add a new one.
We'd also like to figure out what to do about figure-2.svg and the files it depends on. The rules are exactly the same as those for figure-1.svg and its prerequisites; duplicating them is just asking for trouble.
Let's start with the case of three files data-1-1.dat, data-1-2.dat, and data-1-3.dat. As we said at the end of the previous episode, it's easy to write a Make rule to update summary-1.dat whenever any of these, or the stats.py script, changes.
We'd like to do better, though, so let's replace the action in the rule…
…with this line.
Instead of naming summary-1.dat in the rule's action, we use the rather cryptic shorthand $@.
$@ is one of Make's automatic variables.
It means "the target of the current rule". In this rule, for example, it means summary-1.dat.
And no, there isn't a more readable long form of the name: it's just another of Make's many warts.
Using $@ instead of repeating the target's name shortens our rule somewhat, but writing the many prerequisite filenames twice is still redundant.
Let's fix that by replacing our shortened rule command…
…with this.
$^ is another automatic variable: it means "all the prerequisites of this rule". In this case, it's the three raw data files, so when Make expands the variables in stats.py $@ $^, we get back our original command.
There are other automatic variables as well: for example, $< means "the first prerequisite in the list"…
…and $? means "all prerequisites that are out of date". Don't worry if you can't remember them: everyone except the most passionate Make user writes them on a sticky note and puts it on their terminal.
Using the automatic variables $@ and $^ eliminates the redundancy in our rule, but doesn't solve the problem of handling an arbitrary number of prerequisite filenames.
We expect to have more than three data files before this project is done, and as we said before, we don't want to have to rewrite our Makefile each time we run our experiment.
What we really want is something like the shell's * wildcard, which matches any number of characters.
Lo and behold, that actually works: we can use data-1-*.dat as the rule's prerequisite…
…and it behaves just like the corresponding shell wildcard.
When we do this, we must use $^ to refer to the rule's prerequisites in the action: we don't know exactly what filenames will match, so we have to rely on Make to put them in an automatic variable for us on a rule-by-rule basis.
Here's our dependency tree one more time.
And here's our Makefile.
There is still some redundancy: we have exactly the same logical rules for our two data series, but have to write them down separately because the '1' and '2' in their names are different.
We'll see how to fix this in the next episode.
Before then, though, we have one more problem to address: our existing Makefile doesn't capture the fact that summary-1.dat and summary-2.dat depends on stats.py as well as on their corresponding raw data files.
We could try to fix this by adding stats.py to their prerequisite lists.
If we do this, though, stats.py will appear in the value of the automatic variable $^ for those two rules.
So when we run stats.py, our command line will be stats.py summary-1.dat stats.py data-1-1.dat data-1-2.dat and so on—i.e., we'll be telling stats.py to process itself as a data file, which is almost certainly a bad idea.
We could "fix" this by having stats.py ignore files that end in .py, or something like that, but it would be an ugly hack.
A second option would be to move the dependency down, and pretend that the raw data files depend on stats.py.
This is called a false dependency.
The raw data files don't really have to be updated when stats.py is changed, but with this false dependency in our Makefile, Make will update the timestamps on the raw data files when stats.py changes, which will in turn trigger an update of the summary files.
False dependencies do solve some problems, but not this one: if we go down this road, we have to list all our raw data files explicitly once again, which is what we're trying to avoid.
Here's our third option: add additional rules for summary-1.dat and summary-2.dat that add stats.py as a prerequisite, but don't have any actions.
When Make sees multiple rules for the same target, it takes the union of all the prerequisites in those rules as the target's actual set of prerequisites.
However, the automatic variable $^ in the rule is still just that rule's prerequisite list. It's a bit of a hack, but it means that our command line has exactly what we want it to have.
In the next episode, we'll see how we can leverage patterns to write generic rules for Make.