TIL: Monorepo Makefile inheritance with shared variables and targets

Makefiles are handy. When you check out a project, it’s nice to just be able to run some simple make commands and get stuff working. This is true even when you’re not working on a C/C++ project, and the Makefile just contains a bunch of .PHONY calls that actually call a language-specific tool, like Cargo or NPM or Poetry etc. Having a consistent way to build, test and work on projects is a good thing and reduces cognitive overhead.

If you’re using a multi-project repository,¹ sometimes referred to as a “monorepo”, it’s quite common that you will want to share Makefiles across the packages in a sensible way.

Consider a project containing:

Makefile
packages/a/Makefile
packages/b/Makefile
packages/c/Makefile
shared/base.mk

The root Makefile may essentially be something like a big for loop that runs across the component packages, or it may kick off some code that builds the packages, links them, does cross-cutting concerns, builds Docker images or provide toast and light conversation. Interesting, but not our concern here.

The shared/base.mk defines the tasks that run in all the packages. Here’s a simple example using Python.²

some_var="abcd"

check:
	poetry run python check_script.py -- $(some_var)

mypy:
	poetry run mypy

serve:
	poetry run gunicorn -w 4 $(pkg):app

test:
	poetry run pytest

.PHONY: mypy server test

Each individual file, then, defines pkg and includes the shared Makefile.

pkg="a"

include "../shared/base.mk"

When you’re in package a, it’ll run with pkg="a". When you’re in b, it’ll run with pkg="b" etc.

If you need to modify it, you can do it once. No more having to create 10 pull requests for 10 different repos. Hooray! Living the monorepo dream!

But, what about if you have one package that has a specific override?

Section 3.3 of the Make manual, Including Other Makefiles says that the include directive is used for sharing a “common set of variable definitions […] or pattern rules”, not a common set of .PHONY targets you can override.

Section 3.6, Overriding Part of Another Makefile shows another route.

you can use a match-anything pattern rule to say that to remake any target that cannot be made from the information in the containing makefile, make should look in another makefile.

“Great. Let’s do that!”, the developer says with the optimism of someone who has not fully internalised the grim reality that All Computers are Bad.

Let’s say you were to put in packages/a/Makefile something like this:

pkg="a"

check:
	poetry run check_script.py --my-flag -- $(some_var)

test:
	poetry run python my_special_test.py

%:
	@$(MAKE) -f ../shared/base.mk $@

.PHONY: check test

You’re going to have a bad time. The test target will work okay, but the check target won’t, because some_var isn’t being transcluded from the base Makefile because there’s no include directive.

But if you put an include directive in, you’re going to have a bad time that results in…

warning: overriding commands for target `test'

At this point, you have a few options to solve the problem.

Don’t use Make

This is kind of a cheat answer, but it’s a real good one.

For work like this, we’re not using Make for the kind of thing it was designed around, namely compiling bucketloads of C/C++ source code into object code.

But that’s not what a lot of us do all day. A lot of the time, especially with interpreted languages like Python, Ruby, JavaScript, etc., we’re using it as a task runner to fire specific commands off, and we don’t need what Make provides, and we could use a tool that’s newer and doesn’t have the idiosyncracies and design decisions Make comes with. If damn near everything you type after make is a .PHONY target, you may not actually need Make and could use something else.

just looks pretty great in as much as the syntax is very similar to Make and the tool itself is implemented in a sensible programming language, and there’s text editor support for pretty much every editor you’d want to use.

And Task is there if you really like YAML, I guess.

There’s also whatever is loved in your preferred language, like Rake or Ant or Maven or Cargo. In JS-land, you’ve got npm run which might actually just be fine.

This obviously relies on the other humans you write software with being happy to install and learn a new tool, and that tool being available on the platform you’re developing on. (If you’re working in a nuclear missile silo in Wyoming that runs everything off airgapped 286 boxes, you do you, I guess.)

Let’s say they don’t want to do this, how do we do this with Make?

Split the variables and targets out

One approach: split the variables off, use include for those. Put the targets in a separate file, and use % to get those.

Using our previous example, we’d end up with shared/vars.mk to contain our base variables:

some_var="abcd"

And in shared/targets.mk, our shared targets:

check:
	poetry run check_script.py -- $(some_var)

mypy:
	poetry run mypy

serve:
	poetry run gunicorn -w 4 $(pkg):app

test:
	poetry run pytest

.PHONY: check mypy server test

Then in our package Makefile:

pkg="a"

include ../shared/vars.mk

check:
	poetry run check_script.py --my-flag -- $(some_var)

test:
	poetry run python my_special_test.py

%:
	@$(MAKE) -f ../shared/targets.mk $@

.PHONY: check test

This is my preferred solution. You end up with two files, which is a bit more complex, but otherwise it’s pretty clean.

Alternative: “base tasks”

Alternatively, you can create some base tasks that are called by the package-specific Makefiles, avoiding the override problem with include.

Your base Makefile ends up something like this:

some_var="abcd"

check-default:
	poetry run check_script.py -- $(some_var)

mypy-default:
	poetry run mypy

serve-default:
	poetry run gunicorn -w 4 $(pkg):app

test-default:
	poetry run pytest

.PHONY: check-default mypy-default server-default test-default

And your package-specific Makefile ends up just calling those tasks.

pkg="a"

include ../shared/base.mk

check: check-default
mypy: mypy-default
serve: serve-default
test: test-default

.PHONY: check mypy serve test

If you wanted to add a dependency to a task, you don’t end up repeating of the body of the base task in your package-specific Makefile. That’s, as far as I can tell, the major advantage of this solution. And it feels less complex because there’s fewer files, and you’re only using include rather than include and %.

The bad: well, if you add a new base task, you’ve got to add a line to each package-specific Makefile that calls it. This is why I rather preferred the earlier solution even if this one is conceptually simpler.

The ugly: you have to pick a naming convention for your base tasks. I used -default but I don’t like it much. I thought about -base or prototype- or something like that and I don’t like those either.

I considered using a single or perhaps double underscore (“dunder”) prefix, a Python convention used for marking methods and instance variables as private. Obviously, that might not be wholly intuitive for non-Python developers. The aforementioned Just uses this for private recipes… so maybe that’s a point in favour.

As we all know, naming things is hard, so pick something and prepare for people to hate it. (Or just use Just…)

This is a TIL post. What’s that?