My idea of how make should be done

Welcome dear reader!

When I start spending time with a new programming language, I usually want to get a clear picture first of how my workflow should look like. I simply want to do it right from the beginning, using state-of-the-art tools and profiting from others' experiences, so I will be able to work efficiently right away and so that my project evolves around a good structure.

Currently, I am trying out C++ (not for the first time, but this time for real), denying myself using an IDE which would get me started quickly but rendering me entirely planless of what is going on behind the scenes.
Naturally, I was confronted right at the beginning with the decision which build system to use. I started out using the best known one, gnu make, and got comfortable with it. Actually it was even so good, that when I decided to go for cmake instead (which seems to be next to autotools the only  relativley widespread build tool out there), I couldn't tell any improvements over make.

So here I want to give you an overview of the conclusions I've drawn about how to use makefiles in a project-independent, yet absolutely clean manner:

1. of all: The project's top-level structure:
   I wanted a clean top-level project directory with nothing in it but introductory documentation (license, readme...), one folder for source (src) and one to build into (build). Lateron, looking into other projects, I also saw that being able to offer support for several other (maybe ide-specific) build systems would be great, too, so I decided to give each one of them a separate directory - "Make" doing the first step here.
   That leads us getting something like:
• project root
• build
• src
• Make
• ...
• XCode/Eclipse etc.
This way we can also strictly keep the build-tool related files from our code, as another positive sideeffect.


2. how to even make it work:
   Now that we have decided to put all our makefiles into the Make directory, how will make get to see the files it has to build together? It is not even in the project's root directory. That's why we need to get this root directory first - the parent of the folder, our makefile resides in:

    ROOTDIR            = $(realpath $(dir $(realpath $(dir $(lastword $(MAKEFILE_LIST))))))  
   

   I am using here, that calling "realpath" on a directory name will cause the original path to be stripped of trailing slashes. Therefore, another call of "dir" on it will give me the parent directory. I know, this is somewhat hacky, but we just won't mind.
   So next thing is to export our ROOTDIR variable, so that all other makefiles can use it et voilà: Accessing specific directories far away from our "Make" directory won't be a problem any more.
   
   
3. how to reuse our makefiles:
   I far as I could see, it is very common to extend a build system as the actual project grows with the time. The downside of this procedure is obvious: Makefiles written this way tend to be project specific and can hardly be reused. So we need ways to separate project specific attributes from the reoccuring parts of a makefile.
   First thing I'd recommend here is a dedicated defs.mk makefile containing all or most of your definitions. This way you will have all your project-dependent stuff just there and this will be easy to understand for external collaborators. Note that my make_boilerplate contains a nice draft of such a defs.mk file which you can use and modify as you like to.
   In the future, we will just include our definition file to gain access to our whole project's configuration:

    include $(ROOTDIR)/Make/defs.mk  
   

   Yes, I do know about the "MAKEFILES" environment variable, but I think, this way it is just more transparent. So let's just do it this way.
   
   
4. recursive makefiles(?):
   Coming from Java WITH IDEs, I had a pretty detailed vision of how I wanted my project to be structured. Over all, folders play a central role in dividing pieces of code into single units of related functionality. But how to tell make that those source files, distributed over several hierarchies of folders, have to be built into one single binary?
   
   Recursive makefiles - while seeming obvious at first (icu, for example, is doing it this way) - didn't comply with my ideas stated above, since they had to lie in the source directory. Additionally, it relatively quickly turned out, that recursive use of makefiles is actually not desireable. Click here for details.
   
   So I went for good old "find" to get my .cpp files which I would then turn into a list of .o files:

    CXXFILES       = $(shell find $(SRCDIR) -type f -name '*.cpp')  
    CXXOBJECTS     = $(patsubst $(SRCDIR)/%.cpp,$(OBJDIR)/module/%.o,$(CXXFILES))  
   

   Those, as you can see, I put into their respective subdirectory in my object build directory.
   
   
5. where to divide makefiles:
   When working with makefiles in a project, you often have them grow according to your needs while you are concentrating on your code. Therefore one often ends up having huge and unreadable makefiles that are likely to break on minimal changes. I suggest the following here: Have one makefile for each big part of your project. This will not contradict our earlier statement "recursive make considered harmful", since the makefile is still as self-contained as possible and should not call any sub-makes either. We just use the modularity of our project to have clean cuts. I suggest one makefile for every lib or binary of your own project, maybe one makefile for each of your dependencies if they have to be built from source or one single makefile together for all your prebuilt libraries.
   Standard targets that can be "outsourced" include:
   
   • deps.mk (for dependencies)
   • tests.mk
   • <projectname>.mk for your project
   
   
6. handling indirect dependencies:
   We know that make works using a file's dependencies to create the file itself, if those are newer. Therefore we have to specify the dependencies, which we already automated. But we completely left out the fact, that header files included in our .cpp files are also dependencies.
   In order to be able to respond to changes made to only those included files, we will use a trick which I borrowed from here: We exploit the -M option (for gcc and similar), which gives us all files included by a codefile and write them into a seperate makefile, which we will include from now on. These secondary makefiles we will be giving the suffix ".d", as in "dependency". So the rule for .o files will be looking like 

    $(OBJDIR)/%.o: $(SRCDIR)/%.cpp  
        @mkdir -p $(dir $@)  
        @$(COMPILE.cxx) $@ $^  
        @# Create the .d file using gcc's -M or -MM option  
        $(CXX) $(CPPFLAGS) $(CXXFLAGS) -MM $(lastword $^) > $(patsubst %.o,%.d,$@)  
   

   Additionally, we will have to retrieve the existing .d files at the beginning of our makefile and include them:
   

    DEPFILES        = $(CXXOBJECTS:.o=.d)  
    -include $(DEPFILES)  
   

   The "-" in front of include turns off errors on file-not-found. And that's it!
   Make sure though that you include the DEPFILES only after the "all" target, so some object file won't become your default target.
   
   
7. coping with different levels of verbosity:
   In general it has to be said, that make's philosophy can be a bit obstructive when it comes to communicating what is going on. First of all, there is no parse-time option to output messages to the console. Trying it with approaches like "$(shell echo some message)" will fail because make' s goal here is it to grab the shell command's output. So you will not be able to just dump all variable values of interest on make startup before a target is built. You will rather have to have an own "info" target doing that, which is prerequisite of your "all" target or you are calling yourself.
   
   The next problem is, that when you have a simple target that just aggregates other tasks, and you want to announce its start and its end, at least the first of those two cannot be achieved. For example, imagine you want to make sure, all build directories are set up correctly in target "directories: dir1 dir2 dir3". Where will you put the "Start creating directories" message? You will have to live with that limitation or invent a smart way to avoid this.
   
   To have control over what is printed and when, I suggest setting up variables like "PRINT" (echoing always) and "VPRINT" (echo if in verbose mode, otherwise just /bin/true). This can be extended to "VVPRINT" (VERBOSE = 2) etc. if necessary.
   Additionally, the make variable "MAKECMDGOALS" can be helpful to detect if the user ran make with the intention of getting more information as usual about the build process. I, for example, did the following right at the beginning of my main Makefile: 

    ifeq ($(filter info, $(MAKECMDGOALS)),info)
        VERBOSE = YES
        export VERBOSE # for all sub-makes
    endif
   

On these thoughts as a basis, I set up a small github repository I called "make_boilerplate". You will find it by following https://github.com/suluke/make_boilerplate . Maybe things will also be clearer if you take a look into the makefiles themselves.

I've been reading lots and lots of articles and stackoverflow questions on the internet to get all this done in a way I personally can live with. I don't guarantee you, though, that it is bugfree or even trapless. I see a high chance that some professional with 40 years of experience shows up and tells me about a bunch of flaws my makefile design comes with. Personally I don't see any problems so far though - and this is why I posted this. I hope, you will find it helpful, too.

Regards

suluke


Sources:

• Obtaining directory makefile resides in: http://stackoverflow.com/questions/4555120/obtaining-directory-makefile-resides-in
• GNU make Quick reference: http://www.gnu.org/software/make/manual/html_node/Quick-Reference.html
• Recursive make considered harmful: http://aegis.sourceforge.net/auug97.pdf
• Handling of indirect dependencies: http://scottmcpeak.com/autodepend/autodepend.html