build process

usually when starting new project it uses either IDE build-in 'project' that tells compiler how to make it, either uses fast written makefiles to make it flexible or autoconf and automake for generating makefiles for us.

although all mentioned before methods sound familiar there are few problems with each of them:

• IDE-integrated build does not allow easy porting.
• hand written, project-specific makefiles are error-prone and usually not very nice-looking after a while.
• auto-tools are good if you want to prepare release code for user but using them during development, when many things changes in structure over time is not very nice job.
• usually no support for automatic and manual test of code is present.
• (almost?) all proposed solutions build in directory with sources. this is bad because:
  • mixing sources and output files decrease readability of both.
  • when you use different profiles (ex: debug, test and release) you need to clean object files and try to build once again whole project. this is inacceptable in bigger projects that may require tens of minutes or even hours to compile!

because of these, and few more, i've deiced to create generic, easy to port and adopt, build process which i can use in any sort of project.

NOTE: as i'm coding mostly in C++ and C, presented build process works with this type of code.

there are few basic concepts that are implemented. they will be shortly described bellow.

as mentioned in introduction part there is a strong need to separate source files from object files. in proposed solution it is done by creating special 'gen/' directory in project root directory. subdirectories of it are profiles that are build (ex: test, release, debug, etc…). then this directory is a root for project output files. each library and application will have it's own directory for output.

in most cases there is a set of predefined profiles that user may need. in this case following are available:

• debug – build debug information on everything that is possible
• release – build with optimizations and dissable asserts.
• profile – build in profiling mode. this will require gprof installed to read generated stats on execution time of particular calls.
• test – build automatic tests for chosen library/application.
• mtest – build manual tests for chosen library/application.
• doc – generates doxygen documentation.

to make file simpler there is always requirement to test each and every part of software. automatic tests allows to quickly see if all of them works fine. in proposed solution tut framework is used. this creates single binary for each library/application. when you run it it shows all tests that pass and long description of those which failed.

though automatic tests are extremely usefull in every day life, there are some situations when writting automatic test is not possible, or at least very hard and not worth all this time (ex: consider wrapper for system call that show something on the screen). not to leave such a piece of code untouched you should write manual test for this situation. this is binary that calls single, tested functionality and you can see and judge results your self.

to automatically solve dependencies between separate sources is obvious. to go one step ahead dependencies between components (libs/apps) are solved semi-automatically. all you need to do is specify what libraries are needed to build each one and building tree from this is done for you.

some libraries does not provide specific libraries that are need but instead supply some executable that generates flags for given compiler and linker. to cope with those ones there is a backend allowing to place any binary that will produce a list of libraries to link with, link parameters and compile time switches.

there is sometimes need to do something like static polymorphism on source-code level. this happens if some code does not have sense in some cases, or if we need to build different code depending on other circumstances (ex: depending on profile). this mechanism is called split backend. in general, in C and C++ projects it works by specifying single header set and two or more sources that need to be build depending on given mode.

example library tree for having split backend could look like this:

lib1/MyClass.hpp
lib1/FakeHandler/MyClass.cpp
lib1/RealHandler/MyClass.cpp

each time only one MyClass.cpp will be build and linked in.

when specifying library's API you need to show which headers are public and can be used by others and which are library privates. in build process i propose by default all are private. to make them visible to others one must put special marker into header file:

/* public header */

files marked this way will be visible by external libraries and applications. rest is visible only inside library.

example usage can be found in test_app/lib2/Lib2/ObjectToForceLink.cpp file, inside build process' release.

there is a set of makefiles to be used for building library, application, project, etc… theses are ready to use and fully automatic. there is no need for user to change anything inside them. all he have to do is to put symbolic links in proper places.

this allows easy changing build process version. all you need to do is replace old directory with makefiles set with new ones! since all symbolic links point in the same place, but files itself have changed, changes are visible straight away.

in most cases there is no need to go into any Makefile. thought from time to time it happens that some code could be added there to prevent or force doing something. since assumption is not to change any build process files, there is a back door provided. you can put files of special names in project, library or application root directory and they will be automatically included in generic makefiles allowing inserting own code.

the most important of those is config.mk placed in project's root. you can put default setting for build there, so that all you'll need to write to build default mode is:

make

toolchain to be used for building can be easily selected by TC parameter to make (i can be also put in build_config/config.mk). currently available is 'local' that used default toolchain to build everyting. new ones can be provided by creating new toolchain description files under build_process/makefiles/common/toolchains/<tc-name> directory.

currently following toolchains are supported:

1. gcc – uses local, default tools (gcc, g++, ar, etc…)
2. avr – uses tools provided with avr-gcc, and related
3. intel – uses Intel's icc toolchain (icc, xiar, etc…)
4. clang – uses clang C/C++ compiler's suit

new parameter MEM_DEBUG=1 can be passed to make causing linker to use library electric fence to be used as memory manager. it helps to debug many memory-related problems.

integrated possibility to use test data for tests. if test requires ex. reading external file(s), it can be provided with one, that is accessible from test's output directory. dependencies of test data are traced as well as the code, i.e. when you change test data file it will be automatically copied to the destination place.

ccache is now supported out of the box. to run build with ccache simply pass WITH_CCACHE:=1 option when calling 'make', or add it to build_config/config.mk (note: since v1.1.0 it is enabled by default).

distcc support is now available. to use is add WITH_DISTCC:=1 option when calling 'make', or add it to build_config/config.mk.

3 new variables have been added to build_config/config.mk, that allows to specify extra compilation flags for a given project:

1. USER_OPT_FLAGS – flags to be used when doing optimized build
2. USER_DBG_FLAGS – flags to be used when doing debug build
3. USER_PRF_FLAGS – flags to be used when doing profile build

some components can be now build only in some of the profiles now. this is especially useful when building (m)test profiles, since it allows to create test-only components that prepare test data and provide proper stubs, commonly used when testing. these components never appear in release mode and similar.

if you create small project, consisting of just a few files, in most cases it would be an overkill to use full-featured build, that requires some configuration to be done first. instead one may use 'small projects build process', which provides most main features of full-build-process, but consists of only one make file and does not require explicit configuration. it is just enough to put it in sources directory.

example of usage can be seen in test application (small_prj_build_process/test_app/), available in build process' release.

force-link allows forcing linker to include given symbols (i.e. object files) in final binary, even though they would be normally discarded as not used. this is mainly useful when using auto-registration mechanisms for design patterns like abstract factory.

usage is extremely simple – in object file that has registration code, and must be included in final binary, even though linker can remove it, one must include one (C-code) header and use macro to define unique name of this element. see the example:

#include "BuildProcess/ForceLink.hpp"
// ...
FORCE_LINK_THIS_OBJECT(MyNamespace1_MyNamespace2_MyObjectToForceLink)
// ...

build process now allows easy importing of components based on auto-tools (./configure && make && make install). now it is enough to extract such a component to directory with typical structure and link make to AutoToolsMakefile. build process will do the rest!

you can set dependencies for such components as well, which makes it possible for quick and easy inclusion of external libs which have some dependencies between them as well.

since v1.7.0 build process allows 'features', that are more flexible (i.e. scalable) extension of 'modes' concept (see: 'split backend'). features can be build or not, i.e. being turned on/off during compile time, by selecting (or not) given name.

as a simple example, to give an idea how it works, consider two implementations of greetings for your program: normal and slang. lat us assume following directory structure:

mycomponent/MyComponent/hello.hpp            // void hell(void);
mycomponent/MyComponent/Impl/normalHello.cpp // void hello(void) { cout<<"hello world!"<<endl; }
mycomponent/MyComponent/Impl/slangHello.cpp  // void hello(void) { cout<<"yo there!"<<endl; }

to enable features add extra directory 'features' in 'modes' with proper names. for example:

mycomponent/features/modes/default # MyComponent/*.[ch]pp
mycomponent/features/modes/features/normal # MyComponent/Impl/normalHello.cpp
mycomponent/features/modes/features/slang  # MyComponent/Impl/slangHello.cpp

to enable given feature just add it to build parameters, 'FEATURES' variable:

make FEATURES="normal" # builds with 'normal' feature turned on
make FEATURES="slang"  # builds with 'slag' feature turned on

you can specify multiple features separate with spaces. features does not need to be implemented in every component ('modes' does!).

note that 'features' are often very useful along with automatic registration mechanisms like abstract factory, but can be used as a more selective 'split backend' as well (see previous example).

the example of some features can be seen in action in downloaded build process file. there is test_app that is kind of playground/example to see and test.

bellow are some examples of calling build process to build specific source with given parameters.

to build all type:

make PROFILE=release TC=local

or to build single app/lib:

make PROFILE=release TC=local somelib

to build all type:

make PROFILE=debug TC=local

or to build single app/lib:

make PROFILE=debug TC=local somelib

to build all type:

make PROFILE=profile TC=local

or to build single app/lib:

make PROFILE=profile TC=local somelib

to build single app/lib in test mode enter:

make PROFILE=test TC=local somelib

note that all dependencies of 'somelib' will be build in debug profile!

to build single app/lib in test mode enter:

make PROFILE=mtest TC=local somelib

note that all dependencies of 'somelib' will be build in debug profile!

default mode is called simply 'default'. if you need to use other mode, you'll need to specify it explicitly (or via build_config/config.mk). example o building release on mode named 'fakedriver'.

make PROFILE=release TC=local MODE=fakedriver

when creating new project following rules must be respected:

1. all libraries and applications need to be placed in separate directories in project's root.
2. all makefiles should be links to corresponding build process makefiles. in particular main project makefile should be link to ProjectMakefile inside build process.
3. build_config/components.lst file must be created in project root, with spaces separated list of all libraries and applications that are present in projects (use directories names).
4. for each library/application must be build following way:
   1. makefile must be link to proper build process makefile.
   2. sources should be placed in single directory.
   3. 'features' directory should be present with following content:
      1. 'deps' dir with files that have names like profile names. each file specifies dependencies on other libs/apps depending on given profile.
      2. 'modes' dir with files that names correspond to each mode. each file has line separated file expressions to selecting source/headers to compile.
      3. 'testdata' dir – here should be placed all data that will be needed for test and mtest applications (they are copied to gen/ directory).

latest version of build process can be downloaded via github. download it directly via lates releases tab.