Cross platform build tool for C/C++/Obj-C/C#/CUDA/opencl ||||| ||||| ||||| | | ||||| | | | | | || || | | | || || | | | | ||||| | ||| ||||| | | | | | | || || | | | || | || | | | || || ||||| | | ||||| | | Testers/comments/suggestions/feature requests welcome. Email me or make an issue. maiken - mkn - master ====== 1.1 Preamble A minimal YAML based productivity tool. 1.2 License BSD 1.3 Application overview Languages supported Compilers supported - C - gcc/clang/hcc - C++ - MSVC (cl.exe) - Obj-C - nvcc - CUDA - MSVC#(csc.exe) - opencl - Intel C Compilers (icc/icpc) - C# (partially) 1.4 Distributions Binaries available @ https://github.com/mkn/mkn/releases/tag/latest 2.1 Basic project mkn.yaml files 2.1.1 Binary name: exe_project main: cpp.cpp 2.1.2 Library name: lib_project src: ./src 2.2 YAML Schema Values which allow property resolution are marked with ${aaa} name: name version: version # considered git branch for pulling scm: string # Appended to remote URLs for lookup if non concrete URL, used if concrete super: $directory # Inherit properties of project at directory, super (..) or sub directory advisable parent: $directory # See profile inheritence @ 2.2.4 property: - aaa: AAA bbb: ${aaa}.BBB inc: ${aaa} src: ${aaa} path: ${aaa} # extra library paths for linking, generally inadvisable unless project is in sub directory lib: ${aaa} # prefixes/file types automatically deduced e.g. "lib: math" = libmath.a/so math.lib/dll dep: # also supports "with" syntax - name: name # Used in URL lookup if ${scm} is missing version: version # git branch and directory name of dependency under $name if no local tag scm: ${aaa} # Appended to remote URLs for lookup if non concrete URL, used if concrete local: ${aaa} # overrides local repository directory profile: ${aaa} # choose profile from config with: xxx.yyy[profile] # The same as "-w" (See 4.9) if_dep: # optional deps for OS valid values [ bsd / nix / win ] win: - name: name # Used in URL lookup if ${scm} is missing version: version # git branch and directory name of dependency under $name if no local tag scm: ${aaa} # Appended to remote URLs for lookup if non concrete URL, used if concrete local: ${aaa} # overrides local repository directory profile: ${aaa} # choose profile from config main: arg: -DTHIS_PROFILE ${aaa} # Optional additional compiler arguments link: ${aaa} # Optional additional linker arguments, applied to end if_arg: # optional arguments if case is true shared: ${aaa} # Add arguments to compilation if mode is shared static: ${aaa} # Add arguments to compilation if mode is static bin: ${aaa} # Add arguments to compilation if linking is binary lib: ${aaa} # Add arguments to compilation if linking is library bsd: ${aaa} bsd_bin: ${aaa} bsd_lib: ${aaa} bsd_shared: ${aaa} bsd_static: ${aaa} nix: ${aaa} nix_bin: ${aaa} nix_lib: ${aaa} nix_shared: ${aaa} nix_static: ${aaa} win: ${aaa} win_bin: ${aaa} win_lib: ${aaa} win_shared: ${aaa} win_static: ${aaa} if_inc: # optional includes if case is true bsd: ${aaa} nix: ${aaa} win: ${aaa} if_src: # optional sources if case is true bsd: ${aaa} nix: ${aaa} win: ${aaa} if_lib: # optional libraries if case is true bsd: ${aaa} nix: ${aaa} win: ${aaa} env: - name: name mode: prepend/append/replace # optional default is prepend value: ${aaa} profiles: - name: a inc: | , false/no/0 # Don't expose if dependency / private include , true/yes/1 # Include if dependency / public include # Include both if dependency / public includes src: | , false/no/0 # Don't find sources recursively , true/yes/1 # Find/add sources recursively # Find/add sources recursively, add file , 1, -DFlags # Find/add sources recursively, add -DFlags to file compile commands , -DFlagsForThisFile # Add and pass -DFlagsForThisFile to compile command dep: - name: name version: version local: . # Dependency on another profile in same project allowed, cycles detected - name: bbb parent: ${aaa} - name: bsd parent: a - name: nix parent: a - name: win parent: a - name: test self: ${OS} ${aaa} # dependencies on current project profiles, cycles detected sub: project # subprojects, follows "with" rules (4.9) for resolution, requires URL main: test.cpp out: ${aaa} # override default binary/library name install: ${aaa} # install binary or library to directory specified # library naming convention is libname_profile.a/so if linux, name_profile.lib if windows 2.2.1 Caveats 2.2.1.1 Type deduction If no main tag is found, library linking is assumed If no lang tag is found, language is deduced from first main tag found in file else linker inferred from max(file type) If the project is a dependency, even if a main tag is present, library linking is assumed. 2.2.1.2 Mode The mode tag may have three values: static/shared/none If the mode tag is found, it takes precedence for linking If no mode tag is found, the -K and -S args will be used if found. Linking cannot be guaranteed if no mode is used, but shared is generally default. Unless a project requires a mode, it's advised to avoid using one. So it can be overridden with -K or -S. If an application includes both static and shared dependencies, using the mode "none" is advised. 2.2.1.3 Entry points All entry points must be referenced by at least one main tag, otherwise linking cannot be guaranteed. eg. profiles: - name: a main: a.cpp - name: b main: b.cpp 2.2.1.4 Compiler deduction Compiler binares are expected to have their default names e.g. gcc/gcc.exe/cl.exe. This only applies to compilers and not archivers or linkers. Compiler masking is provided to use alternate compiler binaries with the same functionality as the underlying compiler. Masking settings.yml exmaple ``` compiler: mask: g++: mpicxx ``` Each value under "g++:" is considered to be a filesystem binary which supports all flags which are supported by "g++". 2.2.2 Properties System properties: OS - system OS, can be either bsd/nix/win HOME - bsd/nix/win(with msys/cygwin/etc) = ~/ win cmd prompt = %HOMEDRIVE%/%HOMEPATH% DATETIME - DateTime when app launched format %Y-%m-%d-%H:%M:%S TIMESTAMP - Unix timestamp when app launched in seconds MKN_ROOT - Directory of current mkn.yaml file MKN_REPO - if settings.yaml has [local][repo], it is that, otherwise bsd/nix/win(with msys/cygwin/etc) = ~/.maiken/repo win cmd prompt = %HOMEDRIVE%/%HOMEPATH%/maiken/repo 2.2.3 Profile Dependencies Adding a profile as a dependency can do a number of things: Import addition public includes Add libraries while linking. This can happen when a dependency profile has sources so is linked into a library, or This can happen when a dependency has the "lib" tag Import addition library search directories with the "path" tag 2.2.4 Profile Inheritence Using the "parent" tag, one profile may derive a number of values from the selected profile. Generally all values are inherited, however even if this is the case, some values such as the "main" tag or others with only a single value are overridden in the case they exist in the active profile. 2.3 SCM Git supported, SVN planned. 3 How to build maiken 3.1 Windows Prerequisites: - git 2.0 - Microsoft Visual Studio 2015 (Express) - Windows Dev Kit 10+ (http://msdn.microsoft.com/en-us/windows/desktop/ff851942.aspx) Operations: - git clone http://www.github.com/mkn/maiken maiken/master - cd maiken/master - make.bat - Enter directories for MSVS/WinKit when prompted Testing: static : ./mkn clean build -dtKa -EHsc shared : ./mkn clean build -dtSa -EHsc Example Windows settings can be found at: https://github.com/mkn/maiken/wiki 3.2 Unix Prerequisites: - git 2.0 - gcc 5.0 (+) or equivalent 3.2.1 Debian Operations: - git clone http://www.github.com/mkn/maiken maiken/master - cd maiken/master - shared make nix - static make nix LDFLAGS="-pthread -ldl" WITH CLANG make nix CXX=clang++ IF CLANG SECONDARY TO GCC GCC_VER=$(gcc --version | grep ^gcc | sed 's/^.* //g') CXXFLAGS="-O2 -std=c++17 -Wall -I/include/c++/${GCC_VER} -I/include/c++/${GCC_VER}/x86_64-unknown-linux-gnu" LDFLAGS="-L/lib64" LD_LIBRARY_PATH=/lib64 make nix CXX=clang++ CXXFLAGS="${CXXFLAGS}" LDFLAGS="${LDFLAGS}" Testing: ./mkn clean build -dtKOWa "-std=c++17" 3.2.1 Mac/OSX Operations: - git clone http://www.github.com/mkn/maiken maiken/master - cd maiken/master - make bsd CXX=clang++ Testing: ./mkn clean build -dtKOWa "-std=c++17 3.2.3 NetBSD/FreeBSD/etc Operations: - git clone http://www.github.com/mkn/maiken maiken/master - cd maiken/master - gmake bsd LDFLAGS="-pthread -ldl -lexecinfo" Operations: ./mkn clean build -dtKOWa "-std=c++17" -l "-pthread -ldl -lexecinfo" 3.2.3 FreeBSD Operations: - git clone http://www.github.com/mkn/maiken maiken/master - cd maiken/master - gmake bsd CXXFLAGS="-std=c++17 -Os -Wall -D_GLIBCXX_USE_C99" Operations: ./mkn clean build -dtKOWa "-std=c++17 -D_GLIBCXX_USE_C99" -l "-pthread -ldl" 4.1 Logging - [missing] - turns on NONE logging to std out KLOG=1 - turns on INFO logging to std out KLOG=2 - turns on ERROR logging to std out KLOG=3 - turns on DEBUG logging to std out KLOG=4 - turns on OTHER logging to std out KLOG=5 - turns on TRACE logging to std out 4.2 Activating Logging 4.2.1 Windows SET KLOG=3 mkn clean build 4.2.2 Linux/Mac KLOG=3 mkn clean build 4.3 settings.yaml The settings file is the default global configuration for projects. 4.3.1 schema super: $directory # Inherit properties of project at directory, super (..) or sub directory advisable property: - aaa: AAA bbb: ${aaa}.BBB local: repo: # Optional, missing assumed ${4.3.2}/repo bin: # Optional, successfully linked binaries are moved to , overrides install lib: # Optional, successfully linked libraries are moved to , overrides install debugger: # Optional debug command, overrides defaut, overriden by env var MKN_DBG string remote: repo: URL_ROOTA URL_ROOTB # Optional, overrides switch _MKN_REMOTE_REPO_ for incomplete SCM URL lookups inc: # Optional, add include directories compiling path: # Optional, add library path directories linking env: # Optional, useful for compiler specific settings - name: name mode: prepend/append/replace # Optional default prepend value: value compiler: mask: # To use a compiler binary with a name other than the default cl: $var_cl # masking is provided to alter the command line call gcc: $var_gcc # while using the same functionality as the mask key # e.g. gcc: gcc-armhf would allow the compiler tag to contain # gcc-armhf instead of just "gcc" file: # Must include at least one item in list - type: c:cpp:cxx # Sources won't be compiled if the filetype is missing archiver: ${aaa} # Archiver is not used for C# compiler: ${aaa} linker: ${aaa} 4.3.2 Location - Linux - ~/.maiken/settings.yaml - BSD/Mac - ~/.maiken/settings.yaml - Windows - IF CYGWIN/MSYS ~/maiken/settings.yaml IF CMD %HOMEDRIVE%/%HOMEPATH%/maiken/settings.yaml 4.4 Language (HELP WANTED!) English is the default language. To activate another language run: mkn clean build -a MKN_LANG=fr 4.5 Timestamp logging Upon succesful compilation of a project, source file modification timestamps are recoreded under ./bin//.mkn/src_stamp, which are checked on the next build to see if recompilation is required. Similarly, the aggregate of all non-hidden file modification timestamps are recorded under ./bin//.mkn/inc_stamp, which are also checked on the next build to see if recompilation is required. This can be enabled by compiling with the argument -D_MKN_TIMESTAMPS_=1 This is disabled for languages like C# when the source is not compiled. The clean command deletes the timestamp files, and recompiles everything. 4.6 Debugging If the environment variable "MKN_DBG" is set, it is used as the process string for debugging such that the format is: $MKN_DBG ./bin// Otherwise, the settings.yaml variable "local->debugger" is checked, resulting in the same style of string If neither are set the following rules apply Windows : cdb -o ./bin// Unix : gdb ./bin// Launch app with gdb, autorun and print backtrace MKN_DBG='gdb -batch -ex run -ex bt --args' mkn dbg -r "arg0 arg1" 4.7 Initialising scripts After a missing dependency is retrieved for the first time, an initial setup script is run to put the repository in the desired state. The type of file differs per OS, the rules are thus. Windows: ./mkn.bat Linux : ./mkn.nix.sh, or if missing, ./mkn.sh BSD : ./mkn.bsd.sh, or if missing, ./mkn.sh This can be disabled by compiling with the argument "-D_MKN_REMOTE_EXEC_=0" 4.8 Default build command If the first line of mkn.yaml starts with "#! " The subsequent string is used as the default command if mkn is executed with zero arguments or commands (-C is allowed). 4.8 Dependency build rules Using the argument -d / --dep The following rules apply If soley numeric, must be greater than 0, used to limit which dependencies are built, if 1, project immediate dependencies are built, if two, includes dependencies of dependencies. If non-numeric, CSV apply, with the value equating to the yaml root node "name", only those found, plus the current project are built. Project profiles may be specified with square brackets, e.g. "-d project[profile]" If no profile is chosen, all profiles from that project which would normally be active with an argumentless "-d" will be active. Also, if non numeric, special characters include "+", ignores current project. If only value builds all but current project. 4.9 With/out dependency build rules CSV with the value equating to either the current profile yaml root node "name" to includel local profiles OR - full project names as if dependencies i.e. mkn.kul / org.boost Project profiles may be specified with square brackets, e.g. "-w project1[profile1,profile2],project2[profile1]" If an included dependency is not from the current project, in most cases (refine) a define is added to the compilation of the current project in the form of "-D_MKN_WITH_${PROJ.ECT}_" where a project name is turned uppercase and the dot is changed to an underscore example with g++ and "-w mkn.ram" will result in a compilation command like: g++ -D_MKN_WITH_MKN_RAM_ -c file.obj -o file.cpp A concrete git URL can be specified with a string between () brackets - if these are missing the project string is attemted to be used as the URL. Further options can be used to specify the version and the location & - location # - version such that the following commands are valid -w mkn.ram#master&directory(https://github.com/mkn/mkn.ram)[https] -w mkn.ram(https://github.com/mkn/mkn.ram)[https] -w https://github.com/mkn/mkn.ram[https] -w [profile] Similarly dependencies can be removed with the "-T" option, this is for instance if you have a library installed in your system and do not wish to use the mkn configured version. 4.9.1 Add modules on the command line Similar to -w/-T Modules may be added with -m, with the same rules regarding, but with additional properties for nested attributes eg. mkn build -m lang.python3{compile{with:numpy}},lang.pybind11 -w lang.pybind11 is the same as adding the following to mkn.yaml: dep: lang.pybind11 mod: - name: lang.python3 compile: with: numpy - name: lang.pybind11 4.10 Github integration When maiken is compiled with "-w mkn.ram[https]" features are activated when attempting to resolve dependencies when they do not have versions specified. The formula to deduce the branch/tag are as follows: if there are any releases, the most recent release is used, or if there is some tags, the most recent tag is used or the default branch is queried and used finally master is used HTTPS queries are performed to the github API top resolve these, so the remote repo URL must contain the string "github.com" in order for this to work. Which by default it does 4.11 Debug / Optimization and Warning flags Options "-O / -g / -W" are provided to pass compiler and linker specific arguments for Optimizations, Debug symbols and warnings. For each argument "9" is the highest value - and 0 is the lowest and is considered to be "disable" For optimizations - 9 will try to create programs optimized for the compiling machine. For debug - 9 will have the highest debugging - while 0 will be considered to disable. i.e. "-g 0" might have less debugging than without -g at all. For Warnings - 8 is default when no value is given - 9 will include failure for any warnings when possible. Example mkn.yaml -------------- #! clean build -Kl -pthread name: example version: version main: cpp.cpp -------------- 5.1 Usage 5.1.1 Commands Available commands - build - Compiles all supported files to the profile binary (./bin/profile) directory, and creates library/executable depending - clean - deletes all files from the profile directory recursively. - compile - Compiles all supported file to the profile binary directory - link - Creates a library/executable from found compiled files - profiles - displays $(PWD)/mkn.yaml profiles to std::out - then exits (--version takes precendence) - init - Create a minimal project file ${PWD}/mkn.yaml, exits, error if one exists - inc - Print include directories to std out - src - Print found source files to std out [allows -d]. - pack - Copy binary files & library files into bin/$profile/pack - run - Starts the application automatically linking libraries for dependencies on dynamic libraries, supports -a and -p/-D - dbg - Same as run but uses debugger - see 4.6 - tree [$p] - Prints dependency tree for base profile (or profile $p) 5.1.2 Arguments Available arguments: -a --args $arg - Adds $arg to the compile command if compile/build, passes arguments to application if run -A --add $arg - CSV list of additional sources to compile/link, disable recursive finding with "\, 0" -b --binc $s - Add include directories to back of compile command, seperated by standard system PATH environment win=; others=: -B --bpath $s - Add library search directory to back of link command, seperated by standard system PATH environment win=; others=: -C --directory $d - Execute on directory $d rather than current directory -d --dependencies - See 4.8 -D --dump - Write command logs to ./.mkn/logs/$PROFILE -E --env $e - CSV key=value environment variables override format -P "k1=v1,k2=v2" , can be used with run/dbg, backslash escaps , and = -f --finc $s - Add include directories to front of compile command, seperated by standard system PATH environment win=; others=: -F --fpath $s - Add library search directory to front of link command, seperated by standard system PATH environment win=; others=: -g --debug [0-9] - Add compiler/linker debug flags for chosen compiler - 0 = off / 9 = full - number missing = 9 / no default -G --get $k - Returns string value for property K in either local yaml or settings.yaml, failure is no string 0 exit code -h --help - Print help -j --jargs $j - Takes JSON in the form of {"c": "-DC_ARG1 -DC_ARG2", "cpp": "-DCXX_ARG1 -DCXX_ARG2"} passing file-type specific args to compiler -l --linker $t - Adds $t to linking of root project -K --static - Links projects without mode as static -L --all-linker $t - Adds $t to linking of all projects -m --mod $str - Add module(s), see 4.9.1 -M --main $f - Set main file as $f, overrides yaml main tag -n --nodes [$n] - Activate distributed compilation with $n nodes from chosen settings.yaml (-x) -o --out $f - Set output binary/lib name to be $f, overrides yaml out tag -O --optimize [0-9] - Add compiler/linker optimisation flags for chosen compiler - 0 = off / 9 = full - number missing = 9 / no default -p --profile $p - Activate profile p -P --property $p - CSV key=value properties override format -P "k1=v1,k2=v2" , backslash escaps , and = -r --run-args $a - Passes $a to running binary as arguments, suppercedes -a -R --dry-run - Print commands without executing them. -s --scm-status - Display SCM status of project, allows -d -S --shared - Links projects without mode as shared -t --threads $n - Consume $num threads while compiling source files where $n > 0, If $n is missing optimal resolution attempted. -T --without $CSV - Remove profile or dependency on the command line see 4.9 -u --scm-update - Update project from SCM when permitted, allows -d -U --scm-force-update - Force update project from SCM, allows -d -v --version - Displays the current maiken version number -w --with - Add profile or dependency on the command line see 4.9 -W --warn - Add compiler warning flags for chosen compiler - 0 = off / 9 = full - number missing = 8 / no default (Requires KLOG=1) -x --settings $f - Sets settings.yaml in use to $f. Directory missing, $(4.3.1) attempted 5.1.3 Examples mkn clean build -dOtug 0 - update/clean/compile with optimal threads/link/flags in relase mode mkn clean compile link -d -t -u - update/clean/compile with optimal threads/link everything mkn clean build -dtKa -DARG - clean/compile with optimal threads passing -DARG/link everything statically mkn clean build -dtKa "-DARG -DARG1" - clean/compile with optimal threads passing -DARG and -DARG1/link everything statically mkn clean build -d 1 -t 2 -u - update/clean/compile with two threads/link project and immediate dependencies mkn -ds - Display "${scm} status" for everything mkn run -- arg0 arg1 arg2 - Execute binary passing arguments to main as argc/argv 6. Environment Variables Key KLOG Type Number Default 0 Description See section 4 Key MKN_COMPILE_THREADS Type uint Default none Description Global override, if set forces all compile calls to use value. Example, low RAM systems Key MKN_DEFAULT_BRANCH Type String Default "master" Description Default branch for SCM if not given or set Key MKN_DBG Type String Default "" Description Sets the preceding command line string when using "dbg". See seciont 4.6 Key MKN_OBJ Type String Default Linux/BSD="o", Windows="obj" Description Sets the file type to compile object files to Key MKN_LIB_LINK_LIB Type bool Default false Description Links libraries with dependency libraries (gcc and derivatives only) Key MKN_GCC_PREFERRED Type bool Default false Description When attempting automatic settings.yaml creation, use gcc even if cl/ang are found Key MKN_CL_PREFERRED Type bool Default false Description Windows only: when attempting automatic settings.yaml creation, use cl even if clang/gcc are found Key MKN_LIB_EXT Type string Default #ifdef _WIN32 evs["MKN_LIB_EXT"] = "dll"; #else evs["MKN_LIB_EXT"] = "so"; Description The file extention of shared objects being created. Key MKN_LIB_PRE Type string Default #ifdef _WIN32 evs["MKN_LIB_PRE"] = ""; #else evs["MKN_LIB_PRE"] = "lib"; Description The file prefix of shared objects being created. Key MKN_LD_PRELOAD Type string Default not set Description Forwards MKN_LD_PRELOAD as LD_PRELOAD for run/dbg 7. Switches Key _MKN_REP_NAME_DOT_ Type bool Default 1 Description split project name by period if true name: pro.ject version: version Will expect folder ${LOCAL_REPO}/pro/ject/version otherwise ${LOCAL_REPO}/pro.ject/version Key _MKN_REP_VERS_DOT_ Type bool Default 0 Description split project version by period. if true name: project version: ver.si.on Will expect folder ${LOCAL_REPO}/project/ver/si/on otherwise ${LOCAL_REPO}/project/ver.si.on Key _MKN_TIMESTAMPS_ Type bool Default 0 Description Logs timestamps of source/includes to skip files with no changes. Key _MKN_REMOTE_EXEC_ Type bool Default 1 Description Execute mkn.(bat/sh etc) in directory of missing dependencies when retrieved from SCM Key _MKN_REMOTE_REPO_ Type string Default "http://github.com/mkn/" Description Space separated list of URLs to use as roots for non-complete SCM repositories. i.e. scm: http://github.com/mkn/mkn.kul.git # ignored scm: git@github.com:mkn/mkn.kul.git # ignored scm: mkn.kul.git # becomes for(const std::string& s: split(_MKN_REMOTE_REPO_, " ") attempt(s + "mkn.kul.git") if all attempts fail, throw Key _MKN_DISABLE_SCM_ Type flag Default 0 Description disables all SCM Key _MKN_DISABLE_SVN_ Type flag Default 0 Description disables SVN if it was supported Key _MKN_DISABLE_GIT_ Type flag Default 0 Description disables git Key _MKN_DISABLE_MODULES_ Type flag Default 0 Description disables modules from executing at any point Key _MKN_DISABLE_RUN_LIB_PATH_HANDLING_ Type flag Default 0 Description nix: disables LD_LIBRARY_PATH modifications during "run/dbg" commands bsd: if osx: disables LD_LIBRARY_PATH/DYLD_LIBRARY_PATH modifications during "run/dbg" commands else : disables LD_LIBRARY_PATH modifications during "run/dbg" commands win: disables PATH modifications during "run/dbg" commands Key _MKN_GIT_WITH_RAM_DEFAULT_CO_ACTION_ Type uint Default 0 Description default action deduction for retrieving dependences if built with mkn.ram 0 - get default branch 1 - if has release, get, else if has tag, get, else 0 7. With options 1. io.cereal Enables serialization of some classes - requirement for distributed compilation 2. mkn.ram[https] Enables calls to github.com API to deduce most recent release or branch if no dependency version is set