CMake/CTest/CDash


A number of prominent boosters have been advocating moving from BJam to CMake. It has some very attractive features.

  • it has strong support from Kitware who have been active in it's development and maintenance for the last 10 years. They are very responsive on the mailing list they maintain for users.

  • It seems to be well documented. There are lot's web pages dedicated to it's various aspects. There is a published book Mastering CMake which is thick enough to be a complete reference.

  • It includes a system for invoking test suites and posting results to a common website. (CTest and cash) This is a really attractive feature that we absolutely have to have. Having it "out of the box" would be a huge plus.

  • It includes some extra features which we don't really need but might be useful.

    • pack - for creation of installation packages including zip files, tar balls etc.

    • It's works with a wide variety of platforms including the ability to create IDE projects in Microsoft Visual Studio and Eclipse.

  • There are also online videos about these tools. Usually I don't like videos as they don't let me skip through stuff. But in this case I found them well done and helpful.

  • It has the ability to create files for different kind of build systems including Makefiles, Visual Studio Project files for windows, code project files for Mac OS, and Eclipse project files among others. I found this feature extremely compelling as I am a heavy user of different IDEs.

So I was optimistic about experimenting with the CMake "family" of tools. I took a careful look at it using the safe_numerics project as a guinea pig. This turned out to be a lot more difficult than one would expect and I gave up several times. Eventually, I had to dig into CMake in detail for a work project and make it work. After some more work, I was able to Implement CMake for the save_numerics project with satisfactory results. So I'm ready to recommend CMake for "boost-like" libraries submitted on this website. But the information required to do this is distributed all over the web and CMake documentation. Below, I've tried to distill what I've learned (in spite of my lack of desire to understand another "tool" so something understandable. You'll still have to troll the net and CMake reference materials, but hopefully this information - along with the example provided by safe_numerics - will facilitate the process.

Here are websites which have useful information on CMake.

CMakeList.txt Cheat Sheet

The typical way of using CMake consists of sprinkling the directory which contains your code with a files named CMakeList.txt. These permit as sort of "inheritance" from the top down so that commonality doesn't have to be repeated. These files contain CMake commands as described in the CMake Reference Manual. The reference manual is a little hard to figure out so I've take the liberty of making my own "cheat sheet" distilling my experience with this tool.

  • general command

    # display a message when project is being created/generated
    message(STATUS "compiler is ${CMAKE_CXX_COMPILER_ID}" )
  • For top level directories

    # specify cmake version required
    cmake_minimum_required(VERSION 2.8.4)
    
    # set project name
    project( LoggerLibrary ) # unique project name
    
    # boilerplate code to be inherited by all lower level CMakeList.txt files
    
    # optionally specify common directory for all products of the build
    set (CMAKE_ARCHIVE_OUTPUT_DIRECTORY "bin")
    
    # in this example we use Boost everywhere so include it here
    # to save typing
    find_package(Boost)
    
    # display some information on our environment for when things
    # don't work.  Modify to taste
    
    if(Boost_FOUND)
        set(Boost_USE_MULTITHREADED true)
        set(Boost_USE_STATIC_LIBS true)
        message(STATUS "Boost is ${BOOST_ROOT}")
        message(STATUS "Boost directories found at ${Boost_INCLUDE_DIR}")
        message(STATUS "Boost libraries found at ${Boost_LIBRARY_DIR}")
        set(Boost_LIBRARY_DIR "${BOOST_ROOT}/stage64/lib")
        include_directories(${Boost_INCLUDE_DIR})
    elseif()
        message("Boost NOT Found!")
    endif()
    
    #if tests are to be generated, this has to appear at the top of the "tree"
    include( CTest )
  • For directories which contain sub directories

    add_subdirectory( src )
    add_subdirectory( test )
  • For directories containing source code to build shared libraries

    # This defines a pre-processor macro which is used inside the source code.
    # The source code will use this correctly set "visibility" attributes
    # Note that this is specific to the source code - NOT a CMake keyword
    add_definitions( -DLIBRARY_EXPORTS )
    
    # build a SHARED library named "logger" from the indicated source files
    add_library( logger SHARED
       CustomLogLevels.cpp
       EventAppender.cpp
       Logger.cpp
       OutputDebugStringAppend.cpp
    )
  • For directories containing source code to build static libraries

    # build a STATIC library named "logger" from the indicated source files
    add_library( logger STATIC
       CustomLogLevels.cpp
       EventAppender.cpp
       Logger.cpp
       OutputDebugStringAppende
    )
  • For directories containing code which import shared libraries

    link_directories (
       ${WORKSPACE_ROOT}/lib
       ${WORKSPACE_ROOT}/3rdparty/boost/lib
       ${WORKSPACE_ROOT}/3rdparty/log4cplus/lib
       ${CMAKE_ARCHIVE_OUTPUT_DIRECTORY}
    )
    
    # for some inexplicable reason CMake throws an error if a library with a given name is 
    # both imported and exported in different nodes on the tree.  I'm not sure why this is, 
    # but this following if seems to work around it.  So use this for those libraries
    # that we're building AND using
    if (NOT TARGET logger)
       add_library( logger SHARED IMPORTED )
       set_property(
          TARGET logger 
          PROPERTY IMPORTED_IMPLIB ${CMAKE_ARCHIVE_OUTPUT_DIRECTORY}/logger.lib
       )
    endif()
  • For directories containing code which import static libraries

    link_directories (
       ${WORKSPACE_ROOT}/lib
       ${WORKSPACE_ROOT}/3rdparty/boost/lib
       ${WORKSPACE_ROOT}/3rdparty/log4cplus/lib
       ${CMAKE_ARCHIVE_OUTPUT_DIRECTORY}
    )
    
    # indicate what the target must link to.
    # must be after the definition of TestLogger
    target_link_libraries (TestLogger logger verror log4cplus )
  • For directories containing code to build executables

    # create an executable target
    add_executable ( TestLogger TestLogger.cpp )
    
    # optionally place in the IDE folder labeled "applications"
    set_target_properties(TestLogger PROPERTIES FOLDER "applications")
  • For directories containing tests

    # create an test target
    add_test ( TestLogger "TestLogger" COMMAND "TestLogger" )
    
    # optionally place in the IDE folder labeled "tests"
    set_target_properties(TestLogger PROPERTIES FOLDER "tests")
  • For directories containing headers

    # optionally place in the IDE folder labeled "includes"
    # this doesn't affect building, but makes header files easily accessible
    # from the IDE.
    add_custom_target(include SOURCES ${include_files})
    set_target_properties(include PROPERTIES FOLDER "includes")
  • create a list of file names in the current directory

    # create a CMake variable named "include_files" which 
    # consists of names of the files in the current directory
    # whose name ends in "hpp"
    file(GLOB include_files 
      RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}" 
      "*.hpp"
    )

Notes

  • A given CMakeList.txt file may well fit in more than one of the above classifications. In this case it would likely contain code from more than one of the above sections

  • CMakeList.txt files inherit all the code from CMakeList.txt files in parent directories. Hence, common code such as include and link directories can be moved higher in the hierarchy to diminish repetition.

Once one has prepared the CMakeList.txt files and placed them into the directory hierarchy of the project, he can invoke CMake "Configure" from the CMake executable. This parses all the CMakeList.txt files from the top down and highlight which variables such as build type, library locations, etc. etc. need to be defined. Next step is to define the required missing variables and try again. Usually this will require a couple of iterations. Finally one can "Generate" a project file for the target IDE.

The above illustrates the strength and weakness of CMake. Distributing a number of files around the directory allows one to factor out commonality. But it also means that the information required to understand what's going one isn't found is one place. Also part of that information is implicit in the relative position of different CMakeList.txt files in the directory hierarchy. It's a two edged sword.

CMake for "Boost Like" Libraries

Figure 1. XCode Screen shot
XCode Screen shot

Now that we've seen the useful CMake commands, we'll use these commands to craft a CMake project which can be used to create a complete project for you preferred IDE. I've used this to create such a project for Apple Xcode project to run/debug tests and examples for the Boost library submission Safe Numerics. This is a header only library project which consists of

  • a directory named "test" which contains the source code for an extensive test suite. In addition it contains the following CMakeLists.txt file.

    # CMake build control file for safe numerics Library tests
    
    cmake_minimum_required(VERSION 3.0)
    project("SafeIntegersTest")
    
    include("../CMake/CMakeLists.txt" )
    
    include_directories("${Boost_INCLUDE_DIRS}")
    include_directories("../include")
    
    ###########################
    # test targets
    
    message( STATUS "Runtimes are stored in ${CMAKE_CURRENT_BINARY_DIR}" )
    
    file(GLOB test_list
      RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}" 
      "*.cpp"
    )
    foreach(file_name ${test_list})
      string(REPLACE ".cpp" "" base_name ${file_name})
      message(STATUS ${base_name})
      add_executable(${base_name} ${file_name})
      add_test(NAME ${base_name} COMMAND ${base_name})
      set_target_properties(${base_name} PROPERTIES FOLDER "tests")
    endforeach(file_name)
    
    # end test targets
    ####################
    
    ###########################
    # add headers to IDE
    
    file(GLOB include_files 
      RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}" 
      "${CMAKE_CURRENT_SOURCE_DIR}/*.hpp"
    )
    add_custom_target(include SOURCES ${include_files})
    set_target_properties(include PROPERTIES FOLDER "tests")
    
    # end headers in IDE
    ####################
    
  • a directory named "CMake". We use it in this example as a place to hold common code that we've factored out of our other CMake scripts. The directory contains a file named CMakeLists.txt which all the boiler plate script used by all the projects in created by this library. We won't include it here as it's too long but it is available in the source code of the Safe Numerics code repository.

    At this point with can use CMake to create a project for Xcode (or any other IDE) which will build and run the Safe Numerics test suite.

  • a directory named "example" which contains source code for applications which use the library. These are used in the library documentation to support the text. Since this directory is very similar to that the "test" project above, we won't include the whole file here.

    At this point we can CMake to create another project which will build and test the applications in the "example" directory of the Safe Numerics submission. This will be a totally separate project than that for test suite.

  • a project root directory which contains:

    # CMake build control file for safe numerics Library Examples
    
    cmake_minimum_required(VERSION 3.0)
    project("SafeIntegers")
    
    include("CMake/CMakeLists.txt")
    
    include_directories("${Boost_INCLUDE_DIRS}")
    include_directories("include")
    
    add_subdirectory("include")
    add_subdirectory("examples")
    add_subdirectory("test")

    This script creates another project which is the composition of the previous two. In addition it refers to the sub directory "include".

  • a directory named "include" which implements includes all headers use by the library and it's users. In addition, it contains a CMake script file whose only purpose is to add file references to the CMake generated IDE project.

    ####################
    # add include headers to IDE
    
    file(GLOB include_files 
      RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}" 
      "*.hpp"
    )
    add_custom_target(safe_numerics SOURCES ${include_files})
    set_target_properties(safe_numerics PROPERTIES FOLDER "includes")
    
    # add subdirectory which contains headers for type requirements
    add_subdirectory("concept")
    
    # end headers in IDE
    ####################

Now we can generate any one of three IDE projects depending on our needs: SafeIntegersTest, SafeIntegersExample, or SafeIntegers.

CMake Library Template - abbreviated version

The above looks useful - but still a little too "heavyweight" for my taste. It also distributes the CMake files around the project which conflicts with common practice for Boost libraries. The following CMakeList.txt file creates a project for the code IDE to test and edit the safe_numerics library. I haven't tested it on other platforms such as the Visual Studio or Eclipse IDE but hopefully it should require only minimal changes to do so. It's different from the original version in that it is entirely contained in one CMakeList.txt file. Files outside the CMake directory can be referenced with the ../<file name> syntax

# CMake build control file for safe numerics Library tests

cmake_minimum_required(VERSION 3.0)

project("SafeFloat")

#
# Compiler settings - special settings for known compilers
#

message(STATUS "compiler is ${CMAKE_CXX_COMPILER_ID}" )

if( CMAKE_CXX_COMPILER_ID STREQUAL "GNU" )
  add_definitions( -ftemplate-depth=300 )
  set (CMAKE_CXX_FLAGS "-std=c++11" )
elseif( CMAKE_CXX_COMPILER_ID STREQUAL "MSVC" )
  add_definitions( /wd4996 )
elseif( CMAKE_CXX_COMPILER_ID STREQUAL "AppleClang" )
  add_definitions( -ftemplate-depth=300 )
  # include the following if and only if you want to use c++11 features
  set (CMAKE_CXX_FLAGS "-std=c++11" )
  set (CMAKE_CXX_FLAGS_DEBUG "-g -O0" )
  set (CMAKE_CXX_FLAGS_RELWITHDEBINFO "-g -O3" )
  set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -stdlib=libc++ -dead_strip")

endif()

#
# Locate Project Prerequisites 
#

# Boost

# note: we're assuming that boost has been built with:
# ./b2 —-layout=versioned toolset=clang-darwin link=static,shared variant=debug,release stage

###########################
# special notes for Xcode.

# these three should result in CMake setting the variables

# But my current version of CMake doesn't automatically set the library names
# to point to the the libraries to be used.  The variables are created
# but they are not initialized.  So be prepared to set these variables by hand.
# If you want to use the static libraries - point to the boost libraries ending
# in ".a".  If you want to use the shared boost libraries - point to the libraries
# ending in ".dylib".

# But wait - there's more.
# if both lib.a and lib.dylib both exist in the library directory, Xcode will
# automatically chose the *.dylib one - and there's nothing you can do to fix this.
# So my recommendation is 
# a) to place the compiled libraries in two different directories
#    - e.g. stage/lib-static/*.a and stage/lib-shared/*.dylib
#    and set the CMake variable Boost_LIBRARY_DIRS to point to one or the other
# b) create two different CMake build directories - build-static and build-shared
#    and switch between the generated projects as desired.  I like to test both since
#    there are things like dead code elimination and visibility which vary
#    between the two environments.
#
#    caveat - as I write this, I've been unable to get the tests on the shared
#    library to function. Problem is that one needs to either put the shared
#    libraries in a special known place or set an environmental
#    variable which points to the shared library directory.  I prefer the latter
#    but I've been unable to figure out how to get Xcode to do on a global basis
#    and it's not practical to do this for 247 test targets one at a time.

# c) The targets in the project will by default be built as universal 32/64 binaries
#    in debug mode.  When working with the  

# end special note for Xcode
############################

#
# Project settings
#

find_package(Boost REQUIRED COMPONENTS unit_test )

if(Boost_FOUND)
  set(Boost_USE_MULTITHREADED ON)
  set(Boost_USE_STATIC_LIBS ON CACHE BOOL "Link to Boost static libraries")
  # note: it seems that boost builds builds both address models in any case
  # so we can defer this decision to the IDE just as we do for debug/release
  # so we'll not use this now
  # set(Boost_ADDRESS_MODEL 64 CACHE INTEGER "32/64 bits")
  if( CMAKE_HOST_APPLE )
    set(Boost_ADDRESS_MODEL 64 CACHE INTEGER "32/64 bits")
  endif()
  message(STATUS "Boost is ${BOOST_ROOT}")
  message(STATUS "Boost directories found at ${Boost_INCLUDE_DIRS}")
  message(STATUS "Boost libraries found at ${Boost_LIBRARY_DIRS}")
  message(STATUS "Boost component libraries to be linked are ${Boost_LIBRARIES}")
  message(STATUS "Boost version found is ${Boost_VERSION}")
  if(Boost_USE_STATIC_LIBS)
    set(BUILD_SHARED_LIBRARIES OFF)
  else()
    set(BUILD_SHARED_LIBRARIES ON)
  endif()
  message(STATUS "Boost Libraries used are: ${Boost_LIBRARIES}" )
elseif()
    message("Boost NOT Found!")
endif()

include_directories("../include" "${Boost_INCLUDE_DIRS}")
link_directories("${Boost_LIBRARY_DIRS}")

###########################
# testing and submitting test results to the test dashboard

include (CTest)

if(0)

## Create a file named CTestConfig.cmake adjacent to the current file.
## This new file should contain the following:

set(CTEST_PROJECT_NAME "Safe Numerics")
set(CTEST_NIGHTLY_START_TIME "01:00:00 UTC")

set(CTEST_DROP_METHOD "http")
set(CTEST_DROP_SITE "my.cdash.org")
# set(CTEST_DROP_LOCATION "/cdash/submit.php?project=MyProject")
set(CTEST_DROP_LOCATION "/index.php?project=Safe+Numerics")
set(CTEST_DROP_SITE_CDASH TRUE)

endif()

###########################
# library builds

# header library only - nothing to be built
#
# add_library(library_name ../src/sources1.cpp  …)

# end library build
###########################

###########################
# test targets

# the "include (CTest)" above includes enable_testing() 
# so the following line isn't necessary
# enable_testing()

file(GLOB test_list
  RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}" 
  "${CMAKE_CURRENT_SOURCE_DIR}/../test/*.cpp"
)
foreach(file_path ${test_list})
  string(REPLACE "../test/" "" file_name ${file_path})
  string(REPLACE ".cpp" "" base_name ${file_name})
  message(STATUS ${base_name})
  add_executable(${base_name} ${file_path})
  target_link_libraries(${base_name}  ${Boost_LIBRARIES} )
  add_test(NAME ${base_name} COMMAND ${base_name})
endforeach(file_path)

# end test targets
####################

###########################
# examples

file(GLOB example_list
  RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}" 
  "${CMAKE_CURRENT_SOURCE_DIR}/../example/*.cpp"
)
foreach(file_path ${example_list})
  string(REPLACE "../example/" "" file_name ${file_path})
  string(REPLACE ".cpp" "" base_name ${file_name})
  message(STATUS ${base_name})
  add_executable(${base_name} ${file_path})
  target_link_libraries(${base_name}  ${Boost_LIBRARIES} )
  add_test(NAME ${base_name} COMMAND ${base_name})
endforeach(file_path)

# end examples targets
####################

####################
# add include headers to IDE

file(GLOB include_files 
  RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}" 
  "${CMAKE_CURRENT_SOURCE_DIR}/../include/boost/*.hpp"
  "${CMAKE_CURRENT_SOURCE_DIR}/../include/boost/safe_float/*.hpp"
  "${CMAKE_CURRENT_SOURCE_DIR}/../include/boost/safe_float/policy/*.hpp"
)
add_custom_target(include SOURCES ${include_files})

# end headers in IDE
####################

CTest/CDash

We've seen how CTest supports the automatic creation of test scripts. This supports testing of a library by the original developer as well as the user who downloads the library. CDash supports the submission of the test results to a common location. This permits developers, users and reviewers to look at test results for all other users of the library. The CDash system depends upon the installation of a

  • Go to the my.cdash website and invoke "Start my dashboard" (it's free!).

  • Step through the tabbed dialog. It's unclear which information is optional and which is required. Just push on through to the end.

  • Create a file named CTestConfig.cmake in the same directory which contains CMakeList.txt. If you have more than one CMakeList.txt file in different directories, place it in the highest level directory which contains a CMakeList.txt file. The file should be a simple text file which contains the following

    ## This file should be placed in the root directory of your project.
    ## Then modify the CMakeLists.txt file in the root directory of your
    ## project to incorporate the testing dashboard.
    ## # The following are required to uses Dart and the Cdash dashboard
    ##   ENABLE_TESTING()
    ##   INCLUDE(CTest)
    set(CTEST_PROJECT_NAME "Safe Numerics")
    set(CTEST_NIGHTLY_START_TIME "01:00:00 UTC")
    
    set(CTEST_DROP_METHOD "http")
    set(CTEST_DROP_SITE "my.cdash.org")
    set(CTEST_DROP_LOCATION "/submit.php?project=Safe+Numerics")
    set(CTEST_DROP_SITE_CDASH TRUE)
    

    Naturally, your version of this file will contain the name of your project in place of "Safe Numerics". This will be the same name used in the cash project edit dialogs.

  • Build the CMake BUILD_ALL and RUN_TESTS targets as you have been doing.

  • If everything goes well up to this point and you are satisfied with the test results you can submit them to the cash server by building the CMake target named "Experimental".

    Note that besides "Experimental", there are some other test targets which CMake creates - "Continuous", "Nightly", "NightlyMemoryCheck". I'm not sure what these are for. In fact, cash seems to be a lot more ambitious than just maintaining the test results dashboard. It seems that it actually runs the test on - I'm not sure where. It's not at all well explained and extremely confusing. Never the less, I have tested the above procedure and am (almost) totally pleased with the results.

  • Now when you go to your test dash board you should see your submitted test results:

    Figure 2. cdash Screen shot
    cdash Screen shot


  • There is only one small problem left. The default setup of the dashboard only shows the current days results. Since we want a cumulative record of results for different platforms we need to make some changes. Our example here shows what you see after invoking "Show Filters". Modify the filter so that it will display all results for the past year. Then invoke "Create Hyper link". This link can be used to bring up this view anytime necessary.

  • To complete the process, go back to your library submission page and paste this Hyper link into the "Test Results Dashboard" field. Now anyone can review the test history for this particular library.

    Figure 3. Library Page Screen shot
    Library Page Screen shot


This procedure produces everything I want. It should:

  • Distribute the testing load among all those (and only those) users interested in the library.

  • Display of testing results all those (and only those) platforms and compilers which actually use the library.

  • Be easy and painless to maintain.

There are 0 comments and replies

Comment on This Page