repository list
docker image
Buiild job for each repository
run even when previous tasks fail
docker image
repository list
docker image
ROS workspace
ROS workspace
ROS workspace
On Github
On Github
coverage reports
coverage report
Pipelines-to-PPA relation
Single MRS ROS build pipeline - dense view
Full test pipeline with coverage reporting - dense view
Single-repository build action - dense view
Single-repository test action - dense view
ROS1 -> ROS2 repository/branch relation
Release-candidate (with testing)
MRS Build pipeline types
Repositories/branches
Single build job
Branches of source repositories
Origin repositories
Artifact ininitialization
Artifact ininitialization
Pre-building dependencies for coverage report [optional]
Unstable
Build matrix determination
Stable
Test unstable
Test stable
Single test job
Gather coverage
Test matrix determination
Collect artifacts
Delete docker builder image
Dockerhub
Dockerhub
Dockerhub
Dockerhub

Clean build images

  1. delete builder images older than 7 days

Build job

  1. loads the Docker builder image from Github's Docker registry
  2. accesses the repository from the commit
  3. [optional] install git and gitman submodules
  4. Runs the Docker builder image:
    1. installs dependencies via rosdep
    2. determines build order of ROS packages within the repository
    3. for each ROS package:
      1. generates build using bloom-generate for each ROS package
      2. builds .deb package
      3. installs the .deb to satisfy dependencies later
  5. outputs the following into the /tmp/artifacts folder for the later jobs
    1. the new .deb packages

master

ctu-mrs/ppa-unstable

ctu-mrs/ppa-testing

ctu-mrs/ppa-stable

ctu-mrs/ci_scripts (master)

ctu-mrs/buildfarm2

single_package_pipeline.png

release_andidate

ctu-mrs/ppa2-unstable

ros2

ctu-mrs/ppa2-stable

ctu-mrs/ppa2-testing

WHY ALL THIS?

- To be able to install the system via apt-get

- To make sure the system is always tested and working

- To automatically and safely propagate new changes into the stable release

- To provide installation for AMD64 and ARM64 architectures

CI Scripts

https://github.com/ctu-mrs/ci_scripts

  • contains
    • scripts
    • dockerfiles
    • github actions
  • ... for building a single repository within the MRS system
  • contains files that are reused in the Buildfarm

Buildfarm

https://github.com/ctu-mrs/buildfarm

  • contains
    • scripts
    • dockerfiles
    • github actions
  • ... for building MRS PPAs and testing MRS software
  • when executed, clonse CI Scripts into ci_scripts.

ctu-mrs/ci_scripts (ros2)

Docker builder priming

  • Adds MRS PPA depending on the "variant" of the build (unstable, stable, testing)
  • installation of neccesary build tools
  • calling apt update
  • saved to Github's registry

Generic Github action link for the build is triggered

  • includes action from the CI Scripts repo link
  • Triggering Arm64 and Amd64 builds

Build job

  1. loads the Docker builder image from Github's Docker registry
  2. accesses the repository from the commit
  3. [optional] install git and gitman submodules
  4. Runs the Docker builder image:
    1. installs dependencies via rosdep
    2. determines build order of ROS packages within the repository
    3. for each ROS package:
      1. generates build using bloom-generate for each ROS package
      2. builds .deb package
      3. installs the .deb to satisfy dependencies later
  5. outputs the following into the /tmp/artifacts folder for the later jobs
    1. the new .deb packages

Collect the artifacts

  1. .deb packages are collected and commited to the PPA repository

release_candidate

devel

ros2_devel

release

ros2_release

New commit pushed to master

Base docker image

  • contains ros-noetic-desktop-full
  • prebuiilt by hand
  • versioned by date

Clean build images

  1. delete builder images older than 7 days

Base docker image

  • contains ros-noetic-desktop-full
  • prebuiilt by hand
  • versioned by date

New commit pushed to devel

Docker builder priming

  • Adds MRS PPA depending on the "variant" of the build (unstable, stable, testing)
  • installation of neccesary build tools
  • calling apt update
  • saved to Github's registry

Generic Github action link for the build is triggered

  • includes action from the CI Scripts repo link
  • Triggering Arm64 and Amd64 builds
  • [optional] clones the package into the pre-compiled workspace
  • creates a new catkin/colcon workspace when no is passed from the previous jobs
  • compiles the tests
  • runs the tests using rostest

Repository list determination

  • packages that have test: true in the yaml
  • order of testing does not matter

ctu-mrs/buildfarm

  • package that have coverage: true in the yaml
  • software is built with flags to export code coverage reports while running
  • pre-building packages before testing allows recording coverage in package A despite not directly testing package A
  • the whole workspace is built, tared and passed further to each of the testing jobs
    • it is also passed to the final job that generates the coverage webpage (that needs source codes, therefore, passing the workspace)

Docker builder priming

  • Adds MRS PPA depending on the "variant" of the build (unstable, stable, testing)
  • calling apt update
  • installation mrs-uav-system-full
  • saved to Github's registry

/tmp/artifacts

Base docker image

  • contains ros-noetic-desktop-full
  • prebuiilt by hand
  • versioned by date

Repository list

Cotains for each repository

  • name
  • address
  • CPU architectures
  • branches for
    • unstable
    • release candidate
    • relelase
  • If it should be tested
  • If it should be built for test coverage

Repository order determination

  • python script
  • clones all the repositories
  • does topological ordering of repositories containing ROS packages
  • for simplicity and minimizing edge cases treats all dependency types equally

Docker builder priming

  • Adds MRS PPA depending on the "variant" of the build (unstable, stable, testing)
  • installation of neccesary build tools
  • calling apt update
  • saved to Github's registry

/tmp/artifacts

  • rosdep.yaml
    • on-the-go updated list of built ROS packages
    • later commited to the PPA
  • compiled.txt
    • keeps track of what was buiilt, used to determine if a package should be rebuild due to its dependencies being updated
  1. delete builder images older than 7 days

Base docker image

  • contains ros-noetic-desktop-full
  • prebuiilt by hand
  • versioned by date
  1. loads the Docker builder image from Github's Docker registry
  2. clones the repository
  3. [optional] install git and gitman submodules
  4. Runs the Docker builder image:
    2. adds the passing rosdep.yaml from the artifacts
    3. installs dependencies via rosdep
    4. determines build order of ROS packages within the repository
    5. for each ROS package:
    1. determines if the package needs to be built (if the commit changed or if the based image changed or if its dependency was compiled (is in compiled.txt))
    2. generates build using bloom-generate for each ROS package
    3. builds .deb package
    4. installs the .deb to satisfy dependencies later
    5. adds the package to the passing rosdep.yaml
    6. if compiled, ads the package name into compiled.txt
  5. installs the created .deb fiiles into the into the builder image and squashes the image
  6. Pushes the updated Docker buiilder image into the Github's docker registry
  7. outputs the following into the /tmp/artifacts folder for the later jobs
    1. the new .deb packages
    2. the passing rosdep.yml
    3. the passing copiled.txt
  1. .deb packages are collected and commited to the PPA repository
  2. rosdep.yml is commited to the PPA
  • We have 3 pipeline types, two ROS and one non-ROS, each producing *.deb packages into our PPAs.
  • Each pipeline is executed for AMD and ARM architectures .
  • Each pipeline is executed for each PPA variant (Stable, Unstable, Testing)
pipelines.png

Thirdparty (ROS)

Nonbloom (non-ROS)

  • link to the repository list
  • Packages that can not be built using bloom-generate.
  • The pipeline for these is different: the docker build executes a script from each repository (.ci/build_package.sh) that is supposed to build the deb package. This procedure is different for each repository, so it is implemented within each package.
  • Specialities: PX4 is not compatible with bloom-generate, so the ROS package is build by catkin and the build artifacts are extracted from a catkin workspace

MRS (ROS)

  • link to the repository list
  • Packages are either MRS-developed or are in some way dependent on packages from this pipeline (e.g., on mrs_lib or mrs_msgs)

Repository list

Cotains for each repository

  • name
  • address
  • CPU architectures
  • branches for
    • unstable
    • release candidate
    • relelase
  • If it should be tested
  • If it should be built for test coverage

Runs more than 1 hour of simulation BDD integration tests

release-candidate branches

build jobs

  1. mrs_msgs
  2. mrs_lib
  3. ..
  4. ..
  5. ..

Commit to PPA

Generate test matrix

Build packages for testing and coverage generation

test jobs

  1. mrs_lib
  2. mrs_uav_managers
  3. ..
  4. ..

test jobs

  1. mrs_lib
  2. mrs_uav_managers
  3. ..
  4. ..

Generate test matrix

release_pipeline.png
  • runs nightly
  • only core MRS developers can merge to release-candidate
  • runs tests after the build
  • if tests succeed, then it merges release-candidate into release

testing-PPA

unstable-PPA

Initialize artifacts

release branches

master branches

stable-PPA

  • runs nightly
  • only the release-candidate pipeline can merge to release
  • stable-PPA should therefore always be internally consistent and tested
  • runs nightly
  • all MRS developers can merge to master
  • also direct pushes are built into the unstable-PPA
    • therefore, can be broken at any time
    • however, quick fixes can propagate fast and can be quickly installed using apt-get

Generate build matrix

LCOV - MRS UAV System - Test coverage report
test_pipeline.png

Merge release_candidate to release

Generate test matrix

test jobs

  1. mrs_lib
  2. mrs_uav_managers
  3. ..
  4. ..

Publish test coverage

Runs more than 1 hour of simulation BDD integration tests

single_test_pipeline.png