repository list
docker image
Buiild job for each repository
run even when previous tasks fail
docker image
repository list
docker image
ROS workspace
ROS workspace
ROS workspace
On Github
On Github
coverage reports
coverage report
Pipelines-to-PPA relation
Single MRS ROS build pipeline - dense view
Full test pipeline with coverage reporting - dense view
Single-repository build action - dense view
Single-repository test action - dense view
ROS1 -> ROS2 repository/branch relation
Release-candidate (with testing)
MRS Build pipeline types
Repositories/branches
Single build job
Branches of source repositories
Origin repositories
Artifact ininitialization
Artifact ininitialization
Pre-building dependencies for coverage report [optional]
Unstable
Build matrix determination
Stable
Test unstable
Test stable
Single test job
Gather coverage
Test matrix determination
Collect artifacts
Delete docker builder image
Dockerhub
Dockerhub
Dockerhub
Dockerhub

ctu-mrs/ppa-unstable

ctu-mrs/ppa-testing

Base docker image

  • contains ros-noetic-desktop-full
  • prebuiilt by hand
  • versioned by date

New commit pushed to master

Clean build images

  1. delete builder images older than 7 days

ctu-mrs/ppa2-unstable

ctu-mrs/ppa2-testing

WHY ALL THIS?

- To be able to install the system via apt-get

- To make sure the system is always tested and working

- To automatically and safely propagate new changes into the stable release

- To provide installation for AMD64 and ARM64 architectures

release_andidate

master

single_package_pipeline.png
single_test_pipeline.png

Clean build images

  1. delete builder images older than 7 days

Build job

  1. loads the Docker builder image from Github's Docker registry
  2. accesses the repository from the commit
  3. [optional] install git and gitman submodules
  4. Runs the Docker builder image:
    1. installs dependencies via rosdep
    2. determines build order of ROS packages within the repository
    3. for each ROS package:
      1. generates build using bloom-generate for each ROS package
      2. builds .deb package
      3. installs the .deb to satisfy dependencies later
  5. outputs the following into the /tmp/artifacts folder for the later jobs
    1. the new .deb packages

Docker builder priming

  • Adds MRS PPA depending on the "variant" of the build (unstable, stable, testing)
  • installation of neccesary build tools
  • calling apt update
  • saved to Github's registry

Generic Github action link for the build is triggered

  • includes action from the CI Scripts repo link
  • Triggering Arm64 and Amd64 builds

Collect the artifacts

  1. .deb packages are collected and commited to the PPA repository

ctu-mrs/ppa-stable

CI Scripts

https://github.com/ctu-mrs/ci_scripts

  • contains
    • scripts
    • dockerfiles
    • github actions
  • ... for building a single repository within the MRS system
  • contains files that are reused in the Buildfarm

devel

ros2_devel

release

ros2_release

Buildfarm

https://github.com/ctu-mrs/buildfarm

  • contains
    • scripts
    • dockerfiles
    • github actions
  • ... for building MRS PPAs and testing MRS software
  • when executed, clonse CI Scripts into ci_scripts.

ros2

ctu-mrs/ppa2-stable

Base docker image

  • contains ros-<version>-desktop-full
  • prebuiilt by hand
  • versioned by date

New commit pushed to devel / Pull request to master

Test job

  1. loads the Docker builder image from Github's Docker registry
  2. accesses the repository from the commit
  3. [optional] install git and gitman submodules
  4. Runs the Docker builder image:
    1. installs dependencies via rosdep
    2. builds a workspace
    3. runs ROS tests

Docker builder priming

  • Adds MRS PPA depending on the "variant" of the build (unstable, stable, testing)
  • installation of the whole MRS UAV System
  • saved to Github's registry

Generic Github action link for the test is triggered

  • includes action from the CI Scripts repo link
  • Triggering AMD64-based test

release_candidate

  • [optional] clones the package into the pre-compiled workspace
  • creates a new catkin/colcon workspace when no is passed from the previous jobs
  • compiles the tests
  • runs the tests using rostest

Repository list determination

  • packages that have test: true in the yaml
  • order of testing does not matter

ctu-mrs/ci_scripts (master)

ctu-mrs/buildfarm2

ctu-mrs/ci_scripts (ros2)

ctu-mrs/buildfarm

  • package that have coverage: true in the yaml
  • software is built with flags to export code coverage reports while running
  • pre-building packages before testing allows recording coverage in package A despite not directly testing package A
  • the whole workspace is built, tared and passed further to each of the testing jobs
    • it is also passed to the final job that generates the coverage webpage (that needs source codes, therefore, passing the workspace)

Docker builder priming

  • Adds MRS PPA depending on the "variant" of the build (unstable, stable, testing)
  • calling apt update
  • installation mrs-uav-system-full
  • saved to Github's registry

/tmp/artifacts

Repository order determination

  • python script
  • clones all the repositories
  • does topological ordering of repositories containing ROS packages
  • for simplicity and minimizing edge cases treats all dependency types equally

Docker builder priming

  • Adds MRS PPA depending on the "variant" of the build (unstable, stable, testing)
  • installation of neccesary build tools
  • calling apt update
  • saved to Github's registry
  1. loads the Docker builder image from Github's Docker registry
  2. clones the repository
  3. [optional] install git and gitman submodules
  4. Runs the Docker builder image:
    2. adds the passing rosdep.yaml from the artifacts
    3. installs dependencies via rosdep
    4. determines build order of ROS packages within the repository
    5. for each ROS package:
    1. determines if the package needs to be built (if the commit changed or if the based image changed or if its dependency was compiled (is in compiled.txt))
    2. generates build using bloom-generate for each ROS package
    3. builds .deb package
    4. installs the .deb to satisfy dependencies later
    5. adds the package to the passing rosdep.yaml
    6. if compiled, ads the package name into compiled.txt
  5. installs the created .deb fiiles into the into the builder image and squashes the image
  6. Pushes the updated Docker buiilder image into the Github's docker registry
  7. outputs the following into the /tmp/artifacts folder for the later jobs
    1. the new .deb packages
    2. the passing rosdep.yml
    3. the passing copiled.txt

Base docker image

  • contains ros-noetic-desktop-full
  • prebuiilt by hand
  • versioned by date

Base docker image

  • contains ros-noetic-desktop-full
  • prebuiilt by hand
  • versioned by date

Repository list

Cotains for each repository

  • name
  • address
  • CPU architectures
  • branches for
    • unstable
    • release candidate
    • relelase
  • If it should be tested
  • If it should be built for test coverage

/tmp/artifacts

  • rosdep.yaml
    • on-the-go updated list of built ROS packages
    • later commited to the PPA
  • compiled.txt
    • keeps track of what was buiilt, used to determine if a package should be rebuild due to its dependencies being updated
  1. delete builder images older than 7 days
  1. .deb packages are collected and commited to the PPA repository
  2. rosdep.yml is commited to the PPA

Repository list

Cotains for each repository

  • name
  • address
  • CPU architectures
  • branches for
    • unstable
    • release candidate
    • relelase
  • If it should be tested
  • If it should be built for test coverage
  • We have 3 pipeline types, two ROS and one non-ROS, each producing *.deb packages into our PPAs.
  • Each pipeline is executed for AMD and ARM architectures .
  • Each pipeline is executed for each PPA variant (Stable, Unstable, Testing)
pipelines.png

Thirdparty (ROS)

Nonbloom (non-ROS)

  • link to the repository list
  • Packages that can not be built using bloom-generate.
  • The pipeline for these is different: the docker build executes a script from each repository (.ci/build_package.sh) that is supposed to build the deb package. This procedure is different for each repository, so it is implemented within each package.
  • Specialities: PX4 is not compatible with bloom-generate, so the ROS package is build by catkin and the build artifacts are extracted from a catkin workspace

MRS (ROS)

  • link to the repository list
  • Packages are either MRS-developed or are in some way dependent on packages from this pipeline (e.g., on mrs_lib or mrs_msgs)

build jobs

  1. mrs_msgs
  2. mrs_lib
  3. ..
  4. ..
  5. ..

Generate test matrix

Commit to PPA

Build packages for testing and coverage generation

test jobs

  1. mrs_lib
  2. mrs_uav_managers
  3. ..
  4. ..

Initialize artifacts

Generate build matrix

release_pipeline.png

Merge release_candidate to release

Generate test matrix

test jobs

  1. mrs_lib
  2. mrs_uav_managers
  3. ..
  4. ..

release branches

stable-PPA

Runs more than 1 hour of simulation BDD integration tests

test jobs

  1. mrs_lib
  2. mrs_uav_managers
  3. ..
  4. ..

Generate test matrix

  • runs nightly
  • only the release-candidate pipeline can merge to release
  • stable-PPA should therefore always be internally consistent and tested
LCOV - MRS UAV System - Test coverage report
test_pipeline.png

unstable-PPA

release-candidate branches

  • runs nightly
  • only core MRS developers can merge to release-candidate
  • runs tests after the build
  • if tests succeed, then it merges release-candidate into release

testing-PPA

Publish test coverage

Runs more than 1 hour of simulation BDD integration tests

master branches

  • runs nightly
  • all MRS developers can merge to master
  • also direct pushes are built into the unstable-PPA
    • therefore, can be broken at any time
    • however, quick fixes can propagate fast and can be quickly installed using apt-get