repository list
docker image
Buiild job for each repository
run even when previous tasks fail
docker image
repository list
docker image
ROS workspace
ROS workspace
ROS workspace
On Github
On Github
coverage reports
coverage report
Pipelines-to-PPA relation
Single MRS ROS build pipeline - dense view
Full test pipeline with coverage reporting - dense view
Single-repository build action - dense view
Single-repository test action - dense view
ROS1 -> ROS2 repository/branch relation
Release-candidate (with testing)
MRS Build pipeline types
Repositories/branches
Single build job
Branches of source repositories
Origin repositories
Artifact ininitialization
Artifact ininitialization
Pre-building dependencies for coverage report [optional]
Unstable
Build matrix determination
Stable
Test unstable
Test stable
Single test job
Gather coverage
Test matrix determination
Collect artifacts
Delete docker builder image
Dockerhub
Dockerhub
Dockerhub
Dockerhub
  • package that have coverage: true in the yaml
  • software is built with flags to export code coverage reports while running
  • pre-building packages before testing allows recording coverage in package A despite not directly testing package A
  • the whole workspace is built, tared and passed further to each of the testing jobs
    • it is also passed to the final job that generates the coverage webpage (that needs source codes, therefore, passing the workspace)

Repository list determination

  • packages that have test: true in the yaml
  • order of testing does not matter

ctu-mrs/ci_scripts (ros2)

ctu-mrs/ppa2-unstable

ctu-mrs/ppa2-testing

ctu-mrs/ppa2-stable

/tmp/artifacts

Docker builder priming

  • Adds MRS PPA depending on the "variant" of the build (unstable, stable, testing)
  • calling apt update
  • installation mrs-uav-system-full
  • saved to Github's registry

devel

release

release_andidate

master

ctu-mrs/ppa-testing

ros2

  • [optional] clones the package into the pre-compiled workspace
  • creates a new catkin/colcon workspace when no is passed from the previous jobs
  • compiles the tests
  • runs the tests using rostest

ctu-mrs/ppa-stable

ros2_devel

CI Scripts

https://github.com/ctu-mrs/ci_scripts

  • contains
    • scripts
    • dockerfiles
    • github actions
  • ... for building a single repository within the MRS system
  • contains files that are reused in the Buildfarm

Buildfarm

https://github.com/ctu-mrs/buildfarm

  • contains
    • scripts
    • dockerfiles
    • github actions
  • ... for building MRS PPAs and testing MRS software
  • when executed, clonse CI Scripts into ci_scripts.

WHY ALL THIS?

- To be able to install the system via apt-get

- To make sure the system is always tested and working

- To automatically and safely propagate new changes into the stable release

- To provide installation for AMD64 and ARM64 architectures

release_candidate

ros2_release

ctu-mrs/ci_scripts (master)

ctu-mrs/ppa-unstable

ctu-mrs/buildfarm

ctu-mrs/buildfarm2

Base docker image

  • contains ros-noetic-desktop-full
  • prebuiilt by hand
  • versioned by date

Repository list

Cotains for each repository

  • name
  • address
  • CPU architectures
  • branches for
    • unstable
    • release candidate
    • relelase
  • If it should be tested
  • If it should be built for test coverage

Generate test matrix

Build packages for testing and coverage generation

Generate test matrix

testing-PPA

Merge release_candidate to release

unstable-PPA

release-candidate branches

master branches

  • runs nightly
  • all MRS developers can merge to master
  • also direct pushes are built into the unstable-PPA
    • therefore, can be broken at any time
    • however, quick fixes can propagate fast and can be quickly installed using apt-get

Publish test coverage

Commit to PPA

Runs more than 1 hour of simulation BDD integration tests

test jobs

  1. mrs_lib
  2. mrs_uav_managers
  3. ..
  4. ..

Generate test matrix

test jobs

  1. mrs_lib
  2. mrs_uav_managers
  3. ..
  4. ..
  • runs nightly
  • only core MRS developers can merge to release-candidate
  • runs tests after the build
  • if tests succeed, then it merges release-candidate into release

Generate build matrix

release_pipeline.png

release branches

build jobs

  1. mrs_msgs
  2. mrs_lib
  3. ..
  4. ..
  5. ..
  • runs nightly
  • only the release-candidate pipeline can merge to release
  • stable-PPA should therefore always be internally consistent and tested

stable-PPA

test jobs

  1. mrs_lib
  2. mrs_uav_managers
  3. ..
  4. ..
LCOV - MRS UAV System - Test coverage report

Initialize artifacts

test_pipeline.png

Runs more than 1 hour of simulation BDD integration tests

  • We have 3 pipeline types, two ROS and one non-ROS, each producing *.deb packages into our PPAs.
  • Each pipeline is executed for AMD and ARM architectures .
  • Each pipeline is executed for each PPA variant (Stable, Unstable, Testing)
pipelines.png

Thirdparty (ROS)

Nonbloom (non-ROS)

  • link to the repository list
  • Packages that can not be built using bloom-generate.
  • The pipeline for these is different: the docker build executes a script from each repository (.ci/build_package.sh) that is supposed to build the deb package. This procedure is different for each repository, so it is implemented within each package.
  • Specialities: PX4 is not compatible with bloom-generate, so the ROS package is build by catkin and the build artifacts are extracted from a catkin workspace

MRS (ROS)

  • link to the repository list
  • Packages are either MRS-developed or are in some way dependent on packages from this pipeline (e.g., on mrs_lib or mrs_msgs)
  1. .deb packages are collected and commited to the PPA repository
  2. rosdep.yml is commited to the PPA
  1. delete builder images older than 7 days
  1. loads the Docker builder image from Github's Docker registry
  2. clones the repository
  3. [optional] install git and gitman submodules
  4. Runs the Docker builder image:
    2. adds the passing rosdep.yaml from the artifacts
    3. installs dependencies via rosdep
    4. determines build order of ROS packages within the repository
    5. for each ROS package:
    1. determines if the package needs to be built (if the commit changed or if the based image changed or if its dependency was compiled (is in compiled.txt))
    2. generates build using bloom-generate for each ROS package
    3. builds .deb package
    4. installs the .deb to satisfy dependencies later
    5. adds the package to the passing rosdep.yaml
    6. if compiled, ads the package name into compiled.txt
  5. installs the created .deb fiiles into the into the builder image and squashes the image
  6. Pushes the updated Docker buiilder image into the Github's docker registry
  7. outputs the following into the /tmp/artifacts folder for the later jobs
    1. the new .deb packages
    2. the passing rosdep.yml
    3. the passing copiled.txt

Build job

  1. loads the Docker builder image from Github's Docker registry
  2. accesses the repository from the commit
  3. [optional] install git and gitman submodules
  4. Runs the Docker builder image:
    1. installs dependencies via rosdep
    2. determines build order of ROS packages within the repository
    3. for each ROS package:
      1. generates build using bloom-generate for each ROS package
      2. builds .deb package
      3. installs the .deb to satisfy dependencies later
  5. outputs the following into the /tmp/artifacts folder for the later jobs
    1. the new .deb packages

Collect the artifacts

  1. .deb packages are collected and commited to the PPA repository
single_package_pipeline.png

Clean build images

  1. delete builder images older than 7 days

Test job

  1. loads the Docker builder image from Github's Docker registry
  2. accesses the repository from the commit
  3. [optional] install git and gitman submodules
  4. Runs the Docker builder image:
    1. installs dependencies via rosdep
    2. builds a workspace
    3. runs ROS tests

Clean build images

  1. delete builder images older than 7 days
single_test_pipeline.png

Base docker image

  • contains ros-noetic-desktop-full
  • prebuiilt by hand
  • versioned by date

Docker builder priming

  • Adds MRS PPA depending on the "variant" of the build (unstable, stable, testing)
  • installation of neccesary build tools
  • calling apt update
  • saved to Github's registry

New commit pushed to master

Generic Github action link for the build is triggered

  • includes action from the CI Scripts repo link
  • Triggering Arm64 and Amd64 builds

Repository order determination

  • python script
  • clones all the repositories
  • does topological ordering of repositories containing ROS packages
  • for simplicity and minimizing edge cases treats all dependency types equally

Repository list

Cotains for each repository

  • name
  • address
  • CPU architectures
  • branches for
    • unstable
    • release candidate
    • relelase
  • If it should be tested
  • If it should be built for test coverage

Docker builder priming

  • Adds MRS PPA depending on the "variant" of the build (unstable, stable, testing)
  • installation of the whole MRS UAV System
  • saved to Github's registry

Base docker image

  • contains ros-<version>-desktop-full
  • prebuiilt by hand
  • versioned by date

Generic Github action link for the test is triggered

  • includes action from the CI Scripts repo link
  • Triggering AMD64-based test

New commit pushed to devel / Pull request to master

Docker builder priming

  • Adds MRS PPA depending on the "variant" of the build (unstable, stable, testing)
  • installation of neccesary build tools
  • calling apt update
  • saved to Github's registry

Base docker image

  • contains ros-noetic-desktop-full
  • prebuiilt by hand
  • versioned by date

/tmp/artifacts

  • rosdep.yaml
    • on-the-go updated list of built ROS packages
    • later commited to the PPA
  • compiled.txt
    • keeps track of what was buiilt, used to determine if a package should be rebuild due to its dependencies being updated