Skip to main content
Version: 2.0.0

MRS ROS Build Pipeline & CI/CD

This guide provides a comprehensive overview of the MRS Continuous Integration and Continuous Deployment (CI/CD) ecosystem, explaining how individual package repositories, reusable GitHub workflows, central buildfarms, and Ubuntu Personal Package Archives (PPAs) interact to automatically compile, test, and distribute Debian packages for both ROS 1 (Noetic) and ROS 2 (Jazzy) across amd64 and arm64 architectures.


1. Ecosystem Architecture & Inter-relationships​

The MRS software distribution relies on several loosely coupled components:

+-----------------------------------------------------------------------------------------+
| Individual Repositories |
| (e.g., mrs_uav_managers, mrs_lib, mrs_robofly_core) |
| - Development Branches: ros2 / ros1 (legacy: master) |
| - Candidate Branches: ros2_release_candidate / ros1_release_candidate |
| - Release Branches: ros2_release / ros1_release |
| - Local Workflows: .github/workflows/ros_package_build.yml, ros_build_test.yml |
+-----------------------------------------------------------------------------------------+
|
v calls reusable workflows
+-----------------------------------------------------------------------------------------+
| ctu-mrs/ci_scripts |
| - Reusable GitHub Actions workflows (@ros2, @master) |
| - Bloom debian generation, architecture matrix builds (amd64, native/QEMU arm64) |
| - PPA deployment scripts (push_to_ppa.sh) |
+-----------------------------------------------------------------------------------------+
|
v orchestrated & scheduled by
+-----------------------------------------------------------------------------------------+
| ctu-mrs/buildfarm (ROS 1) & ctu-mrs/buildfarm2 (ROS 2) |
| - Package registries: mrs.yaml, nonbloom.yaml, thirdparty.yaml |
| - Dependency graph resolution & topological build order |
| - Overnight matrix builders & automated test dispatchers |
+-----------------------------------------------------------------------------------------+
|
v builds inside primed container images
+-----------------------------------------------------------------------------------------+
| GitHub Container Registry (ghcr.io/ctu-mrs/...) |
| - Builder images with pre-cached toolchains & dependencies |
+-----------------------------------------------------------------------------------------+
|
v publishes packages to
+-----------------------------------------------------------------------------------------+
| Ubuntu PPAs |
| - ppa:ctu-mrs/ppa2-unstable <-- Automatic on-push from ros2 / ros1 |
| - ppa:ctu-mrs/ppa2-testing <-- Automatic on-push from *release_candidate |
| - ppa:ctu-mrs/ppa2-stable <-- Promoted by ctumrsbot upon passing automated tests |
+-----------------------------------------------------------------------------------------+

Individual Package Repositories​

Every repository contains its own source code, package.xml, and CMakeLists.txt. Rather than re-implementing complex packaging scripts in every repository, each repo contains lightweight GitHub workflow files under .github/workflows/ that delegate build tasks to centralized workflows in ctu-mrs/ci_scripts.

ctu-mrs/ci_scripts​


The ci_scripts repository centralizes all packaging and deployment scripts:

  • Standard ROS Bloom packaging (bloom-generate rosdebian)
  • Non-bloom packaging for custom CMake, Autotools, or Make-based libraries
  • Multi-architecture builds (native amd64 and ARM64 via native runners or QEMU)
  • PPA artifact publishing, GPG signing, and changelog updates via push_to_ppa.sh

ctu-mrs/buildfarm (ROS 1) & ctu-mrs/buildfarm2 (ROS 2)​


The buildfarm repositories orchestrate complete automated builds of the entire MRS ecosystem. The pipeline is structured into three distinct sub-pipelines:

  • NonBloom packages (nonbloom.yaml): Packages not built with ROS Bloom; they rely on a generic Debian packaging mechanism for external C/C++ libraries and drivers.
  • Third-party ROS packages (thirdparty.yaml): External ROS dependencies packaged for MRS. They do not change often and do not depend on custom MRS packages.
  • MRS packages (mrs.yaml): The primary MRS UAV packages that form the core autonomous system. They depend on foundation libraries and third-party packages.

The ROS-driven pipelines automatically resolve the dependency graph into topological build orders so that upstream foundations (such as mrs_msgs and mrs_lib) compile and publish before downstream managers and controllers.

Primed Builder Container Images (GHCR)​

To avoid spending hours re-installing hundreds of Debian build dependencies on every workflow run, the buildfarm relies on pre-built builder images on GitHub Container Registry (GHCR):

  • ghcr.io/ctu-mrs/buildfarm2:jazzy_builder_amd_<sha>
  • ghcr.io/ctu-mrs/buildfarm2:jazzy_builder_arm_<sha>

2. The Three-Tier Distribution Model​

MRS packages are distributed through three Launchpad PPAs corresponding to specific branch roles:

Pipeline TierTarget PPAROS 2 Standard BranchROS 1 Modern BranchROS 1 Legacy BranchPurpose & Audience
Unstableppa2-unstableros2ros1masterActive development; continuous on-push builds of the latest commits
Testingppa2-testingros2_release_candidateros1_release_candidaterelease_candidatePre-release staging; daily validation and automated system testing
Stableppa2-stableros2_releaseros1_releasereleaseVerified production releases; physical drones and end users

Clarifying Branch Names (master vs ros1 vs ros2)​

Historically in ROS 1, repositories used master, release_candidate, and release. However, having generic master or main branches causes substantial confusionβ€”especially in newer repositories where a master or main branch might exist as a default repository branch with no ROS 1 / Noetic content whatsoever.

To eliminate ambiguity, the clean standard across the MRS organization is:

  • ROS 2 Standard: ros2, ros2_release_candidate, and ros2_release (with ros2_devel for development/PR feature branches)
  • ROS 1 Standard (Clean): ros1, ros1_release_candidate, and ros1_release (migrating away from legacy master, release_candidate, release, and devel β†’ ros1_devel)
  • Wildcard Ruleset Synergy: Using wildcard patterns like *release and *release_candidate in GitHub rulesets cleanly protects both modern prefixed branches (ros2_release, ros1_release) and legacy branches (release) simultaneously without duplicate rules.

3. Why Are There Two Release Branches? (*release_candidate vs *release)​

A central question often asked by developers is why both a candidate branch and a release branch are needed.

[Developer pushes / merges tested feature]
|
v
*release_candidate
|
v (Builds packages)
ppa-testing PPA
|
v (Runs overnight / on dispatch)
Automated Tests (rostests)
|
+------+------+
| |
Passed Failed
| |
v v
[ctumrsbot merges] [Release blocked!]
*release_candidate Fix issue in dev branch
into *release and re-promote
|
v (Builds packages)
ppa-stable PPA
  1. Human Decision vs Automated Verification:
    • Developers choose when a milestone or feature set is ready by pushing to *release_candidate.
    • The package is built into the testing PPA.
  2. The Test Gatekeeper:
    • Automated system integration tests (rostest) run against the testing PPA via the release pipeline workflow.
    • If (and only if) all tests pass, the automated bot (ctumrsbot) automatically fast-forwards/merges *release_candidate into *release.
    • The stable build triggers exclusively from *release, publishing verified packages to the stable PPA.
  3. Protecting Production:
    • Every commit that lands on *release generates a stable release. If humans pushed directly to *release and introduced a compile failure or subtle regression, it would immediately break the production PPA for all users. The candidate branch acts as an essential buffer.
Critical Rule

Never push directly to *release manually. Only the automated bot account (ctumrsbot) should write to *release branches upon automated test validation.


4. Step-by-Step: Adding a New Package to the Pipeline​

Follow these 5 steps to onboard any repository into the MRS build pipeline.

Step 1: Create the Required Branches​

Create the candidate and release branches from your development branch:

# For a ROS 2 package:
git checkout ros2
git branch ros2_release_candidate
git branch ros2_release
git push -u origin ros2_release_candidate
git push -u origin ros2_release

Step 2: Add Collaborator Teams or Maintainers (Suggested Access Pattern)​

Before configuring GitHub Rulesets, collaborators or teams must have access to the repository. GitHub rulesets cannot exempt actors from rules or add them to bypass lists unless they already have direct repository access.

Access Roles as Guidelines

Team membership, responsibilities, and repository ownership naturally vary across packages. The team roles below reflect a suggested convention and convenient starting point for repositories in the ctu-mrs organization, rather than a rigid mandate. Repository owners are encouraged to adapt these suggestionsβ€”for instance, by substituting individual maintainers or custom teams depending on who actively maintains the package.

Collaborators and Teams Configuration


Step 3: Configure Branch Protection Rulesets​

Modern GitHub uses Rulesets (located in Settings β†’ Rules β†’ Rulesets). Unlike legacy branch protection rules which required duplicating rules per branch, rulesets support wildcard pattern matching across multiple targets.

Legacy Branch Protection vs Rulesets


You can import all three rulesets into any repository in seconds using the pre-configured JSON files:

  1. In your repository on GitHub, navigate to Settings β†’ Rules β†’ Rulesets.
  2. Click New ruleset β†’ Import a ruleset.
  3. Import the three ruleset definitions:
Click to view JSON definitions

delete_force_push.json:

{
"name": "delete_force_push",
"target": "branch",
"enforcement": "active",
"conditions": {
"ref_name": {
"exclude": [],
"include": [
"refs/heads/ros2",
"refs/heads/ros1",
"~DEFAULT_BRANCH",
"refs/heads/*release_candidate",
"refs/heads/*release"
]
}
},
"rules": [
{
"type": "deletion"
},
{
"type": "non_fast_forward"
}
],
"bypass_actors": []
}

release.json:

{
"name": "release",
"target": "branch",
"enforcement": "active",
"conditions": {
"ref_name": {
"exclude": [],
"include": [
"refs/heads/*release"
]
}
},
"rules": [
{
"type": "creation"
},
{
"type": "update",
"parameters": {
"update_allows_fetch_and_merge": false
}
}
],
"bypass_actors": [
{
"actor_id": 154206578,
"actor_type": "User",
"bypass_mode": "always"
}
]
}

release_candidate.json:

{
"name": "release_candidate",
"target": "branch",
"enforcement": "active",
"conditions": {
"ref_name": {
"exclude": [],
"include": [
"refs/heads/*release_candidate"
]
}
},
"rules": [
{
"type": "creation"
},
{
"type": "update",
"parameters": {
"update_allows_fetch_and_merge": false
}
}
],
"bypass_actors": [
{
"actor_id": 3812406,
"actor_type": "Team",
"bypass_mode": "always"
}
]
}

Manual Configuration Reference​

If configuring the rulesets manually through the GitHub Web UI:

Ruleset 1: delete_force_push (Universal Protection)​

Protects all development, staging, and production branches against deletions and force pushes:

  • Ruleset Name: delete_force_push
  • Enforcement status: Active
  • Bypass list: Leave completely empty (no bypass).
  • Target branches:
    • Add target β†’ Default branch (~DEFAULT_BRANCH)
    • Add target β†’ Include by pattern:
      • refs/heads/ros2
      • refs/heads/ros1
      • refs/heads/*release_candidate
      • refs/heads/*release
  • Branch rules:
    • βœ“ Restrict deletions
    • βœ“ Block force pushes
Ruleset 2: release (Strict Bot Protection)​

Protects all production release branches so only ctumrsbot can update them:

Ruleset configuration for release

Ruleset 3: release_candidate (Restricted to Maintainers)​

Restricts candidate promotions so only authorized package maintainers can stage releases:

Ruleset configuration for release candidate


Step 4: Add GitHub Actions Workflows to the Repository​

Create two workflow files in .github/workflows/ of the repository:

1. .github/workflows/ros_package_build.yml​

Builds and uploads Debian packages on push:

name: ros_package_build

on:
push:
branches: [ ros2 ]
paths-ignore:
- '**/README.md'

workflow_dispatch:

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
build:
uses: ctu-mrs/ci_scripts/.github/workflows/ros_package_build.yml@ros2
secrets:
PUSH_TOKEN: ${{ secrets.PUSH_TOKEN }}

2. .github/workflows/ros_build_test.yml​

Tests pull requests and development branches without deploying to PPAs:

For ROS 2 (ros2_devel β†’ ros2):​
name: ros_build_test

on:
push:
branches: [ ros2_devel ]
paths-ignore:
- '**/README.md'

pull_request:
branches: [ ros2 ]

workflow_dispatch:

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
build:
uses: ctu-mrs/ci_scripts/.github/workflows/ros_build_test.yml@ros2

Step 5: Register the Package in the Buildfarm​

  • ROS 2: Edit buildfarm2/mrs.yaml

Example Entry (mrs.yaml):​

mrs_robofly_core:
architecture: [amd64, arm64]
source: https://github.com/ctu-mrs/mrs_robofly_core
gitman: false
git_refs:
unstable: ros2
testing: ros2_release_candidate
stable: ros2_release
generate_apt_package: true
generate_docker_image: false
nix: false
ros_test:
enabled: false
full_test_coverage: false

5. Container Registry (GHCR) Access​

The buildfarm executes builds inside primed Docker containers hosted on the GitHub Container Registry.

The "Error response from daemon: denied" Issue​

If you or a developer attempt to pull a builder image locally:

$ docker pull ghcr.io/ctu-mrs/buildfarm2:jazzy_builder_amd_36126839068
Error response from daemon: denied

GitHub Container Registry Package Settings

Why It Happens & How to Fix It​

GitHub Packages images belong to the organization and are private by default:

  1. Actions Access: In the package settings (github.com/orgs/ctu-mrs/packages/container/<package_name>/settings), the organization admin must ensure repositories running CI workflows are added under Actions repository access.
  2. Member Access: Under Manage access, team members or developer groups must be granted Read access.
  3. Local Docker Authentication: If pulling to a local machine, log in via a GitHub Personal Access Token (PAT) with read:packages scope:
    echo $GITHUB_PAT | docker login ghcr.io -u YOUR_GITHUB_USERNAME --password-stdin

6. Quick Reference Summary​

TaskWhere to ConfigureWho Has Access (Suggested Pattern)
Daily DevelopmentPush to ros2Package Contributors (suggested: Write access)
Propose ReleasePush/Merge to *release_candidatePackage Maintainers (suggested: Maintain access)
Stable DeploymentFast-forward to *releasectumrsbot only (automated upon passing tests)
Collaborator TeamsPackage Repo β†’ Settings β†’ CollaboratorsRepository Admins
Branch RulesetsPackage Repo β†’ Settings β†’ Rules β†’ RulesetsRepository Admins
Register Packagebuildfarm2/mrs.yamlBuildfarm Maintainers
Builder Image AccessGitHub Org β†’ Packages β†’ Package SettingsOrganization Admins