Validate every change in CI

A lint stage on a hosted runner, a lab stage on a self-hosted runner that deploys the topology, renders, pushes, runs a pytest with scrapli and always tears down, artifacts, and a protected main branch that only merges what the pipeline proved.

Three pages of tooling and still one weak point: a human runs it, on a lab that may or may not be in the state they think. This page makes the machine do it, on every change: a pipeline lints the repository, deploys a fresh lab on a runner installed on the lab machine, renders and pushes from NetBox, checks BGP and the pings with pytest, publishes the rendered configs and the test report, and tears the lab down whatever happened. Then main is protected, so nothing reaches it that the pipeline has not proved.

Before you start

$cd 
$git remote add origin 
$git branch -M main
$git push -u origin main

Check

$git -C  remote get-url origin

Expected output

The repository layout

text

/
├── topology.clab.yml        # page 1
├── sot/seed.py              # page 2
├── templates/, configs/     # page 3
├── render.py · push.py      # page 3 (or render.yml · push.yml, inventory/, group_vars/)
├── tests/test_lab.py        # this page
├── Makefile · requirements.txt · .yamllint
└── .github/workflows/lab.yml  or  .gitlab-ci.yml

${LAB_DIR}/Makefile

.RECIPEPREFIX = >
VENV = .venv/bin
.PHONY: deps lint deploy render push test destroy
deps:
> python3 -m venv .venv && $(VENV)/pip install -q -r requirements.txt
lint:
> $(VENV)/yamllint . && $(VENV)/ruff check .
deploy:
> sudo containerlab deploy -t topology.clab.yml --reconfigure
render:
> $(VENV)/python render.py
push:
> $(VENV)/python push.py --commit
test:
> $(VENV)/pytest -q tests --junitxml=report.xml
destroy:
> sudo containerlab destroy -t topology.clab.yml --cleanup

$printf 'report.xml\n' >> .gitignore

Lint locally

${LAB_DIR}/.yamllint

extends: default
rules:
  line-length: { max: 200 }
  truthy: { check-keys: false }
  document-start: disable
  comments: { min-spaces-from-content: 1 }

$make deps
$make lint

Check

$cd  && make lint >/dev/null && echo lint OK

Expected output

lint OK

A runner on the lab machine

$echo "$USER ALL=(ALL) NOPASSWD: /usr/bin/containerlab" | sudo tee /etc/sudoers.d/containerlab
$sudo chmod 440 /etc/sudoers.d/containerlab

In the repository: Settings → Actions → Runners → New self-hosted runner, Linux x64. Copy the download and configure commands from that page (they carry a one-time token), and add the label:

$mkdir -p ~/actions-runner && cd ~/actions-runner
$# curl … | tar xzf …   ← the two lines from the UI, with the current version
$./config.sh --url  --token <one-time token from the UI> --labels  --unattended
$sudo ./svc.sh install && sudo ./svc.sh start

Check

$systemctl list-units 'actions.runner.*' --no-legend | grep -c running

Expected output

The tests

${LAB_DIR}/tests/test_lab.py

import ipaddress
import time
import pytest
import yaml
from scrapli import Scrapli
NODES = yaml.safe_load(open("topology.clab.yml"))["topology"]["nodes"]
LOOPBACKS = list(ipaddress.ip_network("").hosts())
PEERS = {"spine1": 2, "leaf1": 1, "leaf2": 1}
PLATFORM, AUTH, BGP, PING = …
def run(node, command):
    with Scrapli(host=NODES[node]["mgmt-ipv4"], platform=PLATFORM, auth_username=AUTH[0],
                 auth_password=AUTH[1], auth_strict_key=False) as conn:
        return conn.send_command(command).result
def eventually(check, tries=12, wait=5):
    for _ in range(tries):
        if check():
            return True
        time.sleep(wait)
    return False
@pytest.mark.parametrize("node", list(NODES))
def test_bgp_established(node):
    assert eventually(lambda: run(node, BGP[0]).count(BGP[1]) >= PEERS[node])
def test_loopback_ping():
    assert "3 received" in run("leaf1", PING.format(dst=LOOPBACKS[2], src=LOOPBACKS[1]))

python

PLATFORM, AUTH = "nokia_srl", ("admin", "NokiaSrl1!")
BGP = ("info from state network-instance default protocols bgp neighbor * session-state", "session-state established")
PING = "ping -c 3 {dst} -I {src} network-instance default"

$make deploy render push
$make test

Check

$cd  && .venv/bin/pytest -q tests 2>/dev/null | tail -1 | grep -o '[0-9]* passed'

Expected output

4 passed

If the BGP test times out

Run make test a second time: if it passes, the fabric was still converging, and a tries=24 is the fix. If it fails again, the state on the node is what it is: docker exec clab-${LAB_NAME}-spine1 sr_cli 'show network-instance default protocols bgp neighbor', then the troubleshooting box of the previous page. A test that cannot connect at all (ScrapliAuthenticationFailed, timeouts) is a management IP that does not match the topology, or a node still booting: make deploy returns before SR Linux accepts SSH.

The pipeline

In the repository: Settings → Secrets and variables → Actions → New repository secret, name NETBOX_TOKEN.

${LAB_DIR}/.github/workflows/lab.yml

name: lab
on:
  pull_request:
  push:
    branches: [main]
  schedule:
    - cron: "17 3 * * *"
jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: make deps lint
  lab:
    needs: lint
    runs-on: [self-hosted, ]
    timeout-minutes: 30
    concurrency: { group: lab-, cancel-in-progress: false }
    env: { NETBOX_TOKEN: "${{ secrets.NETBOX_TOKEN }}" }
    steps:
      - uses: actions/checkout@v4
      - run: make deps deploy
      - run: make render push
      - run: make test
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: "lab-${{ github.sha }}", path: "configs/\nreport.xml" }
      - if: always()
        run: make destroy

$git add -A && git commit -m "ci: lint, lab and tests" && git push

Check

$cd  && python3 -c 'import yaml; print(sorted(yaml.safe_load(open(".github/workflows/lab.yml"))["jobs"]))'

Expected output

['lab', 'lint']

Check

$gh run list --workflow lab.yml -L 1 --json conclusion -q '.[0].conclusion'

Expected output

success

Protect main

Settings → Branches → Add branch ruleset (or Add classic branch protection rule) on main: Require a pull request before merging, Require status checks to pass with lint and lab as required checks, Do not allow bypassing. With the CLI:

$gh api -X PUT repos/{owner}/{repo}/branches/main/protection --input - <<'EOF'
${"required_status_checks": {"strict": true, "contexts": ["lint", "lab"]},
$ "enforce_admins": true, "required_pull_request_reviews": {"required_approving_review_count": 0},
$ "restrictions": null}
$EOF

Check

$gh api repos/{owner}/{repo}/branches/main/protection -q '.required_status_checks.contexts | sort | join(",")'

Expected output

lab,lint

Done

Every change to the lab is now proved before it lands: lint on the hosted runner, a fresh lab on with configuration rendered from , four tests, artifacts, teardown, and a main branch that only accepts what passed. The nightly run watches the source of truth for drift. From one Linux machine and one YAML file on the first page, this is a network you can rebuild, test and review like software, which is what NetDevOps means.

$git checkout -b try-it && sed -i 's/tries=12/tries=24/' tests/test_lab.py && git commit -am "tests: wait longer for BGP" && git push -u origin try-it

Open the pull request and watch the runner do in five minutes what the first three pages did by hand.

Validate every change in CI

Before you start

The repository layout

Lint locally

A runner on the lab machine

The tests

The pipeline

Protect main

Done

Did everything work?