Maintain and upgrade NetBox

An instance nobody maintains is a liability: the day it breaks is the day you find out the backup was never tested. This page sets up a nightly backup and restores it once, upgrades NetBox to a version you chose on purpose, wires the housekeeping job, and leaves a probe behind that tells you when something is wrong.

Before you start

$systemctl is-active netbox netbox-rq postgresql

active
active
active

Nightly backups

$sudo mkdir -p /var/backups/netbox && sudo chmod 700 /var/backups/netbox

#!/usr/bin/env bash
set -euo pipefail
stamp=$(date +%Y%m%d-%H%M)
dest=/var/backups/netbox
sudo -u postgres pg_dump -Fc netbox > "$dest/netbox-$stamp.dump"
files=(netbox/media netbox/netbox/configuration.py)
for f in netbox/netbox/ldap_config.py local_requirements.txt gunicorn.py; do
  if [ -e "/opt/netbox/$f" ]; then files+=("$f"); fi
done
tar -czf "$dest/netbox-files-$stamp.tar.gz" -C /opt/netbox "${files[@]}"
find "$dest" -name 'netbox-*' -mtime +14 -delete

$sudo chmod 750 /usr/local/sbin/netbox-backup.sh
$sudo systemctl daemon-reload
$sudo systemctl enable --now netbox-backup.timer
$sudo systemctl start netbox-backup.service

$systemctl is-active netbox-backup.timer

active

$ls /var/backups/netbox | sed 's/-[0-9]*-[0-9]*\././' | LC_ALL=C sort -u

netbox-files.tar.gz
netbox.dump

Restore drill

$latest=$(ls -t /var/backups/netbox/netbox-*.dump | head -1)
$sudo -u postgres createdb -O netbox netbox_drill
$sudo -u postgres pg_restore --no-owner --role=netbox -d netbox_drill < "$latest"

$sudo -u postgres psql -d netbox_drill -tAc 'SELECT count(*) > 0 FROM dcim_site'

t

$sudo -u postgres dropdb netbox_drill

Upgrade NetBox

$sudo systemctl start netbox-backup.service
$cd /opt/netbox
$sudo git fetch --tags
$sudo git checkout v4.4.1
$sudo ./upgrade.sh
$sudo systemctl restart netbox netbox-rq

$sudo git -C /opt/netbox describe --tags --exact-match

v4.4.1

$curl -skf -H 'Authorization: Token ${NETBOX_TOKEN}' https://netbox.example.com/api/status/ | python3 -c 'import sys,json; print("v" + json.load(sys.stdin)["netbox-version"])'

v4.4.1

$curl -sk -o /dev/null -w '%{http_code}' https://netbox.example.com/login/

200

Then open https://netbox.example.com/ and click through a device page and a search: the API answering is not the same as the UI rendering, since static files and plugins can break one and not the other.

If a plugin blocks the upgrade

• Check the plugin's compatibility table on its repository. If a compatible release exists, pin it in local_requirements.txt (netbox-bgp==0.15.0) and re-run sudo ./upgrade.sh.
• If none exists, remove the plugin from PLUGINS in configuration.py and from local_requirements.txt, upgrade, and put it back when the plugin catches up. Its tables stay in the database; nothing is lost, the pages just disappear until then.
• If the release notes say the plugin is now a core feature (it happens), migrate the data with the plugin's own instructions before upgrading.

Rollback

$sudo systemctl stop netbox netbox-rq
$cd /opt/netbox
$sudo git checkout $(sudo git describe --tags --abbrev=0 v4.4.1^)
$latest=$(ls -t /var/backups/netbox/netbox-*.dump | head -1)
$sudo -u postgres dropdb netbox && sudo -u postgres createdb -O netbox netbox
$sudo -u postgres pg_restore --no-owner --role=netbox -d netbox < "$latest"
$sudo ./upgrade.sh
$sudo systemctl start netbox netbox-rq

Housekeeping

$sudo ln -sf /opt/netbox/contrib/netbox-housekeeping.sh /etc/cron.daily/netbox-housekeeping
$sudo /opt/netbox/venv/bin/python /opt/netbox/netbox/manage.py housekeeping

$run-parts --test /etc/cron.daily | grep netbox

/etc/cron.daily/netbox-housekeeping

Logs and disk: gunicorn writes to stdout, which systemd captures in the journal; cap it. nginx and Apache rotate their own files through logrotate, shipped with the package.

[Journal]
SystemMaxUse=500M

Weekly, five minutes:

Monitoring basics

$systemctl status netbox netbox-rq --no-pager | grep Active
$curl -skf -H "Authorization: Token ${NETBOX_TOKEN}" https://netbox.example.com/api/status/ | python3 -m json.tool

$curl -skf -H 'Authorization: Token ${NETBOX_TOKEN}' https://netbox.example.com/api/status/ | python3 -c 'import sys,json; print(json.load(sys.stdin)["rq-workers-running"] >= 1)'

True

Done

Every night a dump and an archive land in /var/backups/netbox, you have restored one by hand, NetBox runs v4.4.1, the change log is pruned daily, and a probe knows when the workers stop. The next page stops clicking: an automation user, pynetbox, bulk imports, webhooks and custom scripts.