962af3e24e
Python3.12 (which is the python version on Noble) is not compatible with
our pinned borgbackup version (1.1.18). We get his errors when building
borgbackup on python3.12:
‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
We update to 1.2.8 on Noble which is one of the oldest versions claiming
python3.12 support. We try to use the oldest version to ensure maximum
compatiblity with 1.1.18 on the backup servers. Our CI job should give
us decent coverage and then the new paste02 will be the production
canary for whether or not these versions are compatible enough with each
other. No other servers should be effected in the initial pass.
Note there is an upgrade event horizon for using repos with borg<1.2.5
and borg >=1.2.5. It only affects repos that have archives that lack
TAMs. My read on it is that newer borg can treat those archives as
invalid and unceremoniously delete them. This is a problem if they are
valid archives and don't have a TAM. I suspect we will avoid this
problem because borg >= 1.0.9 creates TAMs with archives and we prune
our archives so older ones should be long gone. More info on this can be
found in these documents and reviewers are encouraged to read them:
https://borgbackup.readthedocs.io/en/1.2-maint/changes.html#pre-1-2-5-archives-spoofing-vulnerability-cve-2023-36811
https://borgbackup.readthedocs.io/en/1.2-maint/changes.html#borg-1-1-x-to-1-2-x
https://borgbackup.readthedocs.io/en/stable/usage/upgrade.html#borg-1-x-y-upgrades
I have left a todo for us to upgrade all of the services to 1.2.8 that
can run it (it requires python3.8 or newer so Focal or newer) but for
now we are taking baby steps.
Change-Id: I0c5ca758149b85aeec5321a704300489a57a3cc1
76 lines
2.5 KiB
Django/Jinja
76 lines
2.5 KiB
Django/Jinja
#!/bin/bash
|
|
|
|
# Flags based on
|
|
# https://borgbackup.readthedocs.io/en/stable/quickstart.html
|
|
|
|
if [ -z "$1" ]; then
|
|
echo "Must specify backup host"
|
|
exit 1
|
|
fi
|
|
|
|
BORG="/opt/borg/bin/borg"
|
|
BORG_CREATE="${BORG} create --verbose --filter AME --list --stats --show-rc --compression lz4 --exclude-caches "
|
|
|
|
# Setting this, so the repo does not need to be given on the commandline:
|
|
export BORG_REPO="ssh://{{ borg_username}}@${1}/opt/backups/{{ borg_username }}/backup"
|
|
|
|
# some helpers and error handling:
|
|
info() { printf "\n%s %s\n\n" "$( date )" "$*" >&2; }
|
|
trap 'echo $( date ) Backup interrupted >&2; exit 2' INT TERM
|
|
|
|
info "Starting backup"
|
|
|
|
# This avoids UI prompts when first accessing the remote repository
|
|
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=1
|
|
|
|
# Backup the most important directories into an archive named after
|
|
# the machine this script is currently running on:
|
|
# TODO(clarkb) Borg 1.2 deprecated exclude paths starting with a leading /
|
|
# Borg 1.2 should strip them off for us, but we should clean up our excludes
|
|
# after everything is running 1.2 or newer.
|
|
${BORG_CREATE} \
|
|
{% for item in borg_backup_excludes + borg_backup_excludes_extra -%}
|
|
--exclude '{{ item }}' \
|
|
{% endfor -%}
|
|
::'{hostname}-filesystem-{now}' \
|
|
{% for item in borg_backup_dirs + borg_backup_dirs_extra -%}
|
|
{{ item }} {{ '\\' if not loop.last }}
|
|
{% endfor -%}
|
|
|
|
backup_exit=$?
|
|
|
|
for f in $(shopt -s nullglob; echo /etc/borg-streams/*)
|
|
do
|
|
stream_name=$(basename $f)
|
|
info "Backing up stream archive $stream_name"
|
|
bash $f | ${BORG_CREATE} --stdin-name ${stream_name} \
|
|
::"{hostname}-${stream_name}-{now}" -
|
|
|
|
_status=( "${PIPESTATUS[@]}" )
|
|
if [[ ${_status[0]} -ne 0 ]]; then
|
|
info "Streaming script ${f} failed!"
|
|
info "Note that problems in the ssh connectivity might cause the streaming script to fail. You may need to check both halves of the streaming backup."
|
|
stream_exit=${_status[0]}
|
|
elif [[ ${_status[1]} -ne 0 ]]; then
|
|
info "Borg failed (rc: ${_status[1]})!"
|
|
stream_exit=${_status[1]}
|
|
else
|
|
stream_exit=0
|
|
fi
|
|
(( backup_exit = backup_exit || stream_exit ))
|
|
done
|
|
|
|
if [ ${backup_exit} -eq 0 ]; then
|
|
info "Backup finished successfully"
|
|
else
|
|
info "Backup finished with errors"
|
|
if [ ${BORG_UNDER_CRON:-0} -eq 1 ]; then
|
|
echo "Backups failed on host $(hostname) at $(date)." | \
|
|
mail -s "ACTION REQUIRED: Backup failed on $(hostname)" infra-root@openstack.org
|
|
fi
|
|
fi
|
|
|
|
|
|
exit ${backup_exit}
|
|
|