15e0d6c7df
Rsyslog on Noble has apparmor rules that restrict rsyslog socket creation to /var/lib/*/dev/log. Previously we were configuring haproxy hosts to create an rsyslog socket for haproxy at /var/haproxy/dev/log which doesn't match the apparmor rule so gets denied. To address this we move all the host side haproxy config from /var/haproxy to /var/lib/haproxy. This allows rsyslog to create the socket. To avoid needing to update docker images (for haproxy statsd) and to continue to make the haproxy container itself happy we don't adjust paths on the target side of our bind mounts. This means some things still refer to /var/haproxy but they should all be within containers. I don't believe this will be impactful to existing load balancer servers. We should deploy new content to /var/lib/haproxy then automatically restart services (rsyslog and haproxy container) because their configs are updating. One potential problem with this is rsyslog will restart before the containers do and its log path will have moved. If we are concerned about this we can configure rsyslog to continue to attempt to create the old path in addition to the new path (this will fail on Noble). Change-Id: I4582e6b2dda188583f76265ab78bcb00a302e375
181 lines
5.8 KiB
ReStructuredText
181 lines
5.8 KiB
ReStructuredText
:title: Gitea
|
|
|
|
.. _gitea:
|
|
|
|
Gitea
|
|
#####
|
|
|
|
Gitea is running on opendev.org
|
|
|
|
At a Glance
|
|
===========
|
|
|
|
:Hosts:
|
|
* https://opendev.org/
|
|
* gitea*.opendev.org
|
|
:Ansible:
|
|
* :git_file:`playbooks/roles/gitea`
|
|
* :git_file:`playbooks/roles/haproxy`
|
|
:Configuration:
|
|
* :git_file:`inventory/service/group_vars/gitea-lb.yaml`
|
|
:Projects:
|
|
* https://gitea.io/
|
|
:Bugs:
|
|
* https://storyboard.openstack.org/#!/project/748
|
|
* https://github.com/go-gitea/gitea/issues
|
|
|
|
Overview
|
|
========
|
|
|
|
The OpenDev Git repositories are hosted on a pool of servers. They
|
|
are served via https using Gitea behind HAProxy which handles load
|
|
balancing across the nodes.
|
|
|
|
.. _gitea-maintenance:
|
|
|
|
Backend Maintenance
|
|
===================
|
|
|
|
To temporarily remove a git backend from the HAProxy load balancer,
|
|
you can put it in "maintenance" mode. This can be done interactively
|
|
on the HAProxy host. Note that long-term changes to the topology
|
|
should be made via configuration management. These commands must be
|
|
run as root.
|
|
|
|
To see the current status of all servers::
|
|
|
|
echo "show stat" | socat /var/lib/haproxy/run/stats stdio
|
|
|
|
To disable a server (eg, gitea09)::
|
|
|
|
echo "disable server balance_git_http/gitea09.opendev.org" | socat /var/lib/haproxy/run/stats stdio
|
|
echo "disable server balance_git_https/gitea09.opendev.org" | socat /var/lib/haproxy/run/stats stdio
|
|
|
|
To re-enable a server::
|
|
|
|
echo "enable server balance_git_http/gitea09.opendev.org" | socat /var/lib/haproxy/run/stats stdio
|
|
echo "enable server balance_git_https/gitea09.opendev.org" | socat /var/lib/haproxy/run/stats stdio
|
|
|
|
To run these commands and others interactively, issue the prompt
|
|
command to haproxy::
|
|
|
|
socat readline /var/lib/haproxy/run/stats
|
|
prompt
|
|
|
|
Deploy a New Backend
|
|
====================
|
|
|
|
Our gitea servers do maintain a small amount of state (they remember
|
|
repo rename redirects) so there is a small amount of process required
|
|
to deploy a new Gitea backend.
|
|
|
|
To deploy a new Gitea backend we add it to the ansible inventory, but
|
|
exclude it from the manage-projects.yaml playbook. This will fully
|
|
provision an empty Gitea server with running Gitea and database processes.
|
|
Then we can manually restore the database from another node, create all of
|
|
the bare git repos, and replicate all of the repo content.
|
|
|
|
When these steps are done the new gitea backend can be added to the
|
|
haproxy config and its exclusion from "Create repos on gitea servers"
|
|
can be removed.
|
|
|
|
Restore the Gitea Database
|
|
--------------------------
|
|
|
|
The first step in restoring the database is to determine which container
|
|
is running the database::
|
|
|
|
docker ps -a
|
|
|
|
Make note of the container id for the container running the mariadb image.
|
|
|
|
Next we stop the gitea services::
|
|
|
|
docker stop $GITEA_CONTAINER_IDS
|
|
|
|
With services stopped and the container id captured we are ready to
|
|
restore the database. First find the database to restore; it is
|
|
backed up in ``/var/backups/gitea-mariadb`` on gitea hosts. You can
|
|
copy and uncompress it.
|
|
|
|
Then restore the database (note we use ``docker exec`` and not
|
|
``docker-compose exec`` for performance reasons)::
|
|
|
|
docker exec -i $DB_CONTAINER_ID bash -c '/usr/bin/mysql -uroot -p"$MYSQL_ROOT_PASSWORD"' < /root/gitea-mariadb.sql
|
|
|
|
When that is completed you can restart the docker containers that were
|
|
stopped::
|
|
|
|
# Check that containers are still stopped
|
|
docker ps -a
|
|
docker start $GITEA_CONTAINER_IDS
|
|
|
|
Create All Bare Git Repos
|
|
-------------------------
|
|
|
|
Gitea's admin dashboard includes a useful button to create all missing
|
|
git repos. At this point in the deployment we have recovered the DB
|
|
contents so Gitea knows there are missing repos and will happily create
|
|
empty replacements if we ask it to.
|
|
|
|
Login to Gitea as ``root`` via ``https://giteaXY.opendev.org:3000/user/login``.
|
|
The credentials can be found in hiera's group vars for the gitea group.
|
|
Navigate to ``https://giteaXY.opendev.org:3000/admin`` and click the run
|
|
button for ``Reinitialize all missing Git repositories for which records exist``.
|
|
|
|
Replicate Git Repo Content from Gerrit
|
|
--------------------------------------
|
|
|
|
First we must accept the RSA host key for the Gerrit server (not the
|
|
ecdsa key)::
|
|
|
|
gerrit2@review.opendev.org$ ssh -o HostKeyAlgorithms=ssh-rsa -p222 git@giteaXY.opendev.org
|
|
|
|
Then we can ask Gerrit to replicate all repo content into our new empty repos::
|
|
|
|
ssh -p 29418 $USER@review.opendev.org gerrit replication start --url giteaXY.opendev.org
|
|
|
|
You can monitor the progress of this with::
|
|
|
|
ssh -p 29418 $USER@review.opendev.org gerrit show-queue
|
|
|
|
Once this is complete, add the server back into the haproxy as
|
|
discussed above.
|
|
|
|
Reset a Corrupted Git Repo
|
|
==========================
|
|
|
|
It is possible for the repos Gitea hosts to become corrupted. Since Gerrit
|
|
is our source of truth the easiest way to handle this is have Gerrit replicate
|
|
the data back to Gitea. Unfortunately, replication will fail with a corrupted
|
|
repo on the destination. To work around this we replace the repo with a new
|
|
empty bare repository and then replicate.
|
|
|
|
First thing to do is :ref:`remove the backend from the haproxy rotation<gitea-maintenance>`.
|
|
|
|
Next we need to stop gitea on the backend::
|
|
|
|
cd /etc/gitea-docker
|
|
docker-compose down
|
|
|
|
Then move aside the old repo and replace it with a new empty bare repo::
|
|
|
|
cd /root/corrupted_repos
|
|
mv /var/gitea/data/git/repositories/org/example.git ./example.git.bak
|
|
git init --bare example.git
|
|
chown -R 1000:1000 example.git
|
|
mv ./example.git /var/gitea/data/git/repositories/org/example.git
|
|
|
|
Now start the gitea service back up again::
|
|
|
|
cd /etc/gitea-docker
|
|
docker-compose up -d mariadb gitea-web
|
|
# Wait for web to be responseive
|
|
docker-compose up -d gitea-ssh
|
|
|
|
Finally trigger a Gerrit replication::
|
|
|
|
ssh -p 29418 user.admin@review.opendev.org replication start --url $giteabackendname org/example
|
|
|
|
Once replication is complete you can add the backend to the haproxy rotation.
|