---
title: "Postmortem - how to completely screw up an update"
date: 2025-10-19T12:05:10+02:00
lastmod: 2025-10-19T16:00:04+02:00
draft: false
image: "uploads/postmortem.png"
categories: [ 'English' ]
tags: [ 'backup', 'postmortem', 'fediverse', 'gotosocial' ]
---

The fediverse instance [gay-pirate-assassins.de](https://gay-pirate-assassins.de) was down for a couple of days. This
postmortem will outline what went wrong and what I did to prevent things from going that wrong in the future.

# Timeline

* 2025-10-05 17:26: [Update announcement](https://gay-pirate-assassins.de/@moanos/statuses/01K6TFQ1HVPAR6AYN08XYQ7XFV)
* 2025-10-05 ~17:45: Update started
* 2025-10-05 ~18:00: Services restart
* 2025-10-05 ~18:00: GoToSocial doesn't come up
* 2025-10-12 ~10:00: Issue is found
* 2025-10-12 10:30: Issue is fixed
* 2025-10-12 10:31: GoToSocial is started, migrations start
* 2025-10-12 15:38: Migrations finished successfully
* 2025-10-12 15:38: Service available again
* 2025-10-12 18:36:[Announcement sent](https://gay-pirate-assassins.de/@moanos/statuses/01K7CMGF7S2TE39792CMADGEPJ)

All times are given in CEST.

## The beginning: An update goes wrong

I run a small fediverse server with a few users called. [gay-pirate-assassins](https://gay-pirate-assassins.de/) which is powered by [GoToSocial](https://gotosocial.org/).
The (amazing) GoToSocial devs released `v0.20.0-rc1` and `v0.20.0-rc2`. As the new features seemed pretty cool, I'm
inpatient and the second release candidate seemed stable,
I decided to update to `v0.20.0-rc2`. So I stared a backup (via borgmatic), waited for it to finish and confirmed it ran
successfully.
Then I changed the version number in the [mash](https://github.com/mother-of-all-self-hosting/mash-playbook)-ansible
playbook I use. Then I pulled the newest version of the playbook and it's roles because I wanted to update all services
that run on the server. I checked
the [Changelog](https://github.com/mother-of-all-self-hosting/mash-playbook/blob/main/CHANGELOG.md),
didn't see anything and then started the update. It went through and GoToSocial started up just fine.

But the instance start page showed me 0 users, 0 posts and 0 federated instances. **Something has gone horribly wrong!**

## Migrations

It was pretty clear to me, that the migrations went wrong.
The [GoToSocial Migration notes](https://codeberg.org/superseriousbusiness/gotosocial/releases/tag/v0.20.0-rc1)
specifically mentioned long-running migrations that could take several hours. I assumed that somehow, during the running
database migration, the service must have restarted and left the DB in a broken state. This issue happened to me before.

Well, that's what backups are for, so let's pull it.

## Backups

Backups for this server are done two ways:

* via postgres-backup: Backups of the database are written to disk
* via [borgmatic](https://torsion.org/borgmatic/): Backups via borg are written to backup nodes, one of them at my home

They run every night automatically, monitored by [Healthchecks](https://healthchecks.io/). I triggered a manual run
before the update so that is the one I mounted using [Vorta](https://vorta.borgbase.com/).

And then the realization.

```

mash-postgres:5432 $ ls -lh
total 2.1M
-r-------- 1 moanos root 418K Oct 05 04:03 gitea
-r-------- 1 moanos root 123K Oct 05 04:03 healthchecks
-r-------- 1 moanos root 217K Oct 05 04:03 ilmo
-r-------- 1 moanos root 370K Oct 05 04:03 notfellchen
-r-------- 1 moanos root  67K Oct 05 04:03 oxitraffic
-r-------- 1 moanos root  931 Oct 05 04:03 prometheus_postgres_exporter
-r-------- 1 moanos root 142K Oct 05 04:03 semaphore
-r-------- 1 moanos root 110K Oct 05 04:03 vaultwarden
-r-------- 1 moanos root 669K Oct 05 04:03 woodpecker_ci_server
```

Fuck. The database gay-pirate-assassins is not there. Why?

To explain that I have to tell you how it *should* work: Services deployed by the mash-playbook are automatically wired
to the database and reverse proxy by a complex set of Ansible variables. This is great, because adding a service can
therefore be as easy as adding

```
healthchecks_enabled: true
healthchecks_hostname: health.hyteck.de
```

to the `vars.yml` file.

This will then configure the postgres database automatically, based on the `group_vars`. They look like this

```
mash_playbook_postgres_managed_databases_auto_itemized:
    - |-
      {{
          ({
          'name': healthchecks_database_name,
          'username': healthchecks_database_username,
          'password': healthchecks_database_password,
          } if healthchecks_enabled and healthchecks_database_hostname == postgres_connection_hostname and healthchecks_database_type == 'postgres' else omit)
      }}

```

Note that a healthchecks database is only added to the managed databases if `healthchecks_enabled` is `True`.

This is really useful for backups because the borgmatic configuration also pulls the list
`mash_playbook_postgres_managed_databases_auto_itemized`. Therefore, you do not need to specify which databases to back
up, it just backs up all managed databases.

However, the database for gay-pirate assassins was not managed. In the playbook it's only possible to configure a
service once. You can not manage multiple GoToSocial instances in the same `vars.yml`. In the past, I had two instances
of GoToSocial running on the server. I therefore
followed [the how-to of "Running multiple instances of the same service on the same host"](https://github.com/mother-of-all-self-hosting/mash-playbook/blob/main/docs/running-multiple-instances.md).

Basically this means that an additional `vars.yml` must be created that is treated as a completely different server.
Databases must be created manually as they are not managed.

With that knowledge you can understand that when I say that the database for gay-pirate-assassins was not managed,
this means it was not included in the list of databases to be backed up. The backup service thought it ran successfully,
because it backed up everything it knew of.

So this left me with a three-month-old backup. Unacceptable.

## Investigating

So the existing database needed to be rescued. I SSHed into the server and checked the database. It looked completely
normal.
I asked the devs if they could me provide me with the migrations as they already did in the past. However, they pointed
out that the migrations are too difficult for that approach. They suggested to delete the oldest migration to force a
re-run of the migrations.

Here is where I was confused, because this was the `bun_migrations` table:

```
gay-pirate-assassins=# SELECT * FROM bun_migrations ORDER BY id DESC LIMIT 5;
 id  |      name      | group_id |          migrated_at          
-----+----------------+----------+-------------------------------
 193 | 20250324173534 |       20 | 2025-04-23 20:00:33.955776+00
 192 | 20250321131230 |       20 | 2025-04-23 19:58:06.873134+00
 191 | 20250318093828 |       20 | 2025-04-23 19:57:50.540568+00
 190 | 20250314120945 |       20 | 2025-04-23 19:57:30.677481+00
```

The last migration ran in April, when I updated to `v0.19.1`. Strange.

At this point I went on vacation and paused investigations, not only because the vacation was great, but also because I
bamboozeld by this state.

---

After my vacation I came back, and did some backups of the database.

```
$ docker run -e PGPASSWORD="XXXX" -it --rm --network mash-postgres postgres pg_dump -U gay-pirate-assassins -h mash-postgres gay-pirate-assassins > manual-backup/gay-pirate-assassins-2025-10-13.sql
```

Then I deleted the last migration, as I was advised

```
DELETE FROM bun_migration WHERE id=193; 
```

and restarted the server. While watching the server come up it hit me in the face:

```
Oct 12 08:31:29 s3 mash-gpa-gotosocial[2251925]: timestamp="12/10/2025 08:31:29.905" func=bundb.sqliteConn level=INFO msg="connected to SQLITE database with address file:/opt/gotosocial/sqlite.db?_pragma=busy_timeout%281800000%29&_pragma=journal_mode%>
Oct 12 13:38:46 s3 mash-gpa-gotosocial[2304549]: timestamp="12/10/2025 13:38:46.588" func=router.(*Router).Start.func1 level=INFO msg="listening on 0.0.0.0:8080"
```

The server is **starting from a completely different database**! That explains why

* the last migration was never done
* the server showed me 0 users, 0 posts and 0 federated instances even though the postgres database had plenty of those

All of a sudden a SQlite database was configured. This happened because
of [this commit](https://github.com/mother-of-all-self-hosting/ansible-role-gotosocial/commit/df34af385f9765bda8f160f6985a47cb7204fe96)
which introduced SQlite support and set it as default. This was not mentioned in
the [Changelog](https://github.com/mother-of-all-self-hosting/mash-playbook/blob/main/CHANGELOG.md).

So what happened is, that the config changed and then the server was restarted and an empty DB was initialized. The
postgres DB never started to migrate.

## Fixing

To fix it, I did the following

1. Configure the playbook to use postgres for GoToSocial:

```
# vars.yml
gotosocial_database_type: postgres
```

2. Run the playbook to configure GoToSocial (but not starting the service)

```
just run-tags install-gotosocial
```

3. Check the configuration is correct
4. Start the service

The migrations took several hours but after that, everything looked stable again. I don't think there are any lasting
consequences. However, the server was unavailable for several days.

## Learnings

I believe the main issue here was not the change in the config that went unnoticed by me. While I'd ideally notice stuff
like this, the server is a hobby, and I'll continue to not check every config option that changed.

The larger issue was the backup. Having a backup would have made this easy to solve. And there are other, less lucky
problems where I'd be completely lost without a backup. So to make sure this doesn't happen again, I did/will do the
following:

### 1. Mainstream the config

As explained, I used a specific non-mainstream setup in the ansible playbook because, in the past, I ran two instances
of GoToSocial on the server. After shutting down one of them, I never moved gay-pirate-assassins to be part of the main
config. This means important parts of the configuration had to be done manually, which I botched.

So in the past week I cleaned up and gay-pirate-assassins is now part of the main `vars.yml` and will benefit from all
relevant automations.

### 2. Checking backups

I was confident in my backups because

* they run every night very consistently. If they fail e.g. because of a network outage I reliably get a warning.
* I verified successfully run of the backup job prior to upgrading

The main problem was me assuming that a successful run of the backup command, meant a successful backup. Everyone will
tell you that a backup that is not tested is not to be trusted. And they are right. However, doing frequent
test-restores
exceeds my time and server capacity. So what I'll do instead is the following:

* mount the backup before an upgrade
* `tail` the backup file as created by postgres-backup and ensure the data is from the same day
* check media folders for the last changed image

This is not a 100% guarantee, but I'd argue it's a pretty good compromise for now. As the frequency of mounting backups
increases and therefore becomes faster, I'll re-evaluate to do a test-restore at least semi-regulary.

## Conclusion

I fucked up, but I was lucky that my error was recoverable and no data was lost. Next time this will hopefully be not due
to luck, but better planning!

Any questions? Let me know!
