Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add rotate backups to documentation #50

Merged
33 changes: 26 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,11 @@ Docker image for performing simple backups of Docker volumes. Main features:

- Mount volumes into the container, and they'll get backed up
- Use full `cron` expressions for scheduling the backups
- Backs up to local disk, [AWS S3](https://aws.amazon.com/s3/), or both
- Backs up to local disk, to remote host available via `scp`, to [AWS S3](https://aws.amazon.com/s3/), or to all of them
- Allows triggering a backup manually if needed
- Optionally stops containers for the duration of the backup, and starts them again afterward, to ensure consistent backups
- Optionally `docker exec`s commands before/after backing up a container, to allow easy integration with database backup tools, for example
- Optionally executes commands before/after backing up inside `docker-volume-backup` container and/or on remote host
- Optionally ships backup metrics to [InfluxDB](https://docs.influxdata.com/influxdb/), for monitoring
- Optionally encrypts backups with `gpg` before uploading

Expand Down Expand Up @@ -276,18 +277,36 @@ If so configured, they can also be shipped to an InfluxDB instance. This allows

## Automatic backup rotation

You probably don't want to keep all backups forever. A more common strategy is to hold onto a few recent ones, and remove older ones as they become irrelevant. There's no built-in support for this in `docker-volume-backup`, but if you transfer your backups via SCP to a remote host, you can trigger the rotate-backups script by means of setting the environmental variable `POST_SCP_COMMAND`.
You probably don't want to keep all backups forever. A more common strategy is to hold onto a few recent ones, and remove older ones as they become irrelevant. There's no built-in support for this in `docker-volume-backup`, but you are enabled to trigger an external Docker container that includes [`rotate-backups`](https://pypi.org/project/rotate-backups/). In the examples, we draw on [docker-rotate-backups](https://github.com/jan-brinkmann/docker-rotate-backups).

### Rotation for local backups
In order to start an external Docker container, access to `docker.sock` has to be granted (as already seen in in the section on [stopping containers while backing up](#stopping-containers-while-backing-up)). Then, `docker-rotate-backups` can be run on local directories as well as on remote directories.

Check out these utilities, for example:
The default rotation scheme implemented in `docker-rotate-backups` preserves seven daily, four weekly, twelve monthly, and every yearly backups. For detailed information on customizing the rotation scheme, we refer to the [documentation](https://github.com/jan-brinkmann/docker-rotate-backups#how-to-customize).

* https://rotate-backups.readthedocs.io/en/latest/
* https://github.com/xolox/python-rotate-backups
### Rotation for local backups

Let `/home/pi/backups` be the path to your local backups. Then, initialize the environmental variable `POST_BACKUP_COMMAND` with the following command.
```
environment:
POST_BACKUP_COMMAND: "docker run --rm -e DRY_RUN=false -v /home/pi/backups:/archive ghcr.io/jan-brinkmann/docker-rotate-backups"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /home/pi/backups:/archive
```

### Rotation for backups tranferred via SCP

If you like to trigger `rotate-backups` on a remote host, install `rotate-backups` on the remote host (i.e., by means of `sudo pip install rotate-backups`). Then, follow the instructions for [backing up to remote host by means of SCP](#backing-up-to-remote-host-by-means-of-scp). Finally, set the environmental variable `POST_SCP_COMMAND: rotate-backups --daily 7 --weekly 4 --monthly 12 --yearly always /backup-directory` (where `/backup-directory` is the directory on the remote host where your backups has been transferred to). The suggested configuration preserves zero hourly, seven daily, four weekly, twelve monthly and unlimited yearly backups.
Here, let `/home/pi/backups` be the backup diretory on a remote host. To run `docker-rotate-backups` on that directory, the command within `POST_BACKUP_COMMAND` has to include all necessary information in order to access the remote host by means of SSH. Remember, if you transfer your [backups by means of SCP](#backing-up-to-remote-host-by-means-of-scp), all the information in `SSH_USER`, `SSH_HOST`, `SSH_ARCHIVE`, and the SSH public key are already there.
```
environment:
SCP_HOST: 192.168.0.42
SCP_USER: pi
SCP_DIRECTORY: /path/to/backups
POST_BACKUP_COMMAND: "docker run --rm -e DRY_RUN=false -e SSH_USER=pi -e SSH_HOST=192.168.0.42 -e SSH_ARCHIVE=/home/pi/backups -v /home/pi/.ssh/id_rsa:/root/.ssh/id_rsa:ro ghcr.io/jan-brinkmann/docker-rotate-backups"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /home/pi/.ssh/id_rsa:/ssh/id_rsa:ro
```

### Rotation for S3 backups

Expand Down
38 changes: 38 additions & 0 deletions test/triggering-rotate-backups/docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
version: "3"

services:

dashboard:
image: grafana/grafana:7.4.5
volumes:
- grafana-data:/var/lib/grafana # This is where Grafana keeps its data

backup-locally:
build: ../..
environment:
BACKUP_CRON_EXPRESSION: "0 * * * *"
# Command that is executed after the backup has been transferred:
# "Trigger external Docker container that includes rotate-backups and disable dry-run option."
POST_BACKUP_COMMAND: "docker run --rm -e DRY_RUN=false -v /home/pi/backups:/archive ghcr.io/jan-brinkmann/docker-rotate-backups"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- grafana-data:/backup/grafana-data:ro # Mount the Grafana data volume (as read-only)
- /home/pi/backups:/archive # Mount the directory where the backups are being stored

backup-scp:
build: ../..
environment:
BACKUP_CRON_EXPRESSION: "30 * * * *"
SCP_HOST: 192.168.0.42 # Remote host IP address
SCP_USER: pi # Remote host user to log in
SCP_DIRECTORY: /home/pi/backups # Remote host directory
# Command that is executed after the backup has been transferred:
# "Trigger external Docker container that includes rotate-backups and disable dry-run option."
POST_BACKUP_COMMAND: "docker run --rm -e DRY_RUN=false -e SSH_USER=pi -e SSH_HOST=192.168.0.42 -e SSH_ARCHIVE=/home/pi/backups -v /home/pi/.ssh/id_rsa:/root/.ssh/id_rsa:ro ghcr.io/jan-brinkmann/docker-rotate-backups"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro # Mount the docker.sock file (as read-only)
- /home/pi/.ssh/id_rsa:/ssh/id_rsa:ro # Mount the SSH private key (as read-only)
- grafana-data:/backup/grafana-data:ro # Mount the Grafana data volume (as read-only)

volumes:
grafana-data: