From 9beedcf4e09369b82285a9e4b5cdce5705718e52 Mon Sep 17 00:00:00 2001 From: Manav Rathi Date: Mon, 18 Mar 2024 10:08:36 +0530 Subject: [PATCH] Move into separate folder --- infra/services/prometheus/README.md | 9 ++- infra/services/promtail/README.md | 2 +- server/README.md | 7 +- server/scripts/deploy/README.md | 73 +++++++++++++++++++ server/scripts/{ => deploy}/museum.service | 0 .../deploy/update-and-restart-museum.sh | 15 ++++ 6 files changed, 98 insertions(+), 8 deletions(-) create mode 100644 server/scripts/deploy/README.md rename server/scripts/{ => deploy}/museum.service (100%) create mode 100755 server/scripts/deploy/update-and-restart-museum.sh diff --git a/infra/services/prometheus/README.md b/infra/services/prometheus/README.md index 79bee13d5b..54ca3e5595 100644 --- a/infra/services/prometheus/README.md +++ b/infra/services/prometheus/README.md @@ -3,9 +3,10 @@ Install `prometheus.service` on an instance if it is running something that exports custom Prometheus metrics. In particular, museum does. -Also install `node-exporter.service` (after installing -[node-exporter](https://prometheus.io/docs/guides/node-exporter/) itself) if it -is a production instance whose metrics (CPU, disk, RAM etc) we want to monitor. +If it is an instance whose metrics (CPU, disk, RAM etc) we want to monitor, also +install `node-exporter.service` after installing +[node-exporter](https://prometheus.io/docs/guides/node-exporter/) itself (Note +that our prepare-instance script already installs node-exporter) . ## Installing @@ -14,7 +15,7 @@ remember to change the hardcoded `XX-HOSTNAME` too in addition to adding the `remote_write` configuration. ```sh -scp -P 7426 services/prometheus/* : +scp services/prometheus/* : nano prometheus.yml sudo mv prometheus.yml /root/prometheus.yml diff --git a/infra/services/promtail/README.md b/infra/services/promtail/README.md index 1bcf448337..389542e177 100644 --- a/infra/services/promtail/README.md +++ b/infra/services/promtail/README.md @@ -9,7 +9,7 @@ Replace `client.url` in the config file with the Loki URL that Promtail should connect to, and move the files to their expected place. ```sh -scp -P 7426 services/promtail/* : +scp services/promtail/* : nano promtail.yaml sudo mv promtail.yaml /root/promtail.yaml diff --git a/server/README.md b/server/README.md index 06fdcb5184..40b479577d 100644 --- a/server/README.md +++ b/server/README.md @@ -95,9 +95,10 @@ setup we ourselves use in production. > [!TIP] > > On our production servers, we wrap museum in a [systemd -> service](scripts/museum.service). Our production machines are vanilla Ubuntu -> images, with Docker and Promtail installed. We then plonk in this systemd -> service, and use `systemctl start|stop|status museum` to herd it around. +> service](scripts/deploy/museum.service). Our production machines are vanilla +> Ubuntu images, with Docker and Promtail installed. We then plonk in this +> systemd service, and use `systemctl start|stop|status museum` to herd it +> around. Some people new to Docker/Go/Postgres might have general questions though. Unfortunately, because of limited engineering bandwidth **we will currently not diff --git a/server/scripts/deploy/README.md b/server/scripts/deploy/README.md new file mode 100644 index 0000000000..40da5183f8 --- /dev/null +++ b/server/scripts/deploy/README.md @@ -0,0 +1,73 @@ +# Production Deployments + +Museum runs using Docker + systemd on production instances, load balanced via +Cloudflare. + +This document outlines how we ourselves deploy museum. Note that this is very +specific to our use case, and while this might be useful as an example, this is +likely overkill for simple self hosted deployments. + +## Overview + +We use museum's Dockerfile to build images which we then run on vanilla Ubuntu +servers (+ Docker installed). For ease of administration, we wrap Docker +commands to start/stop/update it in a systemd service. + +* The production machines are vanilla Ubuntu instances, with Docker and Promtail +installed. + +* There is a [GitHub action](../../../.github/workflows/server-release.yml) to + build museum Docker images using its Dockerfile. + +* We wrap the commands to start and stop containers using these images in a + systemd service. + +* We call this general concept of standalone Docker images that are managed +using systemd as "services". More examples and details +[here](../../../infra/services/README.md). + +* So museum is a "service". You can see its systemd unit definition in + [museum.service](museum.service) + +* On the running instance, we use `systemctl start|stop|status museum` to manage + it. + +* The service automatically updates itself on each start. There's also a + convenience [script](update-and-restart-museum.sh) that pre-downloads the + latest image to further reduce the delay during a restart. + +## Installation + +To bring up an additional museum node: + +* Prepare the instance to run our services + +* Setup [promtail](../../../infra/services/promtail/README.md), [prometheus and node-exporter](../../../infra/services/prometheus/README.md) services + +* Add credentials + + sudo mkdir -p /root/museum/credentials + sudo tee /root/museum/credentials/tls.cert + sudo tee /root/museum/credentials/tls.key + sudo tee /root/museum/credentials/pst-service-account.json + sudo tee /root/museum/credentials/fcm-service-account.json + sudo tee /root/museum/credentials.yaml + +* Copy the service definition and restart script to the new instance. The + restart script can remain in the ente user's home directory. Move the service + definition to its proper place. + + scp /scripts/museum.service : + scp update-and-restart-museum.sh : + + sudo mv museum.service /etc/systemd/system + sudo systemctl daemon-reload + +## Starting + +SSH into the instance, and run + + ./update-and-restart-museum.sh + +This'll ask for sudo credentials, pull the latest Docker image, restart the +museum service and start tailing the logs (as a sanity check). diff --git a/server/scripts/museum.service b/server/scripts/deploy/museum.service similarity index 100% rename from server/scripts/museum.service rename to server/scripts/deploy/museum.service diff --git a/server/scripts/deploy/update-and-restart-museum.sh b/server/scripts/deploy/update-and-restart-museum.sh new file mode 100755 index 0000000000..e4e5b34da6 --- /dev/null +++ b/server/scripts/deploy/update-and-restart-museum.sh @@ -0,0 +1,15 @@ +#!/bin/sh + +# This script is meant to be run on the production instances. +# +# It will pull the latest Docker image, restart the museum process and start +# tailing the logs. + +set -o errexit + +# The service file also does this, but also pre-pull here to minimize downtime. +sudo docker pull rg.fr-par.scw.cloud/ente/museum-prod + +sudo systemctl restart museum +sudo systemctl status museum | more +sudo tail -f /root/var/logs/museum.log