how-to-install-gce-server.txt

## Instructions to set up a GCE Disk Image for a Borealis Fireworker.
##
## NOTE:
## * Non-comment lines can be pasted into an ssh shell one section at a time.
##   Respond to input prompts and watch for errors.
## * `#` lines are instructions to carry out manually.
## * `##` lines are comments.

# * Create a Google Cloud Platform (GCP) project if you don't have one already.
#   If needed, enable billing, Compute Engine, Cloud Storage, StackDriver
#   Logging, IAM, Container Registry, and Cloud Build.
# * In the Google Cloud Platform Console > IAM > Service Accounts
#   https://console.cloud.google.com/iam-admin/serviceaccounts create a
#   Service Account "fireworker" and grant it access to your Cloud project.
# * In the Google Cloud Platform Console > IAM
#   https://console.cloud.google.com/iam-admin/iam grant these permissions to
#   your Compute Engine default service account *and* to the fireworker service
#   account:
#       Service Account User
#       Compute Instance Admin v1
#       Logs Writer
#       Storage Object Admin
#       Project Viewer
# * Decide which GCP Region and Zone you want to use. The cost varies by region.
# * Set up a MongoDB instance in that Region and Zone using either a Google
#   managed MongoDB service, a Compute Engine VM of your own with ordinary
#   MongoDB installation, or a Marketplace installation
#   https://console.cloud.google.com/marketplace/browse?q=mongodb


## =========================================================================
## TO CREATE A COMPUTE ENGINE DISK IMAGE
## This Disk Image will be the basis for creating Compute Engine VM instances.
## It will begin an Image Family to swap in upgraded Images when needed.

# * Create a Compute Engine VM instance via the Google Cloud Platform web
#   console https://console.cloud.google.com/compute/instancesAdd or via a
#   `gcloud compute instances create` command line.
#     Name: fireworker
#     Region & Zone: <where you want to run everything>
#     Machine family/series: N1 n1-standard-1 [or other. These steps create a
#       Disk Image. Later you can instantiate different machine types from this
#       Disk Image.]
#     Boot disk: New 200 GB Standard persistent disk, Ubuntu 20.04 LTS Minimal
#       (or later).
#       [Your needs might differ. You can resize it later but changing the OS
#       image requires creating a new Disk Image from scratch. Do NOT pick "COS"
#       Container-Optimized OS. COS doesn't have a package manager and is not
#       built to install software.]
#     Identity and API access: "Compute Engine default service account" or
#       the "fireworker" service account.
#     Access scopes: "Set access for each API":
#       Cloud Debugger: Enabled [we haven't tested this but it should come in handy]
#       Compute Engine: Read Write
#       Service Control: Enabled [the default]
#       Service Management: Read Write
#       Stackdriver Logging + Monitoring + Trace: Write Only [the default]
#       Storage: Read Write
#   Management > Description: Fireworks worker node
#   [Create]

## Connect to the GCE VM to run the setup steps, below.
gcloud compute ssh fireworker

sudo apt update && sudo apt upgrade && sudo apt autoremove

sudo apt install -y docker.io
sudo apt install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \
    libreadline-dev libsqlite3-dev llvm libncurses5-dev \
    libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python-openssl nano


## Reboot the GCE VM to ensure the apts take effect.
sudo reboot
## Give it a few seconds to reboot, then reconnect.
gcloud compute ssh fireworker


## Reinstall gcloud per https://cloud.google.com/sdk/docs/downloads-interactive
## so it can install updates and the docker-credential-gcr component. (The snap,
## apt, and yum packages don't support that.) (GCR is the Google Container
## Registry where you can store Docker Images.)
sudo snap remove google-cloud-sdk

curl https://sdk.cloud.google.com > install.sh
sudo mkdir /usr/local/bin/sdk
sudo chgrp ubuntu /usr/local/bin/sdk
sudo chmod g+ws /usr/local/bin/sdk
bash install.sh --install-dir=/usr/local/bin/sdk --disable-prompts

echo '' >> .profile
echo '. /usr/local/bin/sdk/google-cloud-sdk/path.bash.inc' >> .profile
echo '. /usr/local/bin/sdk/google-cloud-sdk/completion.bash.inc' >> .profile
. /usr/local/bin/sdk/google-cloud-sdk/path.bash.inc
. /usr/local/bin/sdk/google-cloud-sdk/completion.bash.inc

echo y | gcloud components install docker-credential-gcr

## Put the SDK's main executables on the path for all users.
sudo ln -s /usr/local/bin/sdk/google-cloud-sdk/bin/gcloud /usr/local/bin/
sudo ln -s /usr/local/bin/sdk/google-cloud-sdk/bin/docker-credential-gcloud /usr/local/bin/
sudo ln -s /usr/local/bin/sdk/google-cloud-sdk/bin/docker-credential-gcr /usr/local/bin/

## Enable the docker service.
sudo systemctl enable docker
sudo systemctl start docker

## Join the `docker` group so you can run docker commands without `sudo`.
sudo usermod -aG docker "$USER"

## To make that take effect, disconnect and reconnect.
exit
gcloud compute ssh fireworker

## Test docker
docker --version
docker info

## Set up to authenticate to GCR Docker repositories.
## You can pass specific gcr repo names here, e.g. gcr.io,eu.gcr.io,us.gcr.io,asia.gcr.io
echo y | gcloud auth configure-docker
## NOTE: If Docker auth gives you trouble, you can switch to docker-credential-gcr:
##   echo y | docker-credential-gcr configure-docker

## Pull any Docker Images you want preinstalled for faster task startup.
##
## Tip: If each developer's application Docker Image is built 'FROM' a lower
## level Image that's common to the team (containing runtime components like
## Linux apts, Python, and pips), then preinstalling the runtime Image will
## save Docker Image build time as well as task startup time.
### Example: docker pull python:3.8.6
### Example: docker pull gcr.io/$(gcloud config get-value core/project)/app-runtime

## ------------------------------------------------------------------------
## Create and switch to the fireworker user that'll run the borealis service.
sudo adduser --disabled-password fireworker

sudo usermod -aG docker fireworker
sudo su -l fireworker

## Set up fireworker to authenticate to GCR Docker repositories.
echo y | gcloud auth configure-docker

## Set up the software environment for borealis-fireworker
curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash
{
  echo 'export PATH="$HOME/.pyenv/bin:$PATH"'
  echo 'eval "$(pyenv init -)"'
  echo 'eval "$(pyenv virtualenv-init -)"'; } >> ~/.bash_aliases
source ~/.bash_aliases

mkdir ~/borealis && cd ~/borealis

pyenv install 3.8.6

pyenv global 3.8.6
pyenv local 3.8.6
pip install --upgrade pip setuptools virtualenv virtualenvwrapper virtualenv-clone wheel

pyenv virtualenv fireworker
pyenv local fireworker
pip install --upgrade pip setuptools virtualenv virtualenvwrapper virtualenv-clone wheel borealis-fireworks
pyenv rehash

## Find the borealis setup files that we'll need to copy.
SETUP=$(fireworker --setup)
echo The borealis-fireworker "setup" files are in: $SETUP

## Copy the launchpad yaml file from the borealis setup directory.
cp ${SETUP}/example_my_launchpad.yaml my_launchpad.yaml

## Configure fireworker's my_launchpad.yaml
#
# Edit my_launchpad.yaml with the
#     host:
#     port:
# values needed to connect to your MongoDB instance, e.g. "mongo", "27017".
#
# (The database connection fields
#     name:
#     username:
#     password:
# will normally be overridden by metadata attributes set when launching the GCE
# VMs, but you can set them here for a default database.)
#
# ... and optionally add settings like
#     idle_for_rockets: 900
#     idle_for_waiters: 3600
# if you want to adjust how long Fireworker will idle before exiting. Fireworker
# will idle `idle_for_rockets` seconds (default 15 * 60) for any READY-to-run
# rockets to appear in the queue or `idle_for_waiters` seconds (default 60 * 60,
# >= idle_for_rockets) for WAITING rockets to become READY, that is for queued
# rockets that are just waiting on other upstream rockets to finish. Consider
# making that longer when your workflow contains tasks that have to wait longer
# than that for prerequisite tasks.
#
# ... and optionally set
#     logdir: /home/fireworker/fw/logs
# for local Fireworks log files.
nano my_launchpad.yaml

## Copy the startup.sh script and the service file.
cp ${SETUP}/startup.sh .
cp ${SETUP}/borealis-fireworker.service .

## Logout from the fireworker account.
exit

### ------------------------------------------------------------------------

# Follow the instructions in the file you copied to
# /home/fireworker/borealis/borealis-fireworker.service
# to set up that systemd service.

### ------------------------------------------------------------------------

## MAKE A DISK IMAGE IN THE DISK IMAGE FAMILY "fireworker":

## Start up the service and check its journal for problems:
sudo systemctl start borealis-fireworker
sudo journalctl -u borealis-fireworker -f

## Stop the service and the server.
sudo systemctl stop borealis-fireworker
sudo shutdown -h now

# * Watch the Compute Engine > VM instances page to see when this VM has fully stopped.
# * Click on the disk "fireworker" in the Compute Engine > Disks console page.
#   https://console.cloud.google.com/compute/disks
# * Click "CREATE IMAGE".
# * Name the image like "fireworker-v0", picking the next number in the fireworker series.
# * Set "Family" to "fireworker"  <== MUST DO THIS. The launch-workers script will
#   instantiate workers by the Disk Image Family name.
# * Set "Description" to "Fireworks worker node" or some such for documentation.
# * Click "Create".
# * When it finishes:
#   * Test the Disk Image.
#   * Delete this GCE VM and its boot disk using gcloud or the console.
#   * Consider pruning the number of "fireworker-v*" Disk Images to 2 if you have more.


## =========================================================================
## TO UPDATE THIS DISK IMAGE IN THE FUTURE
## First start a VM from this Disk Image using the gce.py script (or use the
## GCE console or gcloud command line although they won't append "-0" to the
## VM name "fireworker"):
gce fireworker

## Connect to it:
gcloud compute ssh fireworker-0

## Stop the service:
sudo systemctl stop borealis-fireworker

## Upgrade apts:
## (If it says "Resource temporarily unavailable", wait and try again.)
sudo apt update && sudo apt upgrade && sudo apt autoremove

## If you upgraded any apts or if it printed `*** System restart required ***`:
sudo reboot
## ... and wait a few seconds, then reconnect:
gcloud compute ssh fireworker-0
sudo systemctl stop borealis-fireworker

## Trim the service journal.
sudo journalctl --vacuum-time=2d

## Pull any Docker images that you want to preload for speed.
docker pull python:3.8.6  # or similar, including your project's runtime Image

## Prune Docker images:
docker image prune

## Update the fireworker pips:
sudo su -l fireworker
cd ~/borealis
pip install --upgrade pip setuptools virtualenv virtualenvwrapper virtualenv-clone wheel borealis-fireworks
pyenv rehash

## Disconnect:
exit

# * Repeat the "Make a disk image" steps, above.
# * Test the new disk image. If it doesn't work, you can mark it "deprecated" in
#   the Image Family so new launches will use a previous Disk Image.