gobank-deploy

Deployment orchestrator for gobank environments: one Hetzner Cloud server per environment (prod, preprod, demo, …) running PostgreSQL and the Model Bank demo, created, deployed to, reported on and deleted by this tool. Creating a server is an explicit act; deleting it is what stops billing.

Pages

How a version reaches an environment

The code comes from the Forgejo. Releasing gobank there is the build stage: goreleaser attaches the demo binaries to the release, and the appliance, which cannot build, fetches them into its store before a deploy. The laptop builds the demo from its own checkout and can push a build to the appliance by hand. Either deploys to a server.

Release path

What the orchestrator talks to

From either place it runs: the same Hetzner project and Route 53 zone, with secrets from Bitwarden.

Components

Part Lives Role
gobank cmd/demo Forgejo, checked out at ../gobank the Model Bank: what gets deployed
gobank-deploy Forgejo; go install on the laptop, a gokrazy package on hydrogen this orchestrator: up, down, status, ui, build
gobank release Forgejo release of a tag demo-linux-amd64, demo-linux-arm64 from gobank's tp release (goreleaser): the build stage
release store build/releases on the laptop, /perm/gobank-deploy/releases on hydrogen demo for linux amd64 and arm64 per version, plus latest; filled from the gobank release on deploy, or by task push
Hetzner server gobank-<env> Hetzner Cloud project Ubuntu, PostgreSQL, the demo on :1347; a firewall of the same name; labels environment and, for a temporary one, expires
Route 53 record drummonds.net zone <env>.gobank.drummonds.net → the server, TTL 60 because addresses are reused
secrets Bitwarden tp secrets on the laptop; the gokrazy Environment on hydrogen

Networking and firewalls

Every server is on one private network, gobank, and carries a firewall of its own name. The firewalls filter the public interface only; traffic on the private network passes unfiltered. So a loader reaches a demo without the demo being open to the world, and home reaches both over their public addresses. The rules are declared, not created once: every up sets them, so a change here reaches standing environments on their next redeploy, and a server made before the network existed is attached to it then.

Networking

Who reaches what (inbound; admitted is the -admit ranges, home's /29; with -admit empty they are anywhere, as before):

From To Port Allowed Why
admitted demo box 22 ssh yes deploys, the console's token, journald
admitted demo box 1347 demo yes the page's probe and about.json, a browser at home
anywhere else demo box 22, 1347 no nothing outside home drives or browses a demo
anywhere demo box icmp yes ping for diagnosis
admitted loader 22 ssh yes install this program, run the journey, read its result
anywhere else loader any tcp no the loader only answers home
loader (10.128.1.x) demo box 1347 yes, unfiltered the load test over the private network
anywhere demo box 443 not yet Caddy in front of the BFF for TLS: the public face, a later story
Network choice Value Why
network gobank, 10.128.0.0/16 one for the project: every box can reach every other privately; in the upper half of 10/8, clear of home's 10.0 and 10.1 LANs, so a VPN or route between them later cannot clash
subnet 10.128.1.0/24, cloud the servers' addresses; a /24 holds far more boxes than run at once
zone eu-central holds fsn1, nbg1 and hel1, every location up tries
loader location the target's the load crosses one data centre's network, not fsn1 to hel1; another location only when the target's has no capacity, and the log says so
loader type cax11 (arm64) or cx23 (amd64) the smallest box of this host's architecture: the loader runs this program's own executable, copied over

What up does

One sequence for create and redeploy; only the -create box starts billing.

Deploy sequence

The demo workflow

A temporary environment is a gobank-workflow pipeline instance keyed <env>@<expiry>: created with its expiry as a server label, served until then, removed. The page starts the workflow for any temporary server nobody is looking after, so a restart does not leave one billing.

Demo workflow

The upgrade drill

gobank's upgrade drill (ADR-0003) is a second workflow: one instance per environment, release pair and date. Step 3 of the manual drill, the release, stays with the operator; the rest runs from the Drill button.

Drill workflow

Term Meaning
drill one rehearsal on an environment: upgrade from release N to N+1, roll back to N, forward to N+1
hop one redeploy inside a drill: upgrade, rollback, forward
observation the demo as read at a moment (before, upgraded, rolled back, forward): version serving, position, newest restart row
position the demo's day, day count, customers, savings, lending
restart row the demo's own record of a process start: which release it followed, the downtime, whether day and customers matched across the stop
gate a check on a hop's observation that fails the stage

What a hop is gated on, and what is recorded but not gated:

Observation after the hop Outcome
version serving is not the release fetched fail: serving X, expected Y
no restart row fail: the demo kept no record of this start
restart row follows a release other than the one before the hop (an unrecorded one is accepted) fail: follows X, expected Y
downtime unknown fail: the previous release did not stop cleanly
handover not intact fail: day or customers differ across the stop
restart row's day more than one past the last observation fail: the run went on before the stop (a clean stop finishes only the day in progress)
day now more than one past the restart row's fail: the run went on after the start (a restart begins only one day)
release without about.json version checked; position and restart row recorded as absent
customers, savings, lending differ from before recorded side by side, not gated: the generator moves them within a day; the restart row already pins customers across the stop
all of the above hold the hop completes

Every drill is kept with its observations in the page's pglike database beside the workflow runs; the Drills page is the history, the DB Explorer the tables.

The load test

The customer app's journey walked by many customers at once, from a loader box beside the target on the private network, up a ladder of concurrent users until a screen is too slow.

Load workflow

Term Meaning
journey the customer's screens in order, defined as data (load.Customer): login, accounts (the balances), account (one account's details and transactions)
screen one request of the journey, timed body and all
target the environment under load: a fresh one at a scale, its books filled with customers enough for the top rung (500)
loader the box the load comes from, <target>-loader: this program, no demo
rung so many users walking at once, each the journey -reps times (10), every walk the next customer in sequence, cust-001 onwards and round, so nothing is answered from a cache
ladder the rungs in order: 1, 2, 5, 10, 20, 50, 100, 200, 500 users
budget the p95 a screen may reach and the rung still hold: 2s
breach why a rung is unacceptable: a journey failed, or a screen's p95 over the budget
capacity the users at the last rung that held; the climb stops at the first breach
At a rung Outcome
every journey completed, every screen's p95 ≤ budget holds: capacity is now this rung's users; climb on
a journey failed (refused, timed out after 30s, no token) breach: the climb stops, capacity stays at the rung below
a screen's p95 > budget breach: the climb stops, capacity stays at the rung below
the top rung holds capacity is the top rung; the breach is blank
the first rung breaches capacity 0

The simulation is stopped for the load, so the figures are the BFF's alone. The Load tests page charts the capacity against gobank's version, a line per scale; a test's run page has every rung.

The Workflows page

The engine's view of the three workflows: each definition's diagram, steps and code, the count of its instances in every state, and the instances themselves, newest first. An instance page lays the steps as they ran over the definition, so a step not reached shows as not run, and offers the instance as d2 with the steps coloured by state.

Environment states

Server Answers Version vs available State on the page Offered
none – – Not provisioned, no billing Create (scale, keep or remove after)
none, not an -envs name, a finished job – – Finished: job, not provisioned Create, Forget (drops the row; its runs stay)
exists no – Not answering, booting? Redeploy, Down
exists yes same Serving, current Redeploy, Down
exists yes different Serving, vX available Redeploy, Down
exists, expires label any any as above, plus removed at the demo workflow's job, Down
exists yes a newer release on the repo, and a store here Serving, vX available Redeploy, Drill to vX, Down

Where no release can be had (no toolchain and an empty store) only status and Down are offered, and the page says why.