gobank-deploy

Deployment orchestrator for gobank environments. One environment (prod, preprod, ...) is one Hetzner Cloud server running PostgreSQL and the Model Bank demo. The orchestrator creates it, puts a release on it, reports on it and deletes it.

It replaces the deploy/hetzner/*.sh scripts that lived in gobank, with the same behaviour and the same billing rule: creating a server is an explicit act (-create), and deleting it is what stops billing.

Usage

HCLOUD_TOKEN is needed, so run everything through tp secrets:

tp secrets gobank-deploy up preprod -create              # cx23: 2 vCPU / 4 GB
tp secrets gobank-deploy up preprod -create -scale large # ccx33: 8 dedicated vCPU / 32 GB
tp secrets gobank-deploy up preprod                      # redeploy binary only
tp secrets gobank-deploy status preprod
tp secrets gobank-deploy down preprod                    # asks; -y to skip

A web page with the same controls, one row per environment, polling every few seconds while a job runs (Hetzner itself is asked at most once every 30 seconds, and at once after anything this program changes):

tp secrets task ui                       # http://localhost:1348/
tp secrets task ui ENVS=prod,preprod     # rows always shown, even when not provisioned

The rows are the -envs names (default prod,preprod,demo) plus every gobank-* server that exists in the Hetzner project, whoever created it. A form on the page creates a further environment by name; it stays listed while its server exists or its last job is on the page. Once a run has finished and its server has gone, Forget drops the row; the run's records stay. So -envs is only the standing environments worth a "Not provisioned" row.

On a host that cannot build cmd/demo (no Go toolchain or no gobank checkout at -src), the page deploys from a release store filled from the Forgejo, or, without one, shows status and offers Down only. That is how it runs on hydrogen, the LAN gokrazy appliance, as a package of its gok_local instance config:

"git.bytestone.uk/hum3/gobank-deploy/cmd/gobank-deploy": {
  "CommandLineFlags": ["-build", "/perm/gobank-deploy", "-store", "/perm/gobank-deploy/releases",
                       "ui", "-addr", ":1348", "-envs", "prod,preprod,demo"],
  "Environment": ["HCLOUD_TOKEN=${HCLOUD_TOKEN}", "GOBANK_DEPLOY_SSH_KEY=${GOBANK_DEPLOY_SSH_KEY}",
                  "AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}", "AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}"],
  "WaitForClock": true
}

with Caddy proxying gobank-deploy.lan.drummonds.net to :1348. The appliance cannot build cmd/demo, so it deploys from a release store (-store), which it fills from the Forgejo itself: gobank's tp release runs goreleaser (the build stage, .goreleaser.yaml in gobank) and attaches demo-linux-amd64 and demo-linux-arm64 to the release of the tag, and before every Create or Redeploy the page fetches the newest tag's binaries into the store when it lacks them. gobank's release also tells the page straight away: its post_release step (task deploy:fetch) posts to /fetch, which fetches the newest release into the store now and answers with its tag, or an error status the release prints as a warning. So a new gobank version reaches an environment with nothing else released or updated: release gobank, press Redeploy. A tag without binaries (from before the build stage) or a Forgejo out of reach is a notice in the job log, and the deploy carries what the store already has.

The laptop can fill the store by hand for a release the repo has not built, or to roll back:

task build    # cmd/demo for linux amd64+arm64 into build/releases/<version>/, marks it latest
task push     # build, then copy the latest release to hydrogen:/perm/gobank-deploy/releases

Pushing an older version again makes it the store's latest, and since the store still has the newer one it is not fetched again: the next deploy is the rollback. From the store itself, POST /fetch?tag=vX makes vX the next deploy, fetching it first if the store lacks it, so a rollback on hydrogen is:

curl -X POST https://gobank-deploy.lan.drummonds.net/fetch?tag=v0.6.0   # then Redeploy

The unit's TimeoutStopSec=900 lets the demo finish the simulated day in progress before it exits (it does so on SIGTERM); the install script adds it to units from before it. The page and status show gobank's newest tag on the Forgejo beside what this host can deploy, and say whether the next deploy fetches it (a store) or the laptop's checkout needs a pull. Without a store and a toolchain the page offers status and Down only, and says why. The appliance's ssh identity is GOBANK_DEPLOY_SSH_KEY (base64 of the private key in PEM, from the gokrazy secrets note); its public half is registered in the Hetzner project so new servers accept it, and was added to servers that predate it by hand. Pinned host keys live in /perm/gobank-deploy/known_hosts.

The upgrade drill

Drill to vX on an environment's row runs gobank's upgrade drill (ADR-0003) as a workflow: a day long enough to upgrade inside (set to 2h when under 30m, then wait for a day with 10m left), observe the demo, upgrade to the newest release, roll back, forward again, observing after each hop. The demo is read at its /about.json; each hop is gated on the version serving, a restart row that follows the expected release with a known downtime and an intact handover, and the run no further on than a restart takes it (the stop finishes the day in progress, the start begins a new one: one day at each). The day length is set again after every hop, since a release before gobank v0.10.3 forgets it on restart. It is offered where the manual drill runs: a store to roll back from and a newer release than the one serving. A failed hop fails the run and says why; pressing Drill again the same day resumes from that hop. The Drills page is the history, one line per drill ready for the story's record; each run's steps are at /workflows/instances/<id>, the tables at /internal/explorer.

The Workflows pages are gobank-workflow's own (its view package) in this page's layout. Instances are a work queue, those in hand first, filtered by workflow, status, date and key. Definitions each have their diagram, their steps with the code behind each, and their instances. An instance page lays the steps as they ran over the definition, so a step not reached shows as not run. Under it come what only this program records: a drill's observations, a scan's routes, a load test's climb, a perf run's figures. Each instance is also offered as d2 with the steps coloured by state (/workflows/instances/<id>/diagram.d2) for d2 diagram.d2 out.svg.

The page keeps its workflow runs and drills in a pglike (SQLite file) database, -db, default <-build>/gobank-deploy.db: on hydrogen that is /perm/gobank-deploy/gobank-deploy.db, beside the release store, so no flag is needed there. gobank-workflow's tables and this program's drill tables share the file, each component's schema versioned in its own migrations table (schema_migrations, deploy_schema_migrations, perf_schema_migrations).

Performance runs: the perf workflow

Measure on the environments page (one scale) or tp secrets gobank-deploy perf <env> (every scale at once: -scales small,large by default, one server each, named <env>-<scale>, the rows written into gobank's benchmark.md in the -src checkout — -doc to say where, or "" to only print them — for you to commit and release) makes gobank's performance run as a workflow: create (up -create at the scale with an expiry of twice the ceiling and an hour, so a run that hangs is reaped by the page's tick; a server that exists is kept), add (the demo set flat out with room for a million customers and a payment a customer a day, up to 10,000 a day, then batches of 1,000 customers until the books hold 10,000, the rate taken from the demo batch by batch so this program's polling is not in it), days (Run until the bank's day is seven years on or six hours have passed, requesting the staff pages — dashboard, P&L, balance sheet, BBSI, explorer, customers, payments, treasury — in rotation, one a poll, so their counts and p95s at scale land in the record; read account days per 12h and the last day it is made of, Stop), remove (down). The two rates are gobank v0.12's /about.json sim block. Each run is kept in the -db database with where it ran — scale, server type, RAM, gobank version — and the Performance page is the history, each run with its row for gobank's benchmark.md. A run creates a billable server for about half an hour, so it is started on purpose, never by a schedule.

A short run has the same shape in tens of minutes: -customers 100 -days 10 -ceiling 1h adds 100 customers, runs ten days instead of seven years, and gives the servers a three-hour expiry; pass -doc "" so its rows stay out of benchmark.md.

Scenarios and the journal: the scenario workflow

tp secrets gobank-deploy scenario seven-years runs a scenario — a named configuration driven through timed steps on a fresh environment named <scenario>-<scale> — as a workflow: create (up -create at the scale; a server that exists is kept), add (the demo set flat out with room for the customers asked for, -customers 1000, added in batches: added, on the books, customers/s), years (Run flat out until the bank's day is -years 7 on from where it started, give or take a poll: bank days run from and to, bank days/h, account days/h, the last day), wallclock (the day length set to -day-length 24h and the bank waited for until it follows the clock), probe (the dashboard, the customer search by name -search Jones — the name posted in the form, never in a URL, as gobank's search takes it — and the first customer it lists, fetched from outside as the operator at concurrency 1 and 4, twenty times each: p50 and p95; then Stop), remove (down, unless -keep). A perf or scenario run that fails after create removes its box, since a box bills while a failed run sits: create is undone, and the next attempt the same day starts from create rather than resuming; with -keep the box is left, failed or not. The run is kept in the -db database and written as a page of the journal, the notebook in this repository's journal/ folder: journal/<date>-<scenario>-<scale>.md with the configuration, steps and probes tables and an empty Notes section for your hand, and its line at the top of journal/index.md (-journal DIR to say where, "" to only print the page). A page is written once and never touched again; hand-written pages go in the folder and on the index too. The ui's Journal page under About shows the same for runs driven from a host with no checkout. The first scenario, seven-years, is the instrument for gobank's milestone 1 (gobank #151): the capacity of one synchronous system over seven years. At 1,000 customers a run is an afternoon of billable server, so it is started on purpose, never by a schedule.

Two clocks run through a page, and every figure says which it is on, so the figures multiply out: customers added is customers/s times the wall clock, account days/h is bank days/h times accounts.

Term Clock Meaning
wall clock wall how long a step took on the host's clock
bank day bank one business date the bank processes
day length wall the setting: how long the bank spends on one bank day
bank days, from, to bank the days run in the step and its first and last date
last day wall how long the last bank day took
customers/s, bank days/h, account days/h bank per wall what the step did, over its own wall clock, polling and all

The perf run's rates are a different instrument: the demo's own figures (/about.json sim), with this program's polling kept out of them.

The full performance scan: the scan workflow

tp secrets gobank-deploy scan <env> scans an environment as it stands, no server made or removed: survey (the environment must be serving; where the scan is, and the customers on the books), scan (every route the demo's /v1/openapi.json lists that the scan can drive — a GET without path parameters as the operator on the API token, or as a customer signed in on the app password by what the route's security says, and the customer search by its post, -search Jones — loaded at each rung of -ladder 1,4,16, -samples 20 calls each: p50, p95 and the throughput; a route with a path parameter, a post other than the search, or one that fails is listed as not scanned with why, so the coverage is visible). The scan is a run of the journal like a scenario, and it writes the measured document, -doc (default <-src>/api/measured.json in gobank's checkout): the date, gobank version, environment, server and customers, a row per route and rung, and the routes not scanned. gobank's About → API pages show it alongside each route's facts, so the API is documented with what it measured; commit the document and release gobank to publish it. A scan at 16 in flight is load: run it on preprod or a scenario's kept server, not prod.

The page reads the scans under Performance: Scans (/perf/scans) lists them and reads the latest against the one before — the critical routes (failed, not scanned where the previous scan drove them, or p95 at the top rung over a 500 ms budget), the ten worst by p95 at the top rung with the move since the previous scan (better or worse beyond 10%), and every route across the scans, a column per scan — and each scan has its page (/perf/scans/<run>) with every route at each rung, worst route first, the routes not scanned with why. The run page and the journal link to it.

The load test: the load workflow

tp secrets gobank-deploy load <env> -scale large load tests a fresh environment: create (the target at the scale, with an expiry), add (the demo stopped, its books filled to -customers, by default the top rung's users — the journey needs a customer to log in as, not full books), loader (a box <env>-loader in the target's location on the private network, ssh from -admit only, this program's own executable copied onto it — so it is arm64 from hydrogen and amd64 from the laptop), load (from the loader against the target's private address: the customer journey — login, the accounts screen, one account's screen — walked by each rung of -ladder 1,2,5,10,20,50,100,200,500 users -reps 10 times each, every walk the next customer, until a screen's p95 is over -budget 2s or a journey fails; a first rung with no journey completed fails the test rather than recording capacity 0), remove (both boxes, also when a stage fails). The capacity is the last rung that held. The page has a Load test form beside Performance run; Load tests under Performance (/perf/load) charts the capacity against gobank's version, a line per scale, and a test's run page has every rung.

gobank-deploy journey -base URL is the load test alone, from wherever it runs, against any BFF it can reach: the app password from GOBANK_APP_PASSWORD, the result as JSON on stdout, a line a rung on stderr. It needs no token; the loader runs it.

Firewalls and the private network

Every server is on the private network gobank (10.128.0.0/16, created on first use) and has a firewall of its own name, set on every up: ssh (22) and the demo (1347) from the -admit ranges, ping from anywhere. -admit (or GOBANK_DEPLOY_ADMIT) is home's range, comma separated CIDRs; empty admits anywhere, as before. Set it on the laptop and in hydrogen's gokrazy flags alike, or one of them will lock the other out of the boxes it made. Traffic on the private network is not filtered. The decision table is on the docs page.

Temporary environments: the demo workflow

"Remove after" on either Create form runs the demo workflow instead of a plain create: create (up with the removal time as an expires label on the server), serve (until then, or until someone presses Down), remove (down). It is a gobank-workflow pipeline: one keyed instance (<env>@<expiry>) whose stages are recorded, so a failed stage is resumed by the next run and a finished one is final. The run records are in the -db database; what must survive is on the server itself, and every minute the page starts the workflow for any temporary server nobody is looking after, so a demo still goes after the page restarts. demo-workflow.d2, drill-workflow.d2 and perf-workflow.d2 (from go run ./cmd/flowd2) draw the three workflows.

Each environment gets the hostname <env>.gobank.drummonds.net: an A record in Route 53 (TTL 60s, since Hetzner reuses addresses) set on up and removed on down. -dns DOMAIN changes the domain, -dns "" turns it off. It needs AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (tp secrets on the laptop, the gokrazy Environment on the appliance, as gokcaddy); without them DNS is skipped and the start-up says so. The hosted zone is the one whose name is the domain's longest suffix, so drummonds.net. The key's IAM policy must allow route53:ChangeResourceRecordSets on that zone for A records under the domain, UPSERT and DELETE (on top of ListHostedZones, ListHostedZonesByName, ListResourceRecordSets and GetChange); a key scoped to another use, such as the LAN Caddy's _acme-challenge TXT records, is refused with AccessDenied and the deploy carries on by address.

Scale presets: small (cx23), medium (cx33), large (ccx33: 8 dedicated vCPU, 32 GB, so a measurement sees cores that are really there), xl (ccx43: 16 dedicated vCPU, 64 GB), or any hcloud server-type list name. cax* types build for arm64. When Hetzner has no capacity for the type ("error during placement"), create tries fsn1, nbg1 and hel1 in turn, then the next size down the ladder (ccx43, ccx33, cx53, cx43, cx33, cx23) at every location again; the server's actual type is what status and a perf run report. The ladder is a first cut, to be revised once there is performance data.

up sizes the demo's memory to the box: GOBANK_MEMORY_LIMIT is set to half the server type's RAM (PostgreSQL shares the box) in /etc/gobank/deploy.env, which the systemd unit reads, so a redeploy can change it. The demo's own default, 800MB, is what a browser tab holds.

up builds cmd/demo from the gobank checkout given by -src (default ../gobank) with CGO_ENABLED=0, so that checkout's local replace directives still apply. Host keys are pinned per project in build/known_hosts, accept-new style; a freshly created server's IP is forgotten first because Hetzner reuses addresses.

Requirements

  • An SSH key registered in the Hetzner project and loaded in your agent (or at ~/.ssh/id_ed25519 / id_rsa).
  • tp unlock session for the Bitwarden-held HCLOUD_TOKEN.

Layout

  • internal/deploy — the up / down / status sequences against Cloud, Dialer, Builder and Prober interfaces; tested with fakes.
  • internal/hetzner — Cloud on hcloud-go.
  • internal/remote — ssh Dialer with host-key pinning, Go Builder, HTTP Prober, and the demo's Console (its /about.json and settings).
  • internal/ui — the lofigui page: states, create / redeploy / down / cancel, one job per environment with its log; the Drills, Performance and Workflows pages and an About page with the component diagrams; tested against a fake Operator.
  • internal/flows — the workflows on gobank-workflow's pipeline runner: the temporary demo environment, and gobank's upgrade drill.
  • internal/drills — the drill record: each drill with what the demo looked like before and after every hop, in the shared pglike database.
  • internal/route53, internal/store — DNS on Route 53; the release store.
  • cmd/gobank-deploy — the command.