gobank-deploy
Deployment orchestrator for gobank
environments. One environment (prod, preprod, ...) is one Hetzner Cloud
server running PostgreSQL and the Model Bank demo. The orchestrator
creates it, puts a release on it, reports on it and deletes it.
It replaces the deploy/hetzner/*.sh scripts that lived in gobank, with
the same behaviour and the same billing rule: creating a server is an
explicit act (-create), and deleting it is what stops billing.
Usage
HCLOUD_TOKEN is needed, so run everything through tp secrets:
tp secrets gobank-deploy up preprod -create # cx23: 2 vCPU / 4 GB
tp secrets gobank-deploy up preprod -create -scale large # ccx33: 8 dedicated vCPU / 32 GB
tp secrets gobank-deploy up preprod # redeploy binary only
tp secrets gobank-deploy status preprod
tp secrets gobank-deploy down preprod # asks; -y to skip
A web page with the same controls, one row per environment, polling every few seconds while a job runs (Hetzner itself is asked at most once every 30 seconds, and at once after anything this program changes):
tp secrets task ui # http://localhost:1348/
tp secrets task ui ENVS=prod,preprod # rows always shown, even when not provisioned
The rows are the -envs names (default prod,preprod,demo) plus every
gobank-* server that exists in the Hetzner project, whoever created it.
A form on the page creates a further environment by name; it stays listed
while its server exists or its last job is on the page. Once a run has
finished and its server has gone, Forget drops the row; the run's records
stay. So -envs is only the standing environments worth a "Not
provisioned" row.
On a host that cannot build cmd/demo (no Go toolchain or no gobank
checkout at -src), the page deploys from a release store filled from the
Forgejo, or, without one, shows status and offers Down only. That is how it
runs on hydrogen, the LAN gokrazy appliance, as a package of its gok_local
instance config:
"git.bytestone.uk/hum3/gobank-deploy/cmd/gobank-deploy": {
"CommandLineFlags": ["-build", "/perm/gobank-deploy", "-store", "/perm/gobank-deploy/releases",
"ui", "-addr", ":1348", "-envs", "prod,preprod,demo"],
"Environment": ["HCLOUD_TOKEN=${HCLOUD_TOKEN}", "GOBANK_DEPLOY_SSH_KEY=${GOBANK_DEPLOY_SSH_KEY}",
"AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}", "AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}"],
"WaitForClock": true
}
with Caddy proxying gobank-deploy.lan.drummonds.net to :1348. The
appliance cannot build cmd/demo, so it deploys from a release store
(-store), which it fills from the Forgejo itself: gobank's tp release
runs goreleaser (the build stage, .goreleaser.yaml in gobank) and
attaches demo-linux-amd64 and demo-linux-arm64 to the release of the
tag, and before every Create or Redeploy the page fetches the newest tag's
binaries into the store when it lacks them. gobank's release also tells the
page straight away: its post_release step (task deploy:fetch) posts to
/fetch, which fetches the newest release into the store now and answers
with its tag, or an error status the release prints as a warning. So a new
gobank version reaches an environment with nothing else released or
updated: release gobank, press Redeploy. A tag without binaries (from before the build stage) or a Forgejo
out of reach is a notice in the job log, and the deploy carries what the
store already has.
The laptop can fill the store by hand for a release the repo has not built, or to roll back:
task build # cmd/demo for linux amd64+arm64 into build/releases/<version>/, marks it latest
task push # build, then copy the latest release to hydrogen:/perm/gobank-deploy/releases
Pushing an older version again makes it the store's latest, and since the
store still has the newer one it is not fetched again: the next deploy is
the rollback. From the store itself, POST /fetch?tag=vX makes vX the
next deploy, fetching it first if the store lacks it, so a rollback on
hydrogen is:
curl -X POST https://gobank-deploy.lan.drummonds.net/fetch?tag=v0.6.0 # then Redeploy
The unit's TimeoutStopSec=900 lets the demo finish the simulated day in
progress before it exits (it does so on SIGTERM); the install script adds
it to units from before it. The page and status show gobank's newest tag on the Forgejo
beside what this host can deploy, and say whether the next deploy fetches
it (a store) or the laptop's checkout needs a pull. Without a store and a
toolchain the
page offers status and Down only, and says why. The appliance's ssh
identity is GOBANK_DEPLOY_SSH_KEY (base64 of the private key in PEM,
from the gokrazy secrets note); its public half is registered in the
Hetzner project so new servers accept it, and was added to servers that
predate it by hand. Pinned host keys live in /perm/gobank-deploy/known_hosts.
The upgrade drill
Drill to vX on an environment's row runs gobank's
upgrade drill
(ADR-0003) as a workflow: a day long enough to upgrade inside (set to 2h
when under 30m, then wait for a day with 10m left), observe the demo,
upgrade to the newest release, roll back, forward again, observing after
each hop. The demo is read at its /about.json; each hop is gated on the
version serving, a restart row that follows the expected release with a
known downtime and an intact handover, and the run no further on than a
restart takes it (the stop finishes the day in progress, the start begins
a new one: one day at each). The day length is set again after every hop,
since a release before gobank v0.10.3 forgets it on restart. It is offered
where the manual drill runs: a store to roll back
from and a newer release than the one serving. A failed hop fails the run
and says why; pressing Drill again the same day resumes from that hop.
The Drills page is the history, one line per drill ready for the
story's record; each run's steps are at /workflows/instances/<id>, the tables at
/internal/explorer.
The Workflows pages are gobank-workflow's own (its view package) in
this page's layout. Instances are a work queue, those in hand first,
filtered by workflow, status, date and key. Definitions each have their
diagram, their steps with the code behind each, and their instances. An
instance page lays the steps as they ran over the definition, so a step not
reached shows as not run. Under it come what only this program records:
a drill's observations, a scan's routes, a load test's climb, a perf run's
figures. Each instance is also offered as d2 with the steps coloured by state
(/workflows/instances/<id>/diagram.d2) for d2 diagram.d2 out.svg.
The page keeps its workflow runs and drills in a pglike (SQLite file)
database, -db, default <-build>/gobank-deploy.db: on hydrogen that is
/perm/gobank-deploy/gobank-deploy.db, beside the release store, so no
flag is needed there. gobank-workflow's tables and this program's drill
tables share the file, each component's schema versioned in its own
migrations table (schema_migrations, deploy_schema_migrations,
perf_schema_migrations).
Performance runs: the perf workflow
Measure on the environments page (one scale) or tp secrets gobank-deploy perf <env> (every scale at once: -scales small,large by
default, one server each, named <env>-<scale>, the rows written into
gobank's benchmark.md in the -src checkout — -doc to say where, or
"" to only print them — for you to commit and release) makes gobank's
performance run as a
workflow: create (up -create at the scale with an expiry of twice the
ceiling and an hour, so a run that hangs is reaped by the page's tick; a
server that exists is kept), add (the demo set flat out with
room for a million customers and a payment a customer a day, up to
10,000 a day, then batches of 1,000 customers until the
books hold 10,000, the rate taken from the demo batch by batch so this
program's polling is not in it), days (Run until the bank's day is seven
years on or six hours have passed, requesting the staff pages — dashboard,
P&L, balance sheet, BBSI, explorer, customers, payments, treasury — in
rotation, one a poll, so their counts and p95s at scale land in the
record; read account days per 12h and the last day it is made of, Stop),
remove (down). The two rates
are gobank v0.12's /about.json sim block. Each run is kept in the -db
database with where it ran — scale, server type, RAM, gobank version — and
the Performance page is the history, each run with its row for
gobank's benchmark.md. A run creates a billable server for about half an
hour, so it is started on purpose, never by a schedule.
A short run has the same shape in tens of minutes: -customers 100 -days 10 -ceiling 1h adds 100 customers, runs ten days instead of seven
years, and gives the servers a three-hour expiry; pass -doc "" so its
rows stay out of benchmark.md.
Scenarios and the journal: the scenario workflow
tp secrets gobank-deploy scenario seven-years runs a scenario — a named
configuration driven through timed steps on a fresh environment named
<scenario>-<scale> — as a workflow: create (up -create at the scale; a
server that exists is kept), add (the demo set flat out with room for
the customers asked for, -customers 1000, added in batches: added, on
the books, customers/s), years (Run flat out until the bank's day is
-years 7 on from where it started, give or take a poll: bank days run
from and to, bank days/h, account days/h, the last day), wallclock (the day length set to
-day-length 24h and the bank waited for until it follows the clock),
probe (the dashboard, the customer search by name -search Jones — the
name posted in the form, never in a URL, as gobank's search takes it — and
the first customer it lists, fetched from outside as the operator at
concurrency 1 and 4, twenty times each: p50 and p95; then Stop), remove
(down, unless -keep). A perf or scenario run that fails after create
removes its box, since a box bills while a failed run sits: create is
undone, and the next attempt the same day starts from create rather than
resuming; with -keep the box is left, failed or not. The run is kept in the -db database and written
as a page of the journal, the notebook in this repository's journal/
folder: journal/<date>-<scenario>-<scale>.md with the configuration,
steps and probes tables and an empty Notes section for your hand, and its
line at the top of journal/index.md (-journal DIR to say where, ""
to only print the page). A page is written once and never touched again;
hand-written pages go in the folder and on the index too. The ui's
Journal page under About shows the same for runs driven from a host
with no checkout. The first scenario, seven-years, is the instrument for
gobank's milestone 1 (gobank #151): the capacity of one synchronous system
over seven years. At 1,000 customers a run is an afternoon of billable
server, so it is started on purpose, never by a schedule.
Two clocks run through a page, and every figure says which it is on, so the figures multiply out: customers added is customers/s times the wall clock, account days/h is bank days/h times accounts.
| Term | Clock | Meaning |
|---|---|---|
| wall clock | wall | how long a step took on the host's clock |
| bank day | bank | one business date the bank processes |
| day length | wall | the setting: how long the bank spends on one bank day |
| bank days, from, to | bank | the days run in the step and its first and last date |
| last day | wall | how long the last bank day took |
| customers/s, bank days/h, account days/h | bank per wall | what the step did, over its own wall clock, polling and all |
The perf run's rates are a different instrument: the demo's own figures
(/about.json sim), with this program's polling kept out of them.
The full performance scan: the scan workflow
tp secrets gobank-deploy scan <env> scans an environment as it stands,
no server made or removed: survey (the environment must be serving;
where the scan is, and the customers on the books), scan (every route
the demo's /v1/openapi.json lists that the scan can drive — a GET
without path parameters as the operator on the API token, or as a
customer signed in on the app password by what the route's security says,
and the customer search by its post, -search Jones — loaded at each
rung of -ladder 1,4,16, -samples 20 calls each: p50, p95 and the
throughput; a route with a path parameter, a post other than the search,
or one that fails is listed as not scanned with why, so the coverage is
visible). The scan is a run of the journal like a scenario, and it writes
the measured document, -doc (default <-src>/api/measured.json in
gobank's checkout): the date, gobank version, environment, server and
customers, a row per route and rung, and the routes not scanned. gobank's
About → API pages show it alongside each route's facts, so the API is
documented with what it measured; commit the document and release gobank
to publish it. A scan at 16 in flight is load: run it on preprod or a
scenario's kept server, not prod.
The page reads the scans under Performance: Scans (/perf/scans)
lists them and reads the latest against the one before — the critical
routes (failed, not scanned where the previous scan drove them, or p95 at
the top rung over a 500 ms budget), the ten worst by p95 at the top rung
with the move since the previous scan (better or worse beyond 10%), and
every route across the scans, a column per scan — and each scan has its
page (/perf/scans/<run>) with every route at each rung, worst route
first, the routes not scanned with why. The run page and the journal
link to it.
The load test: the load workflow
tp secrets gobank-deploy load <env> -scale large load tests a fresh
environment: create (the target at the scale, with an expiry), add
(the demo stopped, its books filled to -customers, by default the top
rung's users — the journey needs a customer to log in as, not full books), loader (a
box <env>-loader in the target's location on the private network, ssh from -admit only, this
program's own executable copied onto it — so it is arm64 from hydrogen and
amd64 from the laptop), load (from the loader against the target's
private address: the customer journey — login, the accounts screen, one
account's screen — walked by each rung of -ladder 1,2,5,10,20,50,100,200,500
users -reps 10 times each, every walk the next customer, until a
screen's p95 is over -budget 2s or a journey fails; a first rung with
no journey completed fails the test rather than recording capacity 0),
remove (both boxes, also when a stage fails). The capacity is the last rung that held. The page has a Load test
form beside Performance run; Load tests under Performance
(/perf/load) charts the capacity against gobank's version, a line per
scale, and a test's run page has every rung.
gobank-deploy journey -base URL is the load test alone, from wherever
it runs, against any BFF it can reach: the app password from
GOBANK_APP_PASSWORD, the result as JSON on stdout, a line a rung on
stderr. It needs no token; the loader runs it.
Firewalls and the private network
Every server is on the private network gobank (10.128.0.0/16, created on
first use) and has a firewall of its own name, set on every up: ssh
(22) and the demo (1347) from the -admit ranges, ping from anywhere.
-admit (or GOBANK_DEPLOY_ADMIT) is home's range, comma separated
CIDRs; empty admits anywhere, as before. Set it on the laptop and in
hydrogen's gokrazy flags alike, or one of them will lock the other out of
the boxes it made. Traffic on the private network is not filtered. The
decision table is on the docs page.
Temporary environments: the demo workflow
"Remove after" on either Create form runs the demo workflow instead of
a plain create: create (up with the removal time as an expires label on
the server), serve (until then, or until someone presses Down), remove
(down). It is a gobank-workflow
pipeline: one keyed instance (<env>@<expiry>) whose stages are recorded,
so a failed stage is resumed by the next run and a finished one is final.
The run records are in the -db database; what must survive is on the
server itself, and every minute the page starts the workflow for any temporary
server nobody is looking after, so a demo still goes after the page
restarts. demo-workflow.d2, drill-workflow.d2 and perf-workflow.d2
(from go run ./cmd/flowd2) draw the three workflows.
Each environment gets the hostname <env>.gobank.drummonds.net: an A
record in Route 53 (TTL 60s, since Hetzner reuses addresses) set on up
and removed on down. -dns DOMAIN changes the domain, -dns "" turns
it off. It needs AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (tp secrets
on the laptop, the gokrazy Environment on the appliance, as gokcaddy);
without them DNS is skipped and the start-up says so. The hosted zone is
the one whose name is the domain's longest suffix, so drummonds.net. The
key's IAM policy must allow route53:ChangeResourceRecordSets on that zone
for A records under the domain, UPSERT and DELETE (on top of
ListHostedZones, ListHostedZonesByName, ListResourceRecordSets and
GetChange); a key scoped to another use, such as the LAN Caddy's
_acme-challenge TXT records, is refused with AccessDenied and the deploy
carries on by address.
Scale presets: small (cx23), medium (cx33), large (ccx33: 8
dedicated vCPU, 32 GB, so a measurement sees cores that are really there),
xl (ccx43: 16 dedicated vCPU, 64 GB), or any hcloud server-type list
name. cax* types build for arm64. When Hetzner has no capacity for the
type ("error during placement"), create tries fsn1, nbg1 and hel1 in turn,
then the next size down the ladder (ccx43, ccx33, cx53, cx43, cx33, cx23)
at every location again; the server's actual type is what status and a
perf run report. The ladder is a first cut, to be revised once there is
performance data.
up sizes the demo's memory to the box: GOBANK_MEMORY_LIMIT is set to
half the server type's RAM (PostgreSQL shares the box) in
/etc/gobank/deploy.env, which the systemd unit reads, so a redeploy can
change it. The demo's own default, 800MB, is what a browser tab holds.
up builds cmd/demo from the gobank checkout given by -src (default
../gobank) with CGO_ENABLED=0, so that checkout's local replace
directives still apply. Host keys are pinned per project in
build/known_hosts, accept-new style; a freshly created server's IP is
forgotten first because Hetzner reuses addresses.
Requirements
- An SSH key registered in the Hetzner project and loaded in your agent
(or at
~/.ssh/id_ed25519/id_rsa). tp unlocksession for the Bitwarden-heldHCLOUD_TOKEN.
Layout
internal/deploy— the up / down / status sequences againstCloud,Dialer,BuilderandProberinterfaces; tested with fakes.internal/hetzner—Cloudon hcloud-go.internal/remote— sshDialerwith host-key pinning, GoBuilder, HTTPProber, and the demo'sConsole(its/about.jsonand settings).internal/ui— the lofigui page: states, create / redeploy / down / cancel, one job per environment with its log; the Drills, Performance and Workflows pages and an About page with the component diagrams; tested against a fakeOperator.internal/flows— the workflows on gobank-workflow's pipeline runner: the temporary demo environment, and gobank's upgrade drill.internal/drills— the drill record: each drill with what the demo looked like before and after every hop, in the shared pglike database.internal/route53,internal/store—DNSon Route 53; the release store.cmd/gobank-deploy— the command.
Links
- Documentation: https://gobank-deploy.docs.bytestone.uk/ (
tp pagesfor a local preview) - Source: https://git.bytestone.uk/hum3/gobank-deploy
- Mirror: https://github.com/drummonds/gobank-deploy