$ Cost economics ⚙ How it works 🧰 Skills & tech
01 Overview
This isn't a single app — it's a small MLOps layer for on-demand cloud GPU training: rent a GPU only for the minutes a job needs, then throw it away. The local machines couldn't train the models I needed (one is a Pascal-era GTX 1080 Ti desktop; the other, a laptop with an RTX 3080, falls back to CPU on larger weights), so the pattern became: rent the cheapest suitable GPU on Vast.ai or RunPod, bootstrap it, train, pull the weights over scp, and destroy the pod — usually inside 15–105 minutes and for a few cents to a couple of dollars.
It's the machinery behind roughly 20 models across the dental-CV stack — the FDI tooth detector (F1 0.9608), the pathology heads, the SAM fine-tunes — plus a few local LLM fine-tunes. Those models and how they were evaluated live on their own pages; this one is only about the pipeline that trained them.
Vast.ai
The dental-CV workhorse. Offer-search DSL sorted by price, --ssh pods, an RTX 4090 running about $0.24–$0.34/hr in practice; H100/H200 for the heavier retrains.
RunPod
Driven three ways — GraphQL, REST v1, and runpodctl. The secondary backend, where the LLM fine-tunes ran; an H100 lands near $2.79/hr, versus roughly $3.87/hr on Vast.
02 Why I built it
Two forces. First, cost discipline — the whole first run was framed as "$0.30–$1.00, thirty to sixty minutes from launching a pod to holding weights," with the standing warning that a forgotten pod overnight is about $10. Second, iteration speed: the FDI accuracy campaign ran seven training experiments in eight days, each a fresh full retrain. Because each pod cost only pennies and got thrown away afterward, no single run felt too precious to attempt — I could run two versions of a model side by side, or fully retrain on a powerful H100 or H200, as casually as any other experiment. The unattended runs also had to be reliable — the first RunPod attempt burned through six failed pods before one worked — so watchdogs, retries, and self-removing monitors grew out of necessity.
03 What I built & how it works
One control node, one rented GPU, and a loop that always ends with the pod destroyed.
Fig. 1 — one loop: the control node rents the cheapest suitable GPU, ships the dataset, trains, and pulls the weights back — then destroys the pod. Typically 15–105 minutes for cents to a couple of dollars; the weights are the only thing that survives.
- Search —
vastai search offerssorted by $/hr with reliability, bandwidth, and disk filters; pick the cheapest suitable GPU. - Provision & verify — create the instance with
--ssh, wait for SSH, then run a CUDA smoke-test and terminate immediately if the GPU is bad. - Bootstrap — paste one
pod_setupscript: apt the cv2 libraries, extract the tarball, patch thedata.yamlpath, pinnumpy<2, pre-download base weights, launch training detached in tmux. - Train — the detached run tees to a log; I poll it from the home server, or a self-removing cron monitor watches for the checkpoint hands-off.
- Retrieve —
scpthe best weights back to the home server; every candidate is archived off-pod. - Destroy — tear the pod down at once, confirm the account shows zero running instances, and record cost + adopt/skip verdict in the ledger.
04 🧰 Skills & tech used
No frontend — this project is headless by design; its outputs are weight files consumed by other systems.
05 Notable challenges & decisions
Most of the engineering was making a rented, unfamiliar machine behave the same way twice.
The pod has to die the moment the weights land
Billing is by the hour while the pod runs, so a forgotten pod overnight is about $10 — more than a whole campaign. "Destroy immediately after downloading weights" became a standing invariant, verified after every run by confirming the account shows zero running instances. Cheap only stays cheap if teardown is non-negotiable.
Eight hard-won fixes, baked into a 22-line script
A fresh pytorch container fails in the same handful of ways every time: numpy<2 vs the image's torch, missing cv2 shared libraries, Roboflow's absolute data.yaml path, a forgotten --ssh flag, a tmux server that dies after one-shot SSH. Each failure got written back into the bootstrap template through a "runbook errata" loop, so eight fixes now live in a script short enough to paste.
Verify the GPU before trusting it
Some rented machines expose the GPU at a non-zero device node and fail cuInit no matter what; a community-cloud pod was evicted twenty minutes into a run. The response was a mandatory CUDA smoke-test — a tiny matmul on .cuda() — that terminates a bad pod on first connect, plus a SECURE-only rule for anything over an hour.
Small-file transfer was the real bottleneck
Uploading 1,805 JPEGs with scp -r took 30+ minutes — longer than the training. A two-tarball strategy (gzip for code, store-only tar for the already-compressed images) cut it to about 2 minutes. The lesson: on rented pods, moving data is often the cost, not the compute.
One thing I'd change. Several of the early RunPod scripts carry their API key hardcoded in plaintext — convenient at the time, and in tension with the key-custody policy I hold Vast.ai to (key stays on the home server, never mirrored). It's the honest weak spot here: those keys should be rotated and moved to environment variables.
06 Results
The point of the whole thing is the ledger. Every run recorded its GPU, wallclock, and dollar cost.
Choosing the provider per job matters: an H100 runs about $2.79/hr on RunPod versus roughly $3.87/hr on Vast.ai, so the campaign's heavier retrains went wherever they were cheapest that day.
| Run | GPU | Wallclock | Cost |
|---|---|---|---|
| Arm A — yolo11m | RTX 4090 | ~58 min | $0.371 |
| Arm B — imgsz 832 | RTX 4090 | ~93 min | $0.416 |
Wasted pod — missing --ssh | RTX 4090 | ~5 min | $0.042 |
| Cycle 1 retrain | H100 SXM | ~28 min | $2.23 |
| Cycle 2 retrain · adopted → prod | H200 | ~28 min | $1.94 |
| Cycle 3 retrain | H200 NVL | ~30 min | $1.94 |
| Cycle 4 retrain | H200 | ~30 min | $1.86 |
| Campaign total | ~12 pods | $9.13 |
The models this spend paid for are covered on their own pages — FDI detector F1 0.9608, the pathology heads, the SAM fine-tunes. Across the stack — restoration and caries retrains, SAM fine-tunes, the BW/PA classifier — individual runs landed in the $0.05–$2.25 range.
07 Screenshots
Headless work photographs as terminals and tables — no live UI to show.
Cycle 2 cost block — the $1.94 H200 run that became production
~22 lines encoding 8 fixes
create → cuda-verify → train → scp → terminate
finally-guaranteed teardown even on Ctrl-C.08 Honest status
It was built and used heavily from March to May 2026. The pipeline is dormant now, but ready to run. What it did well is the honest claim: it trained the models on the rest of this site, cheaply and repeatably.
mlops-gpu — Rented-GPU Training Loop — Build Recipe
The rent → bootstrap → train → retrieve → destroy workflow behind roughly twenty model training runs on Vast.ai and RunPod, typically at \$0.05–\$2.25 per run, with no local GPU. This is honest about what it is: a working practice plus a small script family — not an app. There is nothing to containerize, so this entry is recipe-only and deviates from the usual 3-tier container template; it is structured instead by what you can actually run.
Status: Assembled and de-identified 2026-08-02 from the original working scripts; not re-executed for this recipe. What was previously verified: Path A (the automated RunPod driver in
train/) ran end-to-end for real on 2026-07-31 as the bw-pa-classifier Tier 3b retrain — rented a SECURE RTX 3090, trained, exported, downloaded the artifacts, and self-terminated (~10–15 min, ≈ \$0.10, reproduced val-acc 1.0000). Path B (Vast.ai) and the monitor/watchdog patterns are verified-as-used from the March–May 2026 training campaign; today's copies were checked by inspection only.Sensitive data: the shipped files contain none. The originals on the author's server hardcoded RunPod API keys in several scripts (a weak spot already disclosed on the project page); every copy here reads keys from environment variables only (see
.env.example), and pod IDs, pod IPs/ports, and home-directory paths were replaced with placeholders.
What it is
The problem: the available local GPUs (a Pascal-era 1080 Ti, a laptop 3080 that falls back to CPU on 8 GB workloads) can't train modern models in reasonable time. The solution: rent the cheapest suitable GPU — typically for 15–105 minutes at a time — and make the rental loop cheap, repeatable, and hard to leave running by accident.
| File | Role |
|---|---|
train/runpod-train-bw-pa.py |
full-lifecycle RunPod REST driver: create pod → wait for SSH → CUDA-verify (abort+terminate on failure) → upload code+data → train → checkpoint-first download → export → terminate in a finally. Shared verbatim with the bw-pa-classifier recipe — its 2026-07-31 Tier 3b retrain is the verified execution of this workflow. |
train/runpod-resume-bw-pa.py |
recovery helper: salvage a still-running (still-billing) pod after an SSH drop instead of paying for a fresh one. Also shared with the bw-pa recipe. |
pod_setup.sh |
the most-iterated artifact: a one-page, one-shot pod bootstrap built around the eight errata (table below) learned across nine script generations — six baked into the script, the other two enforced by the surrounding workflow steps. Upload with your dataset, run once, training starts detached in tmux. |
train_fdi_yolo.py |
standalone YOLOv8 trainer — zero project imports, so it scp's to any pod and runs bare. Has --smoke (2-epoch sanity run — the cheap pod-validation step) and the fliplr=0 comment explaining why radiographs must never be h-flipped. |
pod/train_v15b5.py |
second, smaller trainer example (~60 lines): the clearest illustration of domain-aware augmentation invariants (fliplr=0, copy_paste=0, grayscale-only HSV jitter). |
monitor_pod.sh |
self-removing cron monitor: poll every 30 min → when the checkpoint appears and the process has exited, scp artifacts, write a report, API-terminate the pod, remove itself from crontab. |
runpod-watchdog.sh |
provisioning watchdog: if a pod never comes up within a timeout, terminate it and walk a 7-GPU fallback ladder until one provisions. From the GraphQL API era — shipped as a pattern (see Limitations). |
.env.example |
the only configuration: RUNPOD_API_KEY via environment, Vast.ai key via the CLI's own store. Placeholders only. |
Prerequisites
Everything here assumes none of the author's infrastructure — no home server, no pre-provisioned anything. You need:
- A RunPod account (Path A) and/or a Vast.ai account (Path B), each with a few dollars of credit. A typical run costs well under \$1; a careless forgotten pod costs ~\$10 overnight, which is why half of this recipe is about termination.
ssh,scp,taron your PATH, and an ed25519 keypair (ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519), with the public key registered in the provider's account settings.- Python 3 with
requests(Path A only). - A Linux/macOS shell (or WSL) for Path A: its driver packs the uploads into tarballs at
hardcoded
/tmp/…paths, so it won't run from native Windows. Path B is plainssh/scpand works from any shell with those tools. - No local GPU. That's the point.
Path A — fully automated RunPod run (the verified path)
One command rents, trains, retrieves, and cleans up. The two train/ scripts here are
byte-identical copies of the ones in the bw-pa-classifier recipe, whose
Tier 3b Option A is this exact workflow run against a real training job — follow that
recipe's §3b for the dataset assembly and the full walkthrough. Run the command from the
bw-pa-classifier clone root, not from this folder: the driver resolves its project root
as the folder above train/ and packs src/, requirements-train.txt, and an assembled
data/ from there — none of which exist here. Launched from this folder it would create
(and start billing) a pod, then fail at the packing step; the finally block does
terminate the pod, but you'd still pay for a doomed one.
# from the bw-pa-classifier clone root, with data/ assembled per its §3b:
pip install requests
export RUNPOD_API_KEY="<your-key>" # never hardcoded — see .env.example
python train/runpod-train-bw-pa.py # rents → trains → exports → downloads → terminates
Measured on the 2026-07-31 verification run: ~10–15 min end-to-end (training itself ~2 min;
provisioning + a 432 MB upload dominate), RunPod SECURE RTX 3090 @ \$0.50/hr ≈ \$0.10.
(The ~280MB in the script's own comments is stale next to that 432 MB measurement; the
file is left unchanged to stay byte-identical with the bw-pa-classifier copy.)
The pod terminates in a finally block, so even a mid-run exception can't leak a billing GPU —
but still confirm the dashboard shows zero pods afterward.
If SSH drops mid-run and the pod is still up, train/runpod-resume-bw-pa.py re-attaches:
paste the pod's id/host/port into its EDIT block and it finishes the job on the pod you're
already paying for.
To automate a different training job, the driver is the template: swap the upload set and
the two remote commands (train / export); the lifecycle scaffolding (SSH polling, CUDA gate,
checkpoint-first download, finally-terminate) is job-agnostic.
Path B — scripted-manual Vast.ai run
The FDI detector campaign (~12 pods, \$9.13 total) ran this way. Bring any YOLO-format dataset
(data.yaml + images/ + labels/, tarred). The loop, adapted from the campaign notes
(the search filter and image tag below are an example, not the campaign's exact commands):
# 0. one-time CLI setup
pip install vastai
vastai set api-key <your-vast-api-key>
# 1. find the cheapest suitable machine
vastai search offers 'gpu_name=RTX_3090 reliability>0.98 inet_down>200 disk_space>60' -o 'dph_total'
# 2. rent it — the --ssh flag is NOT optional (errata #4)
vastai create instance <OFFER_ID> --image pytorch/pytorch:2.2.1-cuda12.1-cudnn8-devel \
--disk 60 --ssh --direct
# 3. get the connection details
vastai show instances # note the ssh host/port
# 4. upload dataset + scripts, then bootstrap (edit pod_setup.sh's EDIT block first)
scp -P <port> my_dataset.tar.gz pod_setup.sh train_fdi_yolo.py root@<host>:/root/
ssh -p <port> root@<host> "bash /root/pod_setup.sh"
# 5. (recommended) validate the pod with a quick smoke run first (a few minutes,
# ~$0.02 at these rates — an estimate, not a ledger entry):
# set TRAIN_CMD in pod_setup.sh to include --smoke, confirm 2 epochs complete,
# then relaunch the real run. Cheaper than discovering a broken pod at epoch 40.
# 6. watch, then retrieve
ssh -p <port> root@<host> "tail -f /root/train.log" # detach any time; tmux holds it
ssh -p <port> root@<host> "find /root -name best.pt" # errata #6: path is nested
scp -P <port> root@<host>:/root/<runs-path>/weights/best.pt ./
# 7. DESTROY, then verify the money has stopped
vastai destroy instance <INSTANCE_ID>
vastai show instances # must print nothing
The eight errata behind pod_setup.sh
Each of these cost real money or a failed run to learn. Six are baked into the shipped
script; the other two live in the workflow itself — #4 is a flag on the vastai create
instance line above, and #8 is a planning rule, not code:
| # | Fix | Why (one line) |
|---|---|---|
| 1 | pip install "numpy<2" |
numpy 2.x ABI-clashes with the torch build in stock CUDA images — imports die |
| 2 | apt-get install libgl1 libglib2.0-0 … |
ultralytics imports cv2; slim images lack its shared libs |
| 3 | sed -i 's|^path:.*|…|' data.yaml |
the tarball's path: points at the build machine, not /root/ |
| 4 | --ssh on vastai create instance |
without it the instance comes up unreachable — one \$0.042 pod wasted proving this |
| 5 | tmux start-server before new-session |
on some images there is no running tmux server and new-session -d fails silently |
| 6 | know the nested weights path | best.pt lands under runs/detect/<name>/weights/ (or <project>/<name>/weights/) — find it before scp'ing blind |
| 7 | pre-download base weights before launching | a weights-download failure surfaces in the setup log, not silently inside the detached session |
| 8 | budget 2–3× the optimistic wallclock | provisioning, upload, and queue time dominate short trains |
Plus one gate that isn't in the table because it's absolute: CUDA smoke-test-or-die
(torch.randn matmul before anything else — it once caught a pod whose /dev/nvidia6 was
broken; the only fix was to terminate and re-request). And one transfer trick worth stealing: two-tarball
upload — gzip the code, but tar JPEGs with no compression (already-compressed images
gain nothing and gzip is slow); that took a 1,805-file upload from 30+ minutes to ~2.
Hands-off patterns — read and adapt, not turnkey
Two automation layers from the campaign, shipped as de-identified templates with EDIT blocks. These are patterns you edit per run — they were verified by daily use in March–May 2026, and by inspection (not re-execution) today:
monitor_pod.sh— for trains too long to babysit. A cron entry polls every 30 minutes; on completion it downloads the checkpoint and log, writes a report, API-terminates the pod, and deletes its own crontab line. The self-removal matters: a monitor that outlives its pod is how you get 3 a.m. SSH-failure noise.runpod-watchdog.sh— for the provisioning step itself. If a pod is stuck "waiting" past a timeout it terminates it (never leave a stuck pod billing) and retries down a 7-GPU fallback ladder, writing the winning pod's connection details to a shared state file for the next script in the chain.
Cost & hygiene — the invariants
The numbers below are the real campaign ledger, kept per-run at the time:
| Item | Measured |
|---|---|
| FDI campaign total (~12 pods, Mar–May 2026) | \$9.13 |
| Cycle 2 retrain (H200, ~28 min) → F1 0.9608, promoted to production | \$1.94 |
| Architecture-experiment arms A/B | \$0.371 / \$0.416 |
| Per-crop detector runs (v1.5b5 family) | ~\$0.10 / ~15 min each |
Pod created without --ssh, unusable |
\$0.042 |
| Verified bw-pa reproduction (2026-07-31) | ≈ \$0.10 |
And the number the whole practice guards against: a forgotten pod left up overnight is ≈ \$10 — that figure is the campaign runbook's standing warning, not a measured ledger entry.
Provider arbitrage is real and worth a 2-minute check per run: at the time, an H100 was \$2.79/hr on RunPod vs ~\$3.87/hr on Vast.ai. Prices drift — check both, every time.
The invariants, stated as rules:
- Destroy immediately after downloading weights; then verify —
vastai show instancesmust return nothing / the RunPod dashboard must show zero pods. - CUDA smoke-test before training. A pod that fails the matmul gets terminated, not debugged.
- SECURE cloud only for runs over ~1 hour — a community-cloud eviction once killed a run mid-train; interruptible pricing is only cheaper if you don't lose the work.
- Checkpoint-first download: scp
best.ptthe moment it exists, before export or anything else that could fail — the expensive artifact leaves the pod first. - Keep a per-run ledger (pod id, GPU, \$/hr, duration, outcome). The \$9.13 figure exists because every run was written down.
Honest limitations
- Nothing here was re-executed on 2026-08-02. Path A is verified by construction via the bw-pa retrain (2026-07-31); Path B and the monitor/watchdog reflect real use in March–May 2026 but were only re-read, not re-run, for this recipe.
- GPU prices and offer availability drift constantly. Every dollar figure above is a historical measurement, not a quote.
- Provider APIs change. RunPod already migrated GraphQL → REST during this project's
lifetime;
runpod-watchdog.shis from the GraphQL era and is shipped as an adaptable pattern, not a maintained tool. The REST driver (train/) is the current-era code. - The originals were never git-tracked — they lived as working scripts on a home server and laptop. This recipe is their first assembled, versioned form, which also means there is no commit history to audit; dates come from file mtimes and the campaign ledger.
- The originals hardcoded API keys in several server-side scripts (two distinct RunPod keys over the project's life). That is disclosed on the project page rather than hidden; the shipped copies take keys only from the environment.
- The workflow has been dormant since 2026-05-16 (aside from the 2026-07-31 verification run); a staged dataset tarball from the last campaign is still parked on the home server.
What was de-identified (originals untouched)
- Hardcoded RunPod API keys (in the watchdog and monitor originals) →
RUNPOD_API_KEYread from the environment at runtime, with a hard failure if unset;.env.exampledocuments it with a placeholder. - Live pod IDs, pod public IPs, and SSH ports in the monitor/watchdog →
<paste-…>placeholders in EDIT blocks (they were per-run ephemera anyway). - Home-server paths (
/home/<user>/…) and per-user crontab references →$HOME-relative paths and neutral wording. - Project-specific artifact names in the monitor (checkpoint/log filenames, the landmark-training report body) → EDIT-block variables and a generic completion report.
- The pod name in the watchdog's provisioning call (which embedded the home server's
hostname) →
watchdog-train. pod_setup.shis a merge of two of the originals (FDI Cycle 2 + v1.5b5c) with dataset and session names parameterized; the originals contained no secrets.train_fdi_yolo.py,pod/train_v15b5.py, and the twotrain/scripts required no changes (thetrain/pair was de-identified during the bw-pa-classifier recipe build and is copied from there unmodified).