llm-finetune — Build Recipe

Fine-tune a small LLM to be a "cluster controller" for your own home server: a data flywheel turns the machine's own telemetry and docs into an instruction dataset, a rented cloud GPU does a QLoRA fine-tune of Llama-3.1-8B (served locally as a 4-bit GGUF in Ollama), and an optional cloud track runs LoRA SFT + DPO alignment on Together AI. The result is a local model that answers questions about that specific machine — its services, ports, storage layout, and failure playbooks.

Status: ⚠️ Assembled and de-identified 2026-08-02 from the original working scripts; not re-executed for this recipe. The scripts are the real ones that produced five model variants in March 2026 (numbers below are from those runs). What was re-verified recently: the rent → bootstrap → train → retrieve → terminate GPU-rental pattern these scripts depend on was re-run for real on 2026-07-31 (RunPod SECURE RTX 3090, ~$0.10, 10–15 min) as part of the bw-pa-classifier recipe's Tier-3b retrain — same provider, same pod lifecycle.

Recipe-only project — no runnable clone ships, deliberately. This deviates from the usual 3-tier container template because the two artifacts you'd expect in a clone can't be shipped: the fine-tuned weights encode the original server's real topology (including IPs) by design, and the dataset is machine-specific by definition. You reproduce the workflow on your own machine and accounts, with the real (de-identified) scripts shipped alongside.

Sensitive data: the originals hardcoded a RunPod API key in one script and a local gateway key in the benchmark; shipped copies take all keys from environment variables only (see .env.example). Server hostnames, LAN/ Tailscale IPs, usernames, account IDs, and personal mount paths were replaced with placeholders or neutral paths throughout.

What you can actually do

Three independent workflows. Each stands alone — do any one of them.

Zero-account path: Workflow A runs on any Linux machine with cron and python3 — no keys, no GPU, no cloud. It is the honest starting point, and its output is what B and C consume.

Shipped files

llm-finetune/
├── RECIPE.md                 ← this file
├── .env.example              ← key placeholders (env-vars only, never hardcoded)
├── collect-prep/             [A] build + sterilize + format the dataset
│   ├── td-collector.sh          cron collector: /proc metrics + docker stats → JSONL
│   ├── td-sterilize.py          redact → dedup → self-audit (second independent scan)
│   ├── td-synthesize.py         2,400-line generator: 16 categories of Q&A/scenarios
│   ├── td-format-for-cloud.py   → Llama-3 chat-template "text" format (for B)
│   └── td-format-for-together.py→ OpenAI messages + 5% eval split + token report (for C)
├── train-local-gpu/          [B] rented-GPU QLoRA → GGUF
│   ├── train.py                 Llama-3.1-8B QLoRA (Unsloth, r=16/α=32, 3 ep, seed 3407)
│   ├── train-3b.py              Llama-3.2-3B variant (r=32/α=64, 5 ep, packing)
│   └── runpod-full-pipeline.sh  one script: SSH-wait → deps → upload → train → GGUF export → download → terminate
├── together-dpo/             [C] cloud SFT + DPO chain
│   ├── together-upload.sh       upload train/eval files (SDK, not curl — see gotchas)
│   ├── together-finetune.sh     launch LoRA SFT (--dry-run mode; pricing notes in header)
│   ├── together-sft-optimized.py rerun with tuned hyperparameters (cosine, warmup, r=32)
│   ├── together-dpo-prep.py     build preference pairs: your answers = chosen; base-model @ temp 0.9 = rejected
│   ├── together-dpo-train.py    launch the DPO job (beta 0.1, 3 epochs)
│   ├── together-dpo-coordinator.sh  glue: wait for prep → upload → fire DPO
│   ├── together-live-monitor.py loss-plateau watcher; auto-continues +6-epoch chunks, 30-epoch cap
│   └── together-download-and-convert.sh  merged checkpoint → BF16 → Q4_K_M **entirely on CPU**
└── eval-serve/               serve + check
    ├── Modelfile                Ollama packaging (ctx 4096, temp 0.3, Llama-3 stop tokens)
    ├── test-controller.sh       5-test keyword verification harness (edit tests for your infra)
    └── together-benchmark.py    20-question keyword-scored eval, base vs SFT vs DPO

Not shipped (and why): the trained GGUF/safetensors weights (they memorize the original server's facts, including addresses — that is the whole point of the project, and exactly why they can't be published), the dataset (same reason), the original machine's database extractor (td-extract.sh — every line was specific to that box's apps; write your own against whatever SQLite/logs you have), and a parallel legacy normalizer that the flywheel superseded.

Workflow A — the dataset flywheel (any Linux box, $0)

Prerequisites: a Linux machine you administer, python3 (3.10+), optionally docker (for container stats) and nvidia-smi (for GPU metrics — both degrade gracefully if absent).

1. Collect. Run collect-prep/td-collector.sh once by hand, then via cron every 5 minutes if you want a real time-series:

*/5 * * * * $HOME/llm-finetune/collect-prep/td-collector.sh

It appends envelope-format JSONL (CPU, RAM, swap, load, disk, GPU, per-container stats) to ~/training-data/collected/, rotating files at 50 MB. If your output dir is on a removable mount, set DATA_MOUNT=/your/mount so an unmounted disk skips the run instead of quietly filling the root filesystem (that failure mode is why the guard exists).

1. Sterilize. Edit the REDACTIONS list at the top of collect-prep/td-sterilize.py — add the prefixes of your internal keys, your username, your hostnames (the shipped list has generic patterns plus clearly marked YOUR_... placeholders). Then run it. It redacts, dedups by record id, and finishes with a second, independent secret scan over its own output — keep that check list in sync with your redaction list.

2. Synthesize. This is the heart of the flywheel and the biggest editing job: open collect-prep/td-synthesize.py and replace the KNOWLEDGE BASE constants with your own infra facts (the header comment marks everything to change — container list, node specs, hardware/network/storage facts, and the hand-written Q&A answers). As shipped they are neutralized placeholders; train on them verbatim and your model will confidently describe a server that does not exist. The reusable part is the structure: 16 generator categories turn your telemetry

3. facts into a few thousand instruction/input/output pairs. The original run produced 5,884 raw pairs → 3,927 after dedup (3,534 train / 393 val).

4. Format. Two exporters, two targets:

5. td-format-for-cloud.py → single {"text": ...} JSONL in the Llama-3 chat template (what Workflow B's trainers eat). Edit its SYSTEM_PROMPT to describe your server.
6. td-format-for-together.py → OpenAI messages format with a 5% eval split and a token/cost report (what Workflow C uploads; original split: 3,886/204).

Workflow B — QLoRA on a rented GPU, served in Ollama

Prerequisites: a RunPod account (Vast.ai also works — the scripts are provider-thin: SSH + scp + one terminate call) with a few dollars of credit; an ed25519 keypair registered with the provider (the shipped script expects the private key at ~/.ssh/cloud_gpu — edit SSH_KEY at the top if yours lives elsewhere); ollama installed locally to serve the result. export RUNPOD_API_KEY=... first — the shipped script refuses to run without it. No local GPU needed at any step.

• Automated path: rent a pod (RTX 3090-class, ≥16 GB VRAM, CUDA 12.1 PyTorch image), then

bash ./train-local-gpu/runpod-full-pipeline.sh <pod_id> <pod_ip> <ssh_port>

One script does everything: wait-for-SSH, install Unsloth/TRL, upload your combined_formatted.jsonl, train the 8B (3 epochs — 1,473 steps, ~2–3 h at $0.22/hr on the original run), export to Q4_K_M GGUF on the pod, download, train + export the 3B variant, and terminate the pod on every exit path — an EXIT trap covers timeout, training death (3-strike process check), setup failures, and success alike. A forgotten pod bills all night; the guaranteed-terminate design is not optional politeness.

• Manual path: scp the data up, run train.py (8B) or train-3b.py (3B) yourself in tmux, scp the GGUF home, terminate the pod in the console and verify it's gone. If the GGUF export step at the end of train.py fails (see the transformers 5.x note below), use the export_v3.py approach embedded in runpod-full-pipeline.sh instead.

• Serve locally: put the GGUF next to eval-serve/Modelfile and

bash ollama create homelab-controller -f eval-serve/Modelfile ollama run homelab-controller "what's on this server?"

Then edit the five example tests in eval-serve/test-controller.sh to ask about your facts and run it.

• Original measured results: 8B → ~4.6 GB Q4_K_M GGUF (that size was measured on the Together-track CPU conversion of the same 8B). 3B → 180/180 steps in 38:26 on an RTX 4090 (finished on Vast.ai after a RunPod platform outage forced a provider swap mid-project), final loss 0.26, token accuracy 0.923, 1.9 GB GGUF — small enough to run CPU-only on a modest server.

• Why the pipeline exports the way it does: transformers 5.x broke PeftModel.merge_and_unload, so the embedded export_v3.py loads the LoRA through Unsloth natively, saves merged safetensors, and calls llama.cpp's converter + quantizer directly. That workaround is the pipeline's most transferable lesson.

Workflow C — Together AI SFT + DPO

Prerequisites: a Together AI account; pip install together; your key in ~/.env.together (see .env.example). The .sh helpers source that file themselves, but the Python steps read TOGETHER_API_KEY from the environment — run set -a; source ~/.env.together; set +a (or export TOGETHER_API_KEY=...) before them. No GPU anywhere — even the final quantization runs on CPU.

1. together-upload.sh — uploads train/eval files and records the file IDs. Gotcha: curl multipart against the files endpoint fails; the script uses the Python SDK for exactly that reason.
2. together-finetune.sh (start with --dry-run) — launches the LoRA SFT. Gotcha in the header: Qwen2.5-Coder-32B is not LoRA-supported on Together; the general Qwen2.5-32B-Instruct is. The original 32B run cost $7.11; the Together-trained 8B controller's v2 reached eval loss 0.2891 — about 23% lower than the local 8B track, at the price of living in someone else's cloud.
3. together-live-monitor.py <job_id> <train_file_id> <eval_file_id> — watches eval loss per epoch and auto-continues in +6-epoch chunks until the drop falls under 5% (30-epoch hard cap).
4. DPO: together-dpo-prep.py builds preference pairs — your dataset's answers as chosen, base Llama-3.1-8B-Turbo at temperature 0.9 as rejected (high temp makes the base model drift, which is the contrast you want). The original run kept 497 pairs; DPO then ran at beta 0.1 for 3 epochs (together-dpo-train.py, or together-dpo-coordinator.sh to chain it after prep). Gotchas (both tripped the original build): the shipped prep script emits simple {prompt, chosen, rejected} lines, which Together's validator rejects — convert them to the PREFERENCE_OPENAI structure in the current docs before uploading. And check which models DPO accepts: at the time it applied to base models only, not fine-tuned checkpoints like the SFT output together-dpo-train.py points at.
5. Optional: together-download-and-convert.sh — pulls the ~16 GB merged checkpoint and converts BF16 → Q4_K_M on CPU only (builds llama.cpp locally), so the cloud fine-tune can still end up in your local Ollama.

Evaluating honestly

eval-serve/together-benchmark.py is a keyword-scored recall test: ~20 questions about your server, each with a keyword the right answer must contain. Write your own questions (the shipped ones are placeholders showing the shape). On the original build the fine-tuned model + static-facts injection scored 100% vs 5% for base Llama-3.1-8B — a big gap, but read it for what it is: a narrow, self-authored recall test of facts the model was trained on, not a capability measure. A base model has no way to know a private server's port map; the test confirms the knowledge transferred, nothing more.

Known limitations (stated honestly)

• Nothing here was re-executed for this recipe (2026-08-02 assembly). Costs, step counts, and losses are from the March 2026 runs; cloud pricing and both providers' APIs drift. The GPU-rental pattern itself was re-verified 2026-07-31 (see Status).
• No weights, no dataset ship — both are irreducibly specific to the original machine. Your results depend on the dataset you build in Workflow A.
• The original system served the model through a multi-provider LLM gateway with fallback routing; that layer is deliberately stripped — the recipe serves via plain Ollama (or any OpenAI-compatible endpoint) to stay dependency-free.
• The original's live serving side has been dormant since March 2026; this recipe reproduces the build, not an in-production service.
• td-synthesize.py's value is its structure; the shipped knowledge base is neutralized and must be replaced — there is no shortcut around encoding your own facts.
• Cut from the original on purpose (machine-specific, not needed on a fresh box): a GPU-over-IP sidecar, a local-GPU fallback trainer, and the original's app-database extractors.