asset-harvester — Build Recipe

Take a bare machine to a running game-asset search & organization hub: one web app that searches 20 sources (Openverse, Wikimedia Commons, Internet Archive, NASA, Poly Haven, Kenney, Game-icons.net, Quaternius, OpenGameArt, ccMixter, itch.io, YouTube audio, and more) in one query, downloads assets through a persistent queue with live progress, deduplicates by SHA-256, auto-thumbnails everything (images, audio waveforms, video frames), and organizes results into per-project workspaces. Key-free by default — 13 sources need no API key, and a small demo library is imported on first boot.

Status: ✅ Verified 2026-08-02 — built from this clone on a Linux Docker host (docker compose up -d --build). First boot imported the bundled demo library (9 assets, 15 MB — images, SVG sprites, audio, a texture — with thumbnails and a "Demo Game" project); an all-source key-free search returned 137 results from 7 live sources with zero errors; 9 of 10 real downloads completed end-to-end through the queue (hash → dedup → organized storage → thumbnail; the one failure was a transient Wikimedia rate-limit); a restart keeps the data without re-seeding; a real-browser pass showed the gallery, sources, and project pages working with 0 console errors. All three tiers covered; the prebuilt tarball (docker save asset-harvester-clone) is ~885 MB. The clean clone lives in clone/.

Sensitive data: none. The source held no keys (all env-based, and the live app's stored key fields were empty); the demo seed was harvested from public CC0 / CC-BY / public-domain sources with license + author recorded on every row; personal server references were scrubbed. The 7 bug fixes below were also backported to the original live app.

What it is

A single Node/Express + Socket.IO container:

System deps baked into the image: ffmpeg (waveform/frame thumbnails), yt-dlp (YouTube audio source). No GPU, no build step for the frontend.

Prerequisites

New machine? See ../SETUP.md. Then, per tier:

Configuration — nothing required

Runs key-free: 13 of 20 sources work with no key, and a bundled demo library populates the gallery on first boot. Optional extras (each unlocks more, all documented in clone/.env.example): six per-source API keys, AI auto-tagging via any OpenAI-compatible endpoint, and SearXNG web-image search.

Tier 1 — Run the prebuilt container (recommended)

Availability: the prebuilt image is available on request — it is not published or linked anywhere. Ask Kevin for it, or build from source via Tier 2.

docker load -i asset-harvester-clone-image.tar
cd clone
docker compose up -d                        # → http://localhost:8214

Tier 2 — Build from source

cd clone
docker compose up -d --build                # → http://localhost:8214
curl -s localhost:8214/health               # → {"status":"ok", ...}

Open http://localhost:8214 — the gallery starts populated (demo seed), and a search like castle or nebula across "All sources" returns live results from the keyless sources.

Tier 3 — Bare-metal (one Node process)

Install the two prerequisites first (../SETUP.md covers Docker/Python but not these):

# Node 20+ and ffmpeg —
#   Windows: winget install OpenJS.NodeJS.LTS Gyan.FFmpeg
#   Debian/Ubuntu: sudo apt install -y nodejs npm ffmpeg   (Node 20+ via nodesource if the distro's is older)
#   macOS: brew install node ffmpeg
# optional, for the YouTube source: pip install yt-dlp

cd clone
npm ci --omit=dev
# ffmpeg must be on PATH (thumbnails); yt-dlp optional (YouTube source)
DATA_DIR=./data ASSETS_DIR=./assets THUMBNAILS_DIR=./thumbnails TEMP_DIR=./temp \
  SEED_DIR=./seed node server.js            # → http://localhost:8214

On Windows, set the same variables with set/$env: before node server.js.

Optional extras (all off by default)

• Per-source API keys — Pixabay, Pexels, Unsplash, Freesound, Sketchfab, itch.io. Free-tier signup links are in clone/.env.example; paste a key on the Sources page (or in .env) and toggle the source on.
• AI auto-tagging — point LITELLM_URL at any OpenAI-compatible endpoint and every downloaded asset gets an AI description + tags. Local Ollama works key-free (ollama pull llama3.2; a vision model improves image tagging):

bash # in clone/.env LITELLM_URL=http://host.docker.internal:11434/v1 LITELLM_API_KEY=ollama LITELLM_MODEL=llama3.2

• Web-image meta-search — set SEARXNG_URL to a SearXNG instance (JSON API enabled) and enable the "Web Images (SearXNG)" source.

Provenance — the demo seed

The bundled first-boot library was harvested through this app itself from its keyless sources (Wikimedia Commons, NASA, Openverse, Game-icons.net, Poly Haven, Kenney, and similar) — public-domain and Creative-Commons items only. Every seed asset's source, license, author, and origin URL are recorded in its database row and visible in the asset detail view. No model weights, no datasets.

What was stripped from the personal version (de-identification)

• Home-server User-Agent strings (they identified the author's server) → generic, non-identifying AssetHarvester/1.0 UAs throughout (the YouTube source keeps a stock browser UA for rate-limit reasons, as the original did).
• Private LLM-gateway defaults — the original defaulted to an internal LiteLLM endpoint and a private model alias for auto-tagging. Tagging is now opt-in (LITELLM_URL unset = off) against any OpenAI-compatible endpoint.
• Internal SearXNG dependency — the web-image source assumed a search container on the home network; now opt-in via SEARXNG_URL, off by default.
• A TLS-verification bypass for one source (ccMixter's occasionally incomplete cert chain) was removed — the clone verifies TLS normally and that source simply degrades to empty if the remote chain is broken.
• An orphaned dead-code module (searxng.js, superseded by webimages.js) was deleted; personal hardware references in comments were generalized.

Bugs found and fixed during cloning (faithful, minimal fixes)

1. Every downloaded asset got type other / format tmp — two compounding bugs, visible in the original's own database. The UI sends assetType but the server read asset_type, so the download modal's type selection was silently dropped; and format detection read the extension of the queue's <id>.tmp temp file (always "tmp") instead of the source metadata. Types, formats, thumbnails, and download filenames all work now.
2. 12 sources couldn't be configured — the DB seeded only 9 source_configs rows but 20 sources are registered; the settings API 404s without a row, so most sources could never be toggled or keyed through the UI. The seed now covers every registered source.
3. Two keyed sources had no key input — the UI's needs-a-key list omitted Pexels and Unsplash, so their keys couldn't be entered at all.
4. Saving settings could wipe a stored key — the server masks keys in API responses, but clicking Save sent the mask string back and overwrote the real key. The server now ignores the masked echo.
5. Poly Haven returned nothing under a type filter — its API reports asset types as numeric codes, which the adapter compared against strings; results were mislabeled and filtered out (and download URLs malformed). The adapter now tracks which list each entry came from.
6. Game-icons.net returned nothing — the upstream GitHub repo moved to a flat author/icon.svg layout; the adapter only matched the old author/originals/svg/ paths. It now accepts both.
7. yt-dlp pointed at a hardcoded Node path (/usr/local/bin/node) that only exists in the original image; it now uses the running Node binary (process.execPath), so bare-metal installs work too.

Known limitations (stated honestly)

Personal-LAN tool: no authentication — put it behind a reverse proxy if exposed. Downloads run one at a time by design (original ran on a small shared server). The scraping-based sources are best-effort: Incompetech's modern site is JS-rendered so that source usually returns nothing; Kenney falls back to a fixed pack list if its page layout changes; OpenGameArt sits behind intermittent anti-bot protection. Freesound downloads are the HQ MP3 previews (originals require OAuth2); Sketchfab is search/link-out only. YouTube search and download are subject to rate limits and yt-dlp aging — a rebuilt image months from now may need a yt-dlp bump. The container runs as root (as the original did); hardening is left to the deployer.