gpumon — API + Routing Catalog

Every dashboard chart, badge, livestream row, and NIM-card rpm gauge is sourced from a single table — llm_requests in ParadeDB (Postgres) — populated by exactly one upstream pipeline:

Two write paths populate llm_requests, both via gpumon-writer:

1. ingress-direct — gpumon-ingress posts to /llm-calls on every chat/embed proxy call (sync, fire-and-forget). Carries x-gpumon-* headers verbatim → caller_service, project_id, workflow_stage, doc_id all attributed correctly.
2. litellm-callback — services/litellm-callback/gpumon_callback.py is loaded inside LiteLLM. Every call LiteLLM serves (whether through gpumon-ingress or not) fires the callback, which posts to /llm-calls. This catches anything that hits LiteLLM directly (legacy callers, the watchdog probe set, etc.) and tags it caller_service=unknown if the gpumon headers were not present.

Bypass paths to watch for — any caller that hits a vLLM / Ollama / NIM endpoint without going through LiteLLM and without going through gpumon-ingress will be invisible on the dashboard. As of 2026-05-12 the only deliberate bypass is fraud-watchdog's /health?model=… probes (LiteLLM-specific route, not OpenAI-compat — cannot be proxied by gpumon-ingress).

Per-pool routing recap:

• pool-ocr / pool-vision → nim-passthrough sidecar (spark-1:14000) → NVIDIA NIM hosted API. LiteLLM is skipped here because its nvidia_nim provider strips image_url content.
• /v1/embeddings → doc-skills MCP at 192.168.1.86:2280/mcp → ModernBERT on spark-1. Single-backend pool; only x-gpumon-client-id is required.
• All other pools (pool-fraud-*, pool-summary-*, pool-qwen-*, etc.) → LiteLLM → upstream GPU node (spark-1/spark-2 vLLM, nvidia-one/two/four vLLM or llama.cpp). Per-key RPM tracked by ingress; cooldown applied on 429.

Clients make LLM calls by enqueuing to the gpumon.dispatch AMQP queue via DispatchClient; the scheduler paces NIM/LAN admission, records every call (best_effort), applies ctx-fit, and returns the result.

1. Copy lib/dispatch-client.ts into your service (peer deps: amqplib + native fetch).
2. Set FEDERATION_AMQP_URL (required) and GPUMON_WRITER_BASE (required for poll mode, e.g. http://gpumon-writer:2289).
3. Construct a DispatchClient and call enqueueAndAwait(pool, body, attribution, {mode}).
4. Pick a pool from the Task-Specific Pools table (shows on dashboard + gets ctx-fit + failover).

import { DispatchClient } from "./lib/dispatch-client";

// FEDERATION_AMQP_URL env supplies the broker URL automatically.
// writerBase falls back to GPUMON_WRITER_BASE env (needed for poll mode only).
const dispatch = new DispatchClient({ defaultMode: "reply" });

const result = await dispatch.enqueueAndAwait(
  "pool-qwen-large",
  { messages: [{ role: "user", content: "Summarise this document." }], max_tokens: 512 },
  {
    project_id:     "my-service",
    caller_service: "my-service",
    workflow_stage: "summary-pass1",
    doc_id:         "doc:edgar:12345",
  },
  { mode: "reply", timeoutMs: 30_000 },
);
// result.status (200 = ok)  result.content (assistant text)
// result.upstream_model     result.api_key_alias    result.latency_ms

doc-skills/src/providers/gpumon-llm.ts — reply mode MCP provider.
fraud-heuristics/enrich-worker/src/index.ts — reply mode batch enrichment.

The dashboard Routing Control popup owns the tier cascade that gpumon-scheduler walks for every pooled call. Each pool resolves to an ordered list of tiers — fast (NIM + haiku/litellm) → steady (LAN GPU hosts) → failover. Within a tier, backends are weight-shuffled; a 2xx wins, a 4xx (non-429) is a terminal client error, and 429 / 5xx / network failover to the next backend, then the next tier. When every tier is exhausted the request is saturated (429). One helper — walkAndDispatch() in services/scheduler/src/cascade.ts — is the single source of truth: the production /v1/chat/completions path and the load-test engine call it, so measured numbers are exactly what production does. No drift.

Throughput / tuning drives sustained load through that same cascade to find the highest usable (success-bounded) RPM, then proposes the knobs that lock it in. Dry mode walks the gates and picks the winning backend but skips the upstream call — zero GPU / quota cost, validates routing + distribution only. Real mode issues actual completions. include_nim defaults off so a load test cannot burn the 14-key NVIDIA quota unless explicitly opted in. Auto-tune ramps the offered RPM (coarse ×1.5, then binary-search the breakpoint) until the success bound breaks — success_rate ≥ success_min and err_429_rate ≤ err_429_max — converges on the knee, and emits an Apply-able routing.json patch (chiefly nim.rpm_target = observed NIM-tier RPM at the knee). Every run reports lifecycle to the progress board as workflow gpumon:loadtest.

Scheduler endpoints (port 2293), surfaced to the dashboard popup through nginx as /api/routing/*. One run at a time — a second start returns 409 busy. Hard caps: duration_s ≤ 120, max_concurrency ≤ 64.

Three steps for any new caller: (1) register your project on the writer, (2) optionally register a long-running workload, (3) call /v1/chat/completions on the ingress with attribution headers.

Authorization: Bearer sk-litellm-master-cmem-2026
Content-Type:  application/json
x-gpumon-pool:      <pool-name OR same as model>
x-gpumon-client-id: <my-service-name>          # caller_service on dashboard
x-gpumon-project:   <my-registered-project-id>  # shown in projects card

const INGRESS = process.env.GPUMON_INGRESS_BASE ?? "http://192.168.1.211:4010";
const KEY     = process.env.LITELLM_KEY        ?? "sk-litellm-master-cmem-2026";

async function chat(pool: string, messages: any[], opts: {
  doc_id?: string; workflow?: string; workload?: string; max_tokens?: number;
} = {}) {
  const headers: Record<string,string> = {
    "content-type":       "application/json",
    "authorization":      `Bearer ${KEY}`,
    "x-gpumon-pool":      pool,
    "x-gpumon-client-id": "my-pipeline",
    "x-gpumon-project":   "my-pipeline",
  };
  if (opts.doc_id)   headers["x-gpumon-doc-id"]         = opts.doc_id;
  if (opts.workflow) headers["x-gpumon-workflow-stage"] = opts.workflow;
  if (opts.workload) headers["x-gpumon-workload-id"]    = opts.workload;
  const r = await fetch(`${INGRESS}/v1/chat/completions`, {
    method: "POST", headers,
    body: JSON.stringify({ model: pool, messages, max_tokens: opts.max_tokens ?? 1024, temperature: 0.1 }),
  });
  if (!r.ok) throw new Error(`gpumon ${r.status}: ${await r.text()}`);
  return (await r.json()).choices[0].message.content;
}

import os, requests
INGRESS = os.environ.get("GPUMON_INGRESS_BASE", "http://192.168.1.211:4010")
KEY     = os.environ.get("LITELLM_KEY",         "sk-litellm-master-cmem-2026")

def chat(pool, messages, *, doc_id=None, workflow=None, workload=None, max_tokens=1024):
    headers = {
        "content-type":       "application/json",
        "authorization":      f"Bearer {KEY}",
        "x-gpumon-pool":      pool,
        "x-gpumon-client-id": "my-pipeline",
        "x-gpumon-project":   "my-pipeline",
    }
    if doc_id:   headers["x-gpumon-doc-id"]         = str(doc_id)
    if workflow: headers["x-gpumon-workflow-stage"] = workflow
    if workload: headers["x-gpumon-workload-id"]    = workload
    r = requests.post(
        f"{INGRESS}/v1/chat/completions", headers=headers,
        json={"model": pool, "messages": messages, "max_tokens": max_tokens, "temperature": 0.1},
        timeout=120,
    )
    r.raise_for_status()
    return r.json()["choices"][0]["message"]["content"]

curl -s -X POST http://192.168.1.211:4010/v1/chat/completions \
  -H "authorization: Bearer sk-litellm-master-cmem-2026" \
  -H "content-type: application/json" \
  -H "x-gpumon-pool:      pool-qwen-large" \
  -H "x-gpumon-client-id: smoke-test" \
  -H "x-gpumon-project:   smoke-test" \
  -d '{"model":"pool-qwen-large","messages":[{"role":"user","content":"ok"}],"max_tokens":4}' | jq

from openai import OpenAI
client = OpenAI(
    base_url="http://192.168.1.211:4010/v1",
    api_key="sk-litellm-master-cmem-2026",
    default_headers={
        "x-gpumon-client-id": "my-pipeline",
        "x-gpumon-project":   "my-pipeline",
        "x-gpumon-pool":      "pool-qwen-large",
    },
)
client.chat.completions.create(
    model="pool-qwen-large",
    messages=[{"role":"user","content":"…"}],
    extra_headers={"x-gpumon-workflow-stage": "summary-pass1"},
)

Projects are the dashboard's top-level grouping. Register once per logical pipeline — the registration is idempotent. Workloads are optional sub-groupings for long-running batches (EDGAR sweeps, fraud-heuristics passes); use them when you want a single chart row that shows live progress across thousands of calls.

curl -s -X POST http://192.168.1.211:2289/api/projects/register \
  -H "content-type: application/json" \
  -d '{
        "project_id":"my-pipeline",
        "label":     "EDGAR ingest pipeline",
        "owner":     "M.P.",
        "color":     "#80deea"
      }'

Returns { ok: true, project_id, auth_token }. The auth_token is saved server-side and will gate write endpoints when token enforcement lands. Future calls send it as x-gpumon-project-token.

curl -s -X POST http://192.168.1.211:4010/v1/workloads \
  -H "authorization: Bearer sk-litellm-master-cmem-2026" \
  -H "content-type: application/json" \
  -H "x-gpumon-client-id: edgar-indexer" \
  -H "x-gpumon-project:   edgar-indexer" \
  -d '{
        "workload_id":     "wl_2026_05_13_edgar_sweep",
        "description":     "Sweep 8k 10-K filings, summarize + extract triples",
        "expected_calls":  24000,
        "expected_pools":  ["edgar-summarizer-pool","pool-fraud-triples"]
      }'

# When the batch finishes:
curl -s -X POST http://192.168.1.211:4010/v1/workloads/wl_2026_05_13_edgar_sweep/complete

Then send x-gpumon-workload-id: wl_2026_05_13_edgar_sweep on every chat call in that batch. The dashboard groups them under a single live row with progress, ttft histogram, and cost-to-date.

The ingress can sustain 5,000+ RPM combined across all NIM pools. Here's how to actually hit that from the caller side.

// Keep <=8 in flight; let the pool's cooldown logic drop hot keys for you.
const N = 8;
const queue = docs.slice();
const results: any[] = [];
await Promise.all(Array.from({length: N}, async () => {
  while (queue.length) {
    const doc = queue.shift()!;
    results.push(await chat("pool-qwen-large", [...], { doc_id: doc.id, workflow: "summary" }));
  }
}));

• Routing every input through one pool. Vision/OCR pools sit idle while pool-qwen-large piles up 80% timeouts. Use the right tool — see the table above.
• Skipping x-gpumon-pool. The ingress requires it on chat calls; 400 if missing.
• Calling LiteLLM (:4000) directly. Bypasses cooldown + attribution. Always go through gpumon-ingress.
• Vision through pool-qwen-large. LiteLLM's nvidia_nim provider strips image_url. Use pool-vision or pool-ocr (those hit the nim-passthrough sidecar).
• Tight retry loops on 408/429. Every retry burns a key's RPM. Back off exponentially; fall through to a sibling pool after 2 attempts.
• Hardcoding a direct host alias. Local nodes drop in and out. Pools auto-skip cooled members; direct aliases don't.

This is a recipe Claude follows to write a one-shot script — not a CLI you install. The script walks a folder, OCRs scanned PDFs through the ingress, summarizes each file, and writes a sidecar JSON next to every document. No queue, no SQLite, no embeddings. Re-running re-processes (or you can add a sidecar-exists short-circuit).

• <original>.gpumon.json next to each input file — contains { title, bluebook, sentence_sum, paragraph_sum, ocr_text, model_used, ms_taken }
• enrichment.md in the folder — a single markdown table indexing all sidecars (rebuilt every run)
• Nothing else. Original files untouched.

Save as enrich.ts, run with bun enrich.ts <folder> --project <id>. Claude can adapt this template — change pools, swap output format, add/remove enrichment stages — without changing the call shape.

#!/usr/bin/env bun
// enrich.ts — walk a folder, OCR + summarize each doc via gpumon-ingress.
// Usage: bun enrich.ts <folder> --project <id> [--workers 4]

import { readdir, stat, readFile, writeFile } from "node:fs/promises";
import { join, basename, extname }                  from "node:path";
import { randomUUID }                                 from "node:crypto";

const INGRESS = process.env.GPUMON_INGRESS_BASE ?? "http://192.168.1.211:4010";
const KEY     = process.env.LITELLM_KEY        ?? "sk-litellm-master-cmem-2026";
const SESSION = randomUUID();  // one session-id per run

const [folder, ...args] = process.argv.slice(2);
const project = args[args.indexOf("--project") + 1];
const workers = Number(args[args.indexOf("--workers") + 1]) || 4;
if (!folder || !project) { console.error("usage: bun enrich.ts <folder> --project <id>"); process.exit(1); }

async function chat(pool: string, messages: any[], stage: string, doc_id: string, max_tokens = 1024) {
  const r = await fetch(`${INGRESS}/v1/chat/completions`, {
    method: "POST",
    headers: {
      "content-type":       "application/json",
      "authorization":      `Bearer ${KEY}`,
      "x-gpumon-pool":      pool,
      "x-gpumon-client-id": "folder-enrich",
      "x-gpumon-project":   project,
      "x-gpumon-workflow-stage": stage,
      "x-gpumon-session-id":    SESSION,
      "x-gpumon-doc-id":       doc_id,
    },
    body: JSON.stringify({ model: pool, messages, max_tokens, temperature: 0.1 }),
  });
  if (!r.ok) throw new Error(`gpumon ${r.status} ${stage}: ${await r.text()}`);
  return (await r.json()).choices[0].message.content;
}

// crude PDF-vs-image detection. For real OCR you'd run pdftotext first;
// here we route all PDFs through pool-vision (NIM llama-3.2-90b-vision)
// which handles both text and scanned content.
async function ocrIfScanned(path: string, doc_id: string): Promise<string> {
  const ext = extname(path).toLowerCase();
  if (![".pdf", ".png", ".jpg", ".jpeg", ".tiff"].includes(ext)) {
    return await readFile(path, "utf8");  // plain text input
  }
  const buf = await readFile(path);
  const dataUrl = `data:application/${ext.slice(1)};base64,${buf.toString("base64")}`;
  return await chat("pool-vision", [
    { role: "user", content: [
      { type: "text",      text: "Extract all readable text from this document. Output text only, no commentary." },
      { type: "image_url", image_url: { url: dataUrl } },
    ]},
  ], "ocr", doc_id, 4096);
}

async function enrichOne(path: string) {
  const sidecar = `${path}.gpumon.json`;
  const doc_id = `doc:${basename(path)}`;
  const t0 = Date.now();

  // 1. OCR / text extraction
  const text = await ocrIfScanned(path, doc_id);
  const excerpt = text.slice(0, 12_000);  // keep prompts cheap

  // 2. Title + citation in one shot (small model is fine)
  const meta = JSON.parse(await chat("pool-qwen-14b", [
    { role: "system", content: "Return strict JSON. No prose." },
    { role: "user",   content: `Filename: ${basename(path)}\n\nFirst 12k chars:\n${excerpt}\n\nReturn JSON: { "title": string, "bluebook": string }\nIf the filename follows YYYY-MM-DD-citation.pdf, prefer that citation.` },
  ], "title-citation", doc_id, 300));

  // 3. Sentence summary
  const sentence_sum = await chat("pool-qwen-large", [
    { role: "user", content: `Summarize this document in one sentence:\n\n${excerpt}` },
  ], "sentence", doc_id, 80);

  // 4. Paragraph summary
  const paragraph_sum = await chat("pool-qwen-large", [
    { role: "user", content: `Summarize this document in one paragraph (max 5 sentences):\n\n${excerpt}` },
  ], "paragraph", doc_id, 280);

  const out = {
    path, title: meta.title, bluebook: meta.bluebook,
    sentence_sum: sentence_sum.trim(),
    paragraph_sum: paragraph_sum.trim(),
    ocr_text: text,
    enriched_at: new Date().toISOString(),
    ms_taken: Date.now() - t0,
  };
  await writeFile(sidecar, JSON.stringify(out, null, 2));
  console.log(`  ✓ ${basename(path)}  ${out.ms_taken}ms`);
}

// Walk + bounded-concurrency runner
async function walk(dir: string): Promise<string[]> {
  const ents = await readdir(dir, { withFileTypes: true });
  const out: string[] = [];
  for (const e of ents) {
    const p = join(dir, e.name);
    if (e.isDirectory())               out.push(...await walk(p));
    else if (!e.name.endsWith(".gpumon.json") && !e.name.endsWith(".md")) out.push(p);
  }
  return out;
}

const files = (await walk(folder)).filter(p => [".pdf",".png",".jpg",".jpeg",".tiff",".txt"].includes(extname(p).toLowerCase()));
console.log(`Found ${files.length} files; enriching with ${workers} concurrent workers (session=${SESSION.slice(0,8)})`);
const queue = files.slice();
await Promise.all(Array.from({ length: workers }, async () => {
  while (queue.length) {
    const f = queue.shift()!;
    try { await enrichOne(f); }
    catch (e) { console.error(`  ✗ ${basename(f)}: ${(e as Error).message}`); }
  }
}));

// Single index table for the folder
const rows: string[] = ["| File | Title | Citation | Summary |", "|---|---|---|---|"];
for (const f of files) {
  try {
    const j = JSON.parse(await readFile(`${f}.gpumon.json`, "utf8"));
    rows.push(`| ${basename(f)} | ${j.title} | ${j.bluebook} | ${j.sentence_sum} |`);
  } catch {}
}
await writeFile(join(folder, "enrichment.md"), rows.join("\n") + "\n");
console.log(`Done. Index: ${join(folder, "enrichment.md")}`);

# prerequisite: project registered once
curl -s -X POST http://192.168.1.211:2289/api/projects/register \
  -H "content-type: application/json" \
  -d '{"project_id":"case-docs","label":"Case docs","owner":"M.P."}'

# then run the enrichment
bun enrich.ts ~/Documents/case-files --project case-docs --workers 4

• Different pools per stage: swap pool-qwen-large for edgar-summarizer-pool if you want NIM-only with more headroom; swap pool-vision for pool-ocr if you want cheap nemotron-nano-vl instead of llama-vision.
• Skip-if-already-enriched: add a stat(sidecar) check at the top of enrichOne — if sidecar exists and is newer than the source file, return early.
• More stages: drop in additional chat() calls (entity extraction, classification, translation) and add the fields to the sidecar JSON.
• Different output: replace the sidecar JSON with a single CSV row per file, or a JSONL stream — the call shape is unchanged.
• Cost-aware fallback: wrap chat() in a try/catch that retries on anthropic / haiku-4-5 after 2 failures on the NIM pool.
• Resumability: read the existing sidecar JSON files at startup to skip already-completed work — no DB needed.

v1 of this recipe deliberately uses per-file sidecars: each .gpumon.json sits next to its source, makes the work fully visible in ls, survives rsync, and re-running is a no-op if you add the sidecar-exists check. A central SQLite (with FTS5, sqlite_vec, queue, leases) is a separate v1.5 design — see docs/gpumon-v1-design.md for that path. Most folder enrichment tasks don't need it.

The full operational reference lives in the gpumon Claude skill at ~/.claude/skills/gpumon/SKILL.md. Loading it gives Claude the trigger phrases, pool selection logic, header conventions, scaling patterns, and anti-patterns above — without having to read this whole page.

# mepstudio (this host) — already installed
ls ~/.claude/skills/gpumon/SKILL.md

# Federation hosts — rsync from mepstudio
rsync -a ~/.claude/skills/gpumon/ rooot@spark-1:~/.claude/skills/gpumon/
rsync -a ~/.claude/skills/gpumon/ rooot@spark-2:~/.claude/skills/gpumon/
rsync -a ~/.claude/skills/gpumon/ rooot@node-eleven:~/.claude/skills/gpumon/

• /gpumon
• "call an LLM" · "litellm" · "ingress" · "x-gpumon-*"
• "pool-vision" · "pool-qwen-*" · "edgar-summarizer-pool" · "register a project"

• How to pick a pool vs a direct alias for a given workload
• The full header contract (required + attribution + session)
• Three-step setup: register project → register workload → make calls
• Concurrency budgets per pool (matching the table above)
• Anti-patterns and how to recover from per-key cooldown / pool exhaustion
• How to add a new local endpoint or a new pool to the system