gpumon-ingress — LLM proxy

Why this exists / when to use what

gpumon-ingress is the mandatory entry point for every LLM call made by any agent, script, or service in the federation. It speaks the OpenAI chat-completions wire format on the front, picks a pool from pools.local.yaml / pools2.yaml, routes to NIM first then to LAN GPU nodes on 429 / circuit-breaker open, and ledgers every call via gpumon-writer. Callers never talk to LiteLLM, NIM, or vLLM directly — they talk to ingress, send the x-gpumon-* headers, and ingress does the rest (key rotation, RPM budgeting, cooldown bookkeeping, attribution).

Use this surface when you want to make an LLM call. If you already made one (e.g. you called Anthropic directly) and just want to ledger the outcome, talk to the writer instead — see writer.md.

Endpoints

Addresses

The same image runs in three places (swarm on node-eleven, mepstudio docker-compose, mepstudio host-mode). Pick the address closest to the caller. Auth is none on LAN — egress is firewalled at the UDM.

Required headers

Every request MUST include these. Missing headers degrade attribution to unknown and your project will not appear in the dashboard.

Header names are legacy in a couple of places (ingress forwards them upstream as x-gpumon-service and packs workload_id into x-gpumon-doc-id) because LiteLLM's allow-list is strict. Callers should keep using the names in the table above — ingress handles the renaming.

Pools

The pool name is a logical routing key. Ingress maps it to one or more LiteLLM model IDs (or NIM passthrough sidecars) and picks one per request using a weighted-random policy with per-deployment cooldowns on 429 / auth failures.

Live source of truth: services/ingress/pools.local.yaml (and v2 schema pools2.yaml). Re-verify by hitting GET /v1/models on ingress.

Canonical pools

All of the above are aliases of pool-load-balance with a smaller cap — no source changes needed for legacy callers.

Vision / OCR pools (NIM passthrough, cannot be aliased)

These bypass LiteLLM's chat-completions adapter because LiteLLM's nvidia_nim provider strips image_url content from vision messages.

Fraud-specific pools

Discovery

ssh rooot@node-eleven 'curl -s http://localhost:4000/v1/models | jq -r .data[].id'

Or via ingress directly:

curl http://192.168.1.211:4010/v1/models | jq -r '.data[].id'

Concurrency & 429 semantics

• NIM sustained per-key cap is ~5-6 RPM (not the documented 40). Ingress has 14 keys = ~70 RPM aggregate for vision.
• Per-backend circuit breaker OPENs after ~5 fails in 30s; ingress then routes to the LAN fallback tier (spark-2:11234 + nvidia-2:8080 + nvidia-4:8080).
• Cooldowns are tracked per (model, api_key_alias) and persisted to the writer (/cooldown/:key). Callers do not see 429s under normal load; they see successful responses from the fallback tier.
• Honor Retry-After on the rare 429 that escapes (fallback tier also exhausted). Implement exponential backoff with jitter.

Bun fetch example (20 lines)

const r = await fetch("http://192.168.1.211:4010/v1/chat/completions", {
  method: "POST",
  headers: {
    "content-type": "application/json",
    "x-gpumon-client-id":      "fraud-enricher",
    "x-gpumon-project":        "fraud-heuristics",
    "x-gpumon-pool":           "pool-fraud-triples",
    "x-gpumon-workflow-stage": "extract-triples",
    "x-gpumon-doc-id":         "row-12345",
    "x-gpumon-session-id":     crypto.randomUUID(),
  },
  body: JSON.stringify({
    model: "pool-fraud-triples",
    messages: [{ role: "user", content: "Extract triples from: ..." }],
    max_tokens: 2048,
    stream: false,
  }),
});
if (!r.ok) throw new Error(`ingress ${r.status}: ${await r.text()}`);
const out = await r.json();
console.log(out.choices[0].message.content);

curl example

curl http://192.168.1.211:4010/v1/chat/completions \
  -H 'content-type: application/json' \
  -H 'x-gpumon-client-id: edgar-summarizer' \
  -H 'x-gpumon-project: edgar-cik-cli' \
  -H 'x-gpumon-pool: pool-summary-medium' \
  -H 'x-gpumon-workflow-stage: 10k-summary' \
  -H 'x-gpumon-doc-id: 0000320193-2024-10K' \
  -d '{
    "model": "pool-summary-medium",
    "messages": [{"role":"user","content":"Summarize this 10-K in one paragraph: ..."}],
    "max_tokens": 256
  }'

Cross-references

• Header bookkeeping is closed by the writer — see writer.md.
• Long-running workloads should register first — see writer.md.
• For batch workloads (1000+ calls), queue the unit of work, not the LLM call — see broker.md and client-pattern.md.