CodeRoute Model And Harness Plan

Updated July 2, 2026.

Decisions

• Treat gpt-5.5 as the premium OpenAI frontier target.
• Evaluate hosted full GLM-5.2 before relying on local Q3_K_M for hard coding work.
• Prefer OpenRouter z-ai/glm-5.2 first because it is available as an OpenAI-compatible hosted model with a 1M-token context window.
• Add named OpenRouter :free routes for development, smoke tests, and zero-token-cost fallback candidates. Do not use openrouter/free in normal ladders because it picks a free model at random.
• Check Azure/Microsoft Foundry for GLM-5.2 availability, but do not block on it. Current public catalog evidence shows Microsoft Foundry GPT-5.5 and Fireworks-on-Foundry GLM-5.1, not a clearly listed Azure/Foundry GLM-5.2 route.
• Keep local GLM-5.2 Q3_K_M as an explicit private/offline route only. On the homelab deployment this is exposed as coding-glm-local through LiteLLM model group studio-glm-5.2-q3-k-m, backed by LM Studio model id glm-5.2 on the Mac Studio. coding-auto should prefer the dedicated 4090 Qwen workhorse first and use hosted OpenRouter GLM-5.2 instead of local GLM.
• Add Mac Studio Ornith 1.0 35B as an explicit local/self-hosted agentic coding route. It is exposed as coding-ornith-local through LiteLLM model group studio-ornith-1.0-35b, backed by LM Studio model id ornith-1.0-35b.
• Add harness-aware routing metadata before trying to orchestrate harnesses automatically.

Target Routing

Live Validation Snapshot

Validated against https://coderoute.ai/v1 on July 2, 2026:

The Mac Studio LM Studio server was started with lms server start --bind 0.0.0.0, and pchomelab reached it at http://192.168.2.112:1234/v1. Local GLM-5.2 and Ornith require a realistic output budget because they emit reasoning tokens before visible content; direct smoke tests should use at least max_tokens: 256 for local GLM and max_tokens: 768 for local Ornith.

Adaptive aliases now retry the next allowed candidate after retryable provider failures such as network errors, 429, and 5xx. Fixed aliases stay fixed so an explicit coding-glm-openrouter, coding-glm-local, coding-gpt55, or named *-free request does not silently switch providers.

Live Eval Snapshot

A reproducible eval runner now lives at ../scripts/coderoute-eval-smoke.py. The July 2 direct API run is summarized in eval-results/2026-07-02-live-eval.md, with raw JSONL in eval-results/2026-07-02-direct-api.jsonl.

Summary: coding-gpt55 passed 3/3 compact tasks, hosted GLM-5.2 and local GLM-5.2 each passed smoke plus bugfix but missed the pytest naming task at max_tokens=512, and coding-gpt-oss-120b-free passed 2/3 with one upstream 502. OpenCode 1.17.13 and Codex CLI 0.142.5 both passed a no-edit bugfix prompt through CodeRoute using coding-glm-openrouter.

Azure/Foundry Snapshot

The July 2 Azure/Foundry follow-up is documented in azure-foundry-followup.md. Microsoft Foundry's public catalog lists gpt-5.5, but the local Mac has no az CLI available and the pchomelab CodeRoute container has no Azure endpoint/key/deployment env vars set. The Fireworks-on-Foundry catalog lists FW-GLM-5.1, not GLM-5.2. CodeRoute should not add Azure aliases until an actual deployment endpoint is available.

Model Work

1. Add LiteLLM alias openrouter-glm-5.2 for openrouter/z-ai/glm-5.2. 2. Add or update OpenAI alias cloud-openai-gpt55 for gpt-5.5. Done. 3. If Azure OpenAI deployment exists, add azure-gpt-5.5 and prefer it for enterprise/data-bound routes. Not currently wired: pchomelab has no Azure endpoint/key/deployment env vars set. 4. If Azure or Foundry exposes full GLM-5.2, add azure-glm-5.2; otherwise record Fireworks GLM routes separately and do not label them GLM-5.2. Current Fireworks catalog evidence shows FW-GLM-5.1, not GLM-5.2. 5. Add CodeRoute registry entries:

• openrouter-glm-5.2 / coding-glm-openrouter. Done.
• homelab-frontier / coding-glm-local on pchomelab. Done.
• homelab-agentic-ornith / coding-ornith-local on pchomelab. Done.
• cloud-openai-gpt55 / coding-gpt55. Done.
• azure-openai-gpt55 when deployed; not added until a real Azure deployment is available.

6. Rebalance step ladders so hosted GLM-5.2 handles most implementation and GPT-5.5 handles highest-risk work. Done. 7. Keep these named free OpenRouter routes available for explicit selection and low-cost ladder candidates:

• openrouter-qwen3-coder-free
• openrouter-north-mini-code-free
• openrouter-laguna-xs-free
• openrouter-laguna-m-free
• openrouter-nemotron-super-free
• openrouter-gpt-oss-120b-free
• openrouter-qwen-next-free
• openrouter-gemma-4-26b-free

Harness Work

1. Add client profile detection from metadata, API key labels, and User-Agent:

• opencode
• codex
• cline
• roo
• cursor
• aider
• continue
• claude-code

2. Store (harness, alias, selected_model, task_type, codegraph_signal, outcome) for route evaluation. First-pass client/API-key fields are implemented on request records. 3. Add harness smoke tests:

• OpenCode through coding-glm-openrouter. Done in production on July 2, 2026.
• Codex CLI through coding-glm-openrouter using /v1/responses and wire_api = "responses". Done in production on July 2, 2026.
• Aider through CodeRoute. Done with uvx --python 3.12 --from aider-chat aider 0.86.2 and openai/coding-glm-openrouter; Python 3.13 is not usable for Aider on this host because of audio dependency import errors.
• Cline/Roo manual smoke checklist. Done in the harness guide; GUI execution remains manual.

4. Add an Anthropic-compatible facade only after the OpenAI-compatible path is stable:

• /anthropic/v1/messages. Done.
• streaming compatibility. Done with live pass-through translation of the routed OpenAI SSE stream into Anthropic events (July 2, 2026); the synthesized-SSE path remains only as a fallback for non-streaming upstreams.
• tool use translation. Done for Anthropic tool_use/tool_result and OpenAI function tools.
• Claude Code smoke test. Done with Claude Code 2.1.196, ANTHROPIC_BASE_URL=https://coderoute.ai/anthropic, and --model sonnet.

User Documentation

• Publish /harnesses with setup steps for OpenCode, Codex CLI, Cline, Roo Code, Cursor, Aider, Continue, and Claude Code caveats.
• Link /harnesses from /, /meta, README, and the quick start guide.
• Keep /quick-start focused on the fastest path and /harnesses focused on client-specific setup.

Dashboard Plan

The dashboard should show users both spend and CodeRoute's value, not only a raw token meter.

The working checklist lives in dashboard-todo.md.

Phase 1: Ledger API

1. Finalize streaming requests with provider usage and provider-reported cost when available. 2. Preserve high-precision dollar amounts in request, session, and performance records. 3. Expose /v1/coderoute/usage summary buckets:

• actual provider cost
• completed fallback estimates
• pending max-token estimates
• observed-or-estimated total
• max exposure

4. Expose request/session drill-down with model, alias, step type, tokens, latency, cost accuracy, and status.

Phase 2: Dashboard MVP

1. Build an authenticated /dashboard or /usage page. 2. Add top-line cards for actual spend, fallback estimate, pending max exposure, requests, token volume, cache/usage metadata, and budget remaining. 3. Add charts by time, project, model, alias, step type, and cost accuracy. 4. Add a filterable request table with session drill-down and provider usage JSON. 5. Add CSV/JSON export for billing and debugging.

Phase 3: CodeRoute Value View

1. Show saved cost versus baseline routing and versus always-strong-model routing. Done for selected baseline, always-local, always-strong, and paid-provider comparisons. 2. Attribute savings to local routes, free OpenRouter routes, cache hits, and avoided escalations. 3. Track quality outcomes from feedback: failed test, fixed after retry, accepted, reverted, or manually escalated. Done for dashboard/API feedback summaries. 4. Show provider health, latency, rate-limit pressure, and free-model quota behavior. Done for the dashboard/API first pass. 5. Add budget alerts, anomaly detection, workspace/API-key breakdowns, and monthly reports. Done for dashboard anomaly detection, budget alerts, external alert webhooks, and the monthly report API.

Immediate Next Steps

1. Add and deploy the harness setup guide. 2. Add OpenRouter GLM-5.2 to LiteLLM and run direct smoke tests. Done. 3. Add GPT-5.5 to LiteLLM or update the existing OpenAI cloud alias. Done. 4. Smoke test the named free OpenRouter aliases through LiteLLM and CodeRoute. LiteLLM groups are configured; coding-qwen3-coder-free and coding-qwen-next-free reached CodeRoute but returned upstream 502 during July 2 validation, consistent with OpenRouter free-provider limits. 5. Add CodeRoute registry entries and route ladders for hosted GLM-5.2 and GPT-5.5. Done. 6. Deploy the cost ledger update and run an OpenCode streaming smoke test against coding-glm-openrouter. Done. 7. Run eval tasks comparing local GLM-5.2 Q3_K_M, hosted GLM-5.2, free OpenRouter candidates, and GPT-5.5 through OpenCode and Codex. Done for a compact July 2 eval suite plus OpenCode/Codex bugfix harness checks; broader benchmark tasks can now use scripts/coderoute-eval-smoke.py. 8. Build the dashboard MVP once the ledger API has real streaming usage data in production. Done for the MVP plus provider-health, spend-comparison, feedback-value, and alert-summary slices. 9. Add provider-error fallback logic so transient local/provider failures can retry the next allowed candidate before returning 502. Done for non-streaming adaptive aliases; streaming fallback remains limited to pre-first-byte error handling. 10. Add external notification hooks for budget alerts. Done with POST /v1/coderoute/alerts/notify and optional CODEROUTE_ALERT_* webhook settings. 11. Harden request-record metadata for dashboard and route evaluation. Done with metadata-backed user/workspace attribution, route policy version and config hash, OpenRouter generation IDs when available, provider cache token details, and UTC Z timestamp serialization in request APIs/exports. 12. Add monthly reporting for workspace, project, API-key, client, model, and alias breakdowns. Done with GET /v1/coderoute/reports/monthly.