19 — Every Signal, One Map: Telemetry and Test Tooling in Tables

19 of 19 · the atlas

This stack generates a lot of artifacts — 36 metric families on the proxy alone, another 40-odd on four exporters, spans, log streams, feedback scores, and alerts. And it is tested with four different rigs. This post is the reference map: what is generated, where each thing goes, and how the load and unit tests run. Each section is a tab.

Metrics

1Metrics: five Prometheus exporters, scraped not pushed

Every mlx_* metric is served as a Prometheus text endpoint and scraped by VictoriaMetrics every 10 s — nothing is pushed over OTLP. That split was an explicit audit decision (post 4): the OTLP metrics path was removed from the proxy so each series exists exactly once.

Exporter	Port	What it reports (families)	Scraped at
mlx_metrics_proxy.py (the inference front door)	:8080	36 mlx_* families — volume (mlx_requests_total, tokens), latency (mlx_ttft_seconds, mlx_generation_seconds, mlx_token_rate_* histograms), tool calls, finish reasons, temperature, hallucination heuristics (mlx_hallucination_risk, mlx_heuristic_*), context/KV gauges, model info (mlx_model_info{attr}, mlx_model_arch), and the confidence set (mlx_output_perplexity, mlx_token_entropy_nats, mlx_token_confidence_*, mlx_token_margin_mean, mlx_low_confidence_*, mlx_confidence_scored_total{source}); volume/errors/tokens/calls/finish/temperature/risk carry a kind label (reasoning when enable_thinking is set, else chat)	192.0.2.64:8080
mlx_hw_telemetry.py (both nodes)	:9102	24 gauges — CPU/mem/disk/temp, GPU util/freq/power/mem, ANE & package power, thermal, per-worker CPU/RSS	192.0.2.64:9102 (rank0), 203.0.113.2:9102 (rank1, over Ethernet)
mlx_kv_cache_agent.py	:9104	mlx_kv_cache_sequences/bytes{type}, mlx_kv_cache_utilization, mlx_context_*, mlx_kv_bytes_per_token, mlx_kv_est_bytes_max_context	192.0.2.64:9104
mlx_server_supervisor.py	:9105	Serving truth — mlx_server_state/up/ready, mlx_model_loaded, mlx_server_restarts_total, mlx_server_crashes_total{reason}, mlx_server_uptime_seconds, mlx_server_health_checks_total{result}	192.0.2.64:9105
mlx_server_log_tailer.py	:9106	Log-pipeline health — mlx_log_tailer_up, lines/shipped/dropped/errors, file_bytes, queue_depth	192.0.2.64:9106

ⓘVM runs on this Mac (victoria-metrics:8428 in compose). The rank1 scrape target was repointed to 203.0.113.2:9102 so node B is reachable over the direct Ethernet link even when the Wi-Fi segments are split.

The proxy family detail

The full 36-family set on :8080/metrics (labels in parentheses):

Metric	Type	Meaning
mlx_requests_total model, streaming, kind	Counter	Completion requests (kind = reasoning when the caller sets enable_thinking, else chat)
mlx_tokens_prompt_total / mlx_tokens_completion_total kind	Counter	Token volume by call kind
mlx_ttft_seconds, mlx_generation_seconds, mlx_token_rate_tokens_per_second	Histogram	Latency + throughput
mlx_tool_calls_total kind, tool, type, mlx_finish_reason_total{reason, kind}	Counter	Agentic behaviour by call kind; tool is the function name, type the OpenAI call type (function / code_interpreter / …)
mlx_temperature kind, mlx_hallucination_risk kind, mlx_hallucination_risk_histogram, mlx_hallucination_flag_total	Gauge/Hist/Counter	Sampling + text heuristics
mlx_heuristic_repetition/_refusal/_hedging/_ungrounded	Gauge	Individual heuristic scores
mlx_up, mlx_in_flight, mlx_error_total{status, kind}	Gauge/Counter	Liveness + errors by status class and call kind
mlx_context_length_max_tokens, mlx_context_used_tokens, mlx_context_utilization, mlx_kv_bytes_per_token	Gauge	Context window + KV cache footprint
mlx_model_info{attr}, mlx_model_arch{arch, dtype}	Gauge	Static architecture facts from config.json: layers, KV heads, head dim, hidden size, attention heads, vocab, FFN intermediate, quant bits/group size (feeds the MLX Model dashboard)
mlx_output_perplexity +histogram, mlx_token_entropy_nats +histogram	Gauge	Sampler-flatness signals
mlx_token_confidence_mean/_min/_std, mlx_token_margin_mean, mlx_low_confidence_token_ratio	Gauge	Token confidence (post 18)
mlx_confidence_scored_total{source}, mlx_low_confidence_response_total	Counter	Scoring coverage + low-confidence responses

Logs

2Logs: two producers, one collector, a durable JSONL tail

Producer	Origin	How it leaves the host	Destination
mlx_server_log_tailer.py (:9106)	cluster/logs/server.log — the live mlx_lm.server log (ring, GPU, rank, generation lines; HTTP access lines excluded by default)	OTLP HTTP → 192.0.2.64:4318/v1/logs, classified with attrs mlx.log.severity/category/source/project, header projectName: mlx, rate-limited 50 lines/s	otel-collector logs pipeline → durable JSONL at /var/log/otelcol/mlx-server-logs.jsonl (50 MB × 5 backups) + debug stdout
proxy mlx.proxy.http logger	One access-log line per request	OTLP HTTP → same collector /v1/logs (only when --otlp-endpoint is set)	same JSONL file

Local, unbuffered logs for operators (rotated by the stack, not shipped):

File	Writes
cluster/logs/server.log	mlx_lm.server / ring output (tailer also watches this)
cluster/logs/proxy.log, supervisor.log, hw0.log/hw1.log, kvagent.log, logtailer.log, bootstrap.log	Each component, plus the start script’s own action log

⚠The collector’s traces pipeline only prints to debug. Durable trace storage lives in Opik (next tab); the JSONL log file is the only long-lived artifact the collector itself keeps.

Traces & Opik

3Traces: one span, three sinks — and Opik’s feedback loop

The proxy creates one span mlx.chat.completions (tracer mlx.proxy) per request and stamps it as it passes through:

Phase	Attributes
Request start	http.request.method, url.path, mlx.model, mlx.streaming, mlx.temperature, mlx.tools_provided
Completion	mlx.ttft_seconds, mlx.gen_seconds, mlx.tokens_prompt/completion, mlx.tool_calls, mlx.hallucination_risk, mlx.hallucination_flagged, http.response.status_code
Confidence (only when scored)	mlx.logprobs_source, mlx.perplexity, mlx.token_entropy, mlx.confidence_mean, mlx.confidence_min, mlx.low_confidence_token_ratio
OpenInference (only with --opik-otlp-endpoint)	openinference.span.kind=LLM, input.value, llm.model_name, llm.token_count.*, metadata JSON (incl. _conf_metadata: scored_tokens, confidence_std, token_margin_mean, …)

Where the span goes

Sink	Address	Contents
otel-collector traces	192.0.2.64:4318/v1/traces (gRPC :4317 also open)	debug/stdout only
Opik OTLP ingestion	192.0.2.10:32173/api/v1/private/otel/v1/traces, header projectName: mlx	the same span with OpenInference attrs — this is what populates the mlx project
Opik SDK (when OPIK_ENDPOINT set)	192.0.2.10:32173/api	trace + LLM span, plus in-band feedback scores

Feedback scores (written by the proxy, in-band)

Score	Value	Category
hallucination_quality	1.0 - risk	quality
hallucination_flagged	0.0 if flagged else 1.0	safety
token_confidence	confidence_mean (only when scored)	quality

The judge loop (post 6) — cluster/opik_evaluator.py

Step	How
Read	GET 192.0.2.10:32173/api/v1/private/traces?…
Judge	LLM-as-judge — the judge model is the cluster proxy itself at 127.0.0.1:8080/v1
Write	PUT 192.0.2.10:32173/api/v1/private/traces/feedback-scores with correctness, helpfulness, hallucination_free

Opik dashboards (tools/opik_dashboards.py) turn the traces + judge scores into a project-scoped “mlx: serving health & quality” view at the same base URL (post 17).

Alerts & Dashboards

4Alerts & Dashboards: 46 rules to a webhook, ten dashboards to Grafana

Stage	Component	How
Rules	vmalert (:8880)	46 rules in 6 groups — mlx-hardware (15), mlx-serving (8), mlx-inference (7), mlx-quality (6), mlx-confidence (5, post 18), mlx-slo (5: burn-rate + budget alerts for the SLOs on the SRE dashboard)
Evaluation	vmalert → VM	queries victoria-metrics:8428, writes ALERTS series back via remote-write
Routing	Alertmanager (:9093)	critical → host.containers.internal:9000/hooks/oncall; everything else → .../hooks/alert; inhibit rules (e.g. ProxyDown silences UpstreamDown)
Dashboards	Grafana (:3000)	10 provisioned: MLX Cluster (50 panels), MLX Performance (29), MLX GPU & Power (17), MLX Node (33), MLX Alerts & Serving Health (28), + MLX SRE (SLI/SLA/SLO/error budget), MLX HTTP Requests (20), MLX Call Types (21, reasoning vs chat incl. tool breakdown), MLX Model (25, architecture dims), MLX Tool Calls (14, dynamic tool usage); datasource victoria-metrics:8428

✓Every confidence alert is gated on mlx_confidence_scored_total > 0, and MLXConfidenceScoringStopped fires when traffic flows but nothing is scored — so “blind” and “healthy” never look identical (post 18).

Load tests

5Load tests: four rigs, all against the proxy

Every bench tool targets the proxy (127.0.0.1:8080/v1/chat/completions), streams by default, uses temperature: 0.0, and sends X-Mlx-Trace: 1 so every request lands in Opik. Concurrency is thread-based (stdlib urllib + thread pools), never async.

Rig	Measures	How it runs	Output
tools/bench.py	Streaming TTFT, generation time, tok/s per request	Prompt matrix (short/medium/long) or one fixed-concurrency burst; --iters, --concurrency	table + optional --json
tools/bench_load.py main rig	TTFT/latency/tps p50-95-99, throughput tok/s, error rate	--sweep ramps concurrency 1→2→4→8→16; --sustain N soaks while snapshotting :9102/:9104/:9105 every interval; --profile stream on/off × prompt size. --duration/--settle control each level	JSON report → tools/bench-reports/{label}-{ts}.json
tools/bench_agentic.py	Turns, per-turn TTFT, wall time, tool-call validity, completion-in-budget	Scenarios calculator/tri_tool/planner against a deterministic stub tool registry; streaming chat with tools; full transcript re-sent each turn; --runs × --agents concurrent loops	JSON report → tools/bench-reports/
tools/smoke.py	Ring init + all_sum correctness across ranks	Runs mlx.core on the cluster nodes directly (not HTTP); --backend ring	console

How a load run actually works

• Each concurrency level spawns N worker threads that loop run_one() until a stop event fires after --duration seconds.
• Each run_one returns per-request ttft, total, gen_time, comp_tokens, tps, ok/error — aggregated into percentiles and error rate.
• Results are correlated against VM-scraped cluster metrics (GPU mem, KV util, supervisor state) snapshotted during --sustain.
• No RPS controller — workers fire as fast as they can; there is no rate limiter or think time (documented gap, ghz/k6/locust are drop-in options).

Measured results (from the blogs)

Finding	Source
2-node ring is ~3× slower than single node at 1.7B scale (ring collectives dominate): short prompt 0.8 s/65 tok/s single vs 2.4 s/~18 tok/s ring	README Benchmarks
Peak ring throughput ~28 tok/s at concurrency 4; TTFT degrades linearly as it saturates	post 9
18-request burst: TTFT p50 0.38 s / p95 0.91 s, ~8.8 decode tok/s	post 7
Agentic: calculator 5 turns/4 calls 100% valid; tri_tool 2 turns/3 calls; planner 0 tool calls; 2 concurrent agents crashed the server	post 9
Confidence layer: “what is 2+2” ppl 1.00 / conf_min 1.00 vs obscure factual ppl 1.41 / conf_min 0.119	post 18

Unit tests

6Unit & integration tests: 40, in the repo venv

Run with .venv/bin/python -m pytest tests/ -q (Python 3.14 venv at repo root; both modules import mlx_metrics_proxy directly, no network needed for the confidence tests).

File	Tests	What it proves
tests/test_confidence.py	30 pure unit	perplexity, renormalised top-k entropy (lower-bound), top-2 margin, confidence_stats, score_confidence incl. malformed-input tolerance, strip_logprobs/choice_logprobs, SSE-frame synthesis round-trips (content, tool calls, identity fields), _conf_metadata
tests/test_proxy_rewrite.py	10 e2e	against a StubUpstream that mirrors mlx_lm.server (logprobs on non-streaming only, SSE + [DONE] + connection-close framing, optional tool calls, configurable error status). The proxy runs in-process on an ephemeral port; real HTTP drives it. Verifies: logprobs injected upstream but stripped for the client with matching Content-Length, client-requested logprobs pass through, injection disable, tool calls survive the strip, upstream 503 passthrough, streaming untouched when sampling off, de-streamed requests still parse as SSE, de-streamed tool calls reach the client, and de-streamed requests record no TTFT. Metrics are read back from the in-process proxy’s /metrics.

ⓘThe bench tools themselves are not unit-tested — only the proxy’s confidence/rewrite layer is. The stub upstream is the trick that makes the whole injection/de-streaming path testable without a GPU.

Flow & ports

7Flow & ports: how it all moves

The four data paths in pictures. Moving dots are live traffic; dashed lines are idle by design.

Metrics — five exporters scraped by VictoriaMetrics every 10 s

Logs — OTLP HTTP push to the collector, then a durable JSONL tail

Traces & Opik — one span, three sinks, and the judge loop

Alerts & dashboards — 46 rules to a webhook, ten dashboards to Grafana

ⓘPink dots are forward traffic (scrape, push, fire); teal dots are loopback/return (judge calls, ALERTS remote-write). Dashed gray = deliberately idle: the collector’s metrics remote-write is unused because the proxy exports no OTLP metrics (post 4 audit).

Files & configs

7Files & configurations: where everything lives

File	Role	Key configuration
cluster/cluster.env	Single source of truth for the model + knobs	MLX_MODEL=mlx-community/Qwen3.5-4B-MLX-8bit, MLX_DEFAULT_TEMP=0.0, MLX_LOGPROBS=3, MLX_LOGPROBS_STREAM_SAMPLE=0.05, MLX_LOW_CONFIDENCE=0.5
cluster/start_server.sh	Start/stop/status; wires flags + endpoints	Ports :8081 server, :8080 proxy, :9102/9104/9105/9106 agents; OTLP=192.0.2.64:4318, OPIK_OTLP=192.0.2.10:32173/api/v1/private/otel, RANK1=203.0.113.2 (Ethernet)
cluster/hosts.json	Ring hostfile	rank0 ssh 127.0.0.1 / ips 203.0.113.1; rank1 ssh 203.0.113.2 / ips 203.0.113.2
cluster/mlx_metrics_proxy.py	Public OpenAI API + metrics + spans + scores	36 metric families; kind label (reasoning vs chat); span attrs; 3 in-band feedback scores; flags --otlp-endpoint/--opik-otlp-endpoint/--opik-endpoint/--logprobs*
cluster/mlx_hw_telemetry.py · mlx_kv_cache_agent.py · mlx_server_supervisor.py · mlx_server_log_tailer.py	Hardware / KV / serving / log exporters	Ports :9102 / :9104 / :9105 / :9106; log tailer rate-limit 50/s, JSONL attrs
cluster/opik_evaluator.py	LLM-as-judge feedback loop	judge = proxy 127.0.0.1:8080/v1; reads/writes 192.0.2.10:32173/api
observability/compose.yaml	The observability stack (podman)	victoria-metrics :8428, grafana :3000, otelcol :4317/:4318, vmalert :8880, alertmanager :9093
observability/vm-scrape.yml	Scrape targets (generated from vm-scrape.tmpl.yml by setup.sh)	interval 10s; proxy/hw/kv/supervisor/logtailer targets; rank1 hw at 203.0.113.2:9102
observability/otelcol-config.yaml	Collector pipelines	receivers gRPC :4317/HTTP :4318; logs → JSONL /var/log/otelcol/mlx-server-logs.jsonl (50 MB ×5); traces → debug; metrics → VM remote-write (idle)
observability/vmalert/rules.yml	46 alert rules / 6 groups	hardware 15, serving 8, inference 7, quality 6, confidence 5, slo 5
observability/alertmanager/alertmanager.yml	Notification routing + inhibit	critical → .../hooks/oncall, rest → .../hooks/alert at host.containers.internal:9000
observability/grafana/dashboards/*.json	9 provisioned dashboards	datasource victoria-metrics:8428; + SRE / HTTP / Call Types / Model (this post)
tools/bench.py · tools/bench_load.py · tools/bench_agentic.py · tools/smoke.py	Test rigs	all against proxy :8080/v1; reports → tools/bench-reports/ (git-ignored)
tests/test_confidence.py · tests/test_proxy_rewrite.py	Unit/integration tests	run: .venv/bin/python -m pytest tests/ -q (40 passing)
tools/opik_dashboards.py	Opik dashboard provisioning	base 192.0.2.10:32173/api (needs the /api suffix)

⚠Addresses in this table are the current operational values (rank1 repointed to the Ethernet link after the Wi-Fi split, post 19’s fieldwork). If the LAN changes, observability/setup.sh regenerates vm-scrape.yml; the proxy endpoint flags live in start_server.sh.