How every number Anubis displays is computed, with the source code that produces it. If a measurement isn’t clearly defined, it isn’t trustworthy - so this page lists the formula, the units, and the caveats for each metric, and links to the exact file and line in the open-source repo.
All generated tokens count, including a reasoning model’s thinking tokens — they are decoded by the same hardware at the same speed as visible ones. This matches how everyone else measures generation speed: Ollama’s eval rate (eval_count / eval_duration), llama.cpp’s predicted_per_second, and llmperf / genai-perf output throughput. The thinking-phase rate is still reported separately (see reasoning split).
History (methodology ≤ v2): earlier versions excluded thinking from both numerator and denominator — (completionTokens − reasoningTokens) / (evalDuration − reasoningDuration). When a run was reasoning-dominated, that divided a handful of visible tokens by a near-zero difference of two estimated quantities, producing submissions claiming 5,000–29,000 tok/s. Those rows are now filtered from the leaderboard API and rejected at submission.
When the backend measures throughput inside its own decode loop (oMLX usage extensions, llama.cpp’s timings object, LM Studio’s native /api/v1/chat stats), that figure is used verbatim in preference to client-side wall-clock derivation, which a bursty SSE reader can distort.
tokensPerSecond)eval_count / eval_duration and llama.cpp’s timings are the source of truth when available. OpenAI-compatible endpoints don’t always return token counts, in which case Anubis falls back to counting streamed chunks and measuring wall time client-side, which slightly underestimates true tok/s due to network round-trip overhead.Reported separately from output tok/s because the two phases have wildly different characteristics on Apple Silicon: prefill is compute-bound and benefits from wide GPU arrays, while generation is memory-bandwidth-bound. A chip that decodes at 30 tok/s might prefill at 1,000+ tok/s - or vice versa.
TTFT is end-to-end from the user’s perspective: it includes tokenisation, KV cache setup, prefill, and any network round-trip to the local backend on its loopback port. A high TTFT on a long prompt is mostly prefill.
Model load time is excluded when the backend reports it (methodology v3). Ollama cold-loads models and reports load_duration; subtracting it makes Ollama TTFTs comparable with backends that keep the model resident (before this, Ollama’s leaderboard median TTFT was ~4.9 s against ~0.5 s of actual prefill, nearly all of it load time). Load time is still recorded separately on every session. Backends that JIT-load without reporting it (LM Studio via the /v1 API) still fold that load into TTFT.
For reasoning models, TTFT is the time to the first generated token — thinking counts as generation. A model that thinks for 30 seconds before its first visible token still has a sub-second TTFT if thinking started immediately.
BenchmarkViewModel at the moment the first InferenceChunk with visible content arrives.Thinking tokens are counted in the headline tokens/second (they’re decoded at the same hardware speed), but the split is still tracked so you can see how much of a run was spent thinking. Three rates are reported: prefill (input ingestion), reasoning (thinking phase, if any), and the overall generation rate.
Detection depends on the backend exposing a thinking signal. For Ollama, the thinking field on the chunk and the think request parameter are used. For OpenAI-compatible endpoints, only models that surface thinking via a structured field (e.g. reasoning_content) are split.
System power is the all-domain SoC reading from IOReport (GPU + CPU clusters + ANE + DRAM + uncore). tokensPerSecond is the session-level output rate from the formula above - thinking phase excluded.
Computed once at session completion from the two session-level aggregates, not by averaging per-sample power/tps ratios. Per-sample averaging biases the result high because each sample divides by a cumulative tok/s that is artificially low early in generation (the warm-up samples dominate the arithmetic mean). See changelog entry for issue #22 below for the full story.
Anubis subscribes to two IOReport channel groups: Energy Model (per-component power) and GPU Stats / GPU Performance States (frequency residency). The API is private but stable across recent macOS versions, loaded via dlsym against /usr/lib/libIOReport.dylib - the same approach used by asitop, mactop, and macmon.
Power channels exposed:
Sample cadence: 2 Hz default (500 ms), reduced to 1 Hz when the app is idle (no active benchmark) to save battery. Each sample is a delta over the previous one - the IOReport API returns cumulative energy counters which Anubis differences to derive instantaneous power.
The GPU runs at one of N discrete frequencies (P-states), set dynamically by the SoC. Anubis reads the residency counter for each state from the IOReport GPU Stats channel (channel name GPUPH), and the frequency table from IORegistry. Effective GPU frequency over a sample window is the residency-weighted mean:
The full per-state distribution is captured per sample - you can see whether the GPU is sitting in its highest P-state under load (good) or bouncing between mid states (thermally constrained or memory-stalled).
Per-core ticks are read via host_processor_info(PROCESSOR_CPU_LOAD_INFO). Utilization for each core over a sample window is:
Cores are classified as efficiency or performance using the chip’s reported hw.perflevel0.logicalcpu and hw.perflevel1.logicalcpu from sysctl. On Apple Silicon the convention is E-cores first (indices 0…eCores-1), then P-cores.
Backend process memory uses phys_footprint via proc_pid_rusage, which matches the “Memory” column in Activity Monitor and crucially includes Metal GPU buffer allocations. The basic pti_resident_size alternative undercounts because GPU buffers are mapped out of the process’s resident set.
For backends that span multiple processes (LM Studio is an Electron app with many Node helpers; Ollama runs a parent ollama plus a worker per model), Anubis aggregates two ways:
.app bundle, sum phys_footprint across every running process whose path is under that bundle. Catches scattered Electron helpers and embedded servers regardless of parent-child relationship.llama.cpp, Python servers), recursively walk descendants from the root PID and sum phys_footprint.System-wide memory is reported separately, via host_statistics64(HOST_VM_INFO64) as (active + wired + compressed) × page_size. This matches Activity Monitor’s “Memory Used” figure and represents pressure on the entire machine, not just the backend.
Chip metadata comes from sysctl: brand string (machdep.cpu.brand_string), performance-core count (hw.perflevel0.logicalcpu), efficiency-core count (hw.perflevel1.logicalcpu), and total RAM (ProcessInfo.processInfo.physicalMemory). GPU core count is detected via IOKit. ANE core count and memory bandwidth come from a chip-name lookup table (architectural values that aren’t exposed by any API).
The Mac model marketing name (e.g. “MacBook Pro (14-inch, M4, Nov 2024)”) is resolved from hw.model (e.g. Mac16,1) via a four-tier lookup:
product-name, then the raw hw.model identifier.lm-evaluation-harness alongside.Tokens/second counts all generated tokens. Previous versions reported visible-output throughput: (completionTokens − reasoningTokens) / (evalDuration − reasoningDuration). On reasoning-dominated runs both differences approach zero and the ratio explodes — the public leaderboard accumulated rows claiming 5,000–29,000 tok/s from exactly this, and because the board ranks by tok/s they sat at the top. The metric is now completionTokens / evalDuration, the same definition as Ollama’s eval rate, llama.cpp’s predicted_per_second, and llmperf/genai-perf. The thinking-phase rate remains available separately.
Avg token latency is now exactly 1000 / tok_per_s — previously it was derived from the visible-output quantities and could disagree with the displayed throughput, particularly when a backend reported its own decode rate.
TTFT excludes server-reported model load time and is anchored at request dispatch (it was previously stamped a scheduling-gap late, and on some paths after response headers — which on a JIT-loading server meant total_duration could come out smaller than TTFT). total_duration is now always end-to-end wall time.
llama.cpp timings are parsed. llama-server attaches decode-loop timing (prompt_ms, predicted_ms, predicted_per_second, real token counts) to its final stream chunk; these are now used verbatim instead of client-side wall-clock estimates.
LM Studio benchmarks use its native chat API. When the backend is verified as LM Studio, requests go to /api/v1/chat instead of the OpenAI-compat endpoint (which strips all timing). The server then reports tokens/sec and TTFT measured inside its own decode loop, model load time (recorded separately, excluded from TTFT), and exact input/output/reasoning token counts. Anything the native endpoint refuses falls back to /v1/chat/completions wall-clock measurement as before. LM Studio is the most common backend on the leaderboard, so this replaces client-side estimation for the largest slice of submissions.
Data hygiene: the submission API now rejects runs with zero completion tokens or implausible throughput (≥2,000 tok/s), the app performs the same check before uploading, and the leaderboard API filters historical garbage rows out of rankings by default (raw=1 bypasses the filter for audits). New submissions carry methodology_version: 3.
The IOReport “Energy Model” channel group returns values in different units per channel on Apple Silicon. Anubis previously assumed every channel (GPU, ECPU, PCPU, ANE, DRAM) reported in nanojoules and divided by interval × 10&sup9; to get watts. That assumption is correct for GPU Energy but wrong for the other four, which actually report in millijoules. Treating them as nanojoules produced readings ~10&sup6;× too small — CPU power displayed in microwatts (µW) instead of watts. Verified empirically against powermetrics and against macmon / asitop, which apply per-channel scaling.
Knock-on effect: systemPower = gpu + cpu + ane + dram was effectively just GPU because the other three contributed near-zero. avgWattsPerToken (J/Tok) is derived from systemPower, so every J/Tok value submitted to the leaderboard prior to v3.4 was a systematic undercount of true SoC energy per token. The dominant contributor it missed was the CPU.
What was fixed (v3.4):
parseEnergyDelta: GPU stays on nJ, CPU/ANE/DRAM use mJ. Sanity-check warning fires if any channel ever reads above 200 W — would catch a future Apple unit change.cpu_power_watts / ane_power_watts / dram_power_watts are multiplied by 10&sup6;, per-sample system_power_watts is recomputed as the corrected sum, and per-session avg_system_power_watts / peak_system_power_watts / avg_watts_per_token are re-aggregated from the corrected samples. The pre-migration DB is backed up to anubis.sqlite.pre-v7.bak alongside the live file before the migration runs.methodology_version: 2. Submissions prior to v3.4 are implicitly v1 (CPU/ANE/DRAM at µW, systemPower ≈ GPU, J/Tok undercounted). The leaderboard can segregate or badge per-era submissions for honest cross-version comparisons.Impact on existing leaderboard data: v1 submissions remain on the public board with their original (undercounted) J/Tok and avg-system-power values. The leaderboard does not retroactively rewrite history. Going forward, v2 submissions reflect true SoC energy.
Impact on your local data: the v7 migration corrects all historical session aggregates on first launch of v3.4. Your past J/Tok values will shift upward to reflect the SoC total rather than just GPU. The pre-migration snapshot is preserved on disk if you need to recover the original numbers for any reason.
Two related corrections shipped together (commit b904914):
power ÷ tok/s was displayed as “W/Tk” throughout the UI, reports, and exports. Dimensional analysis: W ÷ (tok/s) = (J/s) ÷ (tok/s) = J/tok, not W/tok. All user-facing labels corrected to J/Tk / J/tok / Energy per Token. Database column names (watts_per_token, avg_watts_per_token) and CSV/JSON export keys are unchanged for downstream compatibility with prior submissions.systemPower by cumulativeTokensPerSecond. Early-generation samples have an artificially low cumulative tok/s (still ramping up), inflating their per-sample ratio. Averaging biased the session aggregate roughly 40% high. Now computed as avgSystemPower ÷ tokensPerSecond directly from session-level aggregates, which matches the physically meaningful J/tok and the value users can derive from the displayed avg power and avg tok/s.Impact: sessions logged before this fix retain their previously-stored (inflated) J/tok value in the local DB; new runs use the corrected formula. Leaderboard rows from prior versions will gradually be displaced by corrected entries as users re-run. The leaderboard does not retroactively rewrite history.
Credit to @taozhiyuai for the thorough bug report including the dimensional analysis and a 44% reproducible discrepancy.