anubis-oss

Anubis anubis_icon (1)

macOS 15+ Swift License: GPL-3.0 GitHub Release Ko-fi

Local LLM Testing & Benchmarking for Apple Silicon Community Leaderboard 
brew install --cask uncsoft/anubis/anubis-oss

Or download the zip directly from the Releases page and drag to /Applications.

🚨 Benchmark analysis is live! Check out the results here, over 400+ community submitted runs analyzed Benchmark Report

Anubis is a native macOS app for benchmarking, comparing, and managing local large language models using any OpenAI-compatible endpoint - Ollama, MLX, oMLX, LM Studio Server, OpenWebUI, Docker Models, etc. Built with SwiftUI for Apple Silicon, it provides real-time hardware telemetry correlated with full, history-saved inference performance - something no CLI tool or chat wrapper offers. Export benchmarks directly without having to screenshot, and export the raw data as .MD or .CSV from the history. You can even OLLAMA PULL models directly within the app.

Screenshot 2026-06-03 at 10 34 25 PM

anubis_demo_thumb
Watch Demo

New in 3.7: the Flow Builder — drag-and-drop sequencer for multi-model / multi-prompt / N-rep benchmark runs, with share-ready 16:9 or 1:1 report cards. Build it once, run it hands-off, post the PNG.

Screenshot 2026-05-28 at 1 15 15 PM

What’s New

Flow Builder — Drag-and-Drop Benchmark Sequencer (New in 3.7)

Build, save, and share repeatable benchmark recipes — a Shortcuts-style editor where you sequence steps like Set Backend → Set Model → Repeat × 5 → Run Benchmark → Unload and play them back hands-off. Every individual run still lands in normal Run History, and the whole sequence gets a single share-ready report.

Why use a flow instead of clicking Run repeatedly?

Editor. Three panes: a left Palette of step blocks, a center Step List rendered as a nestable tree, a right Inspector that swaps controls per step type. Add steps by clicking a palette block or dragging it into precise position. Reorder by drag, by ↑/↓ chevrons, or by editing the tree directly. Containers (Repeat, For Each Model, For Each Prompt) hold child steps with an indented rail.

Live lint. As you build, Anubis flags dangling Set steps (e.g. a Set Model with no following Run Benchmark), empty For Each lists, blank entries, and Run Benchmarks that are missing required prior Sets. The header chip jumps straight to the offending step.

For Each, explicitly. Two rules worth remembering:

  1. For Each replaces the corresponding Set. Inside a For Each Model, the active model is whatever the iterator just picked — any prior Set Model is ignored for that iteration (the lint catches this).
  2. Nested containers multiply. Total runs = outer × inner × deeper. 3 models × 2 prompts × 5 reps = 30 BenchmarkSessions.

Templates. Sidebar + → New from Template ships six starters: Quick Smoke Test, Repeated Run (5×), Cold vs Warm Start, Multi-Model Comparison, Q4 vs Q8 Quantization, Prompt Variants. Each is a real working flow; edit or use as-is.

Run sheet. Live progress with checkmarks per step, a scrollback log, a “Run X of Y” counter, and a per-run latest-session preview. Stop (⌘.) cancels cleanly mid-run.

Reports built for sharing. Post-run, View Report (⇧⌘R) opens a forced-dark, 1920×1080 card with a hero, winner callout, per-model leaderboard with relative bars, per-rep tables, and a methodology footer. Toggle to a 1080×1080 square for Instagram/X. Save as PNG @ 2× retina (3840×2160 or 2160×2160), copy to clipboard, or save the per-model CSV. Past runs live in the sidebar’s Run History section.

Import / export. Right-click any flow → Export… writes a portable .anubisflow JSON file. + → Import .anubisflow… reads it back. Format versioned so older clients refuse newer files instead of silently misreading them.

Open the in-app Help sheet (the ? in the Flows sidebar header) for the full reference, including worked code-style examples of each For Each pattern.

Run History Management + Browse Ollama Models (New in 3.6)

The History window is now genuinely usable, and discovering Ollama models no longer requires leaving the app.

N-Rep Benchmark Groups + Accurate SoC Power (New in 3.5)

Bundle N consecutive runs of the same configuration into one benchmark with mean and 95% bootstrap confidence intervals, plus a significant correction to system-wide power accounting.

Search and Download Ollama Models right from inside the app!

Ollama Thinking Toggle (New in 3.2)

Tri-state control over Ollama’s think request parameter, exposed in the Benchmark Performance disclosure when the Ollama backend is selected.

The choice persists across launches.

Reasoning-Aware Metrics & Prefill Speed (New in 3.1)

Output tokens/sec is now visible-throughput only for reasoning models. Previously, thinking time was charged against TTFT and thinking tokens were counted as output, inflating the numbers. Fixes #17 and #18.

Apple Intelligence Backend (New in 3.0) 🍎

Anubis now benchmarks Apple’s on-device Foundation Model alongside Ollama, MLX, and the rest — no server, no network, no setup. If your Mac supports Apple Intelligence (macOS 26+), it shows up in the backend menu automatically.

Reports Tab — Export (New in 3.0)

Export the per-model performance summary directly from the Reports tab.

Denser Benchmark Dashboard (New in 3.0)

Hardware Stress Testing (New in 2.9)

Push your Apple Silicon to its limits and observe power draw, thermal throttling, and frequency scaling under controlled load - all from within the Monitor.

Floating Monitor HUD (New in 2.9)

Screenshot 2026-05-28 at 4 37 09 PM

A compact, frameless, always-on-top overlay showing live system metrics - launchable from any tab via the sidebar or from the Monitor’s Float button.

15 Benchmark Prompts (New in 2.9)

Five new built-in prompts covering causal reasoning, system design, dialogue writing, historical analysis, and constrained writing - bringing the total to 15 across five categories.

Why Anubis?

The local LLM ecosystem on macOS is fragmented:

Anubis fills that gap - all in a native macOS app.

Leaderboard Submissions Now Available! Submit directly through the app

The dataset is robust and open source - check it out here, please contribute!

Features

Benchmark

Real-time performance dashboard for single-model testing.

Arena

Side-by-side A/B model comparison with the same prompt.

Flows

Drag-and-drop sequencer for repeatable, multi-step benchmark recipes.

System Monitor

Standalone real-time hardware monitoring dashboard - no benchmark required.

Leaderboard

Upload your benchmark results to the community leaderboard and see how your Mac stacks up against other Apple Silicon machines.

Vault

Unified model management across all backends.

Auto-Update

Anubis checks for updates automatically via Sparkle and notifies you when a new version is available.

Screenshots

Reports and Exports Screenshot 2026-05-28 at 4 35 19 PM Screenshot 2026-05-28 at 4 22 51 PM

GPU Core detail Screenshot 2026-02-25 at 4 08 44 PM

Arena Mode Screenshot 2026-02-25 at 4 21 50 PM

Settings (add connections with quick presets) Screenshot 2026-05-28 at 4 33 49 PM

Vault - View model details, unload, and Pull models directly for Ollama Screenshot 2026-02-25 at 4 14 57 PM

Supported Backends

Backend Type Default Port Setup
Apple Intelligence On-device (Foundation Models) — macOS 26+ with Apple Intelligence enabled. No setup; appears in the backend menu when supported.
Ollama Native support 11434 Install from ollama.com - auto-detected on launch
LM Studio OpenAI-compatible 1234 Enable local server in LM Studio settings
mlx-lm OpenAI-compatible 8080 pip install mlx-lm && mlx_lm.server --model <model>
vLLM OpenAI-compatible 8000 Add in Settings
LocalAI OpenAI-compatible 8080 Add in Settings
Docker ModelRunner OpenAI-compatible user selected Add in Settings

Any OpenAI-compatible server can be added through Settings > Add OpenAI-Compatible Server with a name, URL, and optional API key.

Hardware Metrics

Anubis captures Apple Silicon telemetry during inference via IOReport and system APIs:

Metric Source Description
GPU Utilization IOReport GPU active residency percentage
CPU Utilization host_processor_info Usage across all cores
GPU Power IOReport Energy Model GPU power consumption in watts
CPU Power IOReport Energy Model CPU (E-cores + P-cores) power in watts
ANE Power IOReport Energy Model Neural Engine power consumption
DRAM Power IOReport Energy Model Memory subsystem power
GPU Frequency IOReport GPU Stats Weighted average from P-state residency
Process Memory proc_pid_rusage Backend process phys_footprint (includes Metal/GPU allocations)
Thermal State ProcessInfo.thermalState System thermal pressure level

Process Monitoring

Anubis automatically detects which process is serving your model:

Metrics degrade gracefully - if IOReport access is unavailable (e.g., in a VM), Anubis still shows inference-derived metrics.

Requirements

Getting Started

1. Install Ollama (or another backend)

# macOS - install Ollama
brew install ollama

# Start the server
ollama serve

# Pull a model
ollama pull llama3.2:3b

2. Install Anubis

The easiest path is Homebrew — installs the signed, notarized .app from the latest GitHub release:

brew install --cask uncsoft/anubis/anubis-oss

Or download the zip directly from the Releases page and drag to /Applications.

Anubis auto-updates via Sparkle in either case. Once installed, Anubis will auto-detect Ollama on launch. Other backends can be added in Settings.

Updating to a new version

The app updates itself via Sparkle — just open Anubis and accept the update prompt (or Settings → Check for Updates). That’s the recommended path and works for every install method.

To update through Homebrew instead:

brew update
brew upgrade --cask anubis-oss --greedy

The --greedy flag is required because the cask is marked auto_updates (the app updates itself), so a plain brew upgrade intentionally skips it.

Homebrew says Cask 'anubis-oss' is unreadable: syntax errors / shows <<<<<<< conflict markers This only happens if your local tap checkout has uncommitted edits or a stash that conflicts with an update. Reset it to the published cask: ```bash cd "$(brew --repository)/Library/Taps/uncsoft/homebrew-anubis" git fetch origin && git reset --hard origin/main brew upgrade --cask anubis-oss --greedy ```

Or build from source

git clone https://github.com/uncSoft/anubis-oss.git
cd anubis-oss/anubis
open anubis.xcodeproj

In Xcode:

  1. Set your development team in Signing & Capabilities
  2. Build and run (Cmd+R)

3. Run Your First Benchmark

  1. Select a model from the dropdown
  2. Type a prompt or pick one from Presets
  3. Click Run
  4. Watch the metrics light up in real time

4. Submit to the Leaderboard

After a benchmark completes, click the Upload button in the benchmark toolbar to submit your results to the community leaderboard. Enter a display name and your run will appear in the rankings - no account required. Only performance metrics and hardware info are submitted; response text is never uploaded.

Building from Source

# Clone
git clone https://github.com/uncSoft/anubis-oss.git
cd anubis-oss/anubis

# Build via command line
xcodebuild -scheme anubis-oss -configuration Debug build

# Run tests
xcodebuild -scheme anubis-oss -configuration Debug test

# Or just open in Xcode
open anubis.xcodeproj

Dependencies

Resolved automatically by Swift Package Manager on first build:

Package Purpose License
GRDB.swift SQLite database MIT
Sparkle Auto-update framework MIT
Swift Charts Data visualization Apple

Architecture

Anubis follows MVVM with a layered service architecture:

┌─────────────────────────────────────────────────────────────┐
│                    PRESENTATION LAYER                       │
│  BenchmarkView  ArenaView  MonitorView  VaultView  Settings │
├─────────────────────────────────────────────────────────────┤
│                      SERVICE LAYER                          │
│   MetricsService   InferenceService   ModelService   Export │
├─────────────────────────────────────────────────────────────┤
│                    INTEGRATION LAYER                        │
│  OllamaClient  OpenAICompatibleClient  IOReportBridge  ProcessMonitor │
├─────────────────────────────────────────────────────────────┤
│                    PERSISTENCE LAYER                        │
│   SQLite (GRDB)              File System                    │
└─────────────────────────────────────────────────────────────┘

Views display data and delegate to ViewModels. ViewModels coordinate Services. Services are stateless and use async/await. Integrations are thin adapters wrapping external systems (Ollama API, IOReport, etc.).

Project Structure

anubis/
├── App/                    # Entry point, app state, navigation
├── Features/
│   ├── Benchmark/          # Performance dashboard
│   ├── Arena/              # A/B model comparison
│   ├── Flows/              # Flow Builder: editor, executor, report, lint, templates
│   ├── Monitor/            # System monitor, stress tests, floating HUD
│   ├── Reports/            # Cross-run model performance table
│   ├── Vault/              # Model management
│   └── Settings/           # Backend config, about, help, contact
├── Services/               # MetricsService, InferenceService, FlowExecutor, ExportService
├── Integrations/           # OllamaClient, OpenAICompatibleClient, IOReportBridge, ProcessMonitor
├── Models/                 # Data models (BenchmarkSession, Flow, ModelInfo, etc.)
├── Database/               # GRDB setup & migrations
├── DesignSystem/           # Theme, colors, reusable components
├── Demo/                   # Demo mode for App Store review
└── Utilities/              # Formatters, constants, logger, benchmark prompts

Backend Abstraction

All inference backends implement a shared protocol, making it straightforward to add new ones:

protocol InferenceBackend {
    var id: String { get }
    var displayName: String { get }
    var isAvailable: Bool { get async }

    func listModels() async throws -> [ModelInfo]
    func generate(prompt: String, parameters: GenerationParameters)
        -> AsyncThrowingStream<InferenceChunk, Error>
}

Data Storage

All data is stored locally - nothing leaves your machine.

Data Location
Database ~/Library/Application Support/Anubis/anubis.db
Exports Generated on demand (CSV, Markdown)
Preferences UserDefaults

Troubleshooting

Ollama shows “Disconnected”

# Make sure Ollama is running
ollama serve

# Verify it's accessible
curl http://localhost:11434/api/tags

No GPU metrics

High memory usage

Model not appearing

Contributing

Contributions are welcome. A few guidelines:

  1. Follow the existing patterns - MVVM, async/await, guard-let over force-unwrap
  2. Keep files under 300 lines - split if larger
  3. One feature per PR - small, focused changes are easier to review
  4. Test services and integrations - views are harder to unit test, but services should have coverage
  5. Handle errors gracefully - always provide errorDescription and recoverySuggestion

Adding a New Backend

  1. Create a new file in Integrations/ implementing InferenceBackend
  2. Register it in InferenceService
  3. Add configuration UI in Settings/
  4. That’s it - the rest of the app works through the protocol

Support the Project

If Anubis is useful to you, consider buying me a coffee on Ko-fi or sponsoring on GitHub. It helps fund continued development and new features.

A sandboxed, less feature rich version is also available on the Mac App Store if you prefer a managed install.

License

GPL-3.0 License - see LICENSE for details.

Other projects: DevPad · Nabu

This site is open source. Improve this page.