**Put Apple's Foundation Models to the test.** Build an evaluation. Inspect the evidence. See what changed. Intents walkthrough: inspect the workflow trace timeline, span details, and evaluation report.

Intents walkthrough: inspect the workflow trace timeline, span details, and evaluation report.

![macOS 27+](https://img.shields.io/badge/macOS-27%2B-111827?style=flat-square&logo=apple&logoColor=white) ![Swift](https://img.shields.io/badge/Swift-F05138?style=flat-square&logo=swift&logoColor=white) ![Apple Foundation Models](https://img.shields.io/badge/Apple-Foundation_Models-2563eb?style=flat-square) ![MCP](https://img.shields.io/badge/Agent_Integration-MCP-7c3aed?style=flat-square) [![MIT License](https://img.shields.io/badge/License-MIT-16a34a?style=flat-square)](LICENSE) [Install](#install) · [First evaluation](#run-your-first-evaluation) · [Review outputs](docs/EVAL_REVIEW_GUIDE.md) · [Batch runs](docs/PRODUCTION_EVALS_GUIDE.md) · [User wiki](docs/wiki/Home.md) · [Models and tools](#models-and-tools) · [Connect an agent](#connect-an-agent) · [Build from source](#build-from-source)

Intents is a native macOS workbench for testing Apple's Foundation Models. Create repeatable suites, inspect responses and execution traces, and compare saved runs as you refine prompts, settings, and tools. Review saved outputs, turn confirmed failures into regression cases, check a judge against held-out examples, and run resumable batches. Its authenticated MCP server lets a coding agent work with the same projects, suites, evidence, and batch store as the app. ## What you can do | Capability | What it gives you | |---|---| | **Repeatable evaluations** | Test cases with shared instructions, attachments, and multiple repetitions. | | **Flexible scoring** | Exact matches, required text, AI rubrics, or response collection without scoring. | | **Execution traces** | A nested workflow waterfall with measured native stages, tool activity, token usage and a selected-span inspector. [Trace details](docs/workflow-traces.md). | | **Human review and regression cases** | Review outputs with AI judgments initially hidden, tag confirmed failure patterns, and promote verified failures into new cases. [Review guide](docs/EVAL_REVIEW_GUIDE.md). | | **Judge checks** | Separate Development and Held-out examples; inspect false acceptance, false rejection, and unavailable evidence. | | **Production batches** | Immutable datasets, resumable jobs, unattended workers, captured-output review, cohort gates and audited evidence. [Production guide](docs/PRODUCTION_EVALS_GUIDE.md). | | **Saved comparisons** | Run history, baseline comparisons, and JSON reports. | | **Instruction improvement** | Generate two instruction alternatives from Development cases, validate the best on reserved Held-out cases, and review before applying. [Workflow](docs/PROMPT_IMPROVEMENT_PLAN.md). | | **Models and tools** | Apple's on-device model, compatible Core AI models, custom HTTP providers, and configurable tools. | | **Agent integration** | Authenticated controls for the shared workspace, evaluations, batches, review, and developer runners. [MCP guide](docs/MCP_CONTROL_GUIDE.md). | | **App feature runners** | A public Swift package for evaluating real app closures on a paired iPhone, iPad, or Mac. [Integration guide](docs/DEVELOPER_SWIFT_INTEGRATION.md). | | **App Intent tests** | Inspect a built Mac app and run JSON input/result checks with Apple AppIntentsTesting, through the native UI or `intents-native-tests`. [Setup and qualification](docs/wiki/Mac-App-Intent-tests.md). | ## Install Requirements: - macOS 27 or later - For the default on-device model: a Mac that supports Apple Intelligence, with Apple Intelligence enabled and the model downloaded 1. Download the installer from the [latest release](https://github.com/coryparrry/Intents/releases/latest) and open **Intents.dmg**. 2. Drag **Intents** onto the **Applications** shortcut in the window. 3. Eject the **Intents** disk, then open the app from **Applications** and check that the model is ready. The app is signed with Developer ID and notarized by Apple. Or install the same signed app with [Homebrew](https://brew.sh): ```sh brew install --cask coryparrry/tap/intents ``` The cask requires Apple Silicon and macOS 27 or later. New published releases update the tap automatically. To upgrade through Homebrew, run `brew update && brew upgrade --cask --greedy coryparrry/tap/intents`. The app also includes **Check for Updates**. Xcode 27+ is needed to build from source or run Mac App Intent tests. It is not needed for a normal on-device suite. ## Run your first evaluation 1. In **Model evaluations**, create a suite. Open **Cases** and add prompts with an expected response or reference answer where appropriate. 2. Use **Setup → Instructions** for shared instructions and text, JSON, CSV, PDF, or image reference files. 3. Use **Setup → Scoring** to choose a method and number of repetitions. 4. Click **Run**. Open the saved run's **Workflow trace** or **Report** to inspect its response, score, explanation, timing, token usage, and tool activity. 5. Reopen runs from **Results** or the sidebar, use **Compare** for earlier runs, or export a JSON report. See the [user wiki](docs/wiki/Home.md) for detailed steps. | Scoring mode | Use it for | |---|---| | **Exact text** | Matching the complete expected response, ignoring surrounding whitespace. | | **Contains text** | Checking for required text, ignoring case and accents. | | **AI rubric** | Assessing concrete requirements on a 1–4 scale; scores of 3 or 4 pass. | | **Collect only** | Saving responses and traces without assigning a score. | For AI rubrics, write one observable requirement per line and provide a verified reference answer for factual tasks. Inspect the judge's explanation alongside its score. Repetitions help reveal variation; a few runs do not establish statistical significance. ## Review and improve a result Open **Review → Samples** after a saved run. Read the output before revealing its AI judgment, save a human verdict and note, and tag observable failures. **Patterns** groups confirmed failures; **Create regression case** requires a verified expected answer. **Judge checks** replays recorded outputs against human decisions in separate Development and Held-out groups. See the [Review guide](docs/EVAL_REVIEW_GUIDE.md). Human review, judge calibration, and baseline approval are separate decisions. Batch baselines require explicit approval of the current evidence; changed results, reviews, or controls invalidate it. Read the saved [release evidence](docs/wiki/Runs-and-results.md) before relying on a passing gate. ## Models and tools The default provider is Apple's on-device Foundation Model. You can also load a compatible [Core AI model](docs/coreai-provider.md) or connect a [custom local HTTP provider](docs/custom-provider-protocol.md). Use the suite's **Setup** pages to configure custom tools, structured output, streaming, and tool workflows. The [tools and structured output guide](docs/foundation-model-features.md) includes a runnable local HTTP example. To evaluate the production Swift feature inside another app, add the `FoundationEvalsDeveloper` package product and host its explicitly paired runner. The [developer integration guide](docs/DEVELOPER_SWIFT_INTEGRATION.md) covers typed closures, `@Generable` results and tools, iPhone/iPad/Mac setup, trust, cancellation, saved run evidence, and the boundary with Apple's development-only Evaluations framework. For large batches, open **Batch runs** in the sidebar. Freeze the current suite or import JSONL, create a batch, and run it on a matching Mac. The [production guide](docs/PRODUCTION_EVALS_GUIDE.md) covers `intents-evals`, unattended native workers, target matrices, schedules and opt-in capture. ## Test Mac App Intents Select a normally installed, registered Mac app in **App Intent tests**, or use `intents-native-tests catalog --app /path/to/MyApp.app` and `intents-native-tests run` with a JSON case manifest. The command builds a native macOS UI-test runner using Apple AppIntentsTesting. It needs Xcode 27+, macOS 27+, XcodeGen, and the same Apple development team for app and runner; your app needs no Intents SDK. Open the installed app once to register it, then quit it before testing. Each result records the selected build identity and authored cases. The [command-line guide](docs/wiki/Mac-App-Intent-tests.md) covers explicit inputs, expected scalar results and execution-only smoke checks. The [Mac sample](examples/MacIntentSample/README.md) includes build/signing instructions and a seven-case manifest. Seven local native qualification cases passed; that sample result does not establish compatibility with every app or verify Siri. ## Connect an agent Intents includes an authenticated MCP server for projects and suites, references, runs, review suggestions, batch jobs, and developer runners. Changes appear in the same native workspace. The standalone Apple Foundation Evals plugin uses a separate store. To connect Codex: 1. Open **Settings** in Intents. 2. Choose **Connect to Codex**. 3. Restart Codex and keep Intents open. The server provides an agent workflow guide during MCP initialization. Start with workspace state, then use `eval_find_actions` and `eval_describe_action` to load the exact schemas needed for advanced work. Mutations use revisions, confirmations where required, and operation IDs for safe retries. See the [MCP control guide](docs/MCP_CONTROL_GUIDE.md). Its HTTP endpoint is `http://127.0.0.1:17873/mcp` while the connector is running. The connector listens only on this Mac. Intents generates a bearer credential, stores it in the login Keychain, and configures Codex to send it. Connected authenticated local clients can read and change evaluation data; keep the managed configuration private. ## Your data Suites, imported attachments, and run history are saved in `~/Library/Application Support/FoundationEvals/`. The app does not encrypt these files itself. Saved traces and JSON exports can include prompts, responses, reference content, and tool arguments and outputs; review them before sharing. On-device evaluations run locally. Custom HTTP providers and tools receive the content needed for their calls, and those separate services control any onward network use. Optional Private Cloud Compute uses Apple's network service when available. Spotlight tools can make matching local file content available to the selected model. Usage statistics are **on by default** and can be disabled in **Settings → Privacy → Share usage statistics**. They send app sessions, first observed use, fixed screen names, completed features, app/build and macOS versions, and random installation/session identifiers. Development builds, tests, previews and demos are excluded; a host-local policy can also exclude every updated build on a development Mac. **Diagnostic sharing is separate and off by default**; it sends fixed operation names, outcomes, timings, safe error categories, build/platform details and automatic native crash reports with content-free stack addresses and binary IDs. Crash reports are sent on the next launch; readable function names require matching release symbols. Prompts, responses, scores, names, files, provider addresses, credentials and raw error messages are excluded. Privacy settings also show recent issues and let you copy a safe local diagnostic report without enabling sharing. Consent changes cancel uploads and revoke queued events; they do not delete data already received by PostHog. See [telemetry details](docs/telemetry.md). ## Standalone Codex plugin The [Apple Foundation Evals plugin](plugins/apple-foundation-evals/README.md) runs local Apple Foundation Models evaluations and displays their saved traces inside Codex. It bundles its own native engine and does not require the Intents app. Its README covers local installation, testing, and packaging. ## Build from source Use Xcode 27 with its command-line tools selected. From the repository root: ```sh ./script/build_and_run.sh ``` This creates and opens a development build at `dist/Intents.app`. Quit the app before rebuilding. You can also open `FoundationEvals/FoundationEvals.xcodeproj` directly in Xcode. Run native tests on macOS 27 by opening the project in Xcode and choosing **Product > Test** with development signing configured. Run the isolated production package checks and build its CLI without launching the app: ```sh swift test --package-path Packages/ProductionEvals -j 2 swift build --product intents-evals -j 2 python3 script/test_production_evals.py .build/debug/intents-evals ``` Check function docstrings with the same 81% minimum used by CI: ```sh npm ci --prefix Tools/DocstringCoverage --ignore-scripts --no-audit --no-fund python3 script/docstring_coverage.py --minimum 81 ``` See the [coverage guide](docs/DOCSTRING_COVERAGE.md) for the source scope, accepted documentation formats, and missing-function reports. Run the app tests with: ```sh xcodebuild -project FoundationEvals/FoundationEvals.xcodeproj \ -scheme FoundationEvals -configuration Debug \ -destination 'platform=macOS' test ``` UI tests require an interactive Mac and a signed test runner. If macOS rejects a command-line UI runner before launch, run the tests directly from Xcode. The optional Core AI inference test requires compatible model resources. GitHub CI classifies the complete pull request or `main` push diff, including deletions and both sides of renames. Linux workflow linting, shell checks, and Python example syntax checks remain lightweight and always run; unit tests and macOS jobs follow the affected code. | Changed files | Unit tests | Native build | |---|---|---| | Docs, README artwork, demo media, GitHub funding or ownership metadata | None | Skipped | | Release workflows and Python/shell tools | Related script modules; signature checks use macOS when affected | Skipped unless build tooling changes | | Portable scoring, installer, or UI component source | Suites that use the changed source, including shared dependencies | App and unit-test bundle; UI-test bundle when UI is affected | | Other app source | No unrelated portable suites; this code needs the native macOS 27 test environment | App and unit-test bundle; UI-test bundle when UI is affected | | Test files | Changed portable/script suites; app-only tests compile in their native target | Relevant native test target, if applicable | | CI routing, package/project settings, release version files, or unknown paths | Full coverage | Full build | Manual runs, malformed events, and unavailable Git history select full coverage. Release PR version changes still require the complete source checks used by installer publication. The **Route changed files** summary lists the selected suites and the reasons for each decision. Empty selections run no unit tests; a selected Swift suite that cannot be discovered fails instead of silently passing with zero tests. The explicit portable-source dependency map lives in `script/ci_routes.py`. Update it when adding a suite or a shared dependency; catalog tests check it against `Package.swift` and the test suite names. The portable package shares production source and existing tests with Xcode without lowering the app’s deployment target. CI uses the hosted `xcode-27` macOS 27 runner and reuses one native build for core tests and up to three UI jobs. Selected script tests run in one Linux job. When simulator acceptance is selected, one job runs all seven scenarios with one fixture and simulator setup. These checks remain automatic and blocking; the ten-repeat picker stability check is opt-in. Run the interactive UI tests above before releasing. See the [release guide](docs/releasing.md) for signed releases through **Release Me** and local packaging. ## License Intents is released under the [MIT License](LICENSE). Dependencies retain their own licenses; see [third-party notices](FoundationEvals/FoundationEvals/Resources/THIRD_PARTY_NOTICES.txt). Separately supplied model resources have their own terms.