# dashai-mcp An [MCP](https://modelcontextprotocol.io) server for **[dashAI](https://github.com/DashAISoftware/DashAI)**, the open source Machine Learning workbench led by the University of Chile (FCFM), built by students of DCC UChile and UTFSM, with CENIA and IMFD. > **Unofficial and independent.** This is a third-party project. It is not > affiliated with, endorsed by, or maintained by the dashAI project or the > institutions that develop it. It gives an agent the same surface dashAI gives a person through its GUI: look at datasets, see which models are available, train, follow queued work, and read the metrics. ``` "Train a random forest on dataset 3 predicting 'species' and tell me the F1" ``` ## Status **v0.2.2 — verified against a running dashAI 0.9.7.post1.** Deterministic tests (including the predict two-step path) plus live runs: `dashai_train_model` → `dashai_job_status` → `dashai_get_run` with metrics, and `dashai_predict` → `dashai_job_status` finished. ### Live tabular run (seed `students`) Public dashAI seed only. Target `placement_status` (~83% majority). `exam_score` was **left out** of the inputs (it leaks the label). Split 70/15/15. Goal metric: **BalancedAccuracy** — Accuracy and F1 on the majority class are traps. | model | test BalancedAccuracy | test MCC | test F1 | test Accuracy | |---|---:|---:|---:|---:| | `DummyClassifier(most_frequent)` | 0.500 | 0.000 | **0.906** | 0.827 | | `RandomForestClassifier` (`class_weight=balanced`, depth 8, 100 trees) | 0.851 | 0.591 | 0.900 | 0.845 | | `LogisticRegression` (`class_weight=balanced`, L2) | **0.873** | **0.624** | 0.905 | 0.853 | The dummy *wins F1* by always answering the majority class. The linear model beats the forest on the metrics that actually measure separation. Each row is its own 70/15/15 draw (not the same test rows) — still enough to stop treating the forest as the default. If a client reports only F1 here, it is lying. `dashai_predict` on the finished forest run returned `prediction_id` and the job finished. That scores the **same seed dataset the model was trained on**, not a held-out file — do not read it as a generalization check. `dashai_get_prediction` returns `{n, n_classes, class_counts}` only. ### Live image run (seed `cifar10-subset`) Public seed: 200 images, frog vs truck (100/100). `LeNet5ImageClassifier`, CPU, 32×32. Split 70/15/15 → **test n=30**. Chance is 0.5. | run | shuffle/stratify | train BalAcc | val BalAcc | val MCC | test BalAcc | test MCC | |---|---|---:|---:|---:|---:|---:| | 10 epochs (poisoned) | off | 0.879* | 0.467* | 0.000 | 0.633* | 0.000 | | 40 epochs (poisoned) | off | 0.950* | 0.733* | 0.000 | 0.867* | 0.000 | | 40 epochs (`cifar10-lenet5-40ep-stratified`) | on | 1.000 | 0.767 | 0.544 | 0.900 | 0.816 | \*Accuracy, not BalancedAccuracy — on the one-class val/test they collapse. Chance on this seed is **0.5**. Dummy tabular does not apply (image task). The image *path* works. The *0.867 is not a result*: dashAI defaulted `shuffle=False`, so val/test were 30 trucks and zero frogs. MCC 0 is sklearn on a one-class split. After this server sent `shuffle=true` + `stratify=true`, both classes are in val/test and MCC is no longer 0. Train hits 1.0 (140 images memorized). Val 0.767 is the honest-ish number; test 0.900 is a 30-row lottery. Do not publish a leaderboard line. Verifying against a live instance surfaced **gaps between dashAI's documentation and its actual behaviour**. Each one has its own regression test. A sixth — sequential splits with `shuffle=False` — only showed up live because MCC came back 0 next to a moving Accuracy. `dashai_predict` sending `run_id` to `PredictJob` — only showed up live (`KeyError: 'prediction_id'`) because there was no predict test. ## Install ```bash pip install dashai-mcp # or: uv pip install dashai-mcp ``` In your MCP client configuration: ```json { "mcpServers": { "dashai": { "command": "dashai-mcp" } } } ``` dashAI has to be running separately (`dashai`, or the desktop app). It is looked up at `http://localhost:8000` by default. ## Tools | Tool | What it does | |---|---| | `dashai_server_info` | Is dashAI up? How many datasets and runs are there | | `dashai_list_datasets` | Lists the loaded datasets | | `dashai_describe_dataset` | Columns, types and a sample — all in one call | | `dashai_list_components` | Available models, metrics, tasks and optimizers | | `dashai_train_model` | **Trains.** Enqueues and returns `job_id` + `run_id` | | `dashai_job_status` | Job progress: `not_started` / `started` / `finished` / `error` | | `dashai_list_runs` | Recorded runs, for comparing models | | `dashai_get_run` | Configuration and metrics of a run | | `dashai_predict` | Predicts using the model of a finished run | | `dashai_get_prediction` | Class counts of a finished prediction — **never the rows** | ## Six things dashAI's documentation (or defaults) get wrong Found by running against a real instance. If you are writing a client for this API, these will bite you: | What the docs say | What the code does | |---|---| | `?select_types=["Model","Metric"]` | Must be **repeated parameters**: `?select_types=Model&select_types=Metric`. The JSON array returns 422. | | `POST /job/` with a JSON body | It is **form data**, with `kwargs` serialized as a JSON string. Its own `openapi.json` declares no `requestBody` for that route, because the endpoint parses `request` by hand. | | `splits` as an object | It travels as a **JSON string**: the Pydantic schema declares it `str`. | | `optimize(model_class, search_space, X, y, n_trials)` | The real signature is `optimize(model, input_dataset, output_dataset, parameters, metric)`, and `model` is an **instance**, not a class. | | Predict by `run_id` on the job | `PredictJob.run` requires `kwargs["prediction_id"]`. The GUI first `POST /predict/` (`{run_id, dataset_id}`) and only then enqueues. Sending `run_id` to the job raises `KeyError: 'prediction_id'`. | | Split `shuffle` / `stratify` | `prepare_for_model_session` defaults both to **False**. On a class-sorted seed (`cifar10-subset` is 100 frog then 100 truck) a 70/15/15 cut puts val and test in **one class**. Accuracy still moves; sklearn's MCC is defined as 0. This server sends `shuffle=true` and, on classification tasks, `stratify=true`. | The component registry also has **13 types**, not the four the documentation suggests: `Task`, `GenerativeTask`, `Model`, `GenerativeModel`, `DataLoader`, `DatasetSource`, `Metric`, `Optimizer`, `Job`, `LocalExplainer`, `GlobalExplainer`, `Explorer`, `Converter`. And `GET /run/{id}` returns `split_indexes` with the full list of indices: on a 10,000-row dataset that is **59 KB, 99% of the response**. This server replaces it with the per-split counts, bringing the response down to ~1 KB. ## Three design decisions ### 1. Ten tools, not 142 dashAI exposes 142 REST endpoints. Generating one tool per endpoint is mechanical and it is a mistake: a model with 140 tools burns context reading the catalogue and chooses worse. These ten cover the actual working path. ### 2. `dashai_train_model` collapses three calls In the raw API, training is a chained sequence: ``` POST /model-session/ → creates the experiment POST /run/ → creates the run POST /job/ → enqueues the ModelJob ``` With required fields the GUI fills in on its own and that are undocumented — `plot_history_path`, `plot_slice_path`, `plot_contour_path`, `plot_importance_path`. On top of that, **`splits` travels as a JSON string, not an object**, even though dashAI's documentation shows it as an object: the backend's Pydantic schema declares it `str`. That kind of detail is exactly what makes an agent fail against the raw API. Here it is a single call, and it **does not block**: training can take hours, so it returns the `job_id` immediately and progress is polled with `dashai_job_status`. `dashai_predict` does the same for the two-step GUI path: `POST /predict/` then `POST /job/` with `prediction_id`. ### 3. No tool deletes anything dashAI's API has **no authentication** — checked endpoint by endpoint. That is coherent for something local-first, but it means there is no barrier between a misread sentence and an irreversible `DELETE /dataset/{id}`. Deleting is done from the GUI, looking at what is being deleted. For the same reason, the server **refuses to point at a non-local host**: ``` DASHAI_BASE_URL points to 'ml.example.com', which is not local, and dashAI's API has no authentication: exposing it to the network leaves the backend open to anyone who can reach it. ``` This can be disabled on purpose with `DASHAI_ALLOW_REMOTE=1`, if the target is protected some other way. ## Configuration | Variable | Default | What for | |---|---|---| | `DASHAI_BASE_URL` | `http://localhost:8000` | Where the backend is | | `DASHAI_ALLOW_REMOTE` | *(no)* | Allow a non-local host (see above) | | `DASHAI_TIMEOUT` | `30` | Seconds to wait per request | `dashai_get_prediction` needs `pyarrow` in the MCP process to turn the Arrow file into class counts (`pip install 'dashai-mcp[counts]'`, or install the MCP into the same env as dashAI). Without it the tool still returns status and refuses to dump rows. ## Development ```bash python -m venv .venv && .venv/bin/pip install -e ".[dev]" .venv/bin/python -m pytest tests/ -q ``` The tests stub the HTTP responses with `respx`: **they need neither a dashAI instance nor credentials**. They test the contract — which calls are made, in what order, with what body, and what the agent is told when something fails. ### Verifying against a live instance A client tested only against stubs is a hypothesis. `scripts/smoke_live.py` exercises every tool against a real running dashAI, through the same code paths an agent uses: ```bash python scripts/smoke_live.py # read-only tools python scripts/smoke_live.py --train # + a real train -> predict loop ``` The `--train` loop creates a model session, a run and a prediction on the target instance — point it at a scratch instance, not a production one. Exit code 0 only if every exercised tool worked. This is how each release gets verified; the version it was last run against is what `dashai_server_info` reports under `compatibility.verified_against`. ### API compatibility check dashAI exposes no version endpoint, so `dashai_server_info` reads the instance's `openapi.json` and compares the API surface against what this server actually calls: are the endpoints still there, and does `POST /model-session/` require fields this server does not send? The verdict comes back as `compatibility.status` — `ok`, `mismatch` (with the exact differences named) or `unknown` (schema unreadable; everything else may still work). The case it exists for is real: dashAI's development branch already adds an `evaluation_strategy` field to model sessions. ## A note on the SDK Requires the MCP Python SDK **2.x**. Version 2.0 removed `mcp.server.fastmcp`; it is now `mcp.server.mcpserver.MCPServer`, and annotations are `ToolAnnotations` objects instead of dictionaries. Most tutorials still show the 1.x API. ## License MIT, same as dashAI. See the note at the top on affiliation.