# AbstractVision — llms-full

> Single-file, agent-oriented context bundle generated from this repository’s docs and metadata.

How to use:
- If you only need a map of where to look, use `llms.txt`.
- If you need an all-in-one context bundle (for offline models or constrained retrieval), use this file.

Notes:
- This is a **convention** (many projects publish an `llms-full.txt`). It is not part of the core `llms.txt` spec.
- Relative links inside each file section are authored for that file’s original location. Use the `FILE: …` marker to interpret link paths.
- If you change docs or packaging metadata, regenerate this file by running:
  - `python scripts/generate_llms_full.py`

Included files:
- `llms.txt`
- `README.md`
- `docs/README.md`
- `docs/getting-started.md`
- `docs/mlx-gen-local-examples.md`
- `docs/api.md`
- `docs/architecture.md`
- `docs/faq.md`
- `docs/reference/backends.md`
- `docs/reference/configuration.md`
- `docs/reference/capabilities-registry.md`
- `docs/adr/README.md`
- `docs/adr/0005_curated_capability_registry_and_download_catalog.md`
- `docs/adr/0006_operator_control_configuration_precedence_and_explicit_network_use.md`
- `docs/reference/artifacts.md`
- `docs/reference/abstractcore-integration.md`
- `playground/README.md`
- `CONTRIBUTING.md`
- `SECURITY.md`
- `ACKNOWLEDGMENTS.md`
- `CHANGELOG.md`
- `pyproject.toml`

--- 8< --- FILE: llms.txt --- 8< ---
# AbstractVision

> Model-agnostic generative vision API (images, optional video) with a capability registry, artifact-ref outputs, and backends for OpenAI-compatible HTTP, Diffusers, stable-diffusion.cpp, and MLX-Gen.

This repository’s current source of truth is the code under `src/abstractvision/` (docs in `docs/`).

Format note: this file follows the `llms.txt` Markdown spec (H1 + optional summary/details + H2 “file list” sections; the `## Optional` section can be skipped when you need a shorter context). Spec: https://llmstxt.org/#format

Maintenance tips:
- Keep link descriptions concise and unambiguous; avoid unexplained jargon.
- Regenerate `llms-full.txt` after doc/packaging changes: `python scripts/generate_llms_full.py`.

Agent quickstart (choose the path that matches your goal):
- **Use the library (Python / CLI)**: start with `README.md` → `docs/getting-started.md` → `docs/api.md` → `docs/reference/backends.md`; shell commands are `abstractvision t2i` / `abstractvision i2i` / `abstractvision upscale` / `abstractvision t2v` / `abstractvision i2v`, and the interactive path is `abstractvision cli` (legacy alias `abstractvision repl`) with `/t2i` / `/i2i` / `/t2v` / `/i2v`.
- **Integrate with AbstractCore/Runtime**: read `docs/reference/abstractcore-integration.md` and `docs/reference/artifacts.md` (includes local model residency control for in-process backends).
- **Inspect provider model catalogs**: use `abstractvision provider-models`, `VisionManager.list_provider_models(...)`, or `llm.vision.list_provider_models(...)`; catalog listing is explicit and does not select the active model.
- **Inspect local model/task metadata**: use `abstractvision catalog`, `abstractvision show-model`, or `GET /v1/vision/models`; runtime-backed local surfaces only expose tasks that the selected backend currently enables.
- **Need the model curation policy**: read `docs/reference/capabilities-registry.md`, `docs/adr/0005_curated_capability_registry_and_download_catalog.md`, and `docs/adr/0006_operator_control_configuration_precedence_and_explicit_network_use.md`.
- **Need a single file**: open `llms-full.txt` (generated bundle of the core docs).
- **Need local Diffusers**: install `abstractvision[diffusers]`, set `ABSTRACTVISION_BACKEND=diffusers`, pre-download `runwayml/stable-diffusion-v1-5`, or set `ABSTRACTVISION_DIFFUSERS_ALLOW_DOWNLOAD=1`.
- **Need local MLX-Gen**: install `abstractvision[mlx-gen]`, choose exact published model ids such as `AbstractFramework/qwen-image-2512-4bit` for `t2i`, `AbstractFramework/qwen-image-edit-2511-4bit` or `briaai/Fibo-Edit` for `i2i`, `AbstractFramework/seedvr2-3b-8bit` / `AbstractFramework/seedvr2-7b-8bit` for SeedVR2 `upscale`, and `AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit` / `AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit` for Wan A14B `t2v`/`i2v`; use typed `guidance_2` / CLI `--guidance-2` for the A14B second-stage guidance control; pre-download with `abstractvision download ... --provider mlx-gen`.

Reality checks (current shipped behavior, anchored in code):
- Built-in backends implement `text_to_image`, `image_to_image`, `image_upscale`, and backend-dependent `text_to_video` / `image_to_video`.
- Local MLX-Gen 0.18.13+ supports curated q4/q8 Apple Silicon image presets, official FIBO image snapshots, FLUX.2/Qwen multi-reference image edits, canonical SeedVR2 q8/q4 image upscaling packages, and Wan 2.2 TI2V plus task-specific Wan 2.2 A14B `text_to_video` / first-frame `image_to_video`.
- Task-specific Wan A14B video declares typed `guidance_2` defaults in `vision_model_capabilities.json` (`3.0` for T2V, `3.5` for I2V); it is not transported through backend `extra`.
- MLX-Gen progress is surfaced through Python/Core `on_progress(event)` callbacks, opt-in one-shot image generation/edit CLI `--progress`, and denoise-step video/upscale progress by default.
- Local Diffusers `text_to_video` remains experimental and is temporarily disabled from the normal local surfaces.
- OpenAI-compatible HTTP can also provide `text_to_video` / `image_to_video` when video endpoints are configured.
- `multi_view_image` exists in the API but no built-in backend implements it yet.
- Provider `/models` catalogs can be listed explicitly, including through the AbstractCore plugin boundary; they are not used for automatic model selection.
- Backend normalization is shared across `VisionManager`, the playground API, and the AbstractCore plugin, so model-specific parameter constraints are enforced once in the backend layer instead of separately in each UI.
- Image dimensions are provider/model-specific. `width`/`height` are optional request overrides; omit them to let the backend use its default or `auto` behavior, and do not assume `512x512` is universally accepted.

## Documentation

- [llms-full.txt](llms-full.txt): single-file bundle of the core docs (for agent ingestion)
- [README.md](README.md): overview, install, quickstart
- [docs/README.md](docs/README.md): docs index (map)
- [docs/getting-started.md](docs/getting-started.md): first image (OpenAI-compatible HTTP / Diffusers / sdcpp) + Playground
- [docs/mlx-gen-local-examples.md](docs/mlx-gen-local-examples.md): MLX-Gen local image/edit/upscale/video commands with bundled generated outputs
- [docs/troubleshooting.md](docs/troubleshooting.md): symptom-oriented setup/runtime fixes for local backends, playground, and integration issues
- [docs/api.md](docs/api.md): public Python API surface
- [docs/architecture.md](docs/architecture.md): how components fit together (with diagrams)
- [docs/faq.md](docs/faq.md): common questions + troubleshooting
- [docs/reference/backends.md](docs/reference/backends.md): backend support matrix + config notes
- [docs/reference/configuration.md](docs/reference/configuration.md): CLI/REPL commands + `ABSTRACTVISION_*` env vars
- [docs/reference/capabilities-registry.md](docs/reference/capabilities-registry.md): capability registry format + usage
- [docs/adr/README.md](docs/adr/README.md): ADR index for durable engineering policy
- [docs/adr/0005_curated_capability_registry_and_download_catalog.md](docs/adr/0005_curated_capability_registry_and_download_catalog.md): packaged capability registry and cache-backed model catalog policy
- [docs/adr/0006_operator_control_configuration_precedence_and_explicit_network_use.md](docs/adr/0006_operator_control_configuration_precedence_and_explicit_network_use.md): explicit runtime selection, cache-only defaults, and network-use policy
- [docs/reference/artifacts.md](docs/reference/artifacts.md): artifact refs + stores
- [docs/reference/abstractcore-integration.md](docs/reference/abstractcore-integration.md): AbstractCore plugin + tool helpers
- [playground/README.md](playground/README.md): self-contained local web UI/API served by `abstractvision playground`
- [CONTRIBUTING.md](CONTRIBUTING.md): dev setup + tests + contribution guidelines
- [SECURITY.md](SECURITY.md): responsible vulnerability reporting
- [ACKNOWLEDGMENTS.md](ACKNOWLEDGMENTS.md): upstream libraries/projects

## AbstractFramework ecosystem

- [AbstractFramework](https://github.com/lpalbou/AbstractFramework): ecosystem hub (how components fit together)
- [AbstractCore](https://github.com/lpalbou/abstractcore): orchestration + tool calling (AbstractVision integrates via plugin/tools)
- [AbstractRuntime](https://github.com/lpalbou/abstractruntime): runtime services (artifact store integration via adapter)

## Code entry points

- [src/abstractvision/vision_manager.py](src/abstractvision/vision_manager.py): `VisionManager` orchestrator API
- [src/abstractvision/types.py](src/abstractvision/types.py): request/response dataclasses (`ImageGenerationRequest`, `ImageUpscaleRequest`, `GeneratedAsset`, …)
- [src/abstractvision/errors.py](src/abstractvision/errors.py): error types (`CapabilityNotSupportedError`, …)
- [src/abstractvision/backends/base_backend.py](src/abstractvision/backends/base_backend.py): `VisionBackend` contract
- [src/abstractvision/backends/__init__.py](src/abstractvision/backends/__init__.py): lazy imports (keeps `import abstractvision` import-light)
- [src/abstractvision/backends/openai_compatible.py](src/abstractvision/backends/openai_compatible.py): OpenAI-compatible HTTP backend (+ optional video)
- [src/abstractvision/backends/huggingface_diffusers.py](src/abstractvision/backends/huggingface_diffusers.py): local Diffusers backend (local images; experimental local video groundwork)
- [src/abstractvision/backends/stable_diffusion_cpp.py](src/abstractvision/backends/stable_diffusion_cpp.py): stable-diffusion.cpp backend (GGUF via `sd-cli` or python bindings)
- [src/abstractvision/model_capabilities.py](src/abstractvision/model_capabilities.py): capability registry loader + validator
- [src/abstractvision/artifacts.py](src/abstractvision/artifacts.py): artifact refs + stores (`LocalAssetStore`, `RuntimeArtifactStoreAdapter`)
- [src/abstractvision/cli.py](src/abstractvision/cli.py): CLI/REPL (`abstractvision`)
- [src/abstractvision/playground_server.py](src/abstractvision/playground_server.py): self-contained playground server and `/v1/vision/*` jobs
- [src/abstractvision/playground/vision_playground.html](src/abstractvision/playground/vision_playground.html): packaged playground UI asset
- [src/abstractvision/integrations/abstractcore_plugin.py](src/abstractvision/integrations/abstractcore_plugin.py): AbstractCore capability plugin entry point and provider catalog shim
- [src/abstractvision/integrations/abstractcore.py](src/abstractvision/integrations/abstractcore.py): AbstractCore tool helpers (`make_vision_tools`)

## Testing

- [Test suite](tests/): run `python -m unittest discover -s tests -p "test_*.py" -q`
- [Changelog](CHANGELOG.md): release notes
- [pyproject.toml](pyproject.toml): dependencies/extras + entry points
- [scripts/generate_llms_full.py](scripts/generate_llms_full.py): regenerate `llms-full.txt`

## Optional

- [Engineering backlog](docs/backlog/README.md): internal design notes + completion reports
--- >8 --- END FILE: llms.txt --- >8 ---

--- 8< --- FILE: README.md --- 8< ---
# AbstractVision

[![PyPI version](https://img.shields.io/pypi/v/abstractvision.svg)](https://pypi.org/project/abstractvision/)
[![CI](https://github.com/lpalbou/AbstractVision/actions/workflows/ci.yml/badge.svg)](https://github.com/lpalbou/AbstractVision/actions/workflows/ci.yml)
[![Tested Python](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2Flpalbou%2FAbstractVision%2Fmain%2F.github%2Fworkflows%2Fci.yml&query=%24.jobs.test.strategy.matrix%5B%22python-version%22%5D&label=tested%20python&color=blue)](https://github.com/lpalbou/AbstractVision/actions/workflows/ci.yml)
[![license](https://img.shields.io/github/license/lpalbou/AbstractVision)](https://github.com/lpalbou/AbstractVision/blob/main/LICENSE)
[![GitHub stars](https://img.shields.io/github/stars/lpalbou/AbstractVision?style=social)](https://github.com/lpalbou/AbstractVision/stargazers)

Model-agnostic generative vision API (images, optional video) for Python and the Abstract* ecosystem.

## What you get

- A small orchestration API: [`VisionManager`](src/abstractvision/vision_manager.py)
- A packaged capability registry (“what models can do”): [`VisionModelCapabilitiesRegistry`](src/abstractvision/model_capabilities.py) backed by [`vision_model_capabilities.json`](src/abstractvision/assets/vision_model_capabilities.json)
- Shared model metadata that now also drives local catalog surfacing and backend request normalization across the CLI, playground, and AbstractCore paths
- Optional artifact-ref outputs (small JSON refs): [`LocalAssetStore`](src/abstractvision/artifacts.py) and [`RuntimeArtifactStoreAdapter`](src/abstractvision/artifacts.py)
- Built-in backends (execution engines): [`src/abstractvision/backends/`](src/abstractvision/backends/)
  - OpenAI-compatible HTTP: [`openai_compatible.py`](src/abstractvision/backends/openai_compatible.py)
  - Local Diffusers: [`huggingface_diffusers.py`](src/abstractvision/backends/huggingface_diffusers.py)
  - Local stable-diffusion.cpp / GGUF: [`stable_diffusion_cpp.py`](src/abstractvision/backends/stable_diffusion_cpp.py)
  - Local MLX-Gen Apple Silicon backend for curated AbstractFramework MLX presets: [`mflux.py`](src/abstractvision/backends/mflux.py)
- CLI for manual testing (`abstractvision cli`, legacy alias: `abstractvision repl`): [`abstractvision`](src/abstractvision/cli.py)
- Self-contained local Playground UI/API: [`playground/vision_playground.html`](playground/vision_playground.html) (docs: [`playground/README.md`](playground/README.md))

## How it fits together (diagram)

```mermaid
flowchart LR
  Caller[Python / CLI / AbstractCore] --> VM[VisionManager]
  VM --> BE[VisionBackend]
  BE --> VM
  VM -->|optional| Store[MediaStore]
  Store --> Ref[Artifact ref dict]
  VM -->|no store| Asset["GeneratedAsset (bytes + mime)"]
```

## Status (current backend support)

- Development status: **Alpha** (0.x). The public API is stable-by-design, but breaking changes may still happen and will be called out in `CHANGELOG.md`.
- Built-in backends implement images (`text_to_image`, `image_to_image`) plus backend-dependent video (`text_to_video`, `image_to_video`).
- Local MLX-Gen supports `text_to_image` for curated FLUX.2, Qwen Image, Z-Image, ERNIE Image Turbo, FIBO, and Bonsai ternary models; supports `image_to_image` for FLUX.2 klein/base, Qwen Image Edit, ERNIE Image Turbo, FIBO, and FIBO Edit models; and supports Wan 2.2 TI2V plus task-specific Wan 2.2 A14B packages for local `text_to_video` and first-frame `image_to_video`.
- Local Diffusers `text_to_video` remains experimental and is temporarily disabled from the normal local runtime surfaces pending [`docs/backlog/planned/0023_local_runtime_capability_quarantine_for_glm_mflux_and_t2v.md`](docs/backlog/planned/0023_local_runtime_capability_quarantine_for_glm_mflux_and_t2v.md).
- Remote `text_to_video` / `image_to_video` are also supported through the OpenAI-compatible backend **when** endpoints are configured.
- `multi_view_image` is part of the public API (`VisionManager.generate_angles`) but no built-in backend implements it yet.

Details: [`docs/reference/backends.md`](docs/reference/backends.md).

## Installation

```bash
pip install abstractvision
```

The base install is lightweight. It includes the shared API, capability
registry, artifact helpers, CLI, AbstractCore plugin entry point, and the
stdlib OpenAI-compatible HTTP backend. Local inference runtimes are explicit
extras.

Optional extras:

| Extra | Use |
|---|---|
| `abstractvision[openai]` | Official OpenAI provider intent marker; no SDK dependency today. |
| `abstractvision[openai-compatible]` | Generic local/remote OpenAI-shaped endpoint intent marker; stdlib-only today. |
| `abstractvision[models]` | Curated Hugging Face download helpers for cache-backed local quantized vision model presets. |
| `abstractvision[diffusers]` | Install Torch/Diffusers and related packages for local Diffusers generation. |
| `abstractvision[huggingface]` | Compatibility alias for callers that still request the historical Diffusers extra. |
| `abstractvision[sdcpp]` | Install `stable-diffusion-cpp-python` for the pip binding fallback. |
| `abstractvision[mlx-gen]` | Install the optional MLX-Gen Apple Silicon image/video runtime. |
| `abstractvision[mflux]` | Compatibility alias for the MLX-Gen Apple Silicon image/video runtime. |
| `abstractvision[local]` | Convenience for both local backend dependency sets, including `diffusers` and `sdcpp`. |
| `abstractvision[all]` | All runtime backend dependencies, without contributor tooling. |
| `abstractvision[apple]` / `abstractvision[all-apple]` | Native macOS Python profile: Diffusers/Torch MPS, stable-diffusion.cpp bindings, and MLX-Gen. |
| `abstractvision[gpu]` | GPU Diffusers/Torch profile. Install a CUDA/ROCm-enabled PyTorch wheel when needed. |
| `abstractvision[all-gpu]` | Full GPU-relevant local vision profile: Diffusers plus stable-diffusion.cpp bindings. |
| `abstractvision[abstractcore]` | Compatibility marker only; AbstractCore is still supplied by the host application. |

`stable-diffusion-cpp-python` is currently constrained below `0.4.6` because
that release's source distribution is missing vendored CMake files required by
native Linux builds.

Contributor-only extras:

| Extra | Use |
|---|---|
| `abstractvision[diffusers-dev]` / `abstractvision[huggingface-dev]` | Looser dependency pins for newer/unreleased Diffusers pipelines; install Diffusers `main` separately if needed. |
| `abstractvision[test]` | Local test dependencies. |
| `abstractvision[docs]` | Documentation build tooling. |
| `abstractvision[dev]` | Full contributor workflow: tests, docs, build, lint, formatting, and pre-commit. Do not use this as an application runtime profile. |

Note (CUDA): on Windows/Linux, `pip install "abstractvision[diffusers]"` may install a CPU-only PyTorch build. If you want to use an NVIDIA GPU, install a CUDA-enabled PyTorch build first (see <https://pytorch.org/get-started/locally/>) and verify `torch.cuda.is_available()` is `True`.

AbstractCore is not installed by AbstractVision. When an AbstractCore application
has AbstractVision installed in the same environment, AbstractCore can discover
the plugin entry point and use the integration modules lazily.

If you hit “missing pipeline class” errors for newer model families, see [`docs/getting-started.md`](docs/getting-started.md). In that case you may need Diffusers from source (`main`):

```bash
pip install -U "abstractvision[diffusers-dev]"
pip install -U "git+https://github.com/huggingface/diffusers@main"
```

For local development from a repo checkout:

```bash
pip install -e ".[dev]"
```

## Usage

Start here:
- Getting started: [`docs/getting-started.md`](docs/getting-started.md)
- FAQ: [`docs/faq.md`](docs/faq.md)
- Troubleshooting: [`docs/troubleshooting.md`](docs/troubleshooting.md)
- API reference: [`docs/api.md`](docs/api.md)
- Architecture: [`docs/architecture.md`](docs/architecture.md)
- Capability registry + catalog policy: [`docs/reference/capabilities-registry.md`](docs/reference/capabilities-registry.md), [`docs/adr/README.md`](docs/adr/README.md)
- Docs index: [`docs/README.md`](docs/README.md)

### First Apple-local MLX model

For Apple Silicon local image generation, prefer the AbstractFramework
MLX-Gen q4 presets first. They are published in the
[AbstractFramework/mlx-gen Hugging Face collection](https://huggingface.co/collections/AbstractFramework/mlx-gen/)
and are the default recommendation for local memory efficiency. q8 variants are
also listed as separate model repos and should be selected explicitly when quality is more important
than memory footprint. Qwen and ERNIE q4 prepared folders can mix q4 and q8
components, but they remain the default prepared choice.

The downloader stores curated presets in the Hugging Face cache by default and
imports older `~/models/<preset>` trees on first use. Generation stays
cache-only unless you explicitly enable runtime downloads.

```bash
pip install "abstractvision[models,mlx-gen]"
abstractvision model-presets
abstractvision catalog --provider mlx-gen
# Tip: `--provider mlx-gen` implies `--target mlx` (you usually set one or the other).
abstractvision download AbstractFramework/flux.2-klein-4b-4bit --provider mlx-gen
abstractvision download AbstractFramework/qwen-image-2512-4bit --provider mlx-gen
abstractvision download AbstractFramework/qwen-image-edit-2511-4bit --provider mlx-gen
abstractvision download AbstractFramework/z-image-turbo-4bit --provider mlx-gen
abstractvision download AbstractFramework/ernie-image-turbo-8bit --provider mlx-gen
abstractvision download briaai/FIBO --provider mlx-gen
abstractvision download briaai/Fibo-lite --provider mlx-gen
abstractvision download briaai/Fibo-Edit --provider mlx-gen
abstractvision download prism-ml/bonsai-image-ternary-4B-mlx-2bit --provider mlx-gen
abstractvision download Wan-AI/Wan2.2-TI2V-5B-Diffusers --provider mlx-gen
abstractvision download AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit --provider mlx-gen
abstractvision download AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit --provider mlx-gen
abstractvision download AbstractFramework/seedvr2-3b-8bit --provider mlx-gen
abstractvision t2i --provider mlx-gen --model AbstractFramework/flux.2-klein-4b-4bit "a product photo of a matte black espresso machine" --steps 4 --guidance-scale 1.0
abstractvision t2i --provider mlx-gen --model AbstractFramework/flux.2-klein-4b-8bit "a product photo of a matte black espresso machine" --steps 4 --guidance-scale 1.0
abstractvision i2i --provider mlx-gen --model AbstractFramework/qwen-image-edit-2511-4bit --image ./input.png "replace the background with a clean white studio setup" --steps 20 --guidance-scale 2.5 --strength 0.75
abstractvision i2i --provider mlx-gen --model AbstractFramework/qwen-image-edit-2511-4bit --image ./input.png --reference-image ./style.png "apply the style reference while preserving the subject" --progress
abstractvision t2i --provider mlx-gen --model briaai/FIBO "a studio product photo of a white ceramic mug with the AbstractFramework logo" --steps 50 --guidance-scale 4.0
abstractvision t2i --provider mlx-gen --model prism-ml/bonsai-image-ternary-4B-mlx-2bit "a bonsai tree in a quiet ceramic studio" --steps 4 --guidance-scale 1.0
abstractvision i2i --provider mlx-gen --model briaai/Fibo-Edit --image ./input.png "remove the background and keep the object edges clean" --steps 20 --guidance-scale 4.0
abstractvision upscale --provider mlx-gen --model AbstractFramework/seedvr2-3b-8bit --image ./input.png --scale 2x --open
abstractvision t2v --provider mlx-gen --model AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit "a red fox walking through a snowy forest, cinematic" --width 432 --height 240 --frames 41 --fps 10 --steps 20 --guidance-scale 4.0 --guidance-2 3.0
abstractvision i2v --provider mlx-gen --model AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit --image ./first-frame.png "slow camera push-in" --width 432 --height 240 --frames 41 --fps 10 --steps 20 --guidance-scale 3.5 --guidance-2 3.5
```

For a complete local MLX-Gen gallery with copy-paste commands and bundled
generated outputs, see [MLX-Gen local examples](docs/mlx-gen-local-examples.md).

Select q4 or q8 by using the exact published model id, for example
`AbstractFramework/flux.2-klein-4b-4bit` or
`AbstractFramework/flux.2-klein-4b-8bit`. Quantization is metadata of the
published model folder, not a generation-time override.

The shipped MLX-Gen backend currently supports curated q4/q8 prepared folders
for `flux2-klein-4b`, `flux2-klein-9b`, `flux2-klein-base-4b`,
`flux2-klein-base-9b`, `qwen-image`, `qwen-image-edit`, `z-image`, and
`z-image-turbo` families, plus the q4/q8 `ernie-image-turbo` prepared folders.
MLX-Gen 0.18.13+ also runs official runtime snapshots such as `briaai/FIBO`,
`briaai/Fibo-lite`, `briaai/Fibo-Edit`, `briaai/Fibo-Edit-RMBG`,
`prism-ml/bonsai-image-ternary-4B-mlx-2bit`, and
`Wan-AI/Wan2.2-TI2V-5B-Diffusers`, and prepared Wan 2.2 A14B video packages
such as `AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit` and
`AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit`. It also supports SeedVR2
single-image upscaling through the official `ByteDance-Seed/SeedVR2-3B` and
`ByteDance-Seed/SeedVR2-7B` bases plus the canonical
`AbstractFramework/seedvr2-{3b,7b}-{8bit,4bit}` prepared packages. Use
`AbstractFramework/seedvr2-3b-8bit` by default, `AbstractFramework/seedvr2-7b-8bit`
when memory allows, and q4 variants only when memory is tight. Use `--scale 2x`
or an integer `--resolution`; `--quantize` is only needed for official/source
weights. Bonsai is a pre-packed
ternary 2-bit checkpoint, not a q4/q8 prepared folder; use its exact repo id
and keep guidance at 1.0. The binary 1-bit Bonsai checkpoint is not surfaced
because stock MLX cannot run it yet. `image_to_image` is implemented for FLUX.2
klein/base, Qwen Image Edit, ERNIE Image Turbo, FIBO, and FIBO Edit models.
FLUX.2 and Qwen Image Edit can accept additional references with
`--reference-image` (or Python `extra={"reference_images": [...]}`) for
multi-reference edits. FIBO Edit snapshots support mask inputs where the runtime
supports them. Edit strength is passed as `strength` and normalized to
MLX-Gen's `image_strength` parameter where the runtime supports it. For video,
prefer the task-specific prepared Wan 2.2 A14B T2V/I2V packages when memory
allows; MLX-Gen also supports the unified TI2V-5B snapshot. Wan A14B dimensions
must be multiples of 16; `432x240` is a valid low-cost check size, while native
quality checks should use the model's recommended larger resolutions. Wan A14B
uses two guidance stages: `guidance_scale` is the primary/high-noise guidance,
and `guidance_2` (`--guidance-2`) is the second-stage/low-noise guidance.
Registry defaults are `4.0` + `3.0` for T2V and `3.5` + `3.5` for I2V; omit
`guidance_2` for single-stage video models.

SeedVR2 model selection:

| Model id | Use when |
| --- | --- |
| `AbstractFramework/seedvr2-3b-8bit` | Default upscaler: lowest-friction q8 package. |
| `AbstractFramework/seedvr2-7b-8bit` | Higher-quality 7B upscaling when memory allows. |
| `AbstractFramework/seedvr2-3b-4bit` | Lower-memory 3B package. |
| `AbstractFramework/seedvr2-7b-4bit` | Lower-memory 7B package. |
| `ByteDance-Seed/SeedVR2-3B` / `ByteDance-Seed/SeedVR2-7B` | Official source weights; pass `--quantize 8` or `--quantize 4` when runtime quantization is needed. |

If you already have a prepared MLX-Gen folder, pass the folder directly as the
model. AbstractVision infers the SeedVR2 base from names such as
`seedvr2-7b-8bit`:

```bash
abstractvision upscale \
  --provider mlx-gen \
  --model /Users/albou/projects/gh/sbx/mlx-gen/models/seedvr2-7b-8bit \
  --image ./input.png \
  --scale 2x \
  --open
```

One-shot `t2i`, `i2i`, `t2v`, and `i2v` commands store results in the local
asset store and print an artifact ref followed by the local content path. Use
`--open` when you want the generated output opened after storage, or
`--store-dir <dir>` to choose the asset store directory.
Image dimensions are provider/model-specific. `width`/`height` are optional
request overrides; when omitted, AbstractVision lets the selected backend use
its own default or `auto` behavior. Do not treat `512x512` as a universal
default: some older/local models accept it, while newer OpenAI-compatible image
models may only accept larger provider-declared sizes such as `1024x1024`,
`1024x1536`, `1536x1024`, or `auto`. If you pass `--width`/`--height`, the
backend may reject unsupported combinations.
MLX-Gen video and upscale commands report denoise-step progress on stderr by
default; video progress also includes frame context. Pass `--no-progress` when
you need quiet output. MLX-Gen image generation/edit commands are quiet by
default and accept `--progress` for step progress.

Stable Diffusion does not currently have a curated MLX-Gen q4/q8 preset in
AbstractVision, so full Diffusers downloads remain explicit.

Install the Diffusers runtime extra, download a Diffusers snapshot, then select
the Diffusers backend explicitly:

```bash
pip install "abstractvision[models,diffusers]"
abstractvision catalog --provider diffusers
# Tip: `--provider diffusers` implies `--target diffusers` (you usually set one or the other).
abstractvision download stable-diffusion --provider diffusers
abstractvision download sd1.4 --provider diffusers
abstractvision download sd1.5-inpaint --provider diffusers
abstractvision download sdxl-base --provider diffusers
abstractvision download sdxl-inpaint --provider diffusers
abstractvision download sd3-medium --provider diffusers
abstractvision download sd3.5-large --provider diffusers
abstractvision download ernie-image --provider diffusers
abstractvision download qwen-image-edit-2511 --provider diffusers
abstractvision download flux2-dev --provider diffusers
export ABSTRACTVISION_BACKEND=diffusers
export ABSTRACTVISION_MODEL_ID=runwayml/stable-diffusion-v1-5
export ABSTRACTVISION_DIFFUSERS_DEVICE=auto
abstractvision cli
```

Notes:
- `abstractvision download qwen-image-edit-2511 --provider diffusers` downloads the curated official 16-bit Diffusers snapshot.
- `GLM-Image` remains in the packaged registry, but local Diffusers `GLM-Image` is temporarily disabled pending the follow-up tracked in [`docs/backlog/planned/0023_local_runtime_capability_quarantine_for_glm_mflux_and_t2v.md`](docs/backlog/planned/0023_local_runtime_capability_quarantine_for_glm_mflux_and_t2v.md).
- `CogVideoX-2b` downloads are still available for experimentation, but local `text_to_video` is currently marked experimental and disabled from the normal product surfaces.

For a fresh cache, you can also permit the interactive CLI to download missing files:

```bash
ABSTRACTVISION_DIFFUSERS_ALLOW_DOWNLOAD=1 abstractvision cli
```

More recommendations by VRAM: [`docs/getting-started.md`](docs/getting-started.md).

### Capability-driven model selection

```python
from abstractvision import VisionModelCapabilitiesRegistry

reg = VisionModelCapabilitiesRegistry()
assert reg.supports("runwayml/stable-diffusion-v1-5", "text_to_image")
assert reg.supports("Qwen/Qwen-Image-Edit-2511", "image_to_image")

print(reg.list_tasks())
print(reg.models_for_task("text_to_image"))
print(reg.models_for_task("image_to_image"))
```

### Backend wiring + generation (artifact outputs)

The base install is import-light and does not install Torch/Diffusers. Heavy
local backend modules are imported lazily (see [`src/abstractvision/backends/__init__.py`](src/abstractvision/backends/__init__.py)).
Install `abstractvision[diffusers]` for local Diffusers, or
`abstractvision[sdcpp]` for the optional stable-diffusion.cpp python binding
fallback.

```python
from abstractvision import LocalAssetStore, VisionManager, VisionModelCapabilitiesRegistry, is_artifact_ref
from abstractvision.backends import OpenAICompatibleBackendConfig, OpenAICompatibleVisionBackend

reg = VisionModelCapabilitiesRegistry()

backend = OpenAICompatibleVisionBackend(
    config=OpenAICompatibleBackendConfig(
        base_url="http://localhost:1234/v1",
        api_key="YOUR_KEY",      # optional for local servers
        model_id="REMOTE_MODEL", # optional (server-dependent)
    )
)

vm = VisionManager(
    backend=backend,
    store=LocalAssetStore(),         # enables artifact-ref outputs
    model_id="Qwen/Qwen-Image-Edit-2511",  # optional: capability gating
    registry=reg,                   # optional: reuse loaded registry
)

out = vm.generate_image("a cinematic photo of a red fox in snow")
assert is_artifact_ref(out)
print(out)  # {"$artifact": "...", "content_type": "...", ...}

png_bytes = vm.store.load_bytes(out["$artifact"])  # type: ignore[union-attr]
```

When installed next to AbstractCore, AbstractVision is also discovered as a
`llm.vision` capability plugin. The plugin defaults to the official OpenAI
image endpoint (`https://api.openai.com/v1`) and reads `OPENAI_API_KEY`.
Set `OPENAI_BASE_URL` when you need a local or remote compatible `/v1` server,
and use the same `OPENAI_API_KEY` bearer token if that endpoint requires auth.
Set `ABSTRACTVISION_BACKEND=openai-compatible` when you want to force
compatible-endpoint semantics. Set `ABSTRACTVISION_MODEL_ID`,
`OPENAI_IMAGE_MODEL_ID`, or `OPENAI_IMAGE_MODEL` when you need an explicit
image model (static default OpenAI model: `gpt-image-1`). AbstractVision does
not query provider `/models` catalogs to discover or select image models
automatically, but you can inspect them explicitly with
`abstractvision provider-models`, `VisionManager.list_provider_models(...)`,
or the AbstractCore plugin method `llm.vision.list_provider_models(...)`.
After inspection, set the model env var explicitly for newer provider models
when available to your account. Set `ABSTRACTVISION_BACKEND=mlx-gen`,
`ABSTRACTVISION_BACKEND=diffusers`, or `ABSTRACTVISION_BACKEND=sdcpp` when you
want AbstractCore to launch local AbstractVision generation directly. For
MLX-Gen, set `ABSTRACTVISION_MFLUX_MODEL=AbstractFramework/flux.2-klein-4b-4bit`
or use routed model ids such as
`mlx-gen/AbstractFramework/flux.2-klein-4b-4bit`. Legacy `mflux` provider values
remain accepted as compatibility aliases.

### Interactive testing (CLI)

```bash
abstractvision models
abstractvision provider-models --openai --task text_to_image
abstractvision provider-models --base-url http://localhost:1234/v1 --task text_to_image
abstractvision tasks
abstractvision show-model runwayml/stable-diffusion-v1-5

abstractvision cli
```

Inside the interactive CLI:

```text
/t2i "a watercolor painting of a lighthouse" --width 512 --height 512 --steps 10 --open
/i2i --image ./input.png "make it watercolor" --steps 20 --guidance-scale 6.5 --open
```

For a newer but still relatively small local model, try `black-forest-labs/FLUX.2-klein-4B` after installing Diffusers
from source (see [`docs/getting-started.md`](docs/getting-started.md)):

```text
/backend diffusers black-forest-labs/FLUX.2-klein-4B mps float16
/t2i "a product photo of a matte black espresso machine" --steps 4 --guidance-scale 1.0 --open
```

Local Diffusers `text_to_video` remains experimental and is temporarily
disabled from the normal bundled local surfaces. Use MLX-Gen Wan 2.2 A14B for
local Apple Silicon `text_to_video` / `image_to_video`, or use the
OpenAI-compatible backend for remote video endpoints.

For Apple Silicon local generation through MLX-Gen:

```text
/backend mlx-gen AbstractFramework/flux.2-klein-4b-4bit
/t2i "a product photo of a matte black espresso machine" --steps 4 --guidance-scale 1.0 --open
/backend mlx-gen prism-ml/bonsai-image-ternary-4B-mlx-2bit
/t2i "a bonsai tree in a quiet ceramic studio" --steps 4 --guidance-scale 1.0 --open
/backend mlx-gen AbstractFramework/qwen-image-edit-2511-4bit
/i2i --image ./input.png "replace the background with a clean white studio setup" --steps 20 --guidance-scale 2.5 --strength 0.75 --open
/backend mlx-gen AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit
/t2v "a red fox walking through a snowy forest, cinematic" --width 432 --height 240 --frames 41 --fps 10 --steps 20 --guidance-scale 4.0 --guidance-2 3.0 --open
/backend mlx-gen AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit
/i2v --image ./first-frame.png "slow camera push-in" --width 432 --height 240 --frames 41 --fps 10 --steps 20 --guidance-scale 3.5 --guidance-2 3.5 --open
```

`/t2v` and `/i2v` show MLX-Gen denoise-step video progress by default; add
`--no-progress` to suppress progress output in scripts.

OpenAI-compatible server example:

```text
/backend openai http://localhost:1234/v1
/t2i "a watercolor painting of a lighthouse" --width 512 --height 512 --steps 10 --open
```

The CLI/REPL can also be configured via `ABSTRACTVISION_*` env vars; see [`docs/reference/configuration.md`](docs/reference/configuration.md).

### Local web playground

The playground is owned by AbstractVision and runs without AbstractCore. It is
a local/dev testing surface; use AbstractCore/Gateway for production routing,
authentication, and browser-origin policy.

```bash
abstractvision playground --port 8091
```

Open `http://127.0.0.1:8091/vision_playground.html`. The page and the API are served by the same process.

Current behavior:
- The UI is split into task tabs (`Text→Image`, `Image→Image`, `Text→Video`, and a placeholder `Image→Video` tab for later work).
- Each active task tab has its own model selector and unload button. Switching models in a tab unloads the current active backend first to free memory before loading the replacement.
- The Image→Image tab is enabled only for models that both advertise `image_to_image` in the packaged capability registry and remain enabled by the selected backend.
- MLX-Gen FLUX.2 klein/base, Qwen Image Edit, ERNIE Image Turbo, FIBO, and FIBO Edit models are surfaced for `Image→Image` edits when cached.
- The Text→Video tab is experimental in the playground UI. Use shell/REPL or AbstractCore for the most complete MLX-Gen Wan video controls.
- Model-specific request normalization happens at the API/backend layer, not just in the page.
- Local video export packages generated frames into MP4 via an external `ffmpeg` binary on `PATH`.
- Response logs intentionally show only a shortened `b64_json` preview instead of the full base64 image payload.

One-shot commands default to the OpenAI-compatible HTTP backend, but they also support local providers:

```bash
abstractvision t2i --base-url http://localhost:1234/v1 "a studio photo of an espresso machine"
abstractvision i2i --base-url http://localhost:1234/v1 --image ./input.png "make it watercolor"
abstractvision t2i --provider diffusers --model qwen-image "a studio photo of an espresso machine"
```

#### Local GGUF via stable-diffusion.cpp

If you want to run GGUF diffusion models locally, use the stable-diffusion.cpp backend (`sdcpp`). Start with a
single-file Stable Diffusion model when possible; Qwen Image and FLUX GGUF component sets are heavier.

Recommended:
- `abstractvision` auto-installs `sd-cli` into `~/.abstractvision/bin` on first use (set `ABSTRACTVISION_SDCPP_AUTO_INSTALL=0` to disable).
- If you prefer python bindings: install `abstractvision[sdcpp]` (uses `stable-diffusion-cpp-python`).

Alternative (external executable): install `sd-cli` from <https://github.com/leejet/stable-diffusion.cpp/releases>.

In the REPL:

```text
/backend sdcpp /path/to/sd-v1-5.gguf /path/to/sd-cli
/t2i "a watercolor painting of a lighthouse" --width 512 --height 512 --steps 10 --open
```

Curated FLUX/Qwen GGUF bundle example:

```bash
abstractvision download flux2-klein-base-4b --provider sdcpp
abstractvision download qwen-image-edit-2511-gguf --provider sdcpp
```

```text
/backend sdcpp flux2-klein-base-4b /path/to/sd-cli
/t2i "a product photo of a matte black espresso machine" --steps 4 --guidance-scale 1.0 --sampling-method euler --diffusion-fa --offload-to-cpu --open
```

The package resolves the required VAE and text-encoder companions from the cache automatically for curated `sdcpp`
model keys. Manual component wiring remains available for advanced cases.

Extra flags are forwarded via `request.extra`. In CLI mode they are forwarded to `sd-cli`; in python bindings mode, keys are mapped to python binding kwargs when supported and unsupported keys are ignored.

### AbstractCore tool integration (artifact refs)

If you’re using AbstractCore tool calling, AbstractVision can expose vision tasks as tools:

```python
from abstractvision.integrations.abstractcore import make_vision_tools

tools = make_vision_tools(vision_manager=vm, model_id="Qwen/Qwen-Image-Edit-2511")
```

Install `abstractcore` in the host application environment when you use these helpers; it is not pulled in by AbstractVision.

## AbstractFramework ecosystem

AbstractVision is part of the **AbstractFramework** ecosystem and is designed to compose with:

- **AbstractFramework** (project hub): <https://github.com/lpalbou/AbstractFramework>
- **AbstractCore** (orchestration + tool calling): <https://github.com/lpalbou/abstractcore>
- **AbstractRuntime** (runtime services, including artifact storage): <https://github.com/lpalbou/abstractruntime>

In practice:
- AbstractVision standardizes *generative vision outputs* (image/video) behind `VisionManager`.
- AbstractCore can discover and use AbstractVision via the capability plugin (`src/abstractvision/integrations/abstractcore_plugin.py`) or you can expose vision tasks as tools (`src/abstractvision/integrations/abstractcore.py`).
- Artifact refs returned by AbstractVision are designed to travel across processes; `RuntimeArtifactStoreAdapter` bridges to an AbstractRuntime-style artifact store (`src/abstractvision/artifacts.py`).

## Project

- Release notes: [`CHANGELOG.md`](CHANGELOG.md)
- Contributing: [`CONTRIBUTING.md`](CONTRIBUTING.md)
- Security: [`SECURITY.md`](SECURITY.md)
- Acknowledgments: [`ACKNOWLEDGMENTS.md`](ACKNOWLEDGMENTS.md)
- Agent docs: [`llms.txt`](llms.txt) and [`llms-full.txt`](llms-full.txt)

## Requirements

- Python >= 3.9

## License

MIT License - see LICENSE file for details.

## Author

Laurent-Philippe Albou

## Contact

contact@abstractcore.ai
--- >8 --- END FILE: README.md --- >8 ---

--- 8< --- FILE: docs/README.md --- 8< ---
# AbstractVision documentation

This folder contains the user-facing documentation for `abstractvision`.

## Start here (new users)

1) [Project overview + quickstart](../README.md)  
2) [Getting started](getting-started.md) (shell and interactive `t2i`/`i2i`; then Diffusers, MLX-Gen, GGUF, OpenAI-compatible HTTP, Playground)
3) [Architecture](architecture.md) (how the pieces fit together)

## Quick reference

- [FAQ](faq.md)
- [Troubleshooting](troubleshooting.md)
- [API reference](api.md)
- [Architecture decisions](adr/README.md)
- [Backends](reference/backends.md)
- [MLX-Gen local examples](mlx-gen-local-examples.md)
- [Configuration (CLI/REPL env vars + flags)](reference/configuration.md)
- [Capability registry (`vision_model_capabilities.json`)](reference/capabilities-registry.md)
- [Artifacts (artifact refs + stores)](reference/artifacts.md)
- [AbstractCore integration (capability plugin + tools)](reference/abstractcore-integration.md)
- Agent-oriented docs: [`../llms.txt`](../llms.txt) and [`../llms-full.txt`](../llms-full.txt)

## AbstractFramework ecosystem

AbstractVision is part of the **AbstractFramework** ecosystem and is designed to compose with:

- **AbstractFramework** (project hub): <https://github.com/lpalbou/AbstractFramework>
- **AbstractCore** (orchestration + tool calling): <https://github.com/lpalbou/abstractcore>
- **AbstractRuntime** (runtime services, including artifact storage): <https://github.com/lpalbou/abstractruntime>

## Current implementation status (as shipped)

Public API surface: [`VisionManager`](../src/abstractvision/vision_manager.py) exposes:
- `generate_image` (`text_to_image`), `edit_image` (`image_to_image`)
- `generate_video` (`text_to_video`), `image_to_video` (`image_to_video`) (backend-dependent)
- `generate_angles` (`multi_view_image`) (API exists; no built-in backend implements it yet)

Built-in backends implement:
- **Images**: Diffusers, stable-diffusion.cpp, MLX-Gen, OpenAI-compatible HTTP ([`../src/abstractvision/backends/`](../src/abstractvision/backends/))
- **Current local policy**: MLX-Gen supports curated q4/q8 Apple Silicon image presets, official FIBO image snapshots, and Wan 2.2 TI2V/A14B video; local Diffusers `text_to_video` is experimental and temporarily disabled from normal local surfaces.
- **Video**:
  - MLX-Gen for Wan 2.2 local `text_to_video` and first-frame `image_to_video`
  - OpenAI-compatible HTTP for optional `text_to_video` / `image_to_video` when endpoints are configured ([`openai_compatible.py`](../src/abstractvision/backends/openai_compatible.py))

If you’re looking for “what can model X do?”, the single source of truth is the packaged registry:
[`../src/abstractvision/assets/vision_model_capabilities.json`](../src/abstractvision/assets/vision_model_capabilities.json) (loaded by `VisionModelCapabilitiesRegistry` in [`../src/abstractvision/model_capabilities.py`](../src/abstractvision/model_capabilities.py)).
The curation and cross-platform support policy for that registry is governed by
[ADR 0005](adr/0005_curated_capability_registry_and_download_catalog.md).

## Internal engineering notes

[`docs/adr/`](adr/) contains durable engineering policy.

[`docs/backlog/`](backlog/) is an internal log (planned work + completion reports). It is not the normative user documentation surface.

## Project

- Release notes: [`CHANGELOG.md`](../CHANGELOG.md)
- Contributing: [`CONTRIBUTING.md`](../CONTRIBUTING.md)
- Security: [`SECURITY.md`](../SECURITY.md)
- License: [`LICENSE`](../LICENSE)
- Acknowledgments: [`ACKNOWLEDGMENTS.md`](../ACKNOWLEDGMENTS.md)
--- >8 --- END FILE: docs/README.md --- >8 ---

--- 8< --- FILE: docs/getting-started.md --- 8< ---
# Getting Started

This guide helps you generate your first image using AbstractVision with the built-in backends:

- **OpenAI-compatible HTTP**: call a local/remote server that exposes OpenAI-shaped image endpoints
- **Diffusers (local Python)**: Stable Diffusion / Qwen Image / FLUX 2 / other supported Diffusers pipelines
- **MLX-Gen (local Apple Silicon)**: q4/q8 AbstractFramework MLX-optimized image generation via the optional MLX-Gen 0.18.13+ runtime, official FIBO image models, canonical SeedVR2 image upscaling packages, and Wan 2.2 A14B `text_to_video` / first-frame `image_to_video`
- **stable-diffusion.cpp (local GGUF)**: GGUF diffusion models via `sd-cli` (recommended for GPU backends like **Metal**/**CUDA**) or via pip-installable python bindings (often **CPU-only** fallback)
- **Playground (web, optional)**: self-contained AbstractVision UI/API for local model loading and jobs (`/v1/vision/*`)

See also:
- Docs index: [docs/README.md](README.md)
- FAQ: [docs/faq.md](faq.md)
- Troubleshooting: [docs/troubleshooting.md](troubleshooting.md)
- API reference: [docs/api.md](api.md)
- Architecture: [docs/architecture.md](architecture.md)
- Backends: [docs/reference/backends.md](reference/backends.md)
- Configuration (CLI/REPL env vars): [docs/reference/configuration.md](reference/configuration.md)
- Capability registry: [docs/reference/capabilities-registry.md](reference/capabilities-registry.md)
- Artifacts: [docs/reference/artifacts.md](reference/artifacts.md)
- AbstractCore integration: [docs/reference/abstractcore-integration.md](reference/abstractcore-integration.md)

---

## 0) Install

From PyPI:

```bash
pip install abstractvision
```

AbstractVision’s base install is lightweight. It includes the shared API, capability registry, artifact helpers, CLI, AbstractCore plugin entry point, and stdlib OpenAI-compatible HTTP backend. Local inference runtimes are explicit extras: install `abstractvision[diffusers]` for Torch/Diffusers, `abstractvision[sdcpp]` for the stable-diffusion.cpp python binding fallback, `abstractvision[mlx-gen]` for Apple Silicon MLX-Gen, or `abstractvision[all-apple]` for the full native macOS stack. `abstractvision[mflux]` remains available as a compatibility alias for older install instructions.

If you see “missing pipeline class” errors for newer model families, install the `diffusers-dev` extra (or compatibility alias `huggingface-dev`) to get compatible dependencies, then install Diffusers from source (`main`).

For that newer-pipeline workflow from a **repo checkout**, install the `diffusers-dev` extra (compatible deps; does not include Diffusers `main`):

```bash
pip install -e ".[diffusers-dev]"
```

If you're installing **AbstractVision from PyPI**, you can install the extra directly:

```bash
pip install -U "abstractvision[diffusers-dev]"
```

Or install Diffusers from source directly:

```bash
pip install -U "git+https://github.com/huggingface/diffusers@main"
```

Sanity check:

```bash
python -c "import diffusers; print(diffusers.__version__)"
python -c "import diffusers; print('GlmImagePipeline', hasattr(diffusers, 'GlmImagePipeline')); print('Flux2KleinPipeline', hasattr(diffusers, 'Flux2KleinPipeline'))"
```

Offline alternative (if you already have a local Diffusers checkout):

```bash
pip install -U -e /path/to/diffusers
```

Or, from a repo checkout (run in the repo root):

```bash
pip install -e .
```

For contributor tooling from a repo checkout, use:

```bash
pip install -e ".[dev]"
```

For local Diffusers generation, install `abstractvision[diffusers]` before selecting the `diffusers` backend. Use `diffusers-dev` only when you need newer Diffusers-compatible dependency pins, and use `sdcpp` only when you want the optional stable-diffusion.cpp python binding fallback.

Optional extras:

| Extra | Use |
|---|---|
| `openai` | Empty official OpenAI provider intent marker; the HTTP backend is stdlib-only today. |
| `openai-compatible` | Empty local/remote OpenAI-shaped endpoint intent marker; the HTTP backend is stdlib-only today. |
| `diffusers` | Installs Torch/Diffusers and related packages for local Diffusers generation. |
| `sdcpp` | Installs `stable-diffusion-cpp-python` for the stable-diffusion.cpp pip binding fallback. |
| `mlx-gen` | Installs the optional MLX-Gen runtime for Apple Silicon MLX image/video generation. |
| `mflux` | Compatibility alias for the MLX-Gen runtime. |
| `apple` | Native macOS profile: Diffusers/Torch MPS, stable-diffusion.cpp bindings, and MLX-Gen. |
| `gpu` | GPU-friendly profile for Diffusers/Torch (does not include MLX-Gen). |
| `huggingface` | Compatibility alias for the historical Diffusers backend dependency set. |
| `local` | Convenience extra for both local backend dependency sets, including `sdcpp`. |
| `all` | All runtime backend dependencies, without contributor tooling. |
| `all-apple` | Aggregate native macOS profile: Diffusers/Torch MPS, stable-diffusion.cpp, and MLX-Gen. |
| `all-gpu` | Aggregate GPU profile (Diffusers + stable-diffusion.cpp bindings). |
| `abstractcore` | Empty compatibility marker; install AbstractCore in the host application environment. |

Contributor-only extras:

| Extra | Use |
|---|---|
| `diffusers-dev` / `huggingface-dev` | Looser dependency pins for newer/unreleased Diffusers pipelines. Install Diffusers `main` separately when a pipeline is not in the latest release. |
| `test` | Local test dependencies. |
| `docs` | Documentation build tooling. |
| `dev` | Full contributor workflow: tests, docs, packaging, formatting, release checks, and pre-commit. Do not use this as an application runtime profile. |

Optional (recommended): pre-download heavyweight model sets (so first-run doesn’t do surprise multi‑GB downloads):

```bash
python scripts/download_model_sets.py --list
python scripts/download_model_sets.py --plan --set sd15_diffusers
python scripts/download_model_sets.py --plan --set flux2_klein_4b_gguf
python scripts/download_model_sets.py --set sd15_diffusers
```

### 0.1 Hardware quickstart (macOS Metal vs NVIDIA CUDA vs CPU)

AbstractVision can run “locally” via three main routes:

- **Diffusers backend**: uses Torch device selection (`cuda` / `mps` / `cpu`).
- **MLX-Gen backend (`mlx-gen`)**: Apple Silicon MLX generation through the optional MLX-Gen runtime. q4 AbstractFramework model repos are the default recommendation; q8 variants are separate exact model ids for quality-focused runs.
- **stable-diffusion.cpp backend (`sdcpp`)**: runs GGUF diffusion models using:
  - `sd-cli` (**recommended** when you want GPU backends like **Metal** or **CUDA**)
  - or `stable-diffusion-cpp-python` (convenient, but often **CPU-only**, especially on macOS)

#### macOS (Apple Silicon, Metal)

- **Diffusers**: start with Stable Diffusion 1.5, then move up:
  - `/backend diffusers runwayml/stable-diffusion-v1-5 mps float16`
  - `/backend diffusers black-forest-labs/FLUX.2-klein-4B mps float16` (requires Diffusers `main` today)
- **GGUF (`sdcpp`)**: install `sd-cli` from stable-diffusion.cpp releases and use **CLI mode** for Metal speed:
  - Download: <https://github.com/leejet/stable-diffusion.cpp/releases>
  - Pick the Darwin arm64 zip (example asset name: `sd-…-bin-Darwin-macOS-…-arm64.zip`)
  - If macOS blocks execution, clear quarantine: `xattr -dr com.apple.quarantine /path/to/sd-cli`
  - In the REPL, pass the full path as the last arg to `/backend sdcpp …` (see section **6)**).

If you see `Using CPU backend` in logs, you’re on CPU (it will work, but can be extremely slow for large models).

#### NVIDIA (CUDA)

- Install a CUDA-enabled PyTorch wheel first (see <https://pytorch.org/get-started/locally/>).
- Use Diffusers with `cuda` + `float16`:
  - `/backend diffusers runwayml/stable-diffusion-v1-5 cuda float16`
- For GGUF (`sdcpp`) on NVIDIA, use an `sd-cli` build compiled with CUDA (stable-diffusion.cpp releases provide multiple assets depending on tag).

#### CPU-only

- Expect slow inference. Prefer smaller models and lower resolutions/steps.
- `sdcpp` via python bindings is the simplest “no external binary” option, but it will use whatever backend the wheel was compiled with (often CPU).

---

## Recommended default models (VRAM guide)

If you run **locally** (Diffusers backend) and want a reliable starting point, here are practical model picks from the packaged capability registry (`src/abstractvision/assets/vision_model_capabilities.json`).

Notes:
- VRAM needs vary with resolution, dtype, and pipeline implementation. Treat this as a starting point.
- Some models are **gated** on Hugging Face and require accepting terms + setting `HF_TOKEN`.
- If you want a non-gated modern image model, try `black-forest-labs/FLUX.2-klein-4B` (but it currently requires installing Diffusers from source; see the FLUX section below).

| GPU VRAM | Recommended model id | Why | Install / quickstart |
|---:|---|---|---|
| ≤ 16 GB | `runwayml/stable-diffusion-v1-5` | Small, stable, and widely compatible (Windows/Linux CUDA, macOS MPS) | `pip install "abstractvision[diffusers]"` then run the REPL using the snippet below |
| 24-32 GB | `black-forest-labs/FLUX.2-klein-4B` | Newer non-gated model, much smaller than FLUX.2-dev | Install Diffusers `main`, then use the FLUX.2 klein section below |
| 32 GB | `stabilityai/stable-diffusion-3.5-large-turbo` | High-quality still images with low step counts (gated) | Accept model terms on HF, set `HF_TOKEN`, then use the SD3.5 section below |
| 64 GB | `Qwen/Qwen-Image-2512` | Strong prompt following and text rendering (large model) | Same as Diffusers setup; if pipeline import fails, use Diffusers `main` (see install section above) |
| 128 GB | `black-forest-labs/FLUX.2-dev` | Very high quality (very large; non-commercial license; gated) | Accept model terms on HF, set `HF_TOKEN`, then use the FLUX section below |

macOS Metal (Apple Silicon) quick picks:

- If you want **local quantized FLUX.2** on Metal: start with `AbstractFramework/flux.2-klein-4b-4bit`, then use `AbstractFramework/flux.2-klein-4b-8bit` when quality is more important than memory.
- If you want a fast local FLUX.2 for iteration: `AbstractFramework/flux.2-klein-4b-4bit` through MLX-Gen is usually the most practical Apple Silicon starting point.
- If you want strong prompt following + text rendering: use exact MLX-Gen ids such as `AbstractFramework/qwen-image-2512-4bit` or `AbstractFramework/ernie-image-turbo-4bit`, select the matching `-8bit` repo when quality is more important than memory, or use `Qwen/Qwen-Image-2512` / `baidu/ERNIE-Image-Turbo` through Diffusers on `mps` when you want the full Diffusers path.

Recommended default (local, cross-platform) — Stable Diffusion 1.5:

```bash
pip install "abstractvision[diffusers]"
abstractvision download stable-diffusion --provider diffusers
export ABSTRACTVISION_BACKEND=diffusers
export ABSTRACTVISION_MODEL_ID=runwayml/stable-diffusion-v1-5
export ABSTRACTVISION_DIFFUSERS_DEVICE=auto
abstractvision cli
```

Then type a prompt (plain text runs `/t2i`), or use `/t2i "..." --open`.

Jump to detailed recipes:
- Stable Diffusion 1.5: section **1) First local image (Diffusers)**
- FLUX.2-klein-4B: section **2) Next small model (FLUX.2-klein-4B)**
- OpenAI-compatible HTTP: section **2.1) OpenAI-compatible HTTP**
- Apple Silicon MLX-Gen: section **2.2) Apple Silicon MLX-Gen (q4 first)**
- Qwen Image: section **3) Qwen Image (Diffusers)**
- FLUX 2 details: section **4) FLUX 2 (Diffusers)**
- SD3.5: section **5) Stable Diffusion 3.5 (Diffusers, gated)**

---

## 1) First local image (Diffusers)

The REPL is cache-only by default, so it will not download model weights. Download the model separately first:

```bash
abstractvision download stable-diffusion --provider diffusers
```

```bash
# Required for this local Diffusers recipe.
export ABSTRACTVISION_BACKEND=diffusers
export ABSTRACTVISION_MODEL_ID=runwayml/stable-diffusion-v1-5
export ABSTRACTVISION_DIFFUSERS_DEVICE=auto
# auto prefers cuda, then mps, then cpu. You can also set cuda/mps/cpu explicitly.
# Optional: override dtype (auto defaults to float16 on MPS for broad compatibility).
# - `float16` is usually the best speed/compatibility tradeoff on Apple Silicon
# - `bfloat16` can work for some models, but can trigger dtype-mismatch errors in some pipelines
# - `float32` is the most stable, but can require much more memory
# export ABSTRACTVISION_DIFFUSERS_TORCH_DTYPE=bfloat16
# export ABSTRACTVISION_DIFFUSERS_TORCH_DTYPE=float16
# export ABSTRACTVISION_DIFFUSERS_TORCH_DTYPE=float32
```

Quick sanity check (device):

```bash
python -c "import torch; print('mps', torch.backends.mps.is_available(), 'cuda', torch.cuda.is_available())"
```

If you have an NVIDIA GPU but `cuda` is `False`, you likely installed a CPU-only PyTorch build. Follow the PyTorch install guide to install a CUDA-enabled wheel, then re-run the sanity check: <https://pytorch.org/get-started/locally/>.

Start the interactive CLI (`abstractvision repl` remains an alias):

```bash
abstractvision cli
```

With `ABSTRACTVISION_BACKEND=diffusers` and `ABSTRACTVISION_MODEL_ID` set above, the REPL uses `runwayml/stable-diffusion-v1-5`:

```text
/set guidance_scale 7
/set seed 42
/t2i "a cinematic photo of a red fox in snow" --width 512 --height 512 --steps 10 --open
```

Change settings by changing `/set …` values, or pass flags per request:

```text
/t2i "a watercolor painting of a lighthouse" --width 512 --height 512 --steps 20 --seed 123 --guidance-scale 6.5 --open
```

---

## 2) Next small model (FLUX.2-klein-4B)

After Stable Diffusion 1.5 works, `black-forest-labs/FLUX.2-klein-4B` is the next recommended local test. It is
non-gated and much smaller than FLUX.2-dev, but it currently needs Diffusers from source because released Diffusers
may not include `Flux2KleinPipeline`.

```bash
pip install -U "abstractvision[diffusers-dev]"
pip install -U "git+https://github.com/huggingface/diffusers@main"
```

Quick REPL test:

```text
/backend diffusers black-forest-labs/FLUX.2-klein-4B mps float16
/t2i "a product photo of a matte black espresso machine" --width 1024 --height 1024 --steps 4 --guidance-scale 1.0 --open
```

Use `cuda float16` on NVIDIA, or `auto` if you want AbstractVision/Torch to pick the device.

---

## 2.1) OpenAI-compatible HTTP

Use this path if you already have a server that exposes OpenAI-shaped image endpoints (e.g. a local model server).

For unknown or local OpenAI-compatible servers, AbstractVision forwards local extension fields such as `steps`, `seed`, `guidance_scale`, `width`, and `height`. For the real OpenAI API and known GPT image models, it suppresses unsupported local-only fields and sends the narrower OpenAI request shape.

List provider-advertised models explicitly:

```bash
abstractvision provider-models --openai --task text_to_image
abstractvision provider-models --base-url http://localhost:1234/v1 --task text_to_image
```

One-shot (stores output via `LocalAssetStore` and prints an artifact ref + file path):

```bash
abstractvision t2i --base-url http://localhost:1234/v1 "a watercolor painting of a lighthouse" --width 512 --height 512 --steps 10 --open
```

Interactive CLI/REPL:

```bash
abstractvision cli
```

```text
/backend openai http://localhost:1234/v1
/t2i "a watercolor painting of a lighthouse" --width 512 --height 512 --steps 10 --open
```

If your server also supports video endpoints, configure them via `ABSTRACTVISION_TEXT_TO_VIDEO_PATH` / `ABSTRACTVISION_IMAGE_TO_VIDEO_PATH` (see [docs/reference/configuration.md](reference/configuration.md)).

---

## 2.2) Apple Silicon MLX-Gen (q4 first)

Use this path on Apple Silicon when you want local MLX-optimized image/video models
without running a separate server. AbstractVision uses the `mlx-gen` Python API
in-process and expects prepared model folders to exist in the Hugging Face
cache. It does not silently download weights during generation.

```bash
pip install "abstractvision[models,mlx-gen]"
abstractvision catalog --provider mlx-gen
abstractvision download AbstractFramework/flux.2-klein-4b-4bit --provider mlx-gen
abstractvision download AbstractFramework/qwen-image-2512-4bit --provider mlx-gen
abstractvision download AbstractFramework/qwen-image-edit-2511-4bit --provider mlx-gen
abstractvision download AbstractFramework/ernie-image-turbo-4bit --provider mlx-gen
abstractvision download AbstractFramework/ernie-image-turbo-8bit --provider mlx-gen
abstractvision download briaai/FIBO --provider mlx-gen
abstractvision download briaai/Fibo-lite --provider mlx-gen
abstractvision download briaai/Fibo-Edit --provider mlx-gen
abstractvision download prism-ml/bonsai-image-ternary-4B-mlx-2bit --provider mlx-gen
abstractvision download Wan-AI/Wan2.2-TI2V-5B-Diffusers --provider mlx-gen
abstractvision download AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit --provider mlx-gen
abstractvision download AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit --provider mlx-gen
abstractvision download AbstractFramework/seedvr2-3b-8bit --provider mlx-gen
```

The default prepared choices are q4 repos from the
[AbstractFramework/mlx-gen Hugging Face collection](https://huggingface.co/collections/AbstractFramework/mlx-gen/).
Select q8 variants by passing the exact `AbstractFramework/...-8bit` model id
when quality is paramount and memory permits it. Quantization is part of the
published model folder, not a generation-time parameter.
Qwen and ERNIE q4 folders can mix q4 and q8 components, but remain the default
prepared choice.
Bonsai ternary is different: `prism-ml/bonsai-image-ternary-4B-mlx-2bit` is a
pre-packed MLX checkpoint consumed directly by MLX-Gen. Use the exact repo id,
keep guidance at 1.0, and do not use it for image-to-image.
SeedVR2 uses the official `ByteDance-Seed/SeedVR2-3B` and
`ByteDance-Seed/SeedVR2-7B` bases plus canonical prepared packages:
`AbstractFramework/seedvr2-3b-8bit` is the default upscaler,
`AbstractFramework/seedvr2-7b-8bit` is the larger q8 variant, and q4 variants
are available for tighter memory. Use `--scale 2x` for normal upscaling or
`--resolution <pixels>` for a target short edge. Runtime `--quantize` is only
needed for official/source weights.

One-shot shell commands store the output in the local asset store and print the
artifact ref followed by the local content path. Add `--open` to open the output
after generation, or `--store-dir <dir>` to select the store directory.

Text-to-image from the shell:

```bash
abstractvision t2i --provider mlx-gen --model AbstractFramework/qwen-image-2512-4bit "a studio product photo of a white ceramic mug with the AbstractFramework logo" --steps 20 --guidance-scale 1.0 --open
abstractvision t2i --provider mlx-gen --model prism-ml/bonsai-image-ternary-4B-mlx-2bit "a bonsai tree in a quiet ceramic studio" --steps 4 --guidance-scale 1.0 --open
```

Image-to-image/edit from the shell:

```bash
abstractvision i2i --provider mlx-gen --model AbstractFramework/qwen-image-edit-2511-4bit --image ./input.png "replace the background with a clean white studio setup" --steps 20 --guidance-scale 2.5 --strength 0.75 --open
abstractvision i2i --provider mlx-gen --model briaai/Fibo-Edit --image ./input.png "remove the background and keep the object edges clean" --steps 20 --guidance-scale 4.0 --open
```

Image upscaling from the shell:

```bash
abstractvision upscale --provider mlx-gen --model AbstractFramework/seedvr2-3b-8bit --image ./input.png --scale 2x --open
```

Text-to-video and first-frame image-to-video from the shell:

```bash
abstractvision t2v --provider mlx-gen --model AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit "a red fox walking through a snowy forest, cinematic" --width 432 --height 240 --frames 41 --fps 10 --steps 20 --guidance-scale 4.0 --guidance-2 3.0 --open
abstractvision i2v --provider mlx-gen --model AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit --image ./first-frame.png "slow camera push-in" --width 432 --height 240 --frames 41 --fps 10 --steps 20 --guidance-scale 3.5 --guidance-2 3.5 --open
```

Wan 2.2 A14B uses 16px width/height multiples; `432x240` is valid for low-cost
local checks, while larger native sizes are more appropriate for quality review.

Video examples (5s MP4):

<div class="af_media_carousel" aria-label="AbstractVision text-to-video examples">
  <button class="af_media_carousel__btn af_media_carousel__btn--prev" type="button" aria-label="Previous video">‹</button>
  <div class="af_media_carousel__track" tabindex="0">
    <figure class="af_media_carousel__item">
      <video class="af_media_carousel__video" controls muted playsinline loop preload="metadata">
        <source src="../assets/videos/video-river.mp4" type="video/mp4" />
      </video>
      <figcaption class="af_media_carousel__caption"><strong>River</strong> — smooth, clean motion (5s)</figcaption>
    </figure>
    <figure class="af_media_carousel__item">
      <video class="af_media_carousel__video" controls muted playsinline loop preload="metadata">
        <source src="../assets/videos/video-space.mp4" type="video/mp4" />
      </video>
      <figcaption class="af_media_carousel__caption"><strong>Space</strong> — cinematic scene (5s)</figcaption>
    </figure>
    <figure class="af_media_carousel__item">
      <video class="af_media_carousel__video" controls muted playsinline loop preload="metadata">
        <source src="../assets/videos/t2v-example.mp4" type="video/mp4" />
      </video>
      <figcaption class="af_media_carousel__caption"><strong>T2V example</strong> — prior sample (5s)</figcaption>
    </figure>
  </div>
  <button class="af_media_carousel__btn af_media_carousel__btn--next" type="button" aria-label="Next video">›</button>
</div>

MLX-Gen image generation/edit commands accept `--progress` for step progress.
MLX-Gen upscale and video commands print denoise-step progress by default;
video events also include frame context while the video is running. Use
`--no-progress` for quiet video or upscaling scripts. A complete local example
gallery with commands and bundled outputs is available in
[MLX-Gen local examples](mlx-gen-local-examples.md).

Interactive CLI/REPL (`abstractvision cli`; `abstractvision repl` remains an
alias) uses the same backend and request normalization:

```text
/backend mlx-gen AbstractFramework/flux.2-klein-4b-4bit
/t2i "a product photo of a matte black espresso machine" --steps 4 --guidance-scale 1.0 --open

/backend mlx-gen AbstractFramework/qwen-image-edit-2511-4bit
/i2i --image ./input.png "make it watercolor" --steps 20 --guidance-scale 2.5 --open

/backend mlx-gen briaai/FIBO
/t2i "a studio product photo of a white ceramic mug with the AbstractFramework logo" --steps 50 --guidance-scale 4.0 --open

/backend mlx-gen prism-ml/bonsai-image-ternary-4B-mlx-2bit
/t2i "a bonsai tree in a quiet ceramic studio" --steps 4 --guidance-scale 1.0 --open

/backend mlx-gen AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit
/t2v "a red fox walking through a snowy forest, cinematic" --width 432 --height 240 --frames 41 --fps 10 --steps 20 --guidance-scale 4.0 --guidance-2 3.0 --open
/backend mlx-gen AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit
/i2v --image ./first-frame.png "slow camera push-in" --width 432 --height 240 --frames 41 --fps 10 --steps 20 --guidance-scale 3.5 --guidance-2 3.5 --open
```

Interactive `/t2v` and `/i2v` use the same default progress display; add
`--no-progress` to suppress it.

Legacy `mflux` provider names and routed ids still work as aliases, but new
configuration should use `mlx-gen`.

---

## 3) Qwen Image (Diffusers)

Qwen Image models in the registry:

- `Qwen/Qwen-Image` (older)
- `Qwen/Qwen-Image-2512` (newer)

Use the same Diffusers flow:

```text
/backend diffusers Qwen/Qwen-Image-2512 mps float16
/t2i "a poster with the word 'ABSTRACT' rendered perfectly in bold typography" --width 512 --height 512 --steps 10 --guidance-scale 2.5 --open
```

Notes:
- Qwen Image models are **large**.
- For best results, prefer the model card’s recommended sizes (e.g. 1328x1328 for 1:1). For quick tests, 512x512 is fine.
- On Apple Silicon (MPS), start with fp16 (default; best compatibility):
  - `ABSTRACTVISION_DIFFUSERS_TORCH_DTYPE=float16` (or in the REPL: `/backend diffusers Qwen/Qwen-Image-2512 mps float16`)
- If you get NaNs/black images, try fp32 (this can require **very** large peak memory during load):
  - `ABSTRACTVISION_DIFFUSERS_TORCH_DTYPE=float32` (or in the REPL: `/backend diffusers Qwen/Qwen-Image-2512 mps float32`)
- On Apple Silicon (MPS), AbstractVision upcasts the VAE to fp32 when using fp16 to avoid common “black image” issues.
- Automatic fp32 retry on all-black output is enabled by default on MPS (can increase peak memory):
  - disable with `ABSTRACTVISION_DIFFUSERS_AUTO_RETRY_FP32=0`
- In AbstractVision, `--guidance-scale` is mapped to Qwen’s `true_cfg_scale` when using Qwen pipelines (CFG). If you set `--guidance-scale` but don’t provide a `negative_prompt`, AbstractVision passes a placeholder negative prompt (`" "`) so CFG is actually enabled.

Tip: keep `guidance_scale` relatively low for some modern DiT models.

---

## 3.1) LoRA + Rapid-AIO (Diffusers)

AbstractVision can apply LoRA adapters (Diffusers adapter system) and optionally swap in a distilled “Rapid-AIO”
transformer for faster Qwen Image Edit inference.

These features follow the Diffusers download setting. The REPL is cache-only by default, so pre-download adapters or
Rapid-AIO weights separately before using repo ids here. If you intentionally want runtime downloads, set:

```bash
export ABSTRACTVISION_DIFFUSERS_ALLOW_DOWNLOAD=1
```

Tip: Qwen Image Edit pipelines often default to a ~1MP output resolution when `width`/`height` are omitted. AbstractVision
defaults to the input image size for image-to-image edits to avoid unexpected memory spikes. To request a specific output
resolution, pass `--width`/`--height` flags in the REPL (forwarded via `request.extra`) or set them in the playground
Extra JSON field.

LoRA example (REPL; note: `loras_json` is forwarded via `request.extra`):

```text
/backend diffusers Qwen/Qwen-Image-Edit-2511 mps bfloat16
/i2i --image ./input.png "make it watercolor" --steps 8 --guidance-scale 1 --loras_json '[{"source":"lightx2v/Qwen-Image-Edit-2511-Lightning","scale":1.0}]' --open
```

Rapid-AIO example (distilled transformer override; Qwen Image Edit):

```text
/backend diffusers Qwen/Qwen-Image-Edit-2511 mps bfloat16
/i2i --image ./input.png "make it watercolor" --steps 4 --guidance-scale 1 --rapid_aio_repo linoyts/Qwen-Image-Edit-Rapid-AIO --open
```

---

## 4) FLUX 2 (Diffusers)

FLUX 2 models in the registry:

- `black-forest-labs/FLUX.2-klein-4B` (Apache-2.0, not gated)
- `black-forest-labs/FLUX.2-klein-9B` (non-commercial license, gated on Hugging Face)
- `black-forest-labs/FLUX.2-dev` (non-commercial license, gated on Hugging Face)

Sanity check:

```bash
python -c "import diffusers; print(diffusers.__version__)"
```

Notes:
- `FLUX.2-dev` uses Diffusers `Flux2Pipeline` and works on released Diffusers (0.36+).
- `FLUX.2-klein-4B` and `FLUX.2-klein-9B` use `Flux2KleinPipeline`, which is not available in the released Diffusers (0.36.0). It currently
  requires installing Diffusers from source (with the `diffusers-dev` extra for compatible dependency pins):
  - `pip install -U "abstractvision[diffusers-dev]"`
  - `pip install -U "git+https://github.com/huggingface/diffusers@main"`

Recommended first FLUX example (`FLUX.2-klein-4B`, not gated):

```text
/backend diffusers black-forest-labs/FLUX.2-klein-4B mps float16
/t2i "a product photo of a matte black espresso machine" --width 1024 --height 1024 --steps 4 --guidance-scale 1.0 --seed 0 --open
```

Example (`FLUX.2-klein-9B`, gated; requires Diffusers `main` and HF access):

```text
/backend diffusers black-forest-labs/FLUX.2-klein-9B mps float16
/t2i "a minimalist product photo of a matte black espresso machine, studio lighting" --width 1024 --height 1024 --steps 4 --guidance-scale 1.0 --seed 0 --open
```

Example (`FLUX.2-dev`, gated; you must pre-download it into your HF cache first):

```text
/backend diffusers black-forest-labs/FLUX.2-dev mps
/t2i "a minimalist product photo of a matte black espresso machine, studio lighting" --width 1024 --height 1024 --steps 4 --guidance-scale 1.0 --seed 0 --open
```

If you use gated models (like `FLUX.2-dev`), you typically must accept the model’s terms on Hugging Face and set `HF_TOKEN` in your environment.

---

## 5) Stable Diffusion 3.5 (Diffusers, gated)

SD3.5 models (all gated on Hugging Face):

- `stabilityai/stable-diffusion-3.5-large-turbo`
- `stabilityai/stable-diffusion-3.5-large`
- `stabilityai/stable-diffusion-3.5-medium`

1) Accept the model terms on Hugging Face (in your browser).  
2) Export a token:

```bash
export HF_TOKEN=...   # your Hugging Face access token
```

Then in the REPL:

```text
/backend diffusers stabilityai/stable-diffusion-3.5-large-turbo mps
/t2i "a modern product photo of a watch, studio lighting" --width 1024 --height 1024 --steps 6 --guidance-scale 4 --seed 42 --open
```

Turbo models are usually best with low step counts (e.g. ~4–8).

---

## 6) GGUF diffusion models (stable-diffusion.cpp)

If you downloaded a GGUF diffusion model (like Qwen Image GGUF or FLUX.2 GGUF), Diffusers cannot load it. Use the stable-diffusion.cpp backend instead (either via pip-installed python bindings or `sd-cli`).

### 6.1 Install stable-diffusion.cpp runtime

The base `pip install abstractvision` path does not install local inference runtimes. Use one of these explicit stable-diffusion.cpp runtime choices:

```bash
pip install "abstractvision[sdcpp]"
```

This pip binding path is convenient, but it may require a native build or run **CPU-only** depending on how the wheel was built.

Alternative (external executable):

- Download `sd-cli` from: <https://github.com/leejet/stable-diffusion.cpp/releases>
- Ensure `sd-cli` is in your `PATH` (or pass a full path as the last arg to `/backend sdcpp …`).

On macOS (Apple Silicon), **`sd-cli` is the recommended path** to get **Metal** acceleration. If you see `Using CPU backend`,
install `sd-cli` and re-run in CLI mode.

### 6.2 Single-file Stable Diffusion model

This is the lowest-friction `sdcpp` shape: one model file plus an optional `sd-cli` path. Use it for Stable Diffusion
1.x/2.x/SDXL checkpoints or GGUF conversions that stable-diffusion.cpp can load as `--model`.

```bash
abstractvision cli
```

```text
/backend sdcpp /path/to/sd-v1-5.gguf /path/to/sd-cli
/t2i "a watercolor painting of a lighthouse" --width 512 --height 512 --steps 10 --open
```

If `sd-cli` is already in your `PATH`, you can omit the final `/path/to/sd-cli` argument. If it is not available,
AbstractVision falls back to `stable-diffusion-cpp-python` when that package is installed, for example through `pip install "abstractvision[sdcpp]"`.

### 6.3 Curated `sdcpp` bundles

For Qwen Image and FLUX GGUF, external users should not have to manually locate a VAE or text encoder.
Download the curated bundle once, then use the model key everywhere:

```bash
abstractvision download qwen-image --provider sdcpp
abstractvision download flux2-klein-base-4b --provider sdcpp
```

AbstractVision resolves the required side artifacts from the Hugging Face cache automatically:

- CLI one-shot: `abstractvision t2i --provider sdcpp --model flux2-klein-base-4b "..."`
- REPL: `/backend sdcpp flux2-klein-base-4b /path/to/sd-cli`
- Playground / AbstractCore plugin: set the model key and let the package resolve the cached bundle

If a required companion file is missing, the package now fails early with a precise `download ... --provider sdcpp`
hint instead of starting generation and then failing deep in the runtime.

### 6.4 Run Qwen Image with `sdcpp`

```bash
abstractvision cli
```

Then:

```text
/backend sdcpp qwen-image /path/to/sd-cli
/set width 1024
/set height 1024
/t2i "a cinematic photo of a red fox in snow" --sampling-method euler --offload-to-cpu --diffusion-fa --flow-shift 3 --open
```

Any extra `--flag` you pass (like `--sampling-method euler`) is forwarded to the backend as `extra`.
- CLI mode: forwarded to `sd-cli`
- Python bindings mode: keys are mapped to python binding kwargs when supported; unsupported keys are ignored (see [`../src/abstractvision/backends/stable_diffusion_cpp.py`](../src/abstractvision/backends/stable_diffusion_cpp.py))
- Diffusers backend: only forwards kwargs that the pipeline `__call__` accepts; unknown keys are ignored (see [`../src/abstractvision/backends/huggingface_diffusers.py`](../src/abstractvision/backends/huggingface_diffusers.py))

### 6.5 FLUX.2-klein-4B / FLUX.2-klein-base-4B (GGUF) example

The curated `sdcpp` presets now download the main GGUF plus the required companion artifacts and let you refer to the
model by key afterward.

Example:

```text
/backend sdcpp flux2-klein-base-4b /path/to/sd-cli
/t2i "a product photo of a matte black espresso machine" --steps 4 --guidance-scale 1.0 --sampling-method euler --diffusion-fa --offload-to-cpu --open
```

Advanced/manual mode is still supported if you want to wire explicit component paths yourself:

```text
/backend sdcpp /path/to/flux-2-klein-base-4b-Q8_0.gguf /path/to/vae/diffusion_pytorch_model.safetensors /path/to/Qwen3-4B-Q4_K_M.gguf /path/to/sd-cli
```

FLUX.2-dev and Qwen Image GGUF are still heavier follow-ups, but the single-file Stable Diffusion path or the curated
klein bundle above should be the first local test on a fresh machine.

---

## 7) Web UI testing (optional): Playground

This repo includes a self-contained web UI and local API server. It is owned by
AbstractVision and does not require AbstractCore. Treat it as a local/dev
testing surface; use AbstractCore/Gateway for production routing,
authentication, and browser-origin policy.

### 7.1 Start the playground

```bash
abstractvision playground --port 8091
```

Open:

- `http://127.0.0.1:8091/vision_playground.html`

In the UI:
- The API Base URL defaults to the same process that serves the page
- Each task tab has its own model selector and unload button
- Switching models in a tab unloads the previously active backend first to free memory before the replacement model is loaded
- Generate (T2I), upload an input image (I2I), or inspect video-capable backend status. The Playground is still a dev surface; prefer the shell/REPL or AbstractCore for MLX-Gen Wan video smoke tests.

For the endpoint list, see `playground/README.md`.

### 7.2 Local text→video status

The packaged local Diffusers `text_to_video` groundwork remains experimental and
is currently disabled from the normal local surfaces. The practical local Apple
Silicon video path is MLX-Gen Wan 2.2, preferably the task-specific A14B
packages when memory allows.

Current policy:
- use `abstractvision t2v --provider mlx-gen --model AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit` and `abstractvision i2v --provider mlx-gen --model AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit` for local Apple Silicon Wan A14B video;
- use the OpenAI-compatible backend when video is served remotely; and
- keep Diffusers local video behind the experimental follow-up in [`docs/backlog/planned/0023_local_runtime_capability_quarantine_for_glm_mflux_and_t2v.md`](backlog/planned/0023_local_runtime_capability_quarantine_for_glm_mflux_and_t2v.md).
--- >8 --- END FILE: docs/getting-started.md --- >8 ---

--- 8< --- FILE: docs/mlx-gen-local-examples.md --- 8< ---
# MLX-Gen local examples

This page shows local MLX-Gen image, edit, multi-reference edit, SeedVR2
upscale, AbstractCore plugin, and Wan A14B video outputs generated through
AbstractVision. The commands target the current MLX-Gen `0.18.13+` runtime with
cached or prepared model artifacts; the included proof manifests record the
runtime used for each artifact group.

MLX-Gen progress follows denoise steps. For video, the CLI percentage is
`step / total_steps`; frame counts are displayed only as context.

## Setup

Install the MLX-Gen runtime extra and download the models used below:

```bash
pip install "abstractvision[mlx-gen]"

abstractvision download AbstractFramework/flux.2-klein-9b-8bit --provider mlx-gen
abstractvision download AbstractFramework/seedvr2-3b-8bit --provider mlx-gen
abstractvision download AbstractFramework/seedvr2-7b-8bit --provider mlx-gen
abstractvision download AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit --provider mlx-gen
```

SeedVR2 upscaler selection:

| Model id | Use when |
| --- | --- |
| `AbstractFramework/seedvr2-3b-8bit` | Default q8 package. |
| `AbstractFramework/seedvr2-7b-8bit` | Higher-quality 7B package when memory allows. |
| `AbstractFramework/seedvr2-3b-4bit` / `AbstractFramework/seedvr2-7b-4bit` | Lower-memory packages. |
| `ByteDance-Seed/SeedVR2-3B` / `ByteDance-Seed/SeedVR2-7B` | Official source weights; use `--quantize 8` or `--quantize 4` only when runtime quantization is needed. |

Prepared local model folders can also be used directly. For the local folders
from the MLX-Gen checkout:

```bash
abstractvision upscale \
  --provider mlx-gen \
  --model /Users/albou/projects/gh/sbx/mlx-gen/models/seedvr2-7b-8bit \
  --image ./input.png \
  --scale 2x \
  --progress \
  --open
```

Choose an asset store and input path names for the examples:

```bash
STORE_DIR=./mlx-gen-local-examples-store
SOURCE_IMAGE=./race-car-source.png
REFERENCE_IMAGE=./multi-reference-scene.png
UPSCALE_SOURCE=./race-car-source.png
```

The multi-reference input image used for the gallery is available here:
[multi_reference_scene.png](assets/mlx-gen-local-examples/multi_reference_scene.png).

## Commands

Create a source image for edit tests:

```bash
abstractvision t2i \
  --provider mlx-gen \
  --model AbstractFramework/flux.2-klein-9b-8bit \
  --store-dir "$STORE_DIR" \
  --width 768 \
  --height 512 \
  --steps 12 \
  --seed 2401 \
  --progress \
  "red toy race car on clean white studio table"
```

Use the generated source image for a style-preserving edit:

```bash
abstractvision i2i \
  --provider mlx-gen \
  --model AbstractFramework/flux.2-klein-9b-8bit \
  --store-dir "$STORE_DIR" \
  --image "$SOURCE_IMAGE" \
  --width 768 \
  --height 512 \
  --steps 12 \
  --seed 2402 \
  --progress \
  "graphite pencil sketch preserving car pose"
```

Use the same source image for a stronger composition change:

```bash
abstractvision i2i \
  --provider mlx-gen \
  --model AbstractFramework/flux.2-klein-9b-8bit \
  --store-dir "$STORE_DIR" \
  --image "$SOURCE_IMAGE" \
  --width 768 \
  --height 512 \
  --steps 12 \
  --seed 2403 \
  --progress \
  "rain-soaked neon city street, low camera angle"
```

Use a second reference image for multi-reference composition:

```bash
abstractvision i2i \
  --provider mlx-gen \
  --model AbstractFramework/flux.2-klein-9b-8bit \
  --store-dir "$STORE_DIR" \
  --image "$SOURCE_IMAGE" \
  --reference-image "$REFERENCE_IMAGE" \
  --width 768 \
  --height 512 \
  --steps 12 \
  --seed 2404 \
  --progress \
  "use race car + elevated runway/sunset/spotlights reference"
```

Upscale the source image with SeedVR2. The default package is
`AbstractFramework/seedvr2-3b-8bit`; use
`AbstractFramework/seedvr2-7b-8bit` when memory allows, or the matching q4
package when memory is tight. Pass either a scale factor such as `2x` or a
shortest-edge target resolution.

```bash
abstractvision catalog --provider mlx-gen --task upscale

abstractvision upscale \
  --provider mlx-gen \
  --model AbstractFramework/seedvr2-3b-8bit \
  --store-dir "$STORE_DIR" \
  --image "$UPSCALE_SOURCE" \
  --scale 2x \
  --seed 2405 \
  --open

abstractvision upscale \
  --provider mlx-gen \
  --model AbstractFramework/seedvr2-7b-8bit \
  --store-dir "$STORE_DIR" \
  --image "$UPSCALE_SOURCE" \
  --resolution 1024 \
  --seed 2405
```

Generate a low-cost Wan A14B video check at `432x240`, 41 frames, and 8
denoise steps:

```bash
abstractvision t2v \
  --provider mlx-gen \
  --model AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit \
  --store-dir "$STORE_DIR" \
  --width 432 \
  --height 240 \
  --num-frames 41 \
  --fps 24 \
  --steps 8 \
  --guidance-scale 4.0 \
  --guidance-2 3.0 \
  --seed 2406 \
  --progress \
  "A small red toy race car rolls forward on a glossy studio floor, camera locked, soft reflections, smooth motion"
```

## Outputs

The generated gallery below is bundled with the docs. The complete manifest is
available as [summary.json](assets/mlx-gen-local-examples/summary.json), and
the AbstractCore plugin progress event capture is available as
[abstractcore_plugin_t2i_progress_events.json](assets/mlx-gen-local-examples/abstractcore_plugin_t2i_progress_events.json).
The SeedVR2 upscaler proof was run separately against MLX-Gen `0.18.13`; its
manifest is [seedvr2_upscale_proof.json](assets/mlx-gen-local-examples/seedvr2_upscale_proof.json).

![MLX-Gen example proof table](assets/mlx-gen-local-examples/proof_table.png)

<table>
  <thead>
    <tr>
      <th>Output</th>
      <th>Prompt or intent</th>
      <th>Result</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Text to image</td>
      <td><code>red toy race car on clean white studio table</code></td>
      <td><img src="assets/mlx-gen-local-examples/01_t2i_source.png" alt="Text-to-image source result" width="260" /></td>
    </tr>
    <tr>
      <td>Image edit, style</td>
      <td><code>graphite pencil sketch preserving car pose</code></td>
      <td><img src="assets/mlx-gen-local-examples/02_i2i_style_pencil.png" alt="Image edit style result" width="260" /></td>
    </tr>
    <tr>
      <td>Image edit, composition</td>
      <td><code>rain-soaked neon city street, low camera angle</code></td>
      <td><img src="assets/mlx-gen-local-examples/03_i2i_composition_neon.png" alt="Image edit composition result" width="260" /></td>
    </tr>
    <tr>
      <td>Multi-reference edit</td>
      <td><code>use race car + elevated runway/sunset/spotlights reference</code></td>
      <td><img src="assets/mlx-gen-local-examples/04_i2i_multi_reference.png" alt="Multi-reference edit result" width="260" /></td>
    </tr>
    <tr>
      <td>SeedVR2 upscale</td>
      <td><code>seedvr2-7b-8bit</code>, <code>scale=2x</code>, local prepared folder</td>
      <td>
        <img src="assets/mlx-gen-local-examples/seedvr2_upscale_input.png" alt="SeedVR2 input image" width="120" />
        <img src="assets/mlx-gen-local-examples/seedvr2_upscale_2x.png" alt="SeedVR2 2x upscaled result" width="240" />
      </td>
    </tr>
    <tr>
      <td>SeedVR2 upscale, q4 and Core</td>
      <td><code>seedvr2-7b-4bit</code> CLI proof plus Core <code>/v1/images/upscale</code> q8 route proof</td>
      <td>
        <img src="assets/mlx-gen-local-examples/seedvr2_upscale_7b_4bit.png" alt="SeedVR2 7B q4 2x result" width="220" />
        <img src="assets/mlx-gen-local-examples/seedvr2_upscale_core_7b_8bit.png" alt="AbstractCore SeedVR2 7B q8 2x result" width="220" />
      </td>
    </tr>
    <tr>
      <td>AbstractCore plugin T2I</td>
      <td><code>blue abstract cube icon</code> with <code>on_progress(event)</code> capture</td>
      <td><img src="assets/mlx-gen-local-examples/05_abstractcore_plugin_t2i_progress.png" alt="AbstractCore plugin text-to-image result" width="260" /></td>
    </tr>
  </tbody>
</table>

## Video output

The A14B example is the real generated MP4. The contact sheet shows sampled
frames for quick inspection in static contexts.

<video controls muted playsinline preload="metadata" style="max-width: 100%; border-radius: 8px;">
  <source src="assets/mlx-gen-local-examples/wan_a14b_t2v_432x240_race_car.mp4" type="video/mp4" />
</video>

![Wan A14B video contact sheet](assets/mlx-gen-local-examples/video_contact_sheet.png)
--- >8 --- END FILE: docs/mlx-gen-local-examples.md --- >8 ---

--- 8< --- FILE: docs/api.md --- 8< ---
# API reference

This document describes the **public** Python API surface of `abstractvision` (0.x / Alpha) and points to the implementation.

See also:
- Getting started (end-to-end examples): [docs/getting-started.md](getting-started.md)
- Architecture (how the pieces fit): [docs/architecture.md](architecture.md)
- Backends reference (support matrix): [docs/reference/backends.md](reference/backends.md)
- FAQ (common questions): [docs/faq.md](faq.md)

## Public exports

The package exports the following symbols from `abstractvision` (see [`../src/abstractvision/__init__.py`](../src/abstractvision/__init__.py)):

- `VisionManager`
- `ProviderModelInfo`
- `VisionModelCapabilitiesRegistry`
- `LocalAssetStore`
- `RuntimeArtifactStoreAdapter`
- `is_artifact_ref`
- `__version__`

## Core concepts

### Tasks

`VisionManager` exposes one method per task (implementation: [`../src/abstractvision/vision_manager.py`](../src/abstractvision/vision_manager.py)):

- `generate_image(...)` → `text_to_image`
- `edit_image(...)` → `image_to_image`
- `upscale_image(...)` → `image_upscale`
- `generate_video(...)` → `text_to_video` (backend-dependent)
- `image_to_video(...)` → `image_to_video` (backend-dependent)
- `generate_angles(...)` → `multi_view_image` (API exists; no built-in backend implements it yet)

Task names are also used by the capability registry ([`../src/abstractvision/assets/vision_model_capabilities.json`](../src/abstractvision/assets/vision_model_capabilities.json)).

### Backends

Backends are execution engines that implement the `VisionBackend` interface ([`../src/abstractvision/backends/base_backend.py`](../src/abstractvision/backends/base_backend.py)).

Built-in backends live in [`../src/abstractvision/backends/`](../src/abstractvision/backends/):
- `OpenAICompatibleVisionBackend` (HTTP)
- `HuggingFaceDiffusersVisionBackend` (local Diffusers images; local Diffusers `text_to_video` groundwork is currently quarantined)
- `StableDiffusionCppVisionBackend` (local stable-diffusion.cpp / GGUF)
- `MLXGenVisionBackend` / compatibility alias `MFluxVisionBackend` (local Apple Silicon MLX-Gen bridge for curated AbstractFramework q4/q8 MLX presets, official FIBO snapshots, and Wan video)

Backend config classes are re-exported from `abstractvision.backends` via lazy imports (see [`../src/abstractvision/backends/__init__.py`](../src/abstractvision/backends/__init__.py)).

Provider catalog listing is exposed as a backend contract:

```python
from abstractvision.backends import OpenAICompatibleBackendConfig, OpenAICompatibleVisionBackend

backend = OpenAICompatibleVisionBackend(
    config=OpenAICompatibleBackendConfig(base_url="http://localhost:1234/v1")
)
for model in backend.list_provider_models(task="text_to_image"):
    print(model.id)
```

For official OpenAI, use `base_url="https://api.openai.com/v1"` and an API key. Catalog listing is explicit and does not change the configured generation model.

When AbstractVision is loaded as an AbstractCore capability plugin, the plugin shim exposes the
same explicit catalog surface as `llm.vision.list_provider_models(task="text_to_image")`. It
returns JSON-safe dictionaries so Core/Gateway route code can avoid private backend reach-throughs.

### Outputs: bytes vs artifact refs

`VisionManager` returns:

- `GeneratedAsset` (bytes) when no store is configured ([`../src/abstractvision/types.py`](../src/abstractvision/types.py))
- an artifact ref `dict` when `VisionManager.store` is configured (via `MediaStore.store_bytes(...)`)

Artifact helpers and stores are defined in [`../src/abstractvision/artifacts.py`](../src/abstractvision/artifacts.py).

Image sizes are backend/model-specific. `width` and `height` arguments are
optional request overrides; omitting them lets the backend use its default or
`auto` behavior. Passing an unsupported size is expected to fail at the selected
provider/backend boundary rather than being silently rewritten by
AbstractVision.

## VisionManager (orchestrator)

`VisionManager` is intentionally thin: it validates/gates best-effort and delegates to the configured backend.

Signature (see [`../src/abstractvision/vision_manager.py`](../src/abstractvision/vision_manager.py)):
- `backend`: a `VisionBackend` implementation (required to run anything)
- `store`: optional `MediaStore` to enable artifact-ref outputs
- `model_id`: optional capability-gating model id (must exist in the registry)
- `registry`: optional `VisionModelCapabilitiesRegistry` instance (reused when gating is enabled)

### Minimal example (OpenAI-compatible backend + artifact refs)

```python
from abstractvision import LocalAssetStore, VisionManager, is_artifact_ref
from abstractvision.backends import OpenAICompatibleBackendConfig, OpenAICompatibleVisionBackend

backend = OpenAICompatibleVisionBackend(
    config=OpenAICompatibleBackendConfig(base_url="http://localhost:1234/v1")
)
store = LocalAssetStore()
vm = VisionManager(backend=backend, store=store)

ref = vm.generate_image("a studio photo of an espresso machine", width=768, height=768, steps=20)
assert is_artifact_ref(ref)
png_bytes = store.load_bytes(ref["$artifact"])
```

### Local example (Diffusers backend)

Install `abstractvision[diffusers]` before using this backend.

```python
from abstractvision import VisionManager
from abstractvision.backends import HuggingFaceDiffusersBackendConfig, HuggingFaceDiffusersVisionBackend

backend = HuggingFaceDiffusersVisionBackend(
    config=HuggingFaceDiffusersBackendConfig(
        model_id="runwayml/stable-diffusion-v1-5",
        device="auto",
        allow_download=False,
    )
)
vm = VisionManager(backend=backend)
asset = vm.generate_image("a watercolor painting of a lighthouse", width=512, height=512, steps=10)
```

Note: `allow_download=False` is the default. Pre-download model weights separately, or set `allow_download=True` only when you want runtime downloads.

`upscale_image(...)`, `generate_video(...)`, and `image_to_video(...)` are part
of the public API. MLX-Gen 0.18.13+ supports SeedVR2 `image_upscale`, Wan
`text_to_video`, and first-frame `image_to_video`, including A14B task-specific
checkpoints. Local Diffusers video remains experimental and disabled from the
normal local surfaces. Generated MP4 outputs still require an `ffmpeg`
executable on `PATH` whenever a backend returns frame sequences for local
packaging.

### Local example (MLX-Gen backend)

Install `abstractvision[mlx-gen]` and pre-download the exact model repo first,
for example `abstractvision download AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit --provider mlx-gen`.

```python
from pathlib import Path

from abstractvision import VisionManager
from abstractvision.backends import MLXGenBackendConfig, MLXGenVisionBackend

def on_progress(event):
    if event.total_frames:
        print(f"{event.phase}: frame {event.frame}/{event.total_frames}")
    else:
        print(f"{event.phase}: step {event.step}/{event.total_steps}")

image_backend = MLXGenVisionBackend(
    config=MLXGenBackendConfig(model="AbstractFramework/flux.2-klein-9b-8bit")
)
image_vm = VisionManager(backend=image_backend)

image_asset = image_vm.generate_image(
    "a studio product photo of a red toy race car",
    width=768,
    height=512,
    steps=12,
    guidance_scale=1.0,
    on_progress=on_progress,
)

edit_asset = image_vm.edit_image(
    "compose the subject using the second image as a style and layout reference",
    image=Path("./subject.png").read_bytes(),
    steps=12,
    guidance_scale=1.0,
    on_progress=on_progress,
    extra={"reference_images": [Path("./style-reference.png").read_bytes()]},
)

upscale_backend = MLXGenVisionBackend(config=MLXGenBackendConfig(model="AbstractFramework/seedvr2-3b-8bit"))
upscale_vm = VisionManager(backend=upscale_backend)

upscaled_asset = upscale_vm.upscale_image(
    image=Path("./subject.png").read_bytes(),
    scale="2x",
    seed=2405,
    on_progress=on_progress,
)

t2v_backend = MLXGenVisionBackend(
    config=MLXGenBackendConfig(model="AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit")
)
t2v_vm = VisionManager(backend=t2v_backend)

asset = t2v_vm.generate_video(
    "a red fox walking through a snowy forest, cinematic",
    width=432,
    height=240,
    num_frames=41,
    fps=10,
    steps=20,
    guidance_scale=4.0,
    guidance_2=3.0,
    on_progress=on_progress,
    extra={"max_sequence_length": 256},
)

i2v_backend = MLXGenVisionBackend(
    config=MLXGenBackendConfig(model="AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit")
)
i2v_vm = VisionManager(backend=i2v_backend)

first_frame_asset = i2v_vm.image_to_video(
    image=Path("./first-frame.png").read_bytes(),
    prompt="slow camera push-in",
    width=432,
    height=240,
    num_frames=41,
    fps=10,
    steps=20,
    guidance_scale=3.5,
    guidance_2=3.5,
    on_progress=on_progress,
    extra={"max_sequence_length": 256},
)
```

Wan 2.2 A14B has two guidance controls. Use `guidance_scale` for the
primary/high-noise stage and `guidance_2` for the second/low-noise stage. The
registry default is `guidance_2=3.0` for text-to-video A14B and `3.5` for
image-to-video A14B. Other video models should omit `guidance_2` unless their
registry task declares it.

For MLX-Gen, `on_progress` receives an `abstractvision.VideoProgressEvent`.
Image generation/editing/upscaling events carry `phase`, `step`, `total_steps`,
and denoise-step `progress`. Wan video events add `frame`, `total_frames`, and
`frame_progress`. The lower-level `backend.generate_image_with_progress(...)`,
`backend.edit_image_with_progress(...)`, `backend.upscale_image_with_progress(...)`,
`backend.generate_video_with_progress(...)`, and
`backend.image_to_video_with_progress(...)` methods keep the existing
two-argument `(current, total)` callback for backend-agnostic progress bars.
For MLX-Gen, that callback reports denoise step counts; use
`on_progress(event)` when a UI also needs video frame context.

## Passing advanced backend parameters (`extra`)

Request dataclasses include an `extra: dict` field ([`../src/abstractvision/types.py`](../src/abstractvision/types.py)). Use it to pass backend-specific parameters in a controlled way:

```python
asset_or_ref = vm.generate_image(
    "a product photo of a matte black espresso machine",
    steps=8,
    guidance_scale=1.0,
    extra={
        # Example keys used by some Diffusers flows:
        "loras_json": [{"source": "lightx2v/Qwen-Image-Edit-2511-Lightning", "scale": 1.0}],
        "rapid_aio_repo": "linoyts/Qwen-Image-Edit-Rapid-AIO",
    },
)
```

Backends may ignore unknown keys; consult the backend implementation and [docs/reference/backends.md](reference/backends.md).

## Capability registry (what models can do)

The packaged registry is loaded by `VisionModelCapabilitiesRegistry` ([`../src/abstractvision/model_capabilities.py`](../src/abstractvision/model_capabilities.py)).

```python
from abstractvision import VisionModelCapabilitiesRegistry

reg = VisionModelCapabilitiesRegistry()
print(reg.list_tasks())
print(reg.models_for_task("text_to_image"))

reg.require_support("runwayml/stable-diffusion-v1-5", "text_to_image")
```

Optional gating:
- If you construct `VisionManager(model_id=..., registry=...)`, the manager will fail fast on unsupported tasks before calling a backend ([`../src/abstractvision/vision_manager.py`](../src/abstractvision/vision_manager.py)).

Important: the registry is *not* a guarantee that your configured backend can execute a task at runtime.
Use [docs/reference/backends.md](reference/backends.md) for backend support.

## Artifacts and stores

Artifact helpers and store implementations live in [`../src/abstractvision/artifacts.py`](../src/abstractvision/artifacts.py):

- `LocalAssetStore` (standalone local files, default `~/.abstractvision/assets`)
- `RuntimeArtifactStoreAdapter` (duck-typed adapter for an external artifact store)
- `is_artifact_ref(...)` / `make_media_ref(...)`

See: [docs/reference/artifacts.md](reference/artifacts.md).

## Errors you may want to handle

Common exceptions (defined in [`../src/abstractvision/errors.py`](../src/abstractvision/errors.py)):

- `BackendNotConfiguredError` (calling `VisionManager` without a backend)
- `CapabilityNotSupportedError` (task isn’t supported by the model registry or backend)
- `UnknownModelError` (model id isn’t present in the registry)
- `OptionalDependencyMissingError` (backend dependency is missing, e.g. Diffusers/Torch)
--- >8 --- END FILE: docs/api.md --- >8 ---

--- 8< --- FILE: docs/architecture.md --- 8< ---
# AbstractVision architecture

AbstractVision is a model-agnostic Python layer that standardizes **generative vision outputs** behind a small API:
text→image, image→image (and optionally video when a backend supports it).

This document describes the *current code in this repo* and links to the supporting reference docs.

See also:
- Docs index: [docs/README.md](README.md)
- Getting started: [docs/getting-started.md](getting-started.md)
- API reference: [docs/api.md](api.md)
- FAQ: [docs/faq.md](faq.md)
- ADR index: [docs/adr/README.md](adr/README.md)
- Backends: [docs/reference/backends.md](reference/backends.md)
- Capability registry: [docs/reference/capabilities-registry.md](reference/capabilities-registry.md)
- Artifacts: [docs/reference/artifacts.md](reference/artifacts.md)
- AbstractCore integration: [docs/reference/abstractcore-integration.md](reference/abstractcore-integration.md)

## AbstractFramework ecosystem (positioning)

AbstractVision is one component in the **AbstractFramework** ecosystem:

- **AbstractFramework** (project hub): <https://github.com/lpalbou/AbstractFramework>
- **AbstractCore** (orchestration + tool calling): <https://github.com/lpalbou/abstractcore>
- **AbstractRuntime** (runtime services, including artifact storage): <https://github.com/lpalbou/abstractruntime>

Where AbstractVision fits:
- AbstractVision focuses on *producing* images/videos (generators).
- AbstractCore focuses on orchestration, tool calling, and higher-level workflows (it can discover AbstractVision via the plugin entry point in `pyproject.toml` and `src/abstractvision/integrations/abstractcore_plugin.py`).
- AbstractRuntime provides runtime services and an artifact store interface; `RuntimeArtifactStoreAdapter` bridges AbstractVision to an AbstractRuntime-style artifact store (`src/abstractvision/artifacts.py`).

## Scope (and non-goals)

AbstractVision focuses on **producing** images/videos.

It is not the owner of “LLM image/video input attachments” (multimodal inputs to LLMs); those concerns live in higher-level layers (e.g., AbstractCore).

## Key components (with evidence pointers)

- **Orchestrator**: [`VisionManager`](../src/abstractvision/vision_manager.py)
  - Delegates execution to a backend.
  - Optionally gates requests using the capability registry when `model_id` is set.
  - Optionally stores outputs and returns artifact refs when `store` is set.
- **Backend contract**: [`VisionBackend`](../src/abstractvision/backends/base_backend.py)
  - Implementations live in [`../src/abstractvision/backends/`](../src/abstractvision/backends/).
- **Capability registry**: [`VisionModelCapabilitiesRegistry`](../src/abstractvision/model_capabilities.py)
  - Loads packaged data: [`vision_model_capabilities.json`](../src/abstractvision/assets/vision_model_capabilities.json).
  - Governs the model catalog, task metadata, and curated download surfaces as a product surface,
    not just as incidental documentation metadata. See [ADR 0005](adr/0005_curated_capability_registry_and_download_catalog.md).
- **Artifact outputs**: [`MediaStore`](../src/abstractvision/artifacts.py), [`LocalAssetStore`](../src/abstractvision/artifacts.py), [`RuntimeArtifactStoreAdapter`](../src/abstractvision/artifacts.py)
  - Artifact ref helper: `is_artifact_ref()` (see [`../src/abstractvision/artifacts.py`](../src/abstractvision/artifacts.py)).
- **CLI/REPL**: `abstractvision` entrypoint ([`../src/abstractvision/cli.py`](../src/abstractvision/cli.py))
  - Lets you inspect the registry and manually test generation backends.
- **AbstractCore integration**:
  - Capability plugin: [`../src/abstractvision/integrations/abstractcore_plugin.py`](../src/abstractvision/integrations/abstractcore_plugin.py) (registered in `pyproject.toml`)
  - Tool helpers: [`../src/abstractvision/integrations/abstractcore.py`](../src/abstractvision/integrations/abstractcore.py)

## High-level flow (library mode)

```mermaid
flowchart LR
  Caller[Caller<br/>(Python / CLI)] --> VM[VisionManager]
  VM -->|request dataclass| BE[VisionBackend]
  BE -->|GeneratedAsset| VM
  VM -->|store set| Store[MediaStore<br/>(LocalAssetStore / Runtime adapter)]
  Store --> Ref[Artifact ref dict]
  VM -->|store not set| Asset[GeneratedAsset<br/>(bytes + mime)]
```

Notes (anchored in code):
- `VisionManager` creates request dataclasses like `ImageGenerationRequest` / `ImageEditRequest` ([`../src/abstractvision/types.py`](../src/abstractvision/types.py)).
- When `store` is set, `VisionManager._maybe_store()` calls `store.store_bytes(...)` and returns an artifact ref dict ([`../src/abstractvision/vision_manager.py`](../src/abstractvision/vision_manager.py), [`../src/abstractvision/artifacts.py`](../src/abstractvision/artifacts.py)).

## Capability gating (model-level) vs runtime gating (backend-level)

AbstractVision separates two kinds of “can I do this?” checks:

1) **Model-level gating** (optional): “Does model X support task Y?”
   - Implemented by `VisionModelCapabilitiesRegistry.require_support(...)` ([`../src/abstractvision/model_capabilities.py`](../src/abstractvision/model_capabilities.py))
   - Used by `VisionManager._require_model_support(...)` when `VisionManager.model_id` is set ([`../src/abstractvision/vision_manager.py`](../src/abstractvision/vision_manager.py))

2) **Backend-level gating** (best-effort): “Does this configured backend support task Y / mask edits?”
   - Backends may implement `get_capabilities()` returning `VisionBackendCapabilities` ([`../src/abstractvision/types.py`](../src/abstractvision/types.py))
   - Enforced by `VisionManager._require_backend_support(...)` and mask checks in `VisionManager.edit_image(...)` ([`../src/abstractvision/vision_manager.py`](../src/abstractvision/vision_manager.py))

## Backend reality (what runs today)

The public API includes `text_to_video`, `image_to_video`, and `multi_view_image`, but backend support is currently limited:

- Built-in backends implement **images** (`text_to_image`, `image_to_image`):
  - OpenAI-compatible HTTP backend ([`../src/abstractvision/backends/openai_compatible.py`](../src/abstractvision/backends/openai_compatible.py))
  - Diffusers backend ([`../src/abstractvision/backends/huggingface_diffusers.py`](../src/abstractvision/backends/huggingface_diffusers.py))
  - stable-diffusion.cpp backend ([`../src/abstractvision/backends/stable_diffusion_cpp.py`](../src/abstractvision/backends/stable_diffusion_cpp.py))
  - MLX-Gen backend for curated Apple Silicon MLX presets ([`../src/abstractvision/backends/mflux.py`](../src/abstractvision/backends/mflux.py))
- Local MLX-Gen supports `text_to_image` for curated q4/q8 FLUX.2, Qwen Image, Z-Image, ERNIE Image Turbo, and official FIBO snapshots; supports `image_to_image` for FLUX.2 klein/base, Qwen Image Edit, ERNIE Image Turbo, FIBO, and FIBO Edit models; and supports Wan 2.2 TI2V plus task-specific Wan 2.2 A14B packages for `text_to_video` and first-frame `image_to_video`.
- Local Diffusers `text_to_video` remains experimental and is temporarily disabled from the normal local surfaces.
- OpenAI-compatible HTTP can also support `text_to_video` / `image_to_video` when `text_to_video_path` / `image_to_video_path` are configured ([`../src/abstractvision/backends/openai_compatible.py`](../src/abstractvision/backends/openai_compatible.py)).
- No built-in backend implements `multi_view_image` yet (they raise `CapabilityNotSupportedError` in `generate_angles(...)`).

For a detailed support matrix and configuration options, see [docs/reference/backends.md](reference/backends.md).

## Catalog and compatibility policy

AbstractVision treats model discovery, curated downloads, and runtime compatibility as part of the
product surface:

- the registry is global and platform-neutral;
- host-specific recommendation happens later in catalog or preset selection;
- one parent model family may expose multiple engine-specific variants;
- official upstream repos are preferred, with curated community ports used only when they provide
  the best runtime-native artifact for a target engine.

That policy is defined in [ADR 0005](adr/0005_curated_capability_registry_and_download_catalog.md).

## AbstractCore plugin flow (framework integration)

AbstractVision can be discovered by AbstractCore via an entry point:
`[project.entry-points."abstractcore.capabilities_plugins"]` in [`../pyproject.toml`](../pyproject.toml).

```mermaid
flowchart LR
  AC[AbstractCore] -->|loads entry point| Plugin[AbstractVision plugin<br/>register(...)]
  Plugin --> Cap[VisionCapability<br/>(t2i/i2i/t2v/i2v)]
  Cap --> VM[VisionManager]
  VM --> BE{Configured backend}
  BE --> HTTP[OpenAI-compatible HTTP<br/>OpenAI or local /v1 server]
  BE --> HF[Local Diffusers]
  BE --> SDCPP[Local stable-diffusion.cpp]
```

Current plugin behavior (evidence in [`../src/abstractvision/integrations/abstractcore_plugin.py`](../src/abstractvision/integrations/abstractcore_plugin.py)):
- Default: OpenAI HTTP with backend id `abstractvision:openai`; the legacy backend id `abstractvision:openai-compatible` remains registered and preserves compatible-endpoint defaults when selected directly.
- Compatible endpoints should set `OPENAI_BASE_URL`; set `ABSTRACTVISION_BACKEND=openai-compatible` when you want to force compatible-endpoint semantics.
- Local Diffusers and stable-diffusion.cpp are supported when `vision_backend` / `ABSTRACTVISION_BACKEND` selects `diffusers` or `sdcpp`.
- Configuration is read from `owner.config` keys like `vision_base_url`, `vision_model_id`, `vision_backend`, and backend-specific keys, then falls back to `ABSTRACTVISION_*` and standard OpenAI env vars where relevant.

## Extending AbstractVision (practical steps)

- Add a new backend:
  1) Implement `VisionBackend` ([`../src/abstractvision/backends/base_backend.py`](../src/abstractvision/backends/base_backend.py))
  2) Add capability reporting via `get_capabilities()` when you can (optional)
  3) Add tests under [`../tests/`](../tests/)
- Update the registry:
  1) Edit [`../src/abstractvision/assets/vision_model_capabilities.json`](../src/abstractvision/assets/vision_model_capabilities.json)
  2) Validate by running the test suite (validator is wired into the registry loader)
  3) Use `abstractvision show-model <id>` to sanity-check task/param printing ([`../src/abstractvision/cli.py`](../src/abstractvision/cli.py))
--- >8 --- END FILE: docs/architecture.md --- >8 ---

--- 8< --- FILE: docs/faq.md --- 8< ---
# FAQ

See also:
- Getting started: [docs/getting-started.md](getting-started.md)
- API reference: [docs/api.md](api.md)
- Architecture: [docs/architecture.md](architecture.md)
- Backends: [docs/reference/backends.md](reference/backends.md)
- Configuration: [docs/reference/configuration.md](reference/configuration.md)

## What is AbstractVision?

AbstractVision is a small, model-agnostic API for **generative vision outputs** (images, optional video) with:
- a small orchestrator ([`VisionManager`](../src/abstractvision/vision_manager.py))
- pluggable execution engines (“backends”) in [`../src/abstractvision/backends/`](../src/abstractvision/backends/)
- a packaged capability registry ([`vision_model_capabilities.json`](../src/abstractvision/assets/vision_model_capabilities.json))
- optional artifact-ref outputs via stores ([`../src/abstractvision/artifacts.py`](../src/abstractvision/artifacts.py))

## How does AbstractVision fit into AbstractFramework?

AbstractVision is part of the **AbstractFramework** ecosystem:

- **AbstractFramework** (project hub): <https://github.com/lpalbou/AbstractFramework>
- **AbstractCore** (orchestration + tool calling): <https://github.com/lpalbou/abstractcore>
- **AbstractRuntime** (runtime services, including artifact storage): <https://github.com/lpalbou/abstractruntime>

Where AbstractVision fits:
- It standardizes *generative vision outputs* behind `VisionManager` (library mode).
- AbstractCore can discover and use AbstractVision via the capability plugin (see [`../src/abstractvision/integrations/abstractcore_plugin.py`](../src/abstractvision/integrations/abstractcore_plugin.py) and the entry point in [`../pyproject.toml`](../pyproject.toml)).
- Artifact refs are designed to cross process boundaries; `RuntimeArtifactStoreAdapter` bridges to an AbstractRuntime-style artifact store (see [`../src/abstractvision/artifacts.py`](../src/abstractvision/artifacts.py)).

## What does AbstractVision support today?

- Built-in backends implement **images**: `text_to_image` and `image_to_image`.
- Local MLX-Gen supports curated q4/q8 Apple Silicon image presets, official FIBO image snapshots, and Wan 2.2 TI2V plus task-specific Wan 2.2 A14B `text_to_video` / first-frame `image_to_video`.
- Local Diffusers `text_to_video` remains experimental and is temporarily disabled from the normal local runtime surfaces.
- OpenAI-compatible HTTP can also provide `text_to_video` / `image_to_video` **when** video endpoints are configured.
- `multi_view_image` exists in the public API (`VisionManager.generate_angles`) but no built-in backend implements it yet (they raise `CapabilityNotSupportedError`).

Details: [docs/reference/backends.md](reference/backends.md).

## Which backend should I use?

- **OpenAI-compatible HTTP** ([`../src/abstractvision/backends/openai_compatible.py`](../src/abstractvision/backends/openai_compatible.py)): call a server that exposes OpenAI-shaped image endpoints (and optional video endpoints).
- **Diffusers (local)** ([`../src/abstractvision/backends/huggingface_diffusers.py`](../src/abstractvision/backends/huggingface_diffusers.py)): run Diffusers pipelines locally (heavy deps). Local `text_to_video` groundwork exists but is currently quarantined from the normal local surfaces.
- **MLX-Gen (local Apple Silicon)** ([`../src/abstractvision/backends/mflux.py`](../src/abstractvision/backends/mflux.py)): run MLX-optimized image models, FIBO snapshots, and Wan 2.2 video locally.
- **stable-diffusion.cpp (local GGUF)** ([`../src/abstractvision/backends/stable_diffusion_cpp.py`](../src/abstractvision/backends/stable_diffusion_cpp.py)): run GGUF diffusion models via `sd-cli` or `stable-diffusion-cpp-python`.

## What model should I start with (local)?

If you’re running locally via the Diffusers backend and want a reliable starting point, we recommend:

- **Default / ≤16GB VRAM (cross-platform)**: `runwayml/stable-diffusion-v1-5`

Quickstart:

```bash
abstractvision download stable-diffusion --provider diffusers
export ABSTRACTVISION_BACKEND=diffusers
export ABSTRACTVISION_MODEL_ID=runwayml/stable-diffusion-v1-5
export ABSTRACTVISION_DIFFUSERS_DEVICE=auto
abstractvision cli
```

More model recommendations (by VRAM tier) are in [docs/getting-started.md](getting-started.md).

After that works, `black-forest-labs/FLUX.2-klein-4B` is the recommended next local test for a newer non-gated model
(it currently requires Diffusers from source).

## Do the one-shot CLI commands run locally?

Yes. `abstractvision t2i` / `abstractvision i2i` / `abstractvision t2v` default to the OpenAI-compatible HTTP backend, but they also support local providers through `--provider diffusers`, `--provider mlx-gen`, or `--provider sdcpp` ([`../src/abstractvision/cli.py`](../src/abstractvision/cli.py)). The legacy `mflux` provider value is still accepted as an alias.

For interactive local generation, use `abstractvision cli` (legacy alias: `abstractvision repl`) with `/backend diffusers ...`, `/backend mlx-gen ...`, or `/backend sdcpp ...`.

Current local video note:
- local Diffusers `t2v` is currently experimental and disabled from the normal bundled local surfaces;
- local MLX-Gen `t2v` / `i2v` is available through task-specific Wan A14B packages such as `AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit` and `AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit`;
- MLX-Gen denoise-step video progress is shown by default in the shell and interactive CLI and is available to Python/Core callers through `on_progress(event)`;
- remote `text_to_video` / `image_to_video` still depend on the OpenAI-compatible backend being configured with video endpoints.

## Where do generated outputs go?

It depends on whether you configured a store:

- **CLI/REPL**: stores outputs in a local store by default (`LocalAssetStore`), under `~/.abstractvision/assets` unless `ABSTRACTVISION_STORE_DIR` is set ([`../src/abstractvision/artifacts.py`](../src/abstractvision/artifacts.py), [`../src/abstractvision/cli.py`](../src/abstractvision/cli.py)).
- **Python**:
  - if `VisionManager.store` is set, methods return an artifact ref dict (stored via `store.store_bytes(...)`)
  - otherwise they return a `GeneratedAsset` containing bytes ([`../src/abstractvision/types.py`](../src/abstractvision/types.py))

## What is an “artifact ref”?

An artifact ref is a small JSON dict that points to a stored blob. Minimal shape:

```json
{"$artifact":"<id>"}
```

Helpers: `is_artifact_ref()` / `make_media_ref()` in [`../src/abstractvision/artifacts.py`](../src/abstractvision/artifacts.py).

## How do I allow or block Diffusers downloads?

- REPL: cache-only is the default. Pre-download models separately, or set `ABSTRACTVISION_DIFFUSERS_ALLOW_DOWNLOAD=1` when you intentionally want runtime downloads ([`../src/abstractvision/cli.py`](../src/abstractvision/cli.py)).
- Python: `HuggingFaceDiffusersBackendConfig` defaults to `allow_download=False`; set `allow_download=True` only when you want runtime downloads ([`../src/abstractvision/backends/huggingface_diffusers.py`](../src/abstractvision/backends/huggingface_diffusers.py)).

## Why do I get “missing pipeline class” errors (e.g. `GlmImagePipeline`)?

Some newer pipelines may only exist on Diffusers GitHub `main`. Install:

- `pip install -U "abstractvision[diffusers-dev]"` (compatible dependency versions)
- `pip install -U "git+https://github.com/huggingface/diffusers@main"` (Diffusers `main`)

See: [docs/getting-started.md](getting-started.md).

## macOS (MPS): why do I get black images / dtype errors?

The Diffusers backend includes MPS-specific mitigations (e.g. VAE upcast and optional fp32 retry) in [`../src/abstractvision/backends/huggingface_diffusers.py`](../src/abstractvision/backends/huggingface_diffusers.py).

Common fixes:
- set `ABSTRACTVISION_DIFFUSERS_TORCH_DTYPE=float32` (more stable, higher memory)
- disable retry if memory is tight: `ABSTRACTVISION_DIFFUSERS_AUTO_RETRY_FP32=0`
- consider using the stable-diffusion.cpp backend for GGUF diffusion models ([docs/getting-started.md](getting-started.md))

## Windows/Linux (CUDA): why is `torch.cuda.is_available()` false?

On Windows/Linux, `pip install torch` (and packages that depend on `torch`) may install a CPU-only PyTorch build by default.

If you have an NVIDIA GPU and want CUDA acceleration:

1) Install a CUDA-enabled PyTorch wheel using the official selector: <https://pytorch.org/get-started/locally/>  
2) Verify:

```bash
python -c "import torch; print('cuda', torch.cuda.is_available())"
```

## How do I pass advanced flags / parameters?

AbstractVision exposes an `extra` dict on requests ([`../src/abstractvision/types.py`](../src/abstractvision/types.py)), and the REPL forwards unknown `--flags` into `request.extra` ([`../src/abstractvision/cli.py`](../src/abstractvision/cli.py)).

Examples:
- Diffusers backend: accepts extra keys like `loras_json` and `rapid_aio_repo` (used by Qwen Image Edit flows; see [docs/getting-started.md](getting-started.md) and [`../src/abstractvision/backends/huggingface_diffusers.py`](../src/abstractvision/backends/huggingface_diffusers.py)).
- stable-diffusion.cpp backend:
  - CLI mode forwards flags to `sd-cli`
  - python-binding mode maps supported keys to binding kwargs and ignores unsupported keys ([`../src/abstractvision/backends/stable_diffusion_cpp.py`](../src/abstractvision/backends/stable_diffusion_cpp.py))

## What does the capability registry mean (and what does it not mean)?

The registry answers “what a model *claims* to support” (task keys/params) and can be used for **optional gating**:

- `VisionModelCapabilitiesRegistry.supports(...)` / `.require_support(...)` ([`../src/abstractvision/model_capabilities.py`](../src/abstractvision/model_capabilities.py))
- `VisionManager(model_id=...)` uses it to fail fast before calling a backend ([`../src/abstractvision/vision_manager.py`](../src/abstractvision/vision_manager.py))

It does **not** guarantee your configured backend can execute the task; backend support is a separate constraint ([docs/reference/backends.md](reference/backends.md)).

## I only need the HTTP backend. Do I have to install Torch/Diffusers?

No. The base install is lightweight and includes the stdlib OpenAI-compatible HTTP backend without Torch/Diffusers (see [`../pyproject.toml`](../pyproject.toml)). Heavy local backend modules are still imported lazily ([`../src/abstractvision/backends/__init__.py`](../src/abstractvision/backends/__init__.py)).

Install `abstractvision[diffusers]` only when you want local Diffusers generation. Use `abstractvision[sdcpp]` or an external `sd-cli` only when you need stable-diffusion.cpp.

## How do I integrate with AbstractCore?

Two options (details in [docs/reference/abstractcore-integration.md](reference/abstractcore-integration.md)):

- **Capability plugin**: [`../src/abstractvision/integrations/abstractcore_plugin.py`](../src/abstractvision/integrations/abstractcore_plugin.py) supports Diffusers, stable-diffusion.cpp, and OpenAI-compatible backends through env/config.
- **Tool helpers**: `make_vision_tools(...)` in [`../src/abstractvision/integrations/abstractcore.py`](../src/abstractvision/integrations/abstractcore.py) requires `VisionManager.store` for artifact-ref outputs.

AbstractCore is the host package; AbstractVision does not install it as a dependency.

## How do I run tests?

From the repo root:

```bash
python -m unittest discover -s tests -p "test_*.py" -q
```
--- >8 --- END FILE: docs/faq.md --- >8 ---

--- 8< --- FILE: docs/reference/backends.md --- 8< ---
# Backends (execution engines)

AbstractVision executes tasks via a `VisionBackend` adapter ([`../../src/abstractvision/backends/base_backend.py`](../../src/abstractvision/backends/base_backend.py)).
`VisionManager` is intentionally thin and delegates to the configured backend ([`../../src/abstractvision/vision_manager.py`](../../src/abstractvision/vision_manager.py)).

See also:
- Getting started (interactive CLI examples): [docs/getting-started.md](../getting-started.md)
- Configuration (env vars / CLI flags): [docs/reference/configuration.md](configuration.md)

## Support matrix (built-in backends)

| Backend | Implementation | Tasks implemented | Notes |
|---|---|---|---|
| OpenAI-compatible HTTP | [`openai_compatible.py`](../../src/abstractvision/backends/openai_compatible.py) | `text_to_image`, `image_to_image` (+ optional `text_to_video`, `image_to_video`) | Stdlib-only (`urllib`). Video is **opt-in** via configured paths. |
| Diffusers (local) | [`huggingface_diffusers.py`](../../src/abstractvision/backends/huggingface_diffusers.py) | `text_to_image`, `image_to_image` | Requires `abstractvision[diffusers]`. Supports cache-only/offline mode. Local `text_to_video` groundwork exists but is currently experimental and disabled from the normal local surfaces. |
| MLX-Gen (local, Apple Silicon) | [`mflux.py`](../../src/abstractvision/backends/mflux.py) | `text_to_image`, `image_to_image`, `text_to_video`, `image_to_video` | Requires `abstractvision[mlx-gen]` (or compatibility extra `abstractvision[mflux]`, or `abstractvision[all-apple]`). Uses downloaded AbstractFramework q4/q8 MLX-Gen image preset snapshots plus official FIBO/Wan runtime snapshots from the Hugging Face cache; `ABSTRACTVISION_MODEL_DIR` is legacy migration input only. |
| stable-diffusion.cpp (local GGUF/checkpoints) | [`stable_diffusion_cpp.py`](../../src/abstractvision/backends/stable_diffusion_cpp.py) | `text_to_image`, `image_to_image` | Uses external `sd-cli` if present, else `abstractvision[sdcpp]` python bindings. Start with single-file Stable Diffusion models; curated Qwen/FLUX GGUF presets now auto-resolve required VAE + LLM companions from the cache. |

Notes:
- `multi_view_image` (`VisionManager.generate_angles`) is part of the public API, but **no built-in backend implements it yet** (all raise `CapabilityNotSupportedError` today).
- Backends may also expose best-effort `get_capabilities()`, `preload()`, `unload()`, `generate_image_with_progress(...)`, `edit_image_with_progress(...)`, and video progress hooks via the shared `VisionBackend` contract.
- Backends may also implement `normalize_image_generation_request(...)`, `normalize_image_edit_request(...)`, `normalize_video_generation_request(...)`, and `normalize_image_to_video_request(...)`. `VisionManager`, the CLI/REPL, the playground API, and the AbstractCore plugin all route through those hooks so model-specific defaults and constraints are applied consistently instead of being hard-coded in one surface.

## OpenAI-compatible HTTP backend

**When to use**
- You already run a service that exposes OpenAI-shaped endpoints (local or remote).
- You want to keep inference out-of-process.

**Core config**
- `base_url` (required): points to a `/v1`-style root, e.g. `http://localhost:1234/v1`
- `api_key` (optional): sent as `Authorization: Bearer ...`
- `model_id` (optional): forwarded as `model` in requests
- `models_path` (default `/models`): provider catalog path for explicit model listing

Request shape:
- Unknown/local endpoints receive local extension fields when provided, including `steps`, `seed`, `guidance_scale`, `negative_prompt`, `width`, and `height`.
- Real OpenAI-looking endpoints and known OpenAI image models use the narrower OpenAI request shape; GPT image models do not receive unsupported local-only fields such as `steps`, `seed`, or `guidance_scale`.

Provider model catalogs:
- `OpenAICompatibleVisionBackend.list_provider_models(...)` queries `GET /models` by default.
- `VisionManager.list_provider_models(...)` delegates to the configured backend.
- The AbstractCore plugin exposes the same catalog through `llm.vision.list_provider_models(...)`.
- CLI examples: `abstractvision provider-models --openai --task text_to_image` and `abstractvision provider-models --base-url http://localhost:1234/v1 --task text_to_image`.
- Listing is explicit; AbstractVision does not use provider catalogs to silently select a model.

Code pointers:
- Config: `OpenAICompatibleBackendConfig` ([`../../src/abstractvision/backends/openai_compatible.py`](../../src/abstractvision/backends/openai_compatible.py))
- Backend: `OpenAICompatibleVisionBackend` ([`../../src/abstractvision/backends/openai_compatible.py`](../../src/abstractvision/backends/openai_compatible.py))

**Video endpoints (optional)**
`OpenAICompatibleVisionBackend` only enables:
- `text_to_video` if `text_to_video_path` is set
- `image_to_video` if `image_to_video_path` is set

## Diffusers backend (local)

**When to use**
- You want local inference for Diffusers pipelines.
- Start with `runwayml/stable-diffusion-v1-5` for the lowest-risk local test.
- Move to `black-forest-labs/FLUX.2-klein-4B` after that if you want a newer non-gated model and can install Diffusers `main`.

Install:
- `pip install "abstractvision[diffusers]"`
- For newer/unreleased pipeline classes: `pip install "abstractvision[diffusers-dev]"` plus Diffusers from source.

Model downloads (curated):
- See what's downloadable for Diffusers:
  - `abstractvision catalog --provider diffusers` (add `--all-targets` to compare engines)
  - Tip: `--provider diffusers` implies `--target diffusers` (you usually set one or the other).
- Download a curated Diffusers snapshot into the Hugging Face cache (legacy `~/models` trees auto-migrate when encountered):
  - `abstractvision download stable-diffusion --provider diffusers`
  - `abstractvision download sd1.4 --provider diffusers`
  - `abstractvision download sd1.5-inpaint --provider diffusers`
  - `abstractvision download instruct-pix2pix --provider diffusers`
  - `abstractvision download sdxl-base --provider diffusers`
  - `abstractvision download sdxl-refiner --provider diffusers`
  - `abstractvision download sdxl-inpaint --provider diffusers`
  - `abstractvision download sdxl-turbo --provider diffusers`
  - `abstractvision download sd-turbo --provider diffusers`
  - `abstractvision download sd3-medium --provider diffusers`
  - `abstractvision download sd3.5-medium --provider diffusers`
  - `abstractvision download sd3.5-large --provider diffusers`
  - `abstractvision download sd3.5-large-turbo --provider diffusers`
  - `abstractvision download ernie-image --provider diffusers`
  - `abstractvision download qwen-image --provider diffusers`
  - `abstractvision download qwen-image-edit-2511 --provider diffusers`
  - `abstractvision download flux2-dev --provider diffusers`
  - `abstractvision download flux2-klein-4b --provider diffusers`
  - `abstractvision download z-image-turbo --provider diffusers`

One-shot generation (uses the cached snapshot when present):
- `abstractvision t2i --provider diffusers --model qwen-image "a studio photo of a ceramic teapot"`

Code pointers:
- Config: `HuggingFaceDiffusersBackendConfig` ([`../../src/abstractvision/backends/huggingface_diffusers.py`](../../src/abstractvision/backends/huggingface_diffusers.py))
- Backend: `HuggingFaceDiffusersVisionBackend` ([`../../src/abstractvision/backends/huggingface_diffusers.py`](../../src/abstractvision/backends/huggingface_diffusers.py))

**Offline / cache-only mode**
The Python backend and interactive CLI are cache-only by default (`allow_download=False`). Pre-download model weights separately,
or set `allow_download=True` / `ABSTRACTVISION_DIFFUSERS_ALLOW_DOWNLOAD=1` when runtime downloads are desired (see
config/env in [docs/reference/configuration.md](configuration.md)).

Config fields:
- `model_id`, `device`, `torch_dtype`
- `allow_download`, `auto_retry_fp32`
- `cache_dir`, `revision`, `variant`
- `use_safetensors`, `low_cpu_mem_usage`

Runtime behavior notes:
- The Diffusers backend now reads packaged registry task metadata for known models when it normalizes requests.
- Local Diffusers `GLM-Image` is temporarily disabled for both `text_to_image` and `image_to_image` pending the follow-up in [`../backlog/planned/0023_local_runtime_capability_quarantine_for_glm_mflux_and_t2v.md`](../backlog/planned/0023_local_runtime_capability_quarantine_for_glm_mflux_and_t2v.md).
- Local Diffusers `text_to_video` is currently experimental and disabled from the normal local surfaces.
- Local video export still requires an `ffmpeg` executable on `PATH` whenever a local backend emits frames for MP4 packaging.
- This backend is where model-specific defaults and constraints such as packaged step counts, guidance defaults, dimension constraints, and unsupported-parameter dropping are enforced for all callers.

## MLX-Gen backend (local Apple Silicon)

**When to use**
- You are on Apple Silicon and want local quantized MLX generation through the optional MLX-Gen runtime.
- You want the AbstractFramework-published q4/q8 prepared folders from the [AbstractFramework/mlx-gen Hugging Face collection](https://huggingface.co/collections/AbstractFramework/mlx-gen/).
- You want the official MLX-Gen 0.18.13+ FIBO snapshots (`briaai/FIBO`, `briaai/Fibo-lite`, `briaai/Fibo-Edit`, `briaai/Fibo-Edit-RMBG`).
- You want the official Prism ML Bonsai ternary 2-bit checkpoint (`prism-ml/bonsai-image-ternary-4B-mlx-2bit`) for very small local `text_to_image`.
- You want SeedVR2 single-image upscaling through canonical `AbstractFramework/seedvr2-{3b,7b}-{8bit,4bit}` packages or the official `ByteDance-Seed/SeedVR2-*` bases.
- You want local Wan 2.2 video generation through MLX-Gen 0.18.13+, including the task-specific A14B `text_to_video` and first-frame `image_to_video` packages.

Install:
- `pip install "abstractvision[mlx-gen]"` (or `pip install "abstractvision[all-apple]"`)
- `pip install "abstractvision[mflux]"` remains a compatibility alias for older install instructions.

Model presets:
- See what's downloadable for your machine/engine:
  - `abstractvision catalog --provider mlx-gen` (add `--all` for full fallback list)
  - Tip: `--provider mlx-gen` implies `--target mlx` (you usually set one or the other).
- Download an exact published model repo into the Hugging Face cache (legacy `~/models` trees auto-migrate when encountered):
  - `abstractvision download AbstractFramework/flux.2-klein-4b-4bit --provider mlx-gen`
  - `abstractvision download AbstractFramework/flux.2-klein-4b-8bit --provider mlx-gen`
  - `abstractvision download AbstractFramework/qwen-image-2512-4bit --provider mlx-gen`
  - `abstractvision download AbstractFramework/qwen-image-edit-2511-4bit --provider mlx-gen`
  - `abstractvision download AbstractFramework/z-image-4bit --provider mlx-gen`
  - `abstractvision download AbstractFramework/z-image-turbo-4bit --provider mlx-gen`
  - `abstractvision download AbstractFramework/ernie-image-turbo-4bit --provider mlx-gen`
  - `abstractvision download AbstractFramework/ernie-image-turbo-8bit --provider mlx-gen`
  - `abstractvision download briaai/FIBO --provider mlx-gen`
  - `abstractvision download briaai/Fibo-lite --provider mlx-gen`
  - `abstractvision download briaai/Fibo-Edit --provider mlx-gen`
  - `abstractvision download briaai/Fibo-Edit-RMBG --provider mlx-gen`
  - `abstractvision download prism-ml/bonsai-image-ternary-4B-mlx-2bit --provider mlx-gen`
  - `abstractvision download Wan-AI/Wan2.2-TI2V-5B-Diffusers --provider mlx-gen`
  - `abstractvision download AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit --provider mlx-gen`
  - `abstractvision download AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit --provider mlx-gen`
  - `abstractvision download AbstractFramework/seedvr2-3b-8bit --provider mlx-gen`
  - `abstractvision download AbstractFramework/seedvr2-7b-8bit --provider mlx-gen`
- q4 repos are the default recommendation for most generation/edit models. Use the exact
  matching `AbstractFramework/...-8bit` model id when quality is paramount. Qwen
  and ERNIE q4 prepared folders can mix q4 and q8 components, but remain the
  default prepared choice.
- `abstractvision t2i`, `abstractvision i2i`, and Python callers select q4/q8
  by exact model id. Quantization is metadata of the published folder, not a
  generation-time parameter.
- `abstractvision upscale` defaults to `--provider mlx-gen --model AbstractFramework/seedvr2-3b-8bit`.
  Use `--scale 2x` for normal upscaling, `--resolution <pixels>` for a target
  short edge, `AbstractFramework/seedvr2-7b-8bit` when memory allows, and q4
  SeedVR2 packages only when memory is tight. `--quantize` is reserved for
  official/source-weight runs.
- SeedVR2 model chooser:
  - `AbstractFramework/seedvr2-3b-8bit`: default q8 package.
  - `AbstractFramework/seedvr2-7b-8bit`: higher-quality 7B package when memory allows.
  - `AbstractFramework/seedvr2-3b-4bit` / `AbstractFramework/seedvr2-7b-4bit`: lower-memory packages.
  - `ByteDance-Seed/SeedVR2-3B` / `ByteDance-Seed/SeedVR2-7B`: official source weights; pass `--quantize 8` or `--quantize 4` if runtime quantization is needed.
- Prepared local folders work as model values. For example:
  `abstractvision upscale --provider mlx-gen --model /Users/albou/projects/gh/sbx/mlx-gen/models/seedvr2-7b-8bit --image ./input.png --scale 2x --open`.
- Bonsai ternary is a pre-packed low-bit MLX artifact, not a q4/q8 prepared folder. Use the exact repo id; guidance is fixed at 1.0 and negative prompts are ignored. The binary 1-bit Bonsai checkpoint is not surfaced because stock MLX cannot run it yet.
- Current shipped backend coverage includes `text_to_image` for FLUX.2 klein/base, Qwen Image, Z-Image, Z-Image Turbo, ERNIE Image Turbo, FIBO, Fibo-lite, and Bonsai ternary. `image_to_image` edits are implemented for FLUX.2 klein/base, Qwen Image Edit, ERNIE Image Turbo, FIBO, Fibo-lite, Fibo-Edit, and Fibo-Edit-RMBG; FIBO Edit snapshots support masks where the runtime supports them. `image_upscale` is implemented for SeedVR2.

One-shot shell commands:

```bash
abstractvision t2i --provider mlx-gen --model AbstractFramework/qwen-image-2512-4bit "a studio product photo of a white ceramic mug with the AbstractFramework logo" --steps 20 --guidance-scale 1.0 --open
abstractvision i2i --provider mlx-gen --model AbstractFramework/qwen-image-edit-2511-4bit --image ./input.png "replace the background with a clean white studio setup" --steps 20 --guidance-scale 2.5 --strength 0.75 --open
abstractvision t2i --provider mlx-gen --model briaai/FIBO "a studio product photo of a white ceramic mug with the AbstractFramework logo" --steps 50 --guidance-scale 4.0 --open
abstractvision t2i --provider mlx-gen --model prism-ml/bonsai-image-ternary-4B-mlx-2bit "a bonsai tree in a quiet ceramic studio" --steps 4 --guidance-scale 1.0 --open
abstractvision i2i --provider mlx-gen --model briaai/Fibo-Edit --image ./input.png "remove the background and keep the object edges clean" --steps 20 --guidance-scale 4.0 --open
abstractvision upscale --provider mlx-gen --model AbstractFramework/seedvr2-3b-8bit --image ./input.png --scale 2x --open
abstractvision upscale --provider mlx-gen --model /Users/albou/projects/gh/sbx/mlx-gen/models/seedvr2-7b-8bit --image ./input.png --scale 2x --open
abstractvision t2v --provider mlx-gen --model AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit "a red fox walking through a snowy forest, cinematic" --width 432 --height 240 --frames 41 --fps 10 --steps 20 --guidance-scale 4.0 --guidance-2 3.0 --open
abstractvision i2v --provider mlx-gen --model AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit --image ./first-frame.png "slow camera push-in" --width 432 --height 240 --frames 41 --fps 10 --steps 20 --guidance-scale 3.5 --guidance-2 3.5 --open
```

MLX-Gen image, edit, and Wan video progress is surfaced as normalized progress
events. Shell image commands accept `--progress`; shell video commands and the
interactive CLI render denoise-step progress with frame context by default.
Python and AbstractCore callers can pass `on_progress(event)` to receive the
same events. Image events carry denoise-step `progress`; video events also carry
`frame`, `total_frames`, and `frame_progress`.

Interactive CLI/REPL commands:

```text
/backend mlx-gen AbstractFramework/qwen-image-2512-4bit
/t2i "a studio product photo of a white ceramic mug with the AbstractFramework logo" --steps 20 --guidance-scale 1.0 --open
/backend mlx-gen AbstractFramework/qwen-image-edit-2511-4bit
/i2i --image ./input.png "replace the background with a clean white studio setup" --steps 20 --guidance-scale 2.5 --strength 0.75 --open
/backend mlx-gen briaai/FIBO
/t2i "a studio product photo of a white ceramic mug with the AbstractFramework logo" --steps 50 --guidance-scale 4.0 --open
/backend mlx-gen prism-ml/bonsai-image-ternary-4B-mlx-2bit
/t2i "a bonsai tree in a quiet ceramic studio" --steps 4 --guidance-scale 1.0 --open
/backend mlx-gen AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit
/t2v "a red fox walking through a snowy forest, cinematic" --width 432 --height 240 --frames 41 --fps 10 --steps 20 --guidance-scale 4.0 --guidance-2 3.0 --open
/backend mlx-gen AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit
/i2v --image ./first-frame.png "slow camera push-in" --width 432 --height 240 --frames 41 --fps 10 --steps 20 --guidance-scale 3.5 --guidance-2 3.5 --open
```

Config/env:
- `ABSTRACTVISION_PROVIDER=mlx-gen` (alias: `ABSTRACTVISION_BACKEND=mlx-gen`)
- `ABSTRACTVISION_MFLUX_MODEL=AbstractFramework/flux.2-klein-4b-4bit` (or routed ids like `mlx-gen/AbstractFramework/flux.2-klein-4b-4bit`)
- Optional: `ABSTRACTVISION_MFLUX_BASE_MODEL`, `ABSTRACTVISION_MFLUX_ALLOW_DOWNLOAD`, `ABSTRACTVISION_MODEL_DIR` (legacy migration root only)
- Legacy `mflux` provider values and routed ids remain accepted as compatibility aliases.

Non-curated MLX-Gen models:
- If you have an MLX-Gen-compatible Hugging Face repo id that is not in `model-presets`, you can still use it:
  - Pre-download it with `abstractvision download org/name` (HF cache) or `hf download org/name`
  - Set `ABSTRACTVISION_MFLUX_MODEL` to that repo id or local path (base model usually auto-infers; override with `ABSTRACTVISION_MFLUX_BASE_MODEL=qwen-image` if needed).

Runtime behavior notes:
- MLX-Gen request normalization is backend-level, so model constraints such as fixed guidance for turbo/distilled families, minimum step counts, and unsupported negative prompts are handled the same way through the CLI/REPL, playground API, and AbstractCore.
- Local MLX-Gen `image_to_image` is supported for FLUX.2 klein/base, Qwen Image Edit, ERNIE Image Turbo, FIBO, and FIBO Edit models. Edit strength is passed as `strength` and normalized to MLX-Gen's `image_strength` parameter where the runtime supports it.
- Local MLX-Gen video is implemented for `Wan-AI/Wan2.2-TI2V-5B-Diffusers` and the task-specific Wan A14B packages `AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit` / `AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit`. Wan A14B dimensions must be multiples of 16; `432x240` is valid for low-cost routing checks, while larger native sizes should be used for quality review.
- Task-specific Wan A14B uses two guidance controls. `guidance_scale` controls the primary/high-noise stage and `guidance_2` controls the second-stage/low-noise stage. `guidance_2` is a typed video request field, CLI/REPL flag `--guidance-2`, and registry parameter default; it is not passed through `extra`.
- Wan requests can pass `max_sequence_length` through Python `extra={...}` or CLI/REPL `--max-sequence-length`.
- Generation does not silently download model files. Missing-cache errors tell you which `abstractvision download ... --provider mlx-gen` or `mlxgen` preparation step is needed.

Code pointers:
- Config: `MLXGenBackendConfig` / compatibility alias `MFluxBackendConfig` ([`../../src/abstractvision/backends/mflux.py`](../../src/abstractvision/backends/mflux.py))
- Backend: `MLXGenVisionBackend` / compatibility alias `MFluxVisionBackend` ([`../../src/abstractvision/backends/mflux.py`](../../src/abstractvision/backends/mflux.py))

## stable-diffusion.cpp backend (local GGUF/checkpoints)

**When to use**
- You want to run single-file Stable Diffusion checkpoints/GGUF or component-based GGUF diffusion models locally.

Runtime modes (auto-selected):
- **CLI mode** via `sd-cli` (stable-diffusion.cpp executable) when available in `PATH`
- **Python mode** via `stable-diffusion-cpp-python` when `sd-cli` is not available

Notes:
- If you care about **GPU acceleration** (macOS **Metal**, NVIDIA **CUDA**, etc.), prefer **CLI mode** via `sd-cli`.
- Python bindings run whatever backend the installed wheel was built with. On macOS, that often means **CPU-only**, so FLUX/Qwen-class models can be extremely slow.
- Operators who want to hide GGUF presets (and reject GGUF execution) on macOS can set `ABSTRACTVISION_DISABLE_GGUF_ON_MACOS=1`.
- The optional python binding is constrained below `0.4.6` because that sdist
  currently misses vendored CMake files needed by native Linux builds.
- Interactive CLI selection supports both `/backend sdcpp <model_key|model.gguf|model.safetensors> [sd_cli_path]` and
  `/backend sdcpp <diffusion_model.gguf> <vae.safetensors> <llm.gguf> [sd_cli_path]`.
- One-shot CLI, playground, and the AbstractCore plugin can also accept curated `sdcpp` model keys such as
  `flux2-klein-base-4b` or `qwen-image` after `abstractvision download ... --provider sdcpp`.
- Python code and AbstractCore plugin configuration can also pass component paths such as `clip_l`, `clip_g`, `t5xxl`, `llm_vision`, plus `extra_args`, `timeout_s`, and `cwd`.

Code pointers:
- Config: `StableDiffusionCppBackendConfig` ([`../../src/abstractvision/backends/stable_diffusion_cpp.py`](../../src/abstractvision/backends/stable_diffusion_cpp.py))
- Backend: `StableDiffusionCppVisionBackend` ([`../../src/abstractvision/backends/stable_diffusion_cpp.py`](../../src/abstractvision/backends/stable_diffusion_cpp.py))
--- >8 --- END FILE: docs/reference/backends.md --- >8 ---

--- 8< --- FILE: docs/reference/configuration.md --- 8< ---
# Configuration (CLI)

AbstractVision configuration is intentionally simple:

- In Python, you configure backends by instantiating backend config objects (see [docs/reference/backends.md](backends.md)).
- The CLI/playground reads `ABSTRACTVISION_*` environment variables to set defaults ([`../../src/abstractvision/cli.py`](../../src/abstractvision/cli.py), [`../../src/abstractvision/playground_server.py`](../../src/abstractvision/playground_server.py)). The interactive command is `abstractvision cli` (legacy alias: `abstractvision repl`).
- The AbstractCore capability plugin also reads `owner.config` plus a small set of standard OpenAI environment aliases. Plugin-only aliases are called out below.

See also:
- Getting started (examples): [docs/getting-started.md](../getting-started.md)
- Backends: [docs/reference/backends.md](backends.md)

## CLI commands (overview)

Implemented in [`../../src/abstractvision/cli.py`](../../src/abstractvision/cli.py):

- `abstractvision models` — list known registry model ids
- `abstractvision tasks` — list known tasks
- `abstractvision show-model <id>` — print a model’s tasks + params
- `abstractvision provider-models --openai --task text_to_image` — explicitly query the official OpenAI `/models` catalog
- `abstractvision provider-models --base-url http://localhost:1234/v1 --task text_to_image` — explicitly query an OpenAI-compatible provider catalog
- `abstractvision cli` — interactive testing (supports `openai`, `diffusers`, `mlx-gen`, `sdcpp`; legacy alias: `abstractvision repl`; `mflux` is accepted as a compatibility alias)
- `abstractvision playground [--host 127.0.0.1] [--port 8091]` — self-contained local web UI and `/v1/vision/*` API
- `abstractvision serve [--host 127.0.0.1] [--port 8091]` — alias for `abstractvision playground`
- `abstractvision t2i ...` / `abstractvision i2i ...` / `abstractvision t2v ...` / `abstractvision i2v ...` — one-shot commands using the configured provider/backend (`openai`/`openai-compatible` by default; also supports local `diffusers`, `mlx-gen`, and `sdcpp`; MLX-Gen 0.18.13+ provides image progress, multi-reference edits, SeedVR2 upscaling, and Wan `t2v`/`i2v`, including A14B task-specific checkpoints)
- `abstractvision upscale ...` — one-shot SeedVR2 image upscaling; defaults to `--provider mlx-gen --model AbstractFramework/seedvr2-3b-8bit` and supports `--scale`, `--resolution`, `--softness`, `--seed`, optional source-weight `--quantize`, and `--vae-tiling`

Note:
- `abstractvision t2i` / `abstractvision i2i` / `abstractvision t2v` / `abstractvision i2v` default to the OpenAI-compatible HTTP backend, but they also support local providers via `--provider diffusers|mlx-gen|sdcpp` (legacy alias: `--backend`; `mflux` remains accepted).
- Local Diffusers requires `abstractvision[diffusers]`. stable-diffusion.cpp python binding fallback requires `abstractvision[sdcpp]`; external `sd-cli` can be used without the binding.
- One-shot image commands write to `ABSTRACTVISION_STORE_DIR` or `--store-dir` and print JSON artifact metadata followed by the local content path. They do not take an `--output` path flag.
- `width`/`height` are optional request overrides, not global defaults. Leave
  them unset to let the selected provider/model choose its default or `auto`
  size. Some models accept quick-test sizes such as `512x512`; others only
  accept provider-declared sizes such as `1024x1024`, `1024x1536`,
  `1536x1024`, or `auto`.

Examples:

```bash
abstractvision t2i --provider mlx-gen --model AbstractFramework/qwen-image-2512-4bit "a studio product photo of a white ceramic mug" --steps 20 --guidance-scale 1.0 --open
abstractvision i2i --provider mlx-gen --model AbstractFramework/qwen-image-edit-2511-4bit --image ./input.png "replace the background with a clean white studio setup" --steps 20 --guidance-scale 2.5 --strength 0.75 --open
abstractvision t2i --provider mlx-gen --model briaai/FIBO "a studio product photo of a white ceramic mug" --steps 50 --guidance-scale 4.0 --open
abstractvision t2i --provider mlx-gen --model prism-ml/bonsai-image-ternary-4B-mlx-2bit "a bonsai tree in a quiet ceramic studio" --steps 4 --guidance-scale 1.0 --open
abstractvision i2i --provider mlx-gen --model briaai/Fibo-Edit --image ./input.png "remove the background and keep the object edges clean" --steps 20 --guidance-scale 4.0 --open
abstractvision upscale --provider mlx-gen --model AbstractFramework/seedvr2-3b-8bit --image ./input.png --scale 2x --open
abstractvision t2v --provider mlx-gen --model AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit "a red fox walking through a snowy forest, cinematic" --width 432 --height 240 --frames 41 --fps 10 --steps 20 --guidance-scale 4.0 --guidance-2 3.0 --open
abstractvision i2v --provider mlx-gen --model AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit --image ./first-frame.png "slow camera push-in" --width 432 --height 240 --frames 41 --fps 10 --steps 20 --guidance-scale 3.5 --guidance-2 3.5 --open
```

For MLX-Gen Wan, one-shot `t2v` and `i2v` show denoise-step progress with
frame context on stderr by default. `upscale` also shows denoise-step progress
by default. Add `--no-progress` when you need quiet shell output.

## Interactive CLI backend selection

Inside `abstractvision cli`:

- `/backend openai <base_url> [api_key] [model_id]`
- `/provider-models [--task text_to_image] [--json]` — query the configured OpenAI-compatible provider catalog
- `/backend mlx-gen <preset_or_local_path> [base_model]`
- `/backend diffusers <model_id_or_path> [device] [torch_dtype]`
- `/backend sdcpp <model_key|model.gguf|model.safetensors> [sd_cli_path]`
- `/backend sdcpp <diffusion_model.gguf> <vae.safetensors> <llm.gguf> [sd_cli_path]`

Run `/help` in the interactive CLI to see the full command list (generated by `_repl_help()` in [`../../src/abstractvision/cli.py`](../../src/abstractvision/cli.py)).

Interactive generation commands include `/t2i`, `/i2i`, `/t2v`, and `/i2v`. The local Diffusers `text_to_video` path remains experimental and disabled from the normal local surfaces, while MLX-Gen Wan supports local `t2v` and `i2v`. Interactive image commands follow the same size rule as one-shot commands: omit width and height unless you intentionally want a specific backend-supported size.

Examples:

```text
/backend mlx-gen AbstractFramework/qwen-image-2512-4bit
/t2i "a studio product photo of a white ceramic mug" --steps 20 --guidance-scale 1.0 --open
/backend mlx-gen AbstractFramework/qwen-image-edit-2511-4bit
/i2i --image ./input.png "replace the background with a clean white studio setup" --steps 20 --guidance-scale 2.5 --strength 0.75 --open
/backend mlx-gen briaai/FIBO
/t2i "a studio product photo of a white ceramic mug" --steps 50 --guidance-scale 4.0 --open
/backend mlx-gen prism-ml/bonsai-image-ternary-4B-mlx-2bit
/t2i "a bonsai tree in a quiet ceramic studio" --steps 4 --guidance-scale 1.0 --open
/backend mlx-gen AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit
/t2v "a red fox walking through a snowy forest, cinematic" --width 432 --height 240 --frames 41 --fps 10 --steps 20 --guidance-scale 4.0 --guidance-2 3.0 --open
/backend mlx-gen AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit
/i2v --image ./first-frame.png "slow camera push-in" --width 432 --height 240 --frames 41 --fps 10 --steps 20 --guidance-scale 3.5 --guidance-2 3.5 --open
```

Interactive `/t2v` and `/i2v` also show denoise-step video progress by default
and accept `--no-progress`.

Important runtime note:
- Model-specific request adjustments are not implemented only in the interactive UI. The shared backend normalization hooks are applied by `VisionManager` and by the playground API server, so the same parameter corrections reach `abstractvision cli`, the legacy `repl` alias, one-shot CLI commands, the playground, and the AbstractCore plugin.

## Environment variables

The interactive CLI state object (`_ReplState` in [`../../src/abstractvision/cli.py`](../../src/abstractvision/cli.py)) reads the common and backend env vars below. The playground server reads the same backend family plus a few playground-only Diffusers cache/version vars through `PlaygroundServerConfig` in [`../../src/abstractvision/playground_server.py`](../../src/abstractvision/playground_server.py).

### Common

- `ABSTRACTVISION_PROVIDER` — preferred backend selector: `openai`, `openai-compatible`, `diffusers`, `mlx-gen`, or `sdcpp` (alias: `ABSTRACTVISION_BACKEND`; `mflux` remains accepted as a compatibility alias)
  - AbstractCore plugin default: `openai` using `https://api.openai.com/v1` plus `OPENAI_API_KEY`
  - AbstractCore compatibility: selecting `abstractvision:openai-compatible` directly, or setting `OPENAI_BASE_URL` to a non-OpenAI endpoint, keeps compatible-endpoint semantics
  - if unset and `OPENAI_BASE_URL` is set, the interactive CLI/playground default to `openai`
  - if unset and no base URL is configured, no backend is selected until you use `/backend ...` or load a model explicitly
- `ABSTRACTVISION_STORE_DIR` — local artifact output directory (default: `~/.abstractvision/assets`)
- `ABSTRACTVISION_TIMEOUT_S` — HTTP timeout for OpenAI-compatible backend (default: `300`)
- `ABSTRACTVISION_MODEL` — preferred model selector for the current backend/provider (alias: `ABSTRACTVISION_MODEL_ID`)
- `ABSTRACTVISION_MODEL_ID` — model id for the current backend in the interactive CLI (legacy):
  - `openai`: sent as `model` in HTTP requests (optional; server-dependent)
  - `diffusers`: Diffusers model id or local path
- `ABSTRACTVISION_MODEL_DIR` — legacy preset root imported into the Hugging Face cache when older installs are migrated
- `ABSTRACTVISION_CAPABILITIES_MODEL_ID` — optional capability-gating model id (must exist in the registry)

Note:
- Curated `abstractvision download ...` presets are written to the Hugging Face cache by default, and older `~/models/<preset>` trees are imported into that same cache on first use.

### OpenAI / OpenAI-Compatible HTTP Backend

- `OPENAI_BASE_URL` — optional OpenAI-compatible `/v1` endpoint override; when unset, the official OpenAI profile uses `https://api.openai.com/v1`
- `OPENAI_API_KEY` — bearer token for official OpenAI and compatible `/v1` endpoints that require auth
- `ABSTRACTVISION_MODEL_ID` — optional remote model id/name (see also “Common”)
- `OPENAI_IMAGE_MODEL_ID` / `OPENAI_IMAGE_MODEL` — plugin-only OpenAI model aliases when `ABSTRACTVISION_MODEL_ID` / `vision_model_id` are unset
- `ABSTRACTVISION_MODELS_PATH` / `vision_models_path` — provider catalog path for explicit listing (default: `/models`, so a `/v1` base URL queries `/v1/models`)
- Many OpenAI-compatible providers expose `GET /models`. AbstractVision exposes that catalog via `abstractvision provider-models`, `VisionManager.list_provider_models(...)`, and the AbstractCore plugin method `llm.vision.list_provider_models(...)`; it does not call the catalog automatically or use it to select a model. The plugin uses its static default (`gpt-image-1`) unless a model id is configured.
- `ABSTRACTVISION_IMAGES_GENERATIONS_PATH` — default: `/images/generations`
- `ABSTRACTVISION_IMAGES_EDITS_PATH` — default: `/images/edits`
- `ABSTRACTVISION_TEXT_TO_VIDEO_PATH` — optional (enables `text_to_video`)
- `ABSTRACTVISION_IMAGE_TO_VIDEO_PATH` — optional (enables `image_to_video`)
- `ABSTRACTVISION_IMAGE_TO_VIDEO_MODE` — `multipart` (default) or `json_b64`

### Diffusers backend

- `ABSTRACTVISION_DIFFUSERS_DEVICE` — `auto` (default), `cpu`, `cuda`, `mps`, …
- `ABSTRACTVISION_DIFFUSERS_TORCH_DTYPE` — optional (`float16`, `bfloat16`, `float32`)
- `ABSTRACTVISION_DIFFUSERS_ALLOW_DOWNLOAD` — `0` (default/cache-only/offline) or `1` to permit runtime downloads
- `ABSTRACTVISION_DIFFUSERS_AUTO_RETRY_FP32` — `1` (default) or `0` (MPS-only image fallback behavior; local video does not auto-promote above 16-bit on MPS)

Playground-only Diffusers vars:
- `ABSTRACTVISION_ALLOW_DOWNLOAD` — legacy fallback used only when `ABSTRACTVISION_DIFFUSERS_ALLOW_DOWNLOAD` is unset
- `ABSTRACTVISION_DIFFUSERS_CACHE_DIR` — optional Hugging Face cache directory override
- `ABSTRACTVISION_DIFFUSERS_REVISION` — optional model revision
- `ABSTRACTVISION_DIFFUSERS_VARIANT` — optional model variant

### MLX-Gen backend (Apple Silicon)

- `ABSTRACTVISION_MFLUX_MODEL` — exact published MLX-Gen repo id, local model path, or custom repo id (examples: `AbstractFramework/flux.2-klein-4b-4bit`, `AbstractFramework/ernie-image-turbo-8bit`, `briaai/FIBO`, `prism-ml/bonsai-image-ternary-4B-mlx-2bit`, `briaai/Fibo-Edit`, `Wan-AI/Wan2.2-TI2V-5B-Diffusers`, `AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit`, `AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit`, `/path/to/preset-dir`)
- `ABSTRACTVISION_MFLUX_BASE_MODEL` — optional base family for local paths or custom repos (`flux2-klein-4b`, `flux2-klein-9b`, `flux2-klein-base-4b`, `flux2-klein-base-9b`, `bonsai-image-ternary`, `z-image`, `z-image-turbo`, `qwen-image`, `qwen-image-edit-2511`, `ernie-image-turbo`, `fibo`, `fibo-lite`, `fibo-edit`, `fibo-edit-rmbg`, `wan2.2-ti2v-5b`, `wan2.2-t2v-a14b`, `wan2.2-i2v-a14b`)
- `ABSTRACTVISION_MFLUX_ALLOW_DOWNLOAD` — `0` (default) or `1` to permit runtime downloads when a preset/repo is missing from the local cache
- `ABSTRACTVISION_MODEL_DIR` — legacy preset root only; curated downloads now land in the Hugging Face cache
- Canonical provider/model routing is `mlx-gen` / `mlx-gen/<exact-huggingface-repo>`, for example `mlx-gen/AbstractFramework/flux.2-klein-4b-4bit`. Legacy `mflux` provider values, routed ids, and env var names are accepted for compatibility, but the model id itself should be the exact published repo id.

### stable-diffusion.cpp backend

- `ABSTRACTVISION_SDCPP_BIN` — `sd-cli` path/name (default: `sd-cli`)
- `ABSTRACTVISION_SDCPP_MODEL` — optional curated model key or single-file model path. For curated keys such as `flux2-klein-base-4b` or `qwen-image`, AbstractVision resolves cached VAE/LLM companions automatically.
- `ABSTRACTVISION_SDCPP_DIFFUSION_MODEL` — GGUF diffusion model path
- `ABSTRACTVISION_SDCPP_VAE` — VAE safetensors path (required for component-mode models like Qwen Image GGUF and FLUX.2 GGUF)
- `ABSTRACTVISION_SDCPP_LLM` — text encoder path (often GGUF; required for component-mode models like Qwen Image GGUF and FLUX.2 GGUF)
- `ABSTRACTVISION_SDCPP_LLM_VISION` — optional vision encoder GGUF path
- `ABSTRACTVISION_SDCPP_EXTRA_ARGS` — extra `sd-cli` flags (string, split like a shell)
- `ABSTRACTVISION_DISABLE_GGUF_ON_MACOS` — set to `1` to hide/disable GGUF/stable-diffusion.cpp catalog entries and execution on macOS (default: `0` / GGUF enabled)

Curated bundle flow:
- `abstractvision download flux2-klein-base-4b --provider sdcpp`
- `abstractvision t2i --provider sdcpp --model flux2-klein-base-4b "a studio photo of an espresso machine"`

This keeps the UX simple for external users: the package owns the bundle map and resolves side artifacts from the cache.

Tip:
- If you want **Metal** acceleration on macOS (Apple Silicon), install a Metal-capable `sd-cli` binary from
  stable-diffusion.cpp releases and point `ABSTRACTVISION_SDCPP_BIN` at it (or pass the path as the last arg to
  `/backend sdcpp ...` in the REPL). If you don’t, the backend may fall back to python bindings that run CPU-only.
--- >8 --- END FILE: docs/reference/configuration.md --- >8 ---

--- 8< --- FILE: docs/reference/capabilities-registry.md --- 8< ---
# Capability registry (`vision_model_capabilities.json`)

AbstractVision keeps a single packaged “source of truth” for what model families can do and how
AbstractVision understands their downloadable runtime variants:

- Asset: [`../../src/abstractvision/assets/vision_model_capabilities.json`](../../src/abstractvision/assets/vision_model_capabilities.json)
- Loader + validator: `VisionModelCapabilitiesRegistry` / `validate_capabilities_json()` in [`../../src/abstractvision/model_capabilities.py`](../../src/abstractvision/model_capabilities.py)

See also:
- CLI/REPL inspection commands: [docs/reference/configuration.md](configuration.md)
- Backends (execution reality): [docs/reference/backends.md](backends.md)
- ADR policy: [docs/adr/0005_curated_capability_registry_and_download_catalog.md](../adr/0005_curated_capability_registry_and_download_catalog.md)

## Policy boundary

The registry is not just descriptive metadata. In this repo it is the authoritative compatibility
inventory that feeds:

- model discovery;
- task and parameter introspection;
- curated download surfacing;
- backend request normalization.

Under [ADR 0005](../adr/0005_curated_capability_registry_and_download_catalog.md), this inventory is:

- global rather than host-specific;
- organized around parent model families with engine-specific download variants;
- official-upstream-first, with clearly labeled community ports only when they are the best
  runtime-native choice for a target engine;
- explicit about download-only or backend-not-supported states.

## What the registry is used for

- **Discovery**: list known task keys and model ids.
- **Optional safety gating**:
  - `VisionManager(model_id=..., registry=...)` will fail fast if the model doesn’t support a task ([`../../src/abstractvision/vision_manager.py`](../../src/abstractvision/vision_manager.py)).
  - The CLI/REPL can enforce gating via `--capabilities-model-id` (CLI) or `/cap-model` (REPL).
- **Task-aware UI/catalog metadata**:
  - `image_to_image` in a model’s `tasks` map is the explicit “this model supports edits” signal.
  - The playground API surfaces that structured metadata as `task_specs` so local tooling can enable edit-only flows without re-encoding model rules elsewhere.
- **Request normalization hints**:
  - additive parameter metadata such as `default`, `const`, `min`, `multiple_of`, `supported`, and `auto_derived_from_input` can be consumed by backends to keep model-specific defaults and constraints in one packaged source of truth.

Important:
- The registry describes **model capability intent**.
- Your configured backend still needs to implement the task at runtime (see backend support matrix in [docs/reference/backends.md](backends.md)).

## Curation rules

When editing the registry:

- keep one parent entry per real model family;
- keep host-specific or engine-specific artifacts in `downloads`, not as duplicated top-level
  model entries;
- prefer official upstream repos;
- use community repos only for runtime-native artifacts such as GGUF or MLX ports when no
  official equivalent exists;
- do not promote adapter-only or component-only repos as standalone curated models unless the
  runtime can orchestrate them as first-class flows;
- mark non-runnable local entries explicitly when the backend is not shipped yet.

## Minimal Python usage

```python
from abstractvision import VisionModelCapabilitiesRegistry

reg = VisionModelCapabilitiesRegistry()
print(reg.schema_version())
print(reg.list_tasks())

assert reg.supports("runwayml/stable-diffusion-v1-5", "text_to_image")
print(reg.models_for_task("text_to_image"))
```

## JSON shape (high level)

The validator enforces a “soft schema”:

- Top-level keys:
  - `schema_version`
  - `tasks` (keyed by task name; includes human descriptions)
  - `models` (keyed by model id)
- Each model entry includes:
  - `provider` (string)
  - `license` (string; informational)
  - optional `downloads` (list of downloadable artifacts; informational metadata used by `abstractvision show-model`)
  - `tasks` (map of task name → task spec)
- Each task spec includes:
  - `inputs`, `outputs` (lists of strings)
  - `params` (object where each param has `required: bool`, plus additive fields)
  - optional `requires` for dependencies like `base_model_id`

Examples of additive `params` metadata used in this repo:
- `default`: backend fills a missing value
- `const`: backend forces a model-specific fixed value
- `min`: backend clamps the value upward
- `multiple_of`: backend rounds dimensions to a required multiple
- `supported: false`: backend drops an unsupported optional parameter
- `auto_derived_from_input: true`: image-edit backends can infer a missing size from the input image

### Downloads (optional)

Some models include a `downloads` list to document what AbstractVision considers “downloadable” for common engines:

- `key`: the canonical short name used in `model-presets` / `download` (e.g. `qwen-image`, `flux2-klein-4b`)
- `engine`: `mlx-gen`, `diffusers`, `stable-diffusion.cpp`, ...
- `target`: `mlx`, `diffusers`, `gguf`, ...
- `bits`: typically `8` (quantized) or `16` (full snapshot)
- `repo_id`: Hugging Face repo id to download
- optional `source`, `notes`

This metadata is descriptive, but in this repo it is maintained to align with the curated
download surfaces exposed by `abstractvision catalog`, `abstractvision model-presets`, and
`abstractvision download`.

In this repo, we keep `downloads` entries aligned with the curated preset table (`model_downloads._PRESETS`) so `abstractvision show-model` and the docs can reliably describe what the CLI can download.
That alignment is intentionally user-facing: if a variant is curated for download, it should have
clear provenance, a realistic runtime target, and a documented readiness story.
--- >8 --- END FILE: docs/reference/capabilities-registry.md --- >8 ---

--- 8< --- FILE: docs/adr/README.md --- 8< ---
# ADRs

This directory records durable engineering policy for `abstractvision`.

## Accepted

- [ADR 0003: Keep the base package lightweight and make local runtimes explicit](0003_lightweight_base_package_and_explicit_runtime_extras.md)
  Base installs stay dependency-light; heavy local runtimes and model-download helpers remain optional extras.
- [ADR 0004: Keep the orchestrator thin and make model semantics backend-owned](0004_thin_orchestrator_and_backend_owned_model_semantics.md)
  `VisionManager` delegates; backend normalization and runtime-specific behavior stay in the backend/package layer.
- [ADR 0005: Own a curated capability registry and cache-backed model catalog](0005_curated_capability_registry_and_download_catalog.md)
  The packaged registry and curated download surfaces are product-facing compatibility contracts, not incidental metadata.
- [ADR 0006: Keep runtime selection explicit and operator-controlled](0006_operator_control_configuration_precedence_and_explicit_network_use.md)
  Config precedence, cache-only defaults, and model selection must stay explicit across CLI, playground, and AbstractCore.
- [ADR 0007: Disclose fallbacks and degraded modes explicitly](0007_explicit_fallback_and_degraded_mode_disclosure.md)
  Fallbacks that change provenance, quality, or runtime behavior must be surfaced rather than hidden.
- [ADR 0008: Require validation and evidence-based change reporting](0008_validation_and_evidence_based_change_reporting.md)
  Backend, catalog, and performance claims need reproducible checks and clear measured-versus-estimated reporting.
- [ADR 0009: Keep docs, backlog, and ADRs code-first](0009_code_first_docs_backlog_and_adr_discipline.md)
  Code is the operational source of truth; durable policy lives here, and execution history lives in `docs/backlog/`.

## Superseded

- [ADR 0001: AbstractVision owns the curated cross-platform model catalog and compatibility layer](0001_curated_cross_platform_model_catalog.md)
  Superseded by ADR 0005 and ADR 0006.
- [ADR 0002: Vision overlays and adapter composition are package-owned](0002_vision_overlays_and_adapter_composition_are_package_owned.md)
  Superseded by ADR 0004.

## Working Rules

- Use ADRs for durable engineering policy that should constrain future changes.
- Use `docs/backlog/` for planning state, implementation history, and completion reports.
- When code, docs, backlog, and ADRs drift, fix the drift explicitly instead of relying on chat history.
--- >8 --- END FILE: docs/adr/README.md --- >8 ---

--- 8< --- FILE: docs/adr/0005_curated_capability_registry_and_download_catalog.md --- 8< ---
# ADR 0005: Own a curated capability registry and cache-backed model catalog

Status: Accepted.

## Context

AbstractVision does more than expose a generic image API. It also ships a packaged registry,
download presets, model-catalog commands, and compatibility-oriented docs. Users therefore treat
the registry and catalog as product surfaces, not as incidental internal metadata.

The failure mode is predictable when this policy is left implicit:

- host-specific ports crowd out the real parent model family;
- catalog entries imply local support that the backend does not actually provide;
- official upstream weights, runtime-native community conversions, and raw side artifacts get
  mixed together without clear provenance;
- users must manually assemble hidden companion files after following a “curated” flow.

The current code already has the ingredients for a better contract:

- [`vision_model_capabilities.json`](../../src/abstractvision/assets/vision_model_capabilities.json)
  is packaged and validated;
- [`model_downloads.py`](../../src/abstractvision/model_downloads.py) drives curated download
  surfaces and host-aware selection;
- curated `sdcpp` bundle resolution now maps model keys to their required companion artifacts.

## Decision

AbstractVision owns a curated, package-level compatibility inventory and download surface.

The rules are:

1. The packaged capability registry is the authoritative source of truth for parent model families,
   task metadata, and curated download variants.
2. The registry stays global and platform-neutral. Host-specific recommendation belongs in catalog
   and preset selection logic, not in the model-family inventory itself.
3. Each real model family should have one parent entry. Engine-specific variants, quantizations,
   and runtime-native ports belong under that family.
4. Official upstream repositories are preferred. Community repositories are allowed only when they
   provide the best runtime-native artifact for a supported engine, and they must carry clear
   provenance notes.
5. Curated user flows must minimize undocumented manual assembly. If a curated local runtime path
   needs companion artifacts such as VAE or encoder files, AbstractVision should own the mapping
   and resolution for that flow.
6. Readiness claims must match shipped backend reality. Catalog-only or download-only entries are
   allowed, but they must be labeled honestly.
7. The registry and curated download surface must stay aligned. A curated model path should have a
   realistic runtime story, not just a repo id.

## Consequences

### Positive

- Users get a clearer discovery and download story.
- Parent families, runtime-native ports, and side artifacts are modeled more honestly.
- The package can keep improving local UX without turning the registry into an unstructured repo
  dump.

### Negative

- Curating the catalog becomes an explicit maintenance obligation.
- Adding new model families requires provenance review and runtime validation work.
- Some attractive community artifacts must stay out of the curated path until the runtime story is
  honest.

### Neutral

- This ADR does not require every model family to support every engine.
- Download-only entries can still exist when that is the truthful state.

## Enforcement

- Reviewers should reject registry or preset changes that:
  - duplicate a model family as host-specific top-level entries;
  - promote side artifacts or adapters as standalone curated models without first-class runtime
    orchestration;
  - claim runtime readiness without a matching backend path or explicit limitation;
  - use a community artifact without explaining why it is the right runtime-native choice.
- Changes to curated local flows must update both code and user-facing docs in the same pass.
- When a curated `sdcpp` or similar component-based path is added, the package should prefer
  package-owned bundle resolution over manual operator assembly.

## Validation

- Load and validate the packaged registry from a repo checkout.
- Verify representative `model-catalog`, `model-presets`, and `show-model` output.
- For changed curated download paths, verify the preset resolves the intended artifact and that the
  resulting runtime flow is either runnable or explicitly marked as not yet runnable.
- Add or keep focused tests for cache-backed bundle resolution when curated local flows depend on
  companion artifacts.

## Backlog links

- [docs/backlog/completed/003_hf_model_landscape_and_capability_registry.md](../backlog/completed/003_hf_model_landscape_and_capability_registry.md)
- [docs/backlog/completed/004_capability_schema_and_validation.md](../backlog/completed/004_capability_schema_and_validation.md)
- [docs/backlog/completed/016_abstractcore_plugin_catalog_discovery_surface.md](../backlog/completed/016_abstractcore_plugin_catalog_discovery_surface.md)
- [docs/backlog/completed/019_best_effort_preload_warmup_for_local_backends.md](../backlog/completed/019_best_effort_preload_warmup_for_local_backends.md)
- [docs/backlog/planned/020_adapter_aware_model_graph_and_catalog.md](../backlog/planned/020_adapter_aware_model_graph_and_catalog.md)

## Related

- [docs/reference/capabilities-registry.md](../reference/capabilities-registry.md)
- [docs/architecture.md](../architecture.md)
- [src/abstractvision/model_capabilities.py](../../src/abstractvision/model_capabilities.py)
- [src/abstractvision/model_downloads.py](../../src/abstractvision/model_downloads.py)
- [src/abstractvision/assets/vision_model_capabilities.json](../../src/abstractvision/assets/vision_model_capabilities.json)
--- >8 --- END FILE: docs/adr/0005_curated_capability_registry_and_download_catalog.md --- >8 ---

--- 8< --- FILE: docs/adr/0006_operator_control_configuration_precedence_and_explicit_network_use.md --- 8< ---
# ADR 0006: Keep runtime selection explicit and operator-controlled

Status: Accepted.

## Context

AbstractVision can be driven through Python, CLI/REPL, playground, and AbstractCore. Those surfaces
all combine defaults, environment variables, explicit arguments, and backend-specific behavior. The
package also has multiple paths that could silently change operator intent if left unchecked:

- runtime backend selection;
- provider model ids;
- cache-only versus download-enabled local behavior;
- provider catalog inspection versus actual model selection;
- curated bundle resolution for component-based local models.

The current code already has good instincts here:

- local Diffusers and MLX-Gen default to cache-only behavior;
- provider catalogs are inspection-only;
- AbstractCore owner config is read before env vars;
- curated `sdcpp` keys resolve cached companion files and fail clearly when they are missing.

That behavior needs to become durable policy.

## Decision

AbstractVision keeps runtime-affecting choices explicit and operator-controlled.

The rules are:

1. Caller-supplied config for a surface takes precedence over ambient environment variables, and
   ambient environment variables take precedence over baked defaults.
2. In the AbstractCore plugin, `owner.config` is authoritative over `ABSTRACTVISION_*` and related
   environment aliases.
3. Provider model catalogs are explicit inspection only. AbstractVision must not silently select or
   switch image models based on `/models` responses.
4. Local runtime downloads are opt-in. Cache-backed or pre-downloaded behavior remains the default
   unless the operator explicitly enables downloads or runs `download-model`.
5. Curated model keys are allowed for local flows when the package can resolve them to explicit
   cached artifacts. Missing required artifacts must fail early with an actionable message.
6. Legacy aliases and compatibility shims may exist, but they must remain documented and auditable.
7. No surface should silently change backend, network, or download behavior in a way that surprises
   an operator who did not explicitly ask for it.

## Consequences

### Positive

- Operators can reason about backend selection and network use more reliably.
- Cache-only local workflows stay predictable.
- Curated local flows become simpler without hiding what files are actually being used.

### Negative

- Some convenience shortcuts remain intentionally unavailable if they would hide important operator
  choices.
- Config docs and tests must keep pace with precedence rules.

### Neutral

- This ADR still allows smart defaults, as long as those defaults do not silently perform material
  new work such as downloading weights or switching to a different provider model.

## Enforcement

- Reviewers should reject new automatic model-selection behavior driven only by provider catalogs.
- Reviewers should reject runtime downloads enabled by default for local backends unless a future
  ADR changes that policy.
- Config precedence changes must update the docs and tests in the same pass.
- Error messages for missing curated local components must stay actionable and point back to the
  supported download or configuration path.

## Validation

- Keep CLI, playground, and AbstractCore tests passing for config precedence and cache-only local
  behavior.
- Verify provider model listing remains inspection-only.
- Verify curated local key resolution either resolves explicit cached files or fails with a precise
  remediation hint.

## Backlog links

- [docs/backlog/completed/014_lightweight_openai_compatible_packaging.md](../backlog/completed/014_lightweight_openai_compatible_packaging.md)
- [docs/backlog/completed/015_vision_install_profiles_and_pending_defaults.md](../backlog/completed/015_vision_install_profiles_and_pending_defaults.md)
- [docs/backlog/completed/016_abstractcore_plugin_catalog_discovery_surface.md](../backlog/completed/016_abstractcore_plugin_catalog_discovery_surface.md)
- [docs/backlog/completed/018_capability_residency_hooks.md](../backlog/completed/018_capability_residency_hooks.md)

## Related

- [docs/reference/configuration.md](../reference/configuration.md)
- [docs/reference/abstractcore-integration.md](../reference/abstractcore-integration.md)
- [src/abstractvision/cli.py](../../src/abstractvision/cli.py)
- [src/abstractvision/playground_server.py](../../src/abstractvision/playground_server.py)
- [src/abstractvision/integrations/abstractcore_plugin.py](../../src/abstractvision/integrations/abstractcore_plugin.py)
--- >8 --- END FILE: docs/adr/0006_operator_control_configuration_precedence_and_explicit_network_use.md --- >8 ---

--- 8< --- FILE: docs/reference/artifacts.md --- 8< ---
# Artifacts (artifact refs + stores)

AbstractVision supports “artifact-first” outputs: return a small JSON dict that points to a stored blob instead of inlining bytes.

Code pointers:
- Store interface + helpers: [`../../src/abstractvision/artifacts.py`](../../src/abstractvision/artifacts.py)
- Orchestration logic: `VisionManager._maybe_store()` in [`../../src/abstractvision/vision_manager.py`](../../src/abstractvision/vision_manager.py)

See also:
- Getting started (REPL stores outputs by default): [docs/getting-started.md](../getting-started.md)

## Output shapes

`VisionManager` returns:

- **Without a store**: `GeneratedAsset` ([`../../src/abstractvision/types.py`](../../src/abstractvision/types.py))
  - contains bytes (`data`), `mime_type`, and best-effort metadata
- **With a store**: an artifact ref dict (via `MediaStore.store_bytes(...)`)
  - minimum shape: `{"$artifact": "<id>"}` (`is_artifact_ref()` checks this)
  - common fields: `content_type`, `sha256`, `filename`, `size_bytes`, `metadata`

## LocalAssetStore (standalone mode)

`LocalAssetStore` stores files under `~/.abstractvision/assets` by default ([`../../src/abstractvision/artifacts.py`](../../src/abstractvision/artifacts.py)):

- Blob: `~/.abstractvision/assets/<artifact_id>.<ext>`
- Metadata: `~/.abstractvision/assets/<artifact_id>.meta.json`

Minimal usage:

```python
from abstractvision import LocalAssetStore, VisionManager
from abstractvision.backends import OpenAICompatibleBackendConfig, OpenAICompatibleVisionBackend

store = LocalAssetStore()
backend = OpenAICompatibleVisionBackend(config=OpenAICompatibleBackendConfig(base_url="http://localhost:1234/v1"))
vm = VisionManager(backend=backend, store=store)

ref = vm.generate_image("a watercolor painting of a lighthouse")
blob = store.load_bytes(ref["$artifact"])  # type: ignore[index]
```

## RuntimeArtifactStoreAdapter (framework mode)

`RuntimeArtifactStoreAdapter` is a duck-typed adapter for an external artifact store (designed for AbstractRuntime),
so AbstractVision can depend on an artifact store **without** a hard dependency ([`../../src/abstractvision/artifacts.py`](../../src/abstractvision/artifacts.py)).

Related:
- AbstractRuntime: <https://github.com/lpalbou/abstractruntime>
--- >8 --- END FILE: docs/reference/artifacts.md --- >8 ---

--- 8< --- FILE: docs/reference/abstractcore-integration.md --- 8< ---
# AbstractCore integration

AbstractVision offers two integration surfaces for AbstractCore:

1) **Capability plugin** (so `abstractcore` can discover a vision backend)
2) **Tool helpers** (so you can expose vision tasks as tools with artifact-ref outputs)

Code pointers:
- Plugin: [`../../src/abstractvision/integrations/abstractcore_plugin.py`](../../src/abstractvision/integrations/abstractcore_plugin.py)
- Tools: [`../../src/abstractvision/integrations/abstractcore.py`](../../src/abstractvision/integrations/abstractcore.py)
- Entry point registration: [`../../pyproject.toml`](../../pyproject.toml) (`[project.entry-points."abstractcore.capabilities_plugins"]`)

See also:
- Artifacts: [docs/reference/artifacts.md](artifacts.md)
- Backends: [docs/reference/backends.md](backends.md)

## 1) Capability plugin (AbstractCore → VisionCapability)

The plugin registers these backend ids:

- `abstractvision:openai` (default backend id) and `abstractvision:openai-compatible` (legacy compatibility backend id) are registered by [`../../src/abstractvision/integrations/abstractcore_plugin.py`](../../src/abstractvision/integrations/abstractcore_plugin.py). The implementation also supports local backends.

Current behavior:
- Default `abstractvision:openai`: OpenAI HTTP (`https://api.openai.com/v1`). Set `OPENAI_API_KEY`.
- OpenAI model ids are configured, not discovered dynamically. Providers may expose an OpenAI-compatible `GET /models` catalog; AbstractVision exposes it through `abstractvision provider-models`, `VisionManager.list_provider_models(...)`, and the plugin method `llm.vision.list_provider_models(...)`, but the plugin does not call it automatically or use it to select a model. The static plugin default is `gpt-image-1`; set `OPENAI_IMAGE_MODEL_ID`, `OPENAI_IMAGE_MODEL`, `ABSTRACTVISION_MODEL_ID`, or `vision_model_id` for newer provider models.
- Compatible HTTP: set `OPENAI_BASE_URL` to a local/remote compatible `/v1` server. Set `ABSTRACTVISION_BACKEND=openai-compatible` when you want to force compatible-endpoint semantics.
- Legacy `abstractvision:openai-compatible`: keeps compatible-endpoint defaults when that backend id is selected directly.
- Local Diffusers: install `abstractvision[diffusers]`, then set `ABSTRACTVISION_BACKEND=diffusers` with `runwayml/stable-diffusion-v1-5` or another supported Diffusers model. It is cache-only/offline unless `ABSTRACTVISION_DIFFUSERS_ALLOW_DOWNLOAD=1` is set. Local `text_to_video` groundwork exists but is currently experimental and disabled from the normal local plugin surfaces.
- Local MLX-Gen (Apple Silicon): install `abstractvision[mlx-gen]` (or `abstractvision[all-apple]`), then set `ABSTRACTVISION_BACKEND=mlx-gen`. Download exact model repos such as `abstractvision download AbstractFramework/flux.2-klein-4b-4bit --provider mlx-gen`, `abstractvision download AbstractFramework/ernie-image-turbo-8bit --provider mlx-gen`, `abstractvision download briaai/FIBO --provider mlx-gen`, `abstractvision download prism-ml/bonsai-image-ternary-4B-mlx-2bit --provider mlx-gen`, `abstractvision download AbstractFramework/seedvr2-3b-8bit --provider mlx-gen`, `abstractvision download AbstractFramework/seedvr2-7b-8bit --provider mlx-gen`, `abstractvision download Wan-AI/Wan2.2-TI2V-5B-Diffusers --provider mlx-gen`, `abstractvision download AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit --provider mlx-gen`, or `abstractvision download AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit --provider mlx-gen` (stored in the Hugging Face cache; `ABSTRACTVISION_MODEL_DIR` is only a legacy import root). Use routed exact model ids such as `mlx-gen/AbstractFramework/flux.2-klein-4b-4bit`, `mlx-gen/prism-ml/bonsai-image-ternary-4B-mlx-2bit`, `mlx-gen/AbstractFramework/seedvr2-3b-8bit`, `mlx-gen/AbstractFramework/seedvr2-7b-8bit`, `mlx-gen/AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit`, or `mlx-gen/AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit`. q4 repos are the memory-efficient default recommendation for most generation/edit models; SeedVR2 defaults to canonical q8 packages and offers q4 packages for tighter memory. Bonsai ternary is already pre-packed at 2-bit and is selected by its exact repo id. Legacy `mflux` provider values remain accepted as aliases.
- stable-diffusion.cpp: set `ABSTRACTVISION_BACKEND=sdcpp` and configure either a model path or a curated model key such as `flux2-klein-base-4b`. Use an external `sd-cli`, or install `abstractvision[sdcpp]` for the python binding fallback.
- The plugin reads AbstractCore owner config keys when present, then falls back to `ABSTRACTVISION_*` env vars.
- Gateway/Core should pass process-level config or `owner.config` and report readiness; they should not mutate AbstractVision environment variables per request.

Key config keys (owner.config):
- `vision_backend_instance` / `vision_backend_factory` (advanced injection hooks; bypass env-driven backend creation)
- `vision_backend` (`openai`, `openai-compatible`, `diffusers`, `mlx-gen`, or `sdcpp`; default `openai`; `mflux` is accepted as a compatibility alias)
- `vision_model_id` (Diffusers/OpenAI-compatible model id; default `gpt-image-1` only for the official OpenAI profile and `runwayml/stable-diffusion-v1-5` for Diffusers)
- `vision_device` / `vision_torch_dtype` / `vision_allow_download` / `vision_auto_retry_fp32` (Diffusers)
- `vision_base_url` / `vision_api_key` (OpenAI or compatible HTTP)
- `vision_mflux_model` / `vision_mflux_base_model` / `vision_mflux_allow_download` (MLX-Gen; env/config names keep MFLUX compatibility)
- `vision_model_dir` (legacy preset import root used by MLX-Gen compatibility/migration helpers; new downloads land in the Hugging Face cache)
- `vision_sdcpp_model` / `vision_sdcpp_diffusion_model` / `vision_sdcpp_bin` (stable-diffusion.cpp; `vision_sdcpp_model` may be a curated model key such as `flux2-klein-base-4b`)
- `vision_sdcpp_vae` / `vision_sdcpp_llm` / `vision_sdcpp_llm_vision` / `vision_sdcpp_clip_l` / `vision_sdcpp_clip_g` / `vision_sdcpp_t5xxl` / `vision_sdcpp_extra_args` (stable-diffusion.cpp component mode)
- `vision_timeout_s` (optional)
- `vision_models_path` (optional provider catalog path; default `/models`)
- Optional video endpoint keys:
  - `vision_text_to_video_path`
  - `vision_image_to_video_path`
  - `vision_image_to_video_mode`

Env-only aliases:
- `ABSTRACTVISION_DIFFUSERS_MODEL_ID` is accepted for the Diffusers plugin backend before falling back to `ABSTRACTVISION_MODEL_ID`.
- `OPENAI_BASE_URL` is accepted by the OpenAI-shaped HTTP backend.
- Compatible OpenAI-shaped endpoints can use `OPENAI_API_KEY` when they require bearer auth.
- `OPENAI_IMAGE_MODEL_ID` and `OPENAI_IMAGE_MODEL` are accepted when `vision_model_id` / `ABSTRACTVISION_MODEL_ID` are unset.
- `ABSTRACTVISION_SDCPP_CLIP_L`, `ABSTRACTVISION_SDCPP_CLIP_G`, and `ABSTRACTVISION_SDCPP_T5XXL` are accepted for stable-diffusion.cpp component mode.

Examples:

```bash
# Local Diffusers. Pre-download weights first, or explicitly allow runtime downloads.
export ABSTRACTVISION_BACKEND=diffusers
export ABSTRACTVISION_MODEL_ID=runwayml/stable-diffusion-v1-5
export ABSTRACTVISION_DIFFUSERS_DEVICE=auto
```

```python
from abstractcore import create_llm

llm = create_llm("openai", model="gpt-4o-mini")
png_bytes = llm.vision.t2i("a red square", width=512, height=512, steps=20)
```

Per-call routing can select the same backend/model ids used by the
`abstractvision` shell commands. For MLX-Gen, pass the exact published model
repo id; there is no separate quantization parameter.

```python
from pathlib import Path

from abstractcore import create_llm

llm = create_llm("openai", model="gpt-4o-mini")

png_bytes = llm.vision.t2i(
    "a studio product photo of a white ceramic mug with the AbstractFramework logo",
    provider="mlx-gen",
    model="AbstractFramework/qwen-image-2512-4bit",
    steps=20,
    guidance_scale=1.0,
    on_progress=lambda event: print(f"{event.phase}: step {event.step}/{event.total_steps}"),
)

edited_png = llm.vision.i2i(
    "replace the background with a clean white studio setup",
    image="./input.png",
    provider="mlx-gen",
    model="AbstractFramework/qwen-image-edit-2511-4bit",
    steps=20,
    guidance_scale=2.5,
    strength=0.75,
    on_progress=lambda event: print(f"{event.phase}: step {event.step}/{event.total_steps}"),
)

multi_ref_png = llm.vision.i2i(
    "compose the subject using the second image as style and layout reference",
    image="./subject.png",
    provider="mlx-gen",
    model="AbstractFramework/qwen-image-edit-2511-4bit",
    reference_images=[Path("./style-reference.png").read_bytes()],
    steps=20,
    guidance_scale=2.5,
)

def on_video_progress(event):
    print(
        f"{event.phase}: step {event.step}/{event.total_steps}, "
        f"frame {event.frame}/{event.total_frames}"
    )

mp4_bytes = llm.vision.t2v(
    "a red fox walking through a snowy forest, cinematic",
    provider="mlx-gen",
    model="AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit",
    width=432,
    height=240,
    num_frames=41,
    fps=10,
    steps=20,
    guidance_scale=4.0,
    guidance_2=3.0,
    max_sequence_length=256,
    on_progress=on_video_progress,
)

first_frame_mp4 = llm.vision.i2v(
    "./first-frame.png",
    prompt="slow camera push-in",
    provider="mlx-gen",
    model="AbstractFramework/wan2.2-i2v-a14b-diffusers-8bit",
    width=432,
    height=240,
    num_frames=41,
    fps=10,
    steps=20,
    guidance_scale=3.5,
    guidance_2=3.5,
    max_sequence_length=256,
    on_progress=on_video_progress,
)
```

`llm.vision.i2i(...)` accepts `image` as bytes, a local file path, or an
artifact ref dict when an `artifact_store` is supplied. Without an artifact
store the plugin returns bytes; with `artifact_store=...` it returns the stored
artifact ref, matching the runtime/gateway artifact path.
Unknown per-call keyword arguments such as `reference_images`,
`max_sequence_length`, and `on_progress` are preserved in the AbstractVision
request `extra` dict for local backends. `llm.vision.t2i(...)`,
`llm.vision.i2i(...)`, `llm.vision.t2v(...)`, and `llm.vision.i2v(...)` forward
`on_progress` to MLX-Gen. Image/edit events include diffusion-step progress;
Wan video events use diffusion-step `progress` and also include frame counters
and `frame_progress`.

For task-specific Wan A14B video models, `guidance_2` is a typed request
parameter for the second-stage/low-noise guidance path. Pass it directly to
`t2v(...)` / `i2v(...)`; do not wrap it in `extra`.

```bash
# OpenAI API.
export OPENAI_API_KEY=...
export OPENAI_IMAGE_MODEL=gpt-image-1
```

```bash
# Local OpenAI-compatible HTTP server, for example AbstractCore Server.
export ABSTRACTVISION_BACKEND=openai-compatible
export OPENAI_BASE_URL=http://localhost:8000/v1
export ABSTRACTVISION_MODEL_ID=server/default
```

### Provider Catalog Discovery

Core/Gateway hosts can inspect provider-advertised model catalogs through the same capability
object used for generation:

```python
models = llm.vision.list_provider_models(task="text_to_image")
for model in models:
    print(model["id"])
```

The return value is a JSON-safe list of dictionaries serialized from `ProviderModelInfo`. Raw
provider metadata is retained in a bounded `raw` field for diagnostics. This method is explicit
inspection only: it does not mutate the configured backend or select a generation model.

Backends that do not implement provider catalog listing raise a clear AbstractVision error instead
of returning a misleading empty catalog. Local Diffusers and stable-diffusion.cpp model discovery
remain separate local-backend concerns, while MLX-Gen and Diffusers provider listings reflect
cache-backed snapshots rather than a separate `~/models` download tree.

### Local Model Residency Control

Core/Gateway hosts can also control in-process local model residency through the same capability
object:

```python
llm.vision.load_resident_model(
    {"task": "text_to_image", "provider": "mlx-gen", "model": "AbstractFramework/flux.2-klein-4b-4bit"}
)
llm.vision.load_resident_model(
    {"task": "text_to_video", "provider": "mlx-gen", "model": "AbstractFramework/wan2.2-t2v-a14b-diffusers-8bit"}
)

loaded = llm.vision.list_loaded_models()
resident = llm.vision.list_resident_models()

llm.vision.unload_resident_model(
    {"provider": "mlx-gen", "model": "AbstractFramework/flux.2-klein-4b-4bit"}
)
```

Notes:

- this surface is local-only and process-local;
- it controls only AbstractVision-managed in-process backends (`diffusers`, `mlx-gen`, `sdcpp`);
- OpenAI/OpenAI-compatible HTTP backends are intentionally rejected here, even on `localhost`,
  because the plugin cannot honestly control another process's loaded-state.

`list_loaded_models()` reports both explicitly preloaded resident models and transient currently
loaded local models from recent generation requests. `list_resident_models()`
returns the explicit pinned subset.

Each loaded-model entry includes stable routing metadata such as `load_id`, `backend_kind`,
`resident`, and a `tasks` list of observed task aliases using that loaded backend.

For deterministic unload behavior, prefer:

- `load_id`, or
- both `provider`/`backend` and `model`.

Broad unload filters that match multiple loaded models are rejected as ambiguous.

## Shared request normalization

`llm.vision` calls go through `VisionManager`, which applies backend normalization hooks before
execution. That means model-specific constraints are shared with the CLI/REPL and playground API
instead of being reimplemented in the plugin layer.

Current examples:
- Distilled MLX-Gen/FLUX-family models can clamp guidance or minimum steps in the backend.
- Diffusers-backed image requests pick up registry-driven defaults such as recommended step counts,
  guidance defaults, and dimension constraints.
- Unsupported parameters, such as negative prompts on constrained model families, are dropped in
  the backend rather than surfacing as per-surface validation bugs.

## 2) Tool helpers (`make_vision_tools`)

`make_vision_tools(...)` builds AbstractCore `@tool` callables for:
- text→image
- image→image
- image upscaling
- multi-view image
- text→video
- image→video

Important:
- Tool outputs are designed to be **artifact refs**, so `VisionManager.store` must be set ([`../../src/abstractvision/integrations/abstractcore.py`](../../src/abstractvision/integrations/abstractcore.py)).
- This module requires AbstractCore to be installed by the host application. AbstractVision does not install AbstractCore as a dependency.

Tip (framework mode):
- If your runtime provides an artifact store (e.g. AbstractRuntime), use `RuntimeArtifactStoreAdapter` so tool outputs can be stored and referenced across processes (see [docs/reference/artifacts.md](artifacts.md)).
--- >8 --- END FILE: docs/reference/abstractcore-integration.md --- >8 ---

--- 8< --- FILE: playground/README.md --- 8< ---
# AbstractVision Playground (Web)

This is a tiny web UI for testing AbstractVision locally. It is powered by the
self-contained `abstractvision playground` command; it does **not** require an
AbstractCore server.

The playground is a local/dev surface. Do not expose it as an authenticated
production serving boundary; use AbstractCore/Gateway for production routing,
authentication, and browser-origin policy.

## Required API endpoints

The page calls:

- `GET /v1/models` (ping)
- `GET /v1/vision/models` (list cached models + active model)
- `POST /v1/vision/model/load` (load a model into memory)
- `POST /v1/vision/jobs/images/generations` (start a text→image job)
- `POST /v1/vision/jobs/videos/generations` (start a text→video job)
- `POST /v1/vision/jobs/images/edits` (start an image→image job)
- `GET /v1/vision/jobs/{job_id}` (poll job status)
  - on success, the page calls `GET /v1/vision/jobs/{job_id}?consume=1` to fetch-and-consume the result

## 1) Start the local playground server

From an AbstractVision checkout:

```bash
PYTHONPATH=src python -m abstractvision playground --port 8091
```

Or, when installed:

```bash
abstractvision playground --port 8091
```

Quick sanity checks (should return JSON):

```bash
curl -s http://127.0.0.1:8091/v1/models | head
curl -s http://127.0.0.1:8091/v1/vision/models | head
```

## 2) Open the page

Open:

- `http://127.0.0.1:8091/vision_playground.html`

Usage notes:
- The page is split into task tabs (`Text→Image`, `Image→Image`, `Text→Video`, and a placeholder `Image→Video` tab for later work).
- Each active task tab has its own model selector and unload button.
- Selecting a different model unloads the current active backend first, then auto-loads the replacement to keep memory usage bounded.
- Raw Hugging Face model ids such as `runwayml/stable-diffusion-v1-5` load directly; no `diffusers/` provider prefix is required.
- For first tests, prefer a small cached model such as Stable Diffusion 1.5 before loading larger Qwen/FLUX models.
- The Image→Image panel is enabled only for models that both advertise `image_to_image` and remain enabled by backend runtime truth.
- MLX-Gen models are surfaced only for tasks the backend currently enables: curated q4/q8 text-to-image presets plus image-to-image for FLUX.2 klein/base and Qwen Image Edit (no masks yet).
- The bundled local `Text→Video` tab is experimental and currently expected to have no shipped local model options until [`docs/backlog/planned/0023_local_runtime_capability_quarantine_for_glm_mflux_and_t2v.md`](../docs/backlog/planned/0023_local_runtime_capability_quarantine_for_glm_mflux_and_t2v.md) is resolved.
- Response logs intentionally show a shortened `b64_json` preview instead of the full base64 payload.
- “Extra JSON” is forwarded to the server:
  - T2I: merged into the JSON request body
  - I2I: sent as a string field `extra_json` in the multipart form body
- Model-specific request fixes happen in the backend/API layer, so the same normalization rules apply to the playground, CLI/REPL, and AbstractCore integration.
- Local video export requires an `ffmpeg` executable on `PATH` whenever a local backend emits frames for packaging.

## 3) stable-diffusion.cpp / GGUF notes

If your server is configured to run GGUF diffusion models via stable-diffusion.cpp, you’ll typically need:
- a diffusion model (`.gguf`)
- a VAE (`.safetensors`) for some families (e.g. Qwen Image GGUF)
- a text encoder/LLM (`.gguf`) for some families (e.g. Qwen Image GGUF)

Exact configuration is backend-specific; check AbstractVision’s backend docs.
--- >8 --- END FILE: playground/README.md --- >8 ---

--- 8< --- FILE: CONTRIBUTING.md --- 8< ---
# Contributing to AbstractVision

Thanks for taking the time to contribute. This repository aims to stay small, stable-by-design, and easy to integrate.

AbstractVision is part of the **AbstractFramework** ecosystem:
- AbstractFramework: <https://github.com/lpalbou/AbstractFramework>
- AbstractCore: <https://github.com/lpalbou/abstractcore>
- AbstractRuntime: <https://github.com/lpalbou/abstractruntime>

## Ground rules

- Keep the public API stable (`VisionManager` in [`src/abstractvision/vision_manager.py`](src/abstractvision/vision_manager.py)).
- Prefer additive changes (new fields, new models, new backends) over breaking changes.
- Don’t commit model weights, large binaries, or cache artifacts.
- Make docs and examples match the code (the repo is intended to be “readme-first”).
- Keep imports lazy for heavy stacks (see [`src/abstractvision/backends/__init__.py`](src/abstractvision/backends/__init__.py)).

## Development setup

```bash
python -m venv .venv
. .venv/bin/activate
python -m pip install -U pip
python -m pip install -e ".[dev]"
```

Optional (if you work on AbstractCore integration locally):

```bash
python -m pip install abstractcore
```

The `abstractvision[abstractcore]` extra is only a compatibility marker. AbstractCore is intentionally supplied by the host application, not installed by AbstractVision.

## Run tests

```bash
python -m unittest discover -s tests -p "test_*.py" -q
```

## Common contribution types

### 1) Improve documentation

Core entrypoints:
- [`README.md`](README.md)
- [`docs/getting-started.md`](docs/getting-started.md)
- [`docs/architecture.md`](docs/architecture.md)
- [`docs/api.md`](docs/api.md)
- [`docs/faq.md`](docs/faq.md)
- [`docs/adr/README.md`](docs/adr/README.md)

Doc hygiene checklist:
- Commands are copy/pastable.
- Links resolve (relative links are preferred).
- Claims about support status match the current code (see [`docs/reference/backends.md`](docs/reference/backends.md)).
- Major claims are anchored in evidence (link to the relevant `src/` implementation).
- Prefer diagrams in Mermaid when they improve clarity ([`docs/architecture.md`](docs/architecture.md) is the canonical place).

### 2) Add or update models in the capability registry

Source of truth:
- `src/abstractvision/assets/vision_model_capabilities.json`

Validator + loader:
- `src/abstractvision/model_capabilities.py`

Checklist:
- Add/update the model entry in the JSON.
- Keep the model family platform-neutral; add engine-specific artifacts as download variants rather
  than as host-specific duplicate models.
- Prefer official upstream repos. Use community repos only when they provide the best runtime-native
  artifact for a target engine and label them clearly.
- Do not treat adapters or component artifacts as standalone curated models unless the runtime path
  is first-class.
- Keep the change aligned with
  [ADR 0005](docs/adr/0005_curated_capability_registry_and_download_catalog.md),
  [ADR 0006](docs/adr/0006_operator_control_configuration_precedence_and_explicit_network_use.md),
  and [docs/reference/capabilities-registry.md](docs/reference/capabilities-registry.md).
- Run the unit tests (they validate schema + coverage).
- Sanity check CLI output:
  - `abstractvision show-model <model_id>`

### 3) Add a new backend

Backend interface:
- `src/abstractvision/backends/base_backend.py`

Where backends live:
- `src/abstractvision/backends/`

Checklist:
- Implement the `VisionBackend` methods (raise `CapabilityNotSupportedError` for unsupported tasks).
- Keep imports lazy (avoid importing Torch/Diffusers at module import time unless unavoidable).
- Add/extend tests under `tests/`.
- Document the backend in `docs/reference/backends.md` and, if user-facing, add a short section in `docs/getting-started.md`.

## Submitting a change

Please include:
- A short explanation of the change and why it’s needed.
- Test results (`python -m unittest ...`).
- Any doc updates required to keep the repository truthful.

## Questions / discussions

If you’re unsure about scope or design, open an issue with a minimal proposal and a concrete example (inputs/outputs).
--- >8 --- END FILE: CONTRIBUTING.md --- >8 ---

--- 8< --- FILE: SECURITY.md --- 8< ---
# Security policy

We take security issues seriously and appreciate responsible disclosure.

## Reporting a vulnerability

Please **do not** open a public GitHub issue for security reports.

Instead, report privately by email:

- `contact@abstractcore.ai`

Include as much of the following as you can:

- A clear description of the issue and its impact
- Reproduction steps (or a minimal PoC)
- Affected versions / commit hash (if known)
- Any relevant logs, stack traces, or configuration
- Suggested mitigation (if you have one)

If you believe the issue is in an upstream dependency (e.g. Torch/Diffusers), it can still be helpful to notify us so we can assess impact and coordinate messaging for AbstractVision users.

## What to expect

We aim to:

- Acknowledge receipt within **3 business days**
- Provide a status update within **7 business days**

If a coordinated disclosure timeline is needed, please include your preferred timeline in the report.

## Scope

This policy applies to vulnerabilities in this repository’s code and packaging.

For non-security bugs and feature requests, please use the normal issue tracker.
--- >8 --- END FILE: SECURITY.md --- >8 ---

--- 8< --- FILE: ACKNOWLEDGMENTS.md --- 8< ---
# Acknowledgments

AbstractVision stands on the shoulders of excellent open-source projects and communities.

## Optional runtime dependencies (declared as extras)

- **Hugging Face Diffusers** (local pipeline runtime; used by the Diffusers backend): [`src/abstractvision/backends/huggingface_diffusers.py`](src/abstractvision/backends/huggingface_diffusers.py) (declared in the `diffusers`/`local`/`all` extras)
- **PyTorch** (tensor runtime for local inference; used via Diffusers): [`src/abstractvision/backends/huggingface_diffusers.py`](src/abstractvision/backends/huggingface_diffusers.py) (declared in the `diffusers`/`local`/`all` extras)
- **Hugging Face Transformers** (tokenizers/encoders used by some diffusion pipelines; imported by the Diffusers backend): [`src/abstractvision/backends/huggingface_diffusers.py`](src/abstractvision/backends/huggingface_diffusers.py) (declared in the `diffusers`/`local`/`all` extras)
- **Accelerate** (installed for ecosystem compatibility; used transitively by some pipelines): declared in optional extras in `pyproject.toml`
- **Safetensors** (model weight format support; used by Diffusers/Transformers): declared in optional extras in `pyproject.toml`
- **SentencePiece** (T5/tokenizer support for some model families): declared in optional extras in `pyproject.toml`
- **protobuf** (runtime dependency for some tokenizers/pipelines): declared in optional extras in `pyproject.toml`
- **einops** (tensor ops used by some modern architectures): declared in optional extras in `pyproject.toml`
- **PEFT** (LoRA adapter support used by Diffusers): declared in optional extras in `pyproject.toml`
- **Pillow** (image I/O utilities used by local backends): [`src/abstractvision/backends/huggingface_diffusers.py`](src/abstractvision/backends/huggingface_diffusers.py), [`src/abstractvision/backends/stable_diffusion_cpp.py`](src/abstractvision/backends/stable_diffusion_cpp.py) (declared in optional extras in `pyproject.toml`)
- **stable-diffusion-cpp-python** (python bindings used when `sd-cli` is not available): [`src/abstractvision/backends/stable_diffusion_cpp.py`](src/abstractvision/backends/stable_diffusion_cpp.py) (declared in the `sdcpp`/`local`/`all` extras)

## Runtime dependencies (transitive but central)

- **huggingface_hub** (model and adapter downloads; used by Diffusers/Transformers pipelines)

## Upstream projects

- **stable-diffusion.cpp** (upstream project that provides `sd-cli` and the core GGUF runtime wrapped by the bindings): [`src/abstractvision/backends/stable_diffusion_cpp.py`](src/abstractvision/backends/stable_diffusion_cpp.py)

## Referenced model/component publishers

- **Comfy-Org** (component artifacts referenced by docs and download helpers for `stable-diffusion.cpp` examples, notably Qwen Image and FLUX.2 VAE files): [`docs/getting-started.md`](docs/getting-started.md), [`scripts/download_model_sets.py`](scripts/download_model_sets.py)
- **Black Forest Labs** (official FLUX.2 upstream weights and side artifacts referenced by the capability registry and local-backend docs): [`src/abstractvision/assets/vision_model_capabilities.json`](src/abstractvision/assets/vision_model_capabilities.json), [`docs/getting-started.md`](docs/getting-started.md)
- **Qwen** (official Qwen Image / Qwen Image Edit upstream weights referenced by the capability registry and `stable-diffusion.cpp` component-mode docs): [`src/abstractvision/assets/vision_model_capabilities.json`](src/abstractvision/assets/vision_model_capabilities.json), [`docs/getting-started.md`](docs/getting-started.md)
- **Unsloth** (community GGUF conversions and companion encoder files referenced by the capability registry and some GGUF download presets): [`src/abstractvision/assets/vision_model_capabilities.json`](src/abstractvision/assets/vision_model_capabilities.json), [`scripts/download_model_sets.py`](scripts/download_model_sets.py)
- **leejet** (stable-diffusion.cpp maintainer-published runtime and GGUF conversions referenced by the `sdcpp` backend docs and GGUF download presets): [`src/abstractvision/backends/stable_diffusion_cpp.py`](src/abstractvision/backends/stable_diffusion_cpp.py), [`scripts/download_model_sets.py`](scripts/download_model_sets.py)

## Optional integrations

- **AbstractCore** (tool integration helpers + capability plugin): [`src/abstractvision/integrations/`](src/abstractvision/integrations/) (optional dependency in [`pyproject.toml`](pyproject.toml))

## Packaging

- **setuptools** and **wheel** (build system): [`pyproject.toml`](pyproject.toml)

## Community and contributors

Thanks to everyone who reports issues, suggests improvements, and contributes fixes or documentation updates.
--- >8 --- END FILE: ACKNOWLEDGMENTS.md --- >8 ---

--- 8< --- FILE: CHANGELOG.md --- 8< ---
# Changelog

## Unreleased

## 0.3.22 - 2026-06-07

- Runtime dependency: raise the MLX-Gen optional runtime floor to `mlx-gen>=0.18.13,<0.19.0` so Apple Silicon installs pick up the current SeedVR2 upscaler, official base-model support, canonical q8/q4 packages, and the latest image-edit fixes.
- Image upscaling: add first-class `ImageUpscaleRequest`, `VisionManager.upscale_image(...)`, backend `upscale_image_with_progress(...)`, and canonical `image_upscale` capability metadata.
- MLX-Gen SeedVR2: surface official `ByteDance-Seed/SeedVR2-3B` and `ByteDance-Seed/SeedVR2-7B` base models plus canonical `AbstractFramework/seedvr2-{3b,7b}-{8bit,4bit}` packages. Short selectors such as `seedvr2-3b` resolve to the recommended 8-bit package, while exact q4/q8 model ids remain exact.
- CLI/Core integration: add one-shot `abstractvision upscale` with default `mlx-gen/AbstractFramework/seedvr2-3b-8bit`, AbstractCore plugin tool support, and residency/catalog support for upscaler-only models.
- Progress events: relay SeedVR2 denoise-step progress through the same normalized progress event surface used by image/video generation, with `task="image_upscale"`.

## 0.3.21 - 2026-06-04

- Wan A14B guidance controls: promote MLX-Gen `guidance_2` to a first-class `VideoGenerationRequest` / `ImageToVideoRequest` field, CLI `--guidance-2` flag, REPL flag, playground job parameter, and AbstractCore tool parameter.
- Model registry: declare `guidance_2` defaults for Wan 2.2 A14B text-to-video and image-to-video tasks so discovery, tool defaults, and provider catalogs agree.
- OpenAI-compatible video: forward typed `guidance_2` to custom text-to-video and image-to-video endpoints that support the Wan-style two-stage guidance contract.

## 0.3.20 - 2026-06-04

- Runtime dependency: raise the MLX-Gen optional runtime floor to `mlx-gen>=0.18.10,<0.19.0` so Apple Silicon installs pick up the current capability planner, shared image/video progress events, and multi-reference edit routing.
- MLX-Gen progress: surface image generation and image edit progress through backend `*_with_progress(...)`, Python/Core `on_progress(event)` callbacks, and opt-in one-shot CLI `--progress` flags while preserving video progress availability for shell, Python, and AbstractCore callers.
- MLX-Gen progress semantics: align normalized video progress, the CLI display, and backend progress-bar callbacks with MLX-Gen denoise-step progress while retaining video frame counters as contextual event fields.
- MLX-Gen multi-reference edits: allow FLUX.2 and Qwen Image Edit requests to pass additional reference images through `extra.reference_images` / CLI `--reference-image`, and mark the supported model task metadata accordingly.
- Docs/tests: document image/edit/video progress across CLI, Python, and AbstractCore plugin calls; add propagation coverage for manager, CLI, plugin, Flux.2 multi-reference edit paths, and MLX-Gen denoise-step video progress normalization.
- Examples: add a bundled MLX-Gen local example gallery with copy-paste image/edit/video commands, generated PNG/MP4 outputs, contact sheet, manifest, and AbstractCore plugin progress-event capture.

## 0.3.19 - 2026-06-03

- Docs: clarify MLX-Gen `0.18.8` setup, Wan 2.2 A14B text-to-video/image-to-video usage, and generated-media configuration references.
- AI-readable docs: refresh `llms.txt` and `llms-full.txt` for the current AbstractVision Apple profile and video model surface.

## 0.3.18 - 2026-05-31

- Runtime dependency: raise the MLX-Gen optional runtime floor to `mlx-gen>=0.18.8,<0.19.0` so Apple Silicon installs pick up Wan 2.2 A14B text-to-video and image-to-video support.
- Wan A14B catalog: expose `Wan-AI/Wan2.2-T2V-A14B-Diffusers` and `Wan-AI/Wan2.2-I2V-A14B-Diffusers` through the canonical `mlx-gen` provider with task-specific defaults and capability metadata.
- Core propagation: prepare AbstractVision `0.3.18` so AbstractCore can depend on the corrected MLX-Gen runtime contract.

## 0.3.17 - 2026-05-29

- Docs: update the docs-site favicon and add an embedded text-to-video example carousel (5s MP4) to `docs/getting-started.md`.
- Runtime dependency: raise the MLX-Gen optional runtime floor to `mlx-gen>=0.18.7,<0.19.0` so Apple Silicon installs pick up the latest Wan video/model fixes.
- Core propagation: prepare the next AbstractVision package version (`0.3.17`) so AbstractCore can depend on the corrected MLX-Gen runtime contract.

## 0.3.16 - 2026-05-26

- MLX-Gen video progress: surface Wan `text_to_video` / `image_to_video` progress events through one-shot `abstractvision t2v` / `i2v`, interactive `/t2v` / `/i2v`, and Python/Core `on_progress(event)` callbacks.
- Backend API: add normalized `VideoProgressEvent` objects with phase, frame, total frame, step, total step, and progress fields while preserving backend `*_with_progress(...)` two-argument callbacks for generic progress bars.
- Capability filtering: keep Diffusers task support aligned with backend-specific registry requirements so MLX-Gen-only edit paths do not leak into Diffusers capability reports.
- AbstractCore integration: forward `on_progress` through plugin `t2v` / `i2v` requests so gateway/runtime callers can observe local MLX-Gen video runs without hardcoded backend plumbing.
- Docs/tests: document video progress usage across shell, REPL, Python, and AbstractCore examples; expand regression coverage for MLX-Gen progress bridging and quiet CLI flags.

## 0.3.15 - 2026-05-26

- CLI compatibility: canonicalize legacy `mflux/<model>` model prefixes to the `mlx-gen` runtime path for one-shot commands, matching `--provider mflux` and canonical `--provider mlx-gen` behavior.
- Tests: add one-shot `t2i`/`i2i` coverage proving the legacy `mflux` alias routes through the MLX-Gen backend without changing catalog/provider identity.

## 0.3.14 - 2026-05-26

- MLX-Gen migration: switch the Apple Silicon image runtime dependency to `mlx-gen`, expose `mlx-gen` as the canonical provider/engine, and keep `mflux` names as compatibility aliases.
- Catalog/model coverage: prioritize the AbstractFramework Hugging Face MLX-Gen q4/q8 prepared model collection, default q4 for local memory efficiency, and keep q8 variants discoverable for quality-focused runs.
- Local image support: expand the MLX-Gen backend/catalog surface for FLUX.2 klein/base, Qwen Image, Qwen Image Edit, Z-Image, Z-Image Turbo, and ERNIE Image Turbo q4/q8, including Qwen Image Edit `image_to_image` routing.
- Runtime dependency: require `mlx-gen>=0.18.5` for ERNIE Image Turbo q4/q8 prepared-folder support.
- Core/Gateway propagation: surface canonical `mlx-gen` provider/model metadata through AbstractCore and Gateway catalog paths while preserving legacy routed ids.
- Docs/tests: refresh CLI, backend, AbstractCore integration, and agent docs for MLX-Gen naming and q4/q8 guidance.

## 0.3.13 - 2026-05-23

- MFLUX edits: enable `image_to_image` for the curated FLUX.2 klein presets (`flux2-klein-4b`, `flux2-klein-9b`) so they surface in the playground edit UI and can be called through `VisionManager.edit_image(...)` (mask edits are not supported yet).
- Request normalization: map edit `strength` into the underlying MFLUX `image_strength` parameter and infer width/height from the input image when omitted so the UI/CLI behave consistently across backends.
- macOS GGUF policy: re-enable stable-diffusion.cpp GGUF presets on Apple Silicon by default and add an explicit opt-out (`ABSTRACTVISION_DISABLE_GGUF_ON_MACOS=1`) to hide GGUF catalog entries and reject GGUF execution for operators who want that behavior.
- Docs/tests: refresh README + core docs for the new MFLUX edit surface and the GGUF opt-out, and expand regression coverage for catalog surfacing and edit parameter plumbing.

## 0.3.12 - 2026-05-22

- Release packaging: publish the current clean tracked repository state as a fresh patch release.

## 0.3.11 - 2026-05-22

- Hugging Face cache integrity: treat repo-level `.incomplete` blobs as an incomplete snapshot, so interrupted Diffusers downloads no longer masquerade as a usable cached model.
- Qwen Image Edit downloads: `abstractvision download qwen-image-edit-2511 --provider diffusers` now resumes the real 16-bit snapshot instead of incorrectly short-circuiting on a partial cache.

## 0.3.10 - 2026-05-22

- Catalog/handles: make the curated Qwen Image Edit aliases consistent across the registry, downloads, and CLI catalog surfaces (`qwen-image-edit`, `qwen-image-edit-2509`, `qwen-image-edit-2511`), and re-verify all packaged Hugging Face repo ids in `vision_model_capabilities.json`.
- Runtime truth: quarantine unreliable local runtime task paths until they are re-validated. Local Diffusers `GLM-Image` is now hidden/disabled, local MFLUX is surfaced for `text_to_image` only, and local Diffusers `text_to_video` is marked experimental and disabled from the normal local surfaces.
- Playground/catalog UX: keep the per-task playground tabs with per-tab load/unload controls, but filter task selectors through backend-supported runtime truth so disabled local tasks do not surface. Update the `download` command messaging so curated 16-bit Diffusers snapshots are reported honestly instead of looking like a failed 8-bit fallback.
- Docs/backlog/tests: add the planned quarantine follow-up item, refresh the core docs around current local support policy, expand regression coverage for the runtime blacklists and catalog filtering, and regenerate `llms-full.txt`.

## 0.3.9 - 2026-05-21

- Local Diffusers video: implement the first in-process `text_to_video` path through `zai-org/CogVideoX-2b` / `THUDM/CogVideoX-2b`, including registry-backed request normalization, preload warmup, unload support, MPS-safe explicit FP16 device moves, and MP4 artifact outputs packaged through `ffmpeg`.
- Interactive surfaces: add one-shot `abstractvision t2v`, REPL `/t2v`, and playground `Text → Video` generation with backend/task-aware model discovery, progress reporting, and catalog surfacing for local video-capable models.
- AbstractCore integration: route plugin `t2v` / `i2v` calls through the same explicit backend binding path used by image generation, extend residency task aliases for video, and track request-warm local `text_to_video` models in the loaded-model inventory.
- Catalog/tests/docs: add curated capability/download metadata for `zai-org/CogVideoX-2b`, keep local image-edit surfaces intact, expand backend/CLI/playground/plugin regression coverage, refresh README/reference/FAQ/getting-started docs, and regenerate `llms-full.txt`.

## 0.3.8 - 2026-05-20

- Model registry/catalog: expand the curated `vision_model_capabilities.json` inventory with more current official or reputable vision model families and cross-platform download variants, tighten provider provenance, remove adapter-only repos from the standalone model catalog, and expose richer model notes / `task_specs` through the CLI catalog surfaces.
- Apple/MLX semantics: treat `mlx` as a download target rather than a runnable backend, fail fast on generic `mlx` provider/model routing across CLI, playground, and AbstractCore, and limit shipped MFLUX presets/docs to the families the backend actually supports today.
- GGUF / local-runtime routing: improve stable-diffusion.cpp bundle resolution for curated multi-artifact models and legacy Qwen GGUF repo ids, resolve curated `sdcpp` model keys consistently in both one-shot CLI and REPL paths, and make AbstractCore request-scoped `sdcpp` routing explicit while canonicalizing MFLUX aliases for backend reuse.
- Docs/policy/tests: add the ADR set and planned backlog items for curated catalog ownership, adapter-aware model graphs, and the long-term native MLX direction; refresh README/reference/FAQ/agent docs; regenerate `llms-full.txt`; and expand regression coverage for catalog aliases, GLM defaults, MFLUX routing, and local backend bundle selection.

## 0.3.7 - 2026-05-19

- AbstractCore plugin: add explicit local model residency control for in-process `diffusers`, `mflux`, and `sdcpp` backends through `load_resident_model(...)`, `list_loaded_models(...)`, `list_resident_models(...)`, and `unload_resident_model(...)`, with `load_model(...)` / `unload_model(...)` compatibility aliases for Core route adapters.
- Residency semantics: preserve explicit resident models across model switches while keeping the previous unload-on-switch behavior for non-resident request-warm backends, and report stable process-local `load_id` / `backend_kind` / `resident` metadata instead of relying on backend private fields.
- Safety/robustness: reject OpenAI/OpenAI-compatible HTTP backends for residency control even on localhost, normalize provider/task filters consistently, reject ambiguous unload requests, and keep unload behavior deterministic for injected local backends.
- Tests/docs: expand AbstractCore plugin coverage for local preload/list/unload flows, request-warm loaded-state reporting, task-aware loaded-model filters, injected backend residency, and switch-survival behavior; document the new local residency control surface and regenerate `llms-full.txt`.

## 0.3.6 - 2026-05-18

- Model registry/downloads: expand `vision_model_capabilities.json` and curated preset coverage for current Hugging Face text-to-image and image-to-image models, including FLUX.1/2, Qwen Image, ERNIE Image, GLM Image, Z-Image, SDXL, SD Turbo, and SD3.5. The registry now remains the packaged source of truth for both downloadable artifacts and task capability metadata.
- Cache/catalog: keep curated downloads in the Hugging Face cache by default, migrate older `~/models/<preset>` trees into that cache on first use, and align Diffusers/MFLUX cache discovery across CLI, REPL, playground, and AbstractCore so cached local models surface consistently.
- API-level normalization: move model-specific parameter normalization into shared backend hooks used by `VisionManager` and the playground server. Model constraints such as MFLUX distilled `guidance_scale=1.0`, unsupported negative prompts, and GLM `steps/guidance_scale/32-multiple dimensions` now apply consistently through the CLI/REPL, playground, and AbstractCore integration.
- Playground/API: expose structured per-model `task_specs` from `/v1/vision/models`, only enable the edit surface for models that advertise `image_to_image`, switch models by auto-loading the new selection, and truncate `b64_json` in logs instead of dumping the full base64 payload.
- AbstractCore/provider catalogs: keep remote provider/model listing robust when internet or API credentials are unavailable, and keep local provider/model catalogs aligned with the same cache-backed discovery used by the interactive surfaces.
- CLI/docs: keep `abstractvision cli` as the canonical interactive command (with `repl` as a legacy alias), expand model catalog/docs coverage, refresh backend/configuration/AbstractCore/playground references, and regenerate `llms.txt` / `llms-full.txt` for release.

## 0.3.5 - 2026-05-13

- AbstractCore plugin: accept runtime output metadata such as `provider` and `size` without rejecting image generation requests, and forward per-call image model selectors to OpenAI-compatible backends.


## 0.3.4 - 2026-05-09

- Packaging: constrain the optional `stable-diffusion-cpp-python` binding to
  `<0.4.6` for `sdcpp`, `local`, `apple`, `all`, `all-apple`, and `all-gpu`
  extras. Version `0.4.6` currently ships an sdist that fails native Linux
  builds because required vendored libwebm CMake files are missing.

## 0.3.3 - 2026-05-08

- Packaging: add shared native install profiles. `abstractvision[apple]` and
  `abstractvision[all-apple]` install the full local macOS vision stack;
  `abstractvision[gpu]` installs the Diffusers/Torch GPU stack; and
  `abstractvision[all-gpu]` installs Diffusers plus stable-diffusion.cpp
  bindings for full local GPU-capable deployments.
- Docs/tests: document and verify the Apple/GPU profile aliases while keeping
  the base package lightweight and remote/OpenAI-compatible by default.

## 0.3.2 - 2026-05-08

- AbstractCore plugin: switch the default backend id to `abstractvision:openai` with official OpenAI defaults, while keeping `abstractvision:openai-compatible` registered as a legacy-compatible backend id.
- OpenAI configuration: use standard `OPENAI_API_KEY`, `OPENAI_BASE_URL`, `OPENAI_IMAGE_MODEL_ID`, and `OPENAI_IMAGE_MODEL` aliases for the plugin default path; keep compatible endpoints explicit with `ABSTRACTVISION_BACKEND=openai-compatible` and `OPENAI_BASE_URL`.
- Provider catalogs: add OpenAI/OpenAI-compatible `/models` discovery through `VisionManager.list_provider_models(...)`, `abstractvision provider-models`, REPL `/provider-models`, and `llm.vision.list_provider_models(...)` in the AbstractCore plugin.
- API/contracts: add `ProviderModelInfo`, provider catalog backend hooks, task filtering for OpenAI image model catalogs, and JSON-safe bounded provider metadata at the AbstractCore plugin boundary.
- Docs/agent context: refresh AbstractCore integration, backend/configuration/API docs, playground production-boundary notes, completed backlog reports, `llms.txt`, and regenerated `llms-full.txt`.
- Tests: add coverage for OpenAI defaults, legacy compatible aliases, provider catalog parsing/filtering, plugin catalog exposure, CLI/REPL catalog commands, manager delegation, and a credential-gated live OpenAI catalog check.

## 0.3.1 - 2026-05-07

- CI/release: move GitHub Actions checkout/setup-python/artifact/release actions to Node 24-compatible major versions, removing the Node 20 deprecation warnings from release runs.
- Tests: strengthen packaging metadata coverage so Diffusers aliases and `local`/`all` runtime bundles cannot drift from their intended dependency sets.
- Docs: separate contributor-only extras from runtime install profiles and explicitly mark `dev` as unsuitable for application/runtime dependency declarations.

## 0.3.0 - 2026-05-07

- Packaging: make the base install lightweight. `pip install abstractvision` no longer installs Torch, Diffusers, Transformers, Pillow, or local inference runtimes by default.
- Extras: add canonical runtime profiles `abstractvision[openai]`, `abstractvision[openai-compatible]`, `abstractvision[diffusers]`, `abstractvision[sdcpp]`, `abstractvision[local]`, and `abstractvision[all]`; keep `huggingface`/`huggingface-dev` compatibility aliases.
- Runtime defaults: keep AbstractCore plugin and one-shot CLI remote-first, and stop the REPL/playground from silently selecting Diffusers when no backend is configured.
- Errors/tests/CI: improve local-backend missing-extra hints, add stronger packaging/import-light/OpenAI-compatible coverage, and split CI into lightweight base and local Diffusers paths.
- Docs: update install guidance, backend references, and internal backlog policy for explicit local runtime extras.

## 0.2.6 - 2026-05-06

- Docs: refresh install extras, AbstractCore integration, playground ownership, OpenAI-compatible request-shape notes, and backend/config references so the public docs match the current code.
- Agent docs: update `llms.txt`, include playground endpoint docs in the generated `llms-full.txt` bundle, and regenerate the AI-ready documentation from current sources.
- Contributing: clarify that `abstractvision[abstractcore]` is a compatibility marker and that AbstractCore is supplied by the host application.

## 0.2.5 - 2026-05-06

- Packaging: keep the default Diffusers backend installable while moving `stable-diffusion-cpp-python` out of the base dependency set and into the explicit `sdcpp`/`local` extras. This keeps AbstractCore plugin installs from failing on platforms where stable-diffusion.cpp bindings need a local native build.
- AbstractCore plugin: restore the OpenAI-compatible HTTP backend as the default while keeping local `diffusers` and `sdcpp` backends explicit through config/env.
- OpenAI-compatible backend: shape requests correctly for real OpenAI GPT image models while preserving local OpenAI-compatible extensions for unknown model ids.
- Playground: capture the active backend at job submission time so background jobs do not accidentally run on a newly selected model.
- Docs/tests: clarify the default install shape, document when to install `abstractvision[sdcpp]`, and add metadata/OpenAI/playground coverage so these paths do not regress.

## 0.2.4 - 2026-05-06

- Playground: add a self-contained `abstractvision playground` command that serves both the web UI and `/v1/vision/*` API locally, so playground testing no longer depends on an AbstractCore server.
- Playground: package the HTML asset in the wheel, default the UI to the serving origin, and avoid stale persisted API URLs that could keep calling an older AbstractCore endpoint.
- Playground model loading: accept raw Hugging Face ids such as `runwayml/stable-diffusion-v1-5` directly, while still accepting explicit backend prefixes like `diffusers/...`, `sdcpp/...`, and `openai-compatible/...`.
- Packaging/CI: keep AbstractCore out of AbstractVision dependency metadata and test workflows; AbstractCore remains an optional host integration loaded lazily when present.
- Docs/tests: refresh playground docs around the self-contained local server and add coverage for cached model listing, raw model loading, playground jobs, and tool integration without installing AbstractCore.

## 0.2.3 - 2026-05-06

- AbstractCore plugin: support local Diffusers and stable-diffusion.cpp backends through `llm.vision`, not only OpenAI-compatible HTTP. The default plugin path now matches the REPL default: local Diffusers with `runwayml/stable-diffusion-v1-5`, cache-only unless `ABSTRACTVISION_DIFFUSERS_ALLOW_DOWNLOAD=1` is set.
- AbstractCore plugin: keep OpenAI-compatible usage available with `ABSTRACTVISION_BACKEND=openai` plus `OPENAI_BASE_URL`, and preserve artifact-store behavior for generated media outputs.

## 0.2.2 - 2026-05-06

- Release automation: add GitHub Actions CI/release workflows, issue templates, pre-commit config, MkDocs config, PyPI trusted publishing, GitHub Releases, and release-time docs deployment to `gh-pages`.
- Packaging: support Python 3.9-3.13, modernize license metadata, include package data explicitly, and add test/docs/dev extras.
- Defaults: make the REPL default to local Diffusers with `runwayml/stable-diffusion-v1-5`, `ABSTRACTVISION_DIFFUSERS_DEVICE=auto`, and cache-only/offline runtime downloads disabled by default.
- Diffusers backend: keep `allow_download=False` as the default, force Hugging Face offline/cache-only env during loads/calls, disable implicit HF token use offline, and load from cached snapshot paths when present.
- Diffusers backend: ignore unknown REPL `extra` flags that the pipeline `__call__` does not accept, avoiding `unexpected keyword argument` crashes.
- Diffusers backend: add better fp16 variant fallback behavior, MPS dtype/invalid-output retry handling, LoRA/Rapid-AIO offline handling, and clearer missing-local-model errors.
- Capability registry: add `black-forest-labs/FLUX.2-klein-9B` and normalize FLUX license id to `flux-non-commercial-license`.
- Docs: refresh quickstarts around Stable Diffusion 1.5 first, add clearer macOS Metal/NVIDIA CUDA/CPU guidance, expand stable-diffusion.cpp notes, and point users to cache-only local workflows.
- Tooling: add `scripts/download_model_sets.py` for explicit heavyweight model downloads (Stable Diffusion 1.5, FLUX 2 GGUF/Diffusers, and Qwen Image snapshots).
- Cleanup: remove the misspelled duplicate `ACKNOWLEDMENTS.md`.

## 0.2.1

- Documentation refresh for public release:
  - add `docs/api.md` and strengthen cross-linking between README and docs
  - add `CONTRIBUTING.md`, `SECURITY.md`, and `ACKNOWLEDGMENTS.md`
  - add `llms.txt` and generated `llms-full.txt` for agent-oriented context
  - clarify playground/server endpoint expectations (`/v1/vision/*`)

## 0.2.0

- Add stable-diffusion.cpp (`sd-cli`) backend for local GGUF diffusion models.
- REPL: forward unknown `--flags` as backend `extra` parameters.
- Add a tiny web playground (`playground/vision_playground.html`) for testing via AbstractCore Server vision endpoints (`/v1/vision/*`).

## 0.1.0

- Initial MVP: capability registry + schema validation.
- Artifact-first outputs via `LocalAssetStore` and runtime adapter.
- OpenAI-compatible HTTP backend for image generation/editing (optional video endpoints via config).
- Local Diffusers backend for images (opt-in deps).
- AbstractCore tool integration (`make_vision_tools`) with artifact refs.
- CLI/REPL for interactive manual testing.
--- >8 --- END FILE: CHANGELOG.md --- >8 ---

--- 8< --- FILE: pyproject.toml --- 8< ---
```toml
[build-system]
requires = ["setuptools>=77.0.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "abstractvision"
dynamic = ["version"]
description = "Model-agnostic generative vision abstractions (image/video) for the Abstract ecosystem"
readme = "README.md"
license = "MIT"
license-files = ["LICENSE"]
authors = [{name = "Laurent-Philippe Albou", email = "contact@abstractcore.ai"}]
requires-python = ">=3.9"
classifiers = [
  "Development Status :: 3 - Alpha",
  "Intended Audience :: Developers",
  "Operating System :: OS Independent",
  "Programming Language :: Python :: 3",
  "Programming Language :: Python :: 3.9",
  "Programming Language :: Python :: 3.10",
  "Programming Language :: Python :: 3.11",
  "Programming Language :: Python :: 3.12",
  "Programming Language :: Python :: 3.13",
  "Topic :: Multimedia",
  "Topic :: Scientific/Engineering :: Artificial Intelligence",
]
# Base install is intentionally lightweight. Local inference runtimes are
# explicit extras so OpenAI-compatible/AbstractCore plugin hosts do not pull
# Torch, Diffusers, CUDA-adjacent wheels, or local model stacks by default.
dependencies = []

[project.urls]
Homepage = "https://github.com/lpalbou/abstractvision"
Repository = "https://github.com/lpalbou/abstractvision"

[project.scripts]
abstractvision = "abstractvision.cli:main"

[project.entry-points."abstractcore.capabilities_plugins"]
abstractvision = "abstractvision.integrations.abstractcore_plugin:register"

[project.optional-dependencies]
# Official OpenAI image endpoint intent. The backend is stdlib-only today and
# does not require the OpenAI SDK.
openai = []

# Generic OpenAI-shaped HTTP endpoint intent (local or remote /v1 servers).
# The backend is stdlib-only today; keep this extra for readable install commands.
openai-compatible = []

# Local image generation via Hugging Face Diffusers/Torch.
diffusers = [
  "diffusers>=0.36.0",
  "torch>=2.0,<3.0.0",
  "torchvision>=0.15.0",
  "transformers>=4.0,<6.0.0",
  "accelerate>=0.0",
  "safetensors>=0.0",
  "huggingface_hub>=0.24.0",
  # Needed by T5 tokenizers used in SD3/FLUX and some other diffusion pipelines.
  "sentencepiece>=0.1.99",
  # Some HF tokenizers/pipelines require protobuf at runtime.
  "protobuf>=3.20.0",
  # Used by some modern diffusion architectures.
  "einops>=0.7.0",
  # LoRA adapter support in Diffusers.
  "peft>=0.10.0",
  # GGUF transformer loading in Diffusers.
  "gguf>=0.10.0",
  "Pillow>=9.0",
]

# Local generation via stable-diffusion.cpp python bindings (pip-installable).
sdcpp = [
  "stable-diffusion-cpp-python>=0.4.2,<0.4.6",
  "huggingface_hub>=0.24.0",
  "Pillow>=9.0",
]

# Native Apple image generation via MLX-Gen. Kept optional because it pulls
# Apple-specific local runtime dependencies.
mlx-gen = [
  "mlx-gen>=0.18.13,<0.19.0; platform_system == 'Darwin' and python_version >= '3.10'",
]

# Compatibility extra for callers that still install the historical MFLUX extra.
# It intentionally installs MLX-Gen, not the retired standalone mflux package.
mflux = [
  "mlx-gen>=0.18.13,<0.19.0; platform_system == 'Darwin' and python_version >= '3.10'",
]

# Native Apple Python install profile. Docker cannot expose Apple's Metal stack,
# but direct macOS installs can use PyTorch/MPS, stable-diffusion.cpp/Metal, and MLX-Gen/MLX.
apple = [
  "diffusers>=0.36.0",
  "torch>=2.0,<3.0.0",
  "torchvision>=0.15.0",
  "transformers>=4.0,<6.0.0",
  "accelerate>=0.0",
  "safetensors>=0.0",
  "huggingface_hub>=0.24.0",
  "sentencepiece>=0.1.99",
  "protobuf>=3.20.0",
  "einops>=0.7.0",
  "peft>=0.10.0",
  "gguf>=0.10.0",
  "stable-diffusion-cpp-python>=0.4.2,<0.4.6",
  "mlx-gen>=0.18.13,<0.19.0; platform_system == 'Darwin' and python_version >= '3.10'",
  "Pillow>=9.0",
]

gpu = [
  "diffusers>=0.36.0",
  "torch>=2.0,<3.0.0",
  "torchvision>=0.15.0",
  "transformers>=4.0,<6.0.0",
  "accelerate>=0.0",
  "safetensors>=0.0",
  "huggingface_hub>=0.24.0",
  "sentencepiece>=0.1.99",
  "protobuf>=3.20.0",
  "einops>=0.7.0",
  "peft>=0.10.0",
  "gguf>=0.10.0",
  "Pillow>=9.0",
]

# Compatibility extra for callers that still request the historical Diffusers extra.
huggingface = [
  "diffusers>=0.36.0",
  "torch>=2.0,<3.0.0",
  "torchvision>=0.15.0",
  "transformers>=4.0,<6.0.0",
  "accelerate>=0.0",
  "safetensors>=0.0",
  "huggingface_hub>=0.24.0",
  "sentencepiece>=0.1.99",
  "protobuf>=3.20.0",
  "einops>=0.7.0",
  "peft>=0.10.0",
  "gguf>=0.10.0",
  "Pillow>=9.0",
]

# Convenience: installs both local backends (Diffusers + stable-diffusion.cpp python bindings).
local = [
  "diffusers>=0.36.0",
  "torch>=2.0,<3.0.0",
  "torchvision>=0.15.0",
  "transformers>=4.0,<6.0.0",
  "accelerate>=0.0",
  "safetensors>=0.0",
  "huggingface_hub>=0.24.0",
  "sentencepiece>=0.1.99",
  "protobuf>=3.20.0",
  "einops>=0.7.0",
  "peft>=0.10.0",
  "gguf>=0.10.0",
  "stable-diffusion-cpp-python>=0.4.2,<0.4.6",
  "Pillow>=9.0",
]

# Convenience: installs every runtime backend, but not contributor tooling.
all = [
  "diffusers>=0.36.0",
  "torch>=2.0,<3.0.0",
  "torchvision>=0.15.0",
  "transformers>=4.0,<6.0.0",
  "accelerate>=0.0",
  "safetensors>=0.0",
  "huggingface_hub>=0.24.0",
  "sentencepiece>=0.1.99",
  "protobuf>=3.20.0",
  "einops>=0.7.0",
  "peft>=0.10.0",
  "gguf>=0.10.0",
  "stable-diffusion-cpp-python>=0.4.2,<0.4.6",
  "mlx-gen>=0.18.13,<0.19.0; platform_system == 'Darwin' and python_version >= '3.10'",
  "Pillow>=9.0",
]

# Full native Apple profile: every runtime backend relevant to macOS/Apple
# Silicon, excluding NVIDIA/CUDA-specific stacks.
all-apple = [
  "diffusers>=0.36.0",
  "torch>=2.0,<3.0.0",
  "torchvision>=0.15.0",
  "transformers>=4.0,<6.0.0",
  "accelerate>=0.0",
  "safetensors>=0.0",
  "huggingface_hub>=0.24.0",
  "sentencepiece>=0.1.99",
  "protobuf>=3.20.0",
  "einops>=0.7.0",
  "peft>=0.10.0",
  "gguf>=0.10.0",
  "stable-diffusion-cpp-python>=0.4.2,<0.4.6",
  "mlx-gen>=0.18.13,<0.19.0; platform_system == 'Darwin' and python_version >= '3.10'",
  "Pillow>=9.0",
]

all-gpu = [
  "diffusers>=0.36.0",
  "torch>=2.0,<3.0.0",
  "torchvision>=0.15.0",
  "transformers>=4.0,<6.0.0",
  "accelerate>=0.0",
  "safetensors>=0.0",
  "huggingface_hub>=0.24.0",
  "sentencepiece>=0.1.99",
  "protobuf>=3.20.0",
  "einops>=0.7.0",
  "peft>=0.10.0",
  "gguf>=0.10.0",
  "stable-diffusion-cpp-python>=0.4.2,<0.4.6",
  "Pillow>=9.0",
]

# NOTE: PyPI rejects VCS/direct URL dependencies in package metadata.
# If you need Diffusers "main" for unreleased pipelines, install it explicitly *after*:
#   pip install "abstractvision[diffusers-dev]"
#   pip install "diffusers @ git+https://github.com/huggingface/diffusers@main"
diffusers-dev = [
  "diffusers>=0.36.0",
  "torch>=2.0,<3.0.0",
  "torchvision>=0.15.0",
  "transformers>=5.0",
  "accelerate>=0.0",
  "safetensors>=0.0",
  "huggingface_hub>=0.24.0",
  "sentencepiece>=0.1.99",
  "protobuf>=3.20.0",
  "einops>=0.7.0",
  "peft>=0.10.0",
  "gguf>=0.10.0",
  "Pillow>=9.0",
]

# Compatibility extra for callers that still request the historical dev extra.
huggingface-dev = [
  "diffusers>=0.36.0",
  "torch>=2.0,<3.0.0",
  "torchvision>=0.15.0",
  "transformers>=5.0",
  "accelerate>=0.0",
  "safetensors>=0.0",
  "huggingface_hub>=0.24.0",
  "sentencepiece>=0.1.99",
  "protobuf>=3.20.0",
  "einops>=0.7.0",
  "peft>=0.10.0",
  "gguf>=0.10.0",
  "Pillow>=9.0",
]

# Compatibility extra only. AbstractVision is loaded by AbstractCore as a plugin,
# so it deliberately does not install AbstractCore as a dependency.
abstractcore = []

# Local model download helpers. Kept optional so the base package remains
# dependency-free for OpenAI-compatible proxy hosts.
models = [
  "huggingface_hub>=0.24.0",
]

# Test dependencies used by local contributors. CI installs the package with
# `--no-deps` and then adds only these light test-time dependencies.
test = [
  "pytest>=7.0.0",
  "Pillow>=9.0",
  "torch>=2.0,<3.0.0",
]

# Documentation dependencies.
docs = [
  "mkdocs>=1.5.0",
  "mkdocs-material>=9.0.0",
]

# Complete local development environment.
dev = [
  "pytest>=7.0.0",
  "Pillow>=9.0",
  "torch>=2.0,<3.0.0",
  "diffusers>=0.36.0",
  "torchvision>=0.15.0",
  "transformers>=4.0,<6.0.0",
  "accelerate>=0.0",
  "safetensors>=0.0",
  "sentencepiece>=0.1.99",
  "protobuf>=3.20.0",
  "einops>=0.7.0",
  "peft>=0.10.0",
  "mkdocs>=1.5.0",
  "mkdocs-material>=9.0.0",
  "build>=1.0.0",
  "twine>=4.0.0",
  "ruff>=0.5.7",
  "black>=23.0.0",
  "pre-commit>=3.0.0",
]

[tool.setuptools]
packages = [
  "abstractvision",
  "abstractvision.assets",
  "abstractvision.backends",
  "abstractvision.integrations",
  "abstractvision.playground",
]

[tool.setuptools.package-dir]
"" = "src"

[tool.setuptools.dynamic]
version = {attr = "abstractvision.__version__"}

[tool.setuptools.package-data]
abstractvision = ["assets/*.json", "assets/*.metal", "playground/*.html"]

[tool.black]
line-length = 100
target-version = ["py39"]
include = '\.pyi?$'
extend-exclude = '''
/(
  \.eggs
  | \.git
  | \.hg
  | \.mypy_cache
  | \.tox
  | \.venv
  | build
  | dist
)/
'''

[tool.isort]
profile = "black"
multi_line_output = 3
line_length = 100
known_first_party = ["abstractvision"]

[tool.ruff]
target-version = "py39"
line-length = 100

[tool.ruff.lint]
select = [
  "E",
  "W",
  "F",
  "I",
  "B",
  "C4",
  "UP",
]
ignore = [
  "E501",
  "B008",
  "B904",
]

[tool.ruff.lint.per-file-ignores]
"__init__.py" = ["F401"]
"tests/**/*" = ["B011"]

[tool.pytest.ini_options]
pythonpath = ["src"]
minversion = "7.0"
addopts = "-ra --tb=short --strict-markers --strict-config --disable-warnings -p no:cacheprovider"
testpaths = ["tests"]
python_files = ["test_*.py"]
python_functions = ["test_*"]
python_classes = ["Test*"]
```
--- >8 --- END FILE: pyproject.toml --- >8 ---
