---
title: Agent Guide
description: Self-contained spec for an AI coding agent implementing nyris visual search in any language.
---

# Agent Guide: implement nyris visual search

You are implementing visual search against the nyris Find API. Follow this file. Do not invent endpoints. Do not call the Data Management API (`/manage/v1`) unless the user explicitly asked to edit the catalog.

Scope: adding nyris takes two steps, and this file covers only the second.

1. **Import the product data into the nyris index.** Not your task. See [Product Data & Images](https://docs.nyris.io/#submit-your-product-data). Do not upload data yourself.
2. **Integrate nyris into a user-facing frontend:** the website or app where users search. A thin backend that only forwards requests is in scope too.

The steps are independent. Do not stop if the user's data is not in nyris yet. Build against a demo index instead: the user can ask [support@nyris.io](mailto:support@nyris.io) for an API key to a demo index with sample products. Keep the key in configuration so it can be swapped for the production key later. Tell the user that step 1 is still open.

Reference implementation: the [nyris example web app](https://github.com/nyris/nyris-webapp/tree/main/packages/nyris-webapp). Where this file says "the example web app" or "the example client", it means that app.

The human version of this guide is [Integration (for humans)](https://docs.nyris.io/#add-nyris-visual-search). This file is enough on its own.

## Inputs you must have

Ask before coding if any of these are missing:

| Input | Required | Default |
| --- | --- | --- |
| Find API key | yes | environment variable `NYRIS_API_KEY` |
| Base URL | no | `https://api.nyris.io` |
| Target | yes | web, iOS, Android, React Native, Flutter, or backend |
| Text search | no | off |
| Filters | no | none |
| Existing text search engine and results page (Algolia, Elasticsearch, ...) | no | none. If there is one, follow [Existing text search](#existing-text-search) instead of building a separate results page. |

Never hard-code the key. Never print it. Never commit it.

The key is sent from the client as `X-Api-Key`. It must be a Find key. Refuse to embed a Data Management key in a client app.

## Endpoints

| Name | Request |
| --- | --- |
| Find | `POST {base}/find/v1.1` |
| Regions | `POST {base}/find/v2/regions` |
| Feedback | `POST {base}/feedback/v1` |
| Filters | `GET {base}/find/v1/filters?limit=100` |
| Recommend | `GET {base}/recommend/v1/{sku}` |
| Product images | `GET {base}/find/v1.1/images/{sku}` |

Filters: use `/find/v1/filters`. The response is a JSON array of `{ "key": "color", "values": ["Red", "Blue"] }` (empty if the index has no filters). To search the values of one key, call `GET {base}/find/v1/filters/{key}/{value}?limit=100`, where `{value}` is the search term; it returns `string[]`.

CAD search (`POST {base}/cad/find/v0.2`, multipart field `file`) is not part of the public API reference. Use it only if the user confirms that nyris enabled CAD search for their account.

## Request body formats

Find and Regions accept the image in three formats. The results are the same.

| Format | `Content-Type` | Body | Use when |
| --- | --- | --- | --- |
| Multipart (default) | `multipart/form-data` | part `image`, optional `text`, optional filter fields | You need text or filters. Browsers and mobile HTTP clients support it natively. |
| Binary | `image/jpeg` or `image/png` | the raw image bytes | Image only, no text or filters. Smallest request. |
| JSON (base64) | `application/json` | `{"image": "<base64>", "text": "...", "filters": [...]}` | The client or platform only speaks JSON (low-code tools, serverless functions, tool-calling backends, some SDK wrappers). |

JSON body schema:

| Field | Type | Required | Notes |
| --- | --- | --- | --- |
| `image` | string | yes, unless the search is text only | Plain base64 (`/9j/4AAQ...`) or a data URI (`data:image/jpeg;base64,/9j/...`). |
| `text` | string | no | Max 512 characters. |
| `filters` | array | no | `[{"key": "MachineSerial", "values": ["SN-40821"]}]`. Same AND/OR rules as multipart. |

Regions JSON takes only `image`.

Base64 makes the request about 33% larger. The 2 MB limit applies to the decoded image, not the encoded body. Still resize before encoding exactly as in step 4. Default to multipart. Use JSON when the user asks for it or the platform cannot send multipart or binary bodies.

Do the base64 encoding in code. If you are an agent that can call HTTP tools, never put image base64 into your own output or tool arguments by hand. It wastes tokens and gets corrupted.

## Normative steps

Implement in this order.

1. **Capture.** User picks a file or a camera frame. Show that image immediately. Do not wait for the network to show the preview.
2. **Normalize and keep.** Decode the bitmap. Apply EXIF orientation. Flatten transparency onto white (`rgb(255,255,255)`). Keep this normalized original at full resolution in memory until the user removes the image. Show a screen-sized preview derived from it. Never crop the preview or the step 4 JPEG; crops come from this original only.
3. **Convert formats.**
   - JPEG, PNG, WebP, GIF first frame, AVIF, BMP: decode as a bitmap.
   - HEIC/HEIF: decode with the platform, `createImageBitmap`, or `heic-to` / libheif, then continue as a bitmap.
   - TIFF: first page only. If pixels are 3-channel RGB, append alpha 255. Most browsers cannot decode TIFF natively; on the web, add a TIFF decoder or reject TIFF with a clear message. Do not list it in the file picker if you cannot decode it.
   - PDF: rasterize page 1, then continue as a bitmap.
   - Extensions `.stp`, `.step`, `.stl`, `.obj`, `.glb`, `.gltf`: do not send them to `/find/v1.1`. If CAD search is enabled for the account, POST the original file to `/cad/find/v0.2` as field `file`. Otherwise treat it as unsupported.
   - Try the native decoder first for every other file. Route by extension only for CAD. Do not reject HEIC, AVIF, or other image types by name; the platform may already decode them.
   - Anything that still fails to decode: stop and tell the user the format is unsupported.
   - Always send JPEG. The API itself accepts only JPEG, PNG, and WebP; anything else returns 415 (binary or multipart) or 400 (JSON base64). Base64 does not change this: the bytes inside must still be JPEG.
4. **Resize.** Check the source (or the crop) in original pixels. If its shortest edge is under 100 px, still search, but show a non-blocking hint that results may be less accurate. Scale with the area formula below so the output area is `1024 * 1024` (the example client always does this, including small sources). Encode JPEG at quality `0.9`. Target size is about 100 kB; keep it at most 1 MB for latency. If the blob is still above ~200 kB, re-encode at `0.8`. The API rejects images over 2 MB (2,097,152 bytes) with 413.
5. **Regions.** `POST /find/v2/regions` with `Content-Type: image/jpeg` and the raw JPEG as the body. Multipart (part `image`) and JSON base64 also work; see [Request body formats](#request-body-formats). Map pixel boxes to normalized 0..1 coordinates. If the list is empty, the crop is the full frame. If Regions fails, continue with the full frame; never block Find on Regions. As soon as Regions returns, auto-select the best region, mark its marker active, and run step 6 with that crop without waiting for a click. Draw the overlay as specified in [Region overlay](#region-overlay).
6. **Find.** `POST /find/v1.1` as `multipart/form-data` (or binary / JSON base64, see [Request body formats](#request-body-formats)). Field `image` is the JPEG. If there is a selected region that is not the full frame, crop the normalized original from step 2 (not the step 4 JPEG), then resize per step 4, then send that JPEG. Header `Accept: application/items.complete+json` for product fields; without it you get the minimal `{ sku, score }` format. Header `X-Api-Key`.
7. **Text.** If the user typed a query, append multipart field `text` (or JSON field `text`). Keep it under 512 characters. Do not put `text` in the query string; a query-string `text` returns `results: []`. Exception: with an existing text search engine that handles the keyword, do not send `text` to nyris (see [Existing text search](#existing-text-search)).
8. **Render.** Read `results[]`. Identity is `sku`, else `key`, else `objectID`. Show a skeleton list while the request is in flight. On a new search, clear the previous list. HTTP 200 with an empty list is an empty state, not an error. Keep the order of `results[]`; it is the visual similarity order. **Do not show `score` (or a percentage) on result cards.** It is an internal ranking signal and misleads users. If the site already has a text search results page, render there; see [Existing text search](#existing-text-search).
9. **Stale responses.** If the user searches again, ignore or abort the older request.
10. **Session (optional).** If the Find response has `session`, send it as header `X-Session` (or query `?session=`) on follow-up searches, for example a crop search after a full-frame search. Only send ids the API returned; an invented value returns HTTP 400.
11. **Feedback (optional).** `POST /feedback/v1` with `Content-Type: application/json`. Required: `request_id` (Find `id`), `timestamp` (ISO 8601 UTC), `event` (`click`, `conversion`, `feedback`, or `region`). Optional: `session_id` (Find `session`) and an event-specific `data` object. See [Feedback event](#feedback-event). The ids are also in the response headers `X-Matching-Request` and `X-Session`, but browsers may not expose those headers, so read the body.

Without an existing text search engine, you may start Find (full frame) and Regions in parallel after step 4. When the user changes the box, run step 6 again with the crop. Cancel the in-flight Find first.

### Feedback event

```json
{
  "request_id": "<id from find>",
  "session_id": "<session from find>",
  "timestamp": "2026-09-29T12:00:00.000Z",
  "event": "click",
  "data": { "positions": [0], "product_ids": ["ABC-100"] }
}
```

`data` by event:

| `event` | `data` |
| --- | --- |
| `click` | `positions` (zero-based) and/or `product_ids` |
| `conversion` | optional: `subtype` (`add_to_cart`, `purchase`, `support_request`), `product_ids`, `positions`, `quantity`, `revenue` `{ amount, currency, order_id }` |
| `feedback` | `success` (boolean), optional `comment` |
| `region` | `rect` `{ x, y, w, h }`, each 0..1 |

A valid event returns HTTP 2xx.

## Resize formula

```text
aspect = width / height
targetArea = 1024 * 1024
outWidth = sqrt(targetArea * aspect)
outHeight = outWidth / aspect
```

Round to integers. Crop in normalized coordinates before this resize:

```text
sx = originalWidth * x1
sy = originalHeight * y1
sw = originalWidth * (x2 - x1)
sh = originalHeight * (y2 - y1)
```

`x1,y1` is top-left, `x2,y2` is bottom-right, each in 0..1.

Regions come back in pixels of the JPEG you posted:

```text
x1 = left / sentWidth
x2 = right / sentWidth
y1 = top / sentHeight
y2 = bottom / sentHeight
```

## Region overlay

Do not draw every detected rectangle at once. Overlapping boxes make the photo unreadable. A row of "Object 1 / Object 2" buttons under the image is useless: those labels have no spatial link to the photo.

Required UI:

1. Auto-select the highest-confidence region (full frame if none). Mark that object's center marker as active immediately, then search that crop. Do not wait for another click.
2. Place a small **dot at the center** of every detected box: `((x1+x2)/2, (y1+y2)/2)`.
3. Draw a bounding box **only** for the selected region (auto-selected, or the one the user last chose by clicking a dot or dragging a crop).
4. Dim the rest of the image with a semi-transparent black overlay outside that box so the selected object is the focus. If the selection is the full frame, do not dim.
5. Clicking a dot selects that object and re-runs Find with its crop. Dragging on the image may set a custom crop. Do not add object-index buttons.

## Existing text search

If the site already has a search results page backed by a text search engine (Algolia, Elasticsearch, OpenSearch, Typesense, ...), do not build a second results page. Full guide: [Combine with your text search](https://docs.nyris.io/#combine-with-your-text-search). Rules:

1. **One results page.** The query image becomes a removable chip in the search box, and the region preview sits on the same results page. Facets, filters, product cards, and pagination are the existing components.
2. **nyris order is final.** Visual search is exact search; the most similar product comes first. Do not let the engine re-rank: turn off merchandising rules, pinned or promoted items, personalization, dynamic re-ranking, and variant grouping for visual queries. Do not boost or sink by stock, margin, or popularity. Out-of-stock products stay visible in their position. The only sort is "Most similar"; hide the sort dropdown while an image is active. Never interleave recommendations or sponsored items.
3. **One search chain.** Regions, auto-select, then one Find with the crop and `X-Options: limit=100`. Do not run a parallel full-frame Find. Treat nyris plus the engine query as one search for stale-response handling.
4. **Join by SKU.** Query the engine with an empty text query, restricted to the returned SKUs, plus the user's facet filters and the page's fixed filters (market, visibility; `AND` them, never replace them). Fetch all of them in one page with facet counts. Then sort the records by nyris position in your own code. Engine order is never trusted. Paginate in the client.
5. **Facets narrow, never reorder.** Changing a facet re-queries the engine with the same SKUs and does not call nyris again. Changing the crop calls nyris again and clears facets.
6. **Missing products are dropped, not replaced.** A nyris SKU the engine does not return is removed and reported as a sync problem. Never backfill with text results or bestsellers.
7. **A keyword narrows.** Pick one approach per site: use the keyword as the engine query inside the SKU restriction (and do not send `text` to nyris), or send it to nyris as `text`. Either way keep the nyris order. Do not merge a separate full-catalog text search.
8. **Feedback positions** are the zero-based index in the nyris `results` list, not the filtered page position. Always include `product_ids`.
9. **Engine failure** shows an error with retry. Show nyris products directly only if the shop has no per-market or per-customer visibility rules.

## Demo styling

For a demo or test site, match the layout and density of [test.nyris.app](https://test.nyris.app/): white header, light gray page (`#fafafa`), Source Sans 3, compact drop zone. Palette is white, black, and one or two neutrals (borders, muted text). Buttons may be black-on-white or the inverse. Do not use decorative colored page backgrounds.

## Find request

```http
POST /find/v1.1 HTTP/1.1
Host: api.nyris.io
X-Api-Key: $NYRIS_API_KEY
Accept: application/items.complete+json
Content-Type: multipart/form-data; boundary=----nyris

------nyris
Content-Disposition: form-data; name="image"; filename="query.jpg"
Content-Type: image/jpeg

<jpeg bytes>
------nyris
Content-Disposition: form-data; name="text"

optional query
------nyris--
```

Filters, when used (AND across keys, OR across values):

```text
filters[0].filterType = category
filters[0].filterValues[0] = pumps
```

Do not set `Content-Type: multipart/form-data` manually if the HTTP client generates the boundary (browser `FormData`, OkHttp `MultipartBody`). Set it only when you build the body yourself (curl, raw sockets).

### Same request as JSON (base64)

```http
POST /find/v1.1 HTTP/1.1
Host: api.nyris.io
X-Api-Key: $NYRIS_API_KEY
Accept: application/items.complete+json
Content-Type: application/json

{
  "image": "/9j/4AAQSkZJRg...",
  "text": "optional query",
  "filters": [{ "key": "category", "values": ["pumps"] }]
}
```

Encode the **resized** JPEG bytes, not the original file. Omit `text` and `filters` when unused.

### Same request as binary (image only)

```http
POST /find/v1.1 HTTP/1.1
Host: api.nyris.io
X-Api-Key: $NYRIS_API_KEY
Accept: application/items.complete+json
Content-Type: image/jpeg

<jpeg bytes>
```

### Example success body (`items.complete`)

```json
{
  "results": [
    {
      "sku": "ABC-100",
      "score": 0.91,
      "title": "Example",
      "images": ["https://img.nyris.io/..."]
    }
  ],
  "id": "request-id",
  "session": "session-id"
}
```

Product fields besides `sku` and `score` depend on the index. Do not require a field the index does not have. For a thumbnail, use `image` if present, else `images[0]`. `session` may be missing; treat it as optional.

## Regions request

```http
POST /find/v2/regions HTTP/1.1
X-Api-Key: $NYRIS_API_KEY
Content-Type: image/jpeg

<jpeg bytes>
```

JSON alternative: `Content-Type: application/json` with body `{"image": "<base64>"}`.

```json
{
  "regions": [
    {
      "confidence": 0.9,
      "region": { "left": 10, "top": 20, "right": 200, "bottom": 300 }
    }
  ]
}
```

Coordinates are floats. Ignore fields not shown here.

## Errors

| Status | Action |
| --- | --- |
| 200, empty `results` | Empty state. Do not retry. |
| 400 | Malformed request (bad JSON or base64, invented `X-Session`). Do not retry; fix the request. |
| 401 | Key missing or wrong type. Stop. |
| 413 | Image over 2 MB. Lower quality or size, retry once. |
| 415 | Unsupported format reached the API (for example HEIC or TIFF). Fix step 3 so everything is converted to JPEG. Do not retry. |
| 429, 5xx, network | Retry at most twice with backoff, unless a newer search replaced this one. |

If the key is missing locally, do not call the API. Show a configuration error.

## Language notes

### TypeScript (browser)

```typescript
const body = new FormData();
body.append("image", jpegBlob, "query.jpg");
if (text) body.append("text", text);

const response = await fetch(`${baseUrl}/find/v1.1`, {
  method: "POST",
  headers: {
    "X-Api-Key": apiKey,
    Accept: "application/items.complete+json",
  },
  body,
  signal,
});
```

JSON variant, Node.js only (`Buffer` does not exist in browsers or most edge runtimes):

```typescript
const image = Buffer.from(jpegBytes).toString("base64");
const response = await fetch(`${baseUrl}/find/v1.1`, {
  method: "POST",
  headers: {
    "X-Api-Key": apiKey,
    Accept: "application/items.complete+json",
    "Content-Type": "application/json",
  },
  body: JSON.stringify(text ? { image, text } : { image }),
});
```

In the browser, `FileReader.readAsDataURL(jpegBlob)` gives a data URI that you can send as `image` unchanged.

### Python

Build the multipart body yourself or use `requests` / `httpx` `files={"image": ("query.jpg", jpeg_bytes, "image/jpeg")}` and `data={"text": text}`. Header `X-Api-Key`. Do not pass `text` as a URL query parameter.

JSON variant:

```python
payload = {"image": base64.b64encode(jpeg_bytes).decode("ascii")}
if text:
    payload["text"] = text
response = httpx.post(f"{base_url}/find/v1.1", json=payload, headers=headers)
```

### Swift

Draw `UIImage` into an opaque renderer filled white, `jpegData(compressionQuality: 0.9)`, upload with `URLSession` multipart part name `image`.

### Kotlin

`BitmapFactory` with `inSampleSize`, EXIF rotate, draw onto a white `Canvas`, `compress(JPEG, 90)`, OkHttp part name `image`.

## Edge cases

| Case | Required behavior |
| --- | --- |
| Image shorter than 100 px on one side | Search anyway. Show a non-blocking hint that small images give less accurate results. |
| Multi-page TIFF or PDF | Use page 1 only. |
| CMYK JPEG | Convert to RGB before encode if the decoder gives CMYK. |
| PNG with alpha | White background, then JPEG. |
| Rotated phone photo | Apply EXIF before crop math, or boxes will not match the preview. |
| User clears the image | Cancel the request and return to the start screen. |
| Text only | Multipart field `text` with no `image` part, or JSON `{"text": "..."}`. |
| Platform can only send JSON | Use the JSON base64 body. Resize first; base64 adds ~33%. |
| Base64 string has line breaks | Strip whitespace before sending. Some encoders wrap lines (`openssl base64` at 64 characters, MIME at 76). |
| Image plus keyword | Either send both fields, or search by image and filter titles locally. Do not do both unless the user asked. With an existing text search engine, see [Existing text search](#existing-text-search). |
| HEIC in Chrome or Firefox | Native decode fails. Convert with `heic-to` or libheif (load it lazily; it is large), then continue as a bitmap. |
| HEIC without extension or MIME type | Native decode fails and the name gives no hint. Try the HEIC decoder before rejecting the file. |
| Empty key | Configuration error, not an empty result list. |

## Acceptance checklist

All of these must pass before you claim the integration is done:

- [ ] Key is read from the environment or a config file that is gitignored.
- [ ] A text search uses multipart field `text` and returns HTTP 200.
- [ ] An image search sends a JPEG scaled with the area formula (about 1024×1024 pixels of area).
- [ ] If the JSON base64 body is used, it encodes the resized JPEG and returns HTTP 200.
- [ ] Transparent PNG becomes a JPEG on a white background.
- [ ] EXIF orientation is applied so a portrait phone photo stays portrait.
- [ ] JPEG, PNG, WebP, GIF, AVIF, and HEIC uploads all reach Find as JPEG (HEIC also in Chrome).
- [ ] Regions response boxes are stored as normalized 0..1. The preview shows a center dot per region and a rectangle only on the selected region, with the rest of the image dimmed. No object-index buttons.
- [ ] The auto-selected region's marker is active as soon as Regions returns, before Find responds.
- [ ] Result cards show no score or percentage, and appear in the order of `results[]`.
- [ ] Changing the box sends a new Find with the cropped JPEG.
- [ ] The crop is cut from the full-resolution original, then resized. A crop of a 4000 × 3000 photo is not cut from the 1024² JPEG or the preview.
- [ ] A second search does not show the first search’s products after the second response arrives.
- [ ] HTTP 200 with `results: []` renders an empty state.
- [ ] Missing API key does not call the network.
- [ ] 401 and 413 each show a distinct message.
- [ ] A source under 100 px still searches and shows the small-image hint.
- [ ] No Data API write calls were added.
- [ ] If an existing text search engine is used: visual results render on the existing results page; the rendered order equals the nyris order (tested); rules, personalization, re-ranking, and variant grouping are off for visual queries only; the page's fixed filters still apply; facets narrow without calling nyris; missing SKUs are dropped, never backfilled. Full list: [Combine with your text search](https://docs.nyris.io/#checklist).

## Test images

Use a photo of a product that is in the index behind the key. With a demo index key, ask the user which sample products it contains. If you have no photo, ask the user for one. Do not download random images from the web.

Expected: HTTP 200 from Find. The number of products depends on the index behind the key, so assert the status and that `results` is an array, not a specific SKU.

## Out of scope

- Creating or deleting index items.
- Tuning `X-Options` unless the user named a threshold or a limit, or you combine with a text search engine (then `limit=100`). Index defaults are already applied server-side.
- Reading response headers that echo resolved options. Do not depend on them. (`X-Matching-Request` and `X-Session` are fine to read.)