For AI

Every command has a JSON form: its name and its input. commands() lists them for a model, and run takes a model’s call back to the editor. The examples call the handle editor. See Get the handle.

run()

run: (input: { command: string; input?: unknown }) => Promise<unknown>;

Runs any command by its name, group.verb, with its input. run({ command: 'menu.select', input: { item: 'FILTER' } }) is menu.select({ item: 'FILTER' }): the same rules, the same Promise, the same errors.

string

required

The command’s name, as commands() lists it: 'history.undo', 'menu.select', 'read.state'. A name the handle doesn’t have rejects with INVALID_COMMAND, and props is ['command'].

unknown

The command’s input, checked against its schema. A command that takes nothing takes no input, or {}.

A read through run waits its turn in the queue, after the commands called before it, and resolves to its value.

run.ts

await editor.run({ command: 'menu.select', input: { item: 'FILTER' } });

const state = await editor.run({ command: 'read.state' });

commands()

commands: () => EmbedEditorCommand[];

type EmbedEditorCommand = { command: string; description: string; input: Record<string, unknown> };

Lists every command run takes in this edition:

string

Its name, run’s command.

string

What it does, written for a model to read: what it does, as which control, when it rejects, and an example call.

Record<string, unknown>

Its input as a JSON Schema (draft 2020-12, with no $schema key, which some model providers reject), always an object: one with no properties for a command that takes nothing.

commands() lists menu.select as:

commands.json

{
	"command": "menu.select",
	"description": "Shows a menu item, as a tap on it in the carousel: its tool opens on the stage (ADJUST the crop, FILTER the filters, an adjustment its ruler…). The item already shown stays as it is. Rejects NOT_AVAILABLE for an item the host turned off or the edition doesn't have, and while Annotate is open (its carousel is not on screen then: annotate.close first). Example: run({ command: 'menu.select', input: { item: 'FILTER' } }).",
	"input": {
		"type": "object",
		"properties": {
			"item": {
				"type": "string",
				"enum": ["ADJUST", "RESIZE", "OBFUSCATE", "FRAME", "WATERMARK", "SIGNATURE", "FILTER", "AUTO", "EXPOSURE", "BRILLIANCE", "HIGHLIGHTS", "SHADOWS", "CONTRAST", "BRIGHTNESS", "BLACK_POINT", "SATURATION", "VIBRANCE", "WARMTH", "SHARPNESS", "VIGNETTE"]
			}
		},
		"required": ["item"],
		"additionalProperties": false
	}
}

The list holds only what this edition runs: in the Free and the Lite, the Full’s commands are left out.

Give a Model the Editor

Map commands() to the model’s tools, and run each tool call it makes with run. This example is for the Claude API’s tool use. For another model’s API, map the same list to its own tool format.

tools.ts

import { EmbedEditorError } from '@embededitor/full/react';

// the editor's commands as the Claude API's tools, less Save, the export and the screenshot (they answer with a File,
// which doesn't travel as JSON) and Close
const tools = editor
	.commands()
	.filter(({ command }) => {
		return !['editor.close', 'output.exportFile', 'output.save', 'read.screenshot'].includes(command);
	})
	.map(({ command, description, input }) => {
		// a tool's name takes no dot
		return { description, input_schema: input, name: command.replace('.', '_') };
	});

// a tool_use block of the model's reply, run on the editor: the tool_result to send back
const runTool = async ({ id, input, name }: { id: string; input: unknown; name: string }) => {
	try {
		const res = await editor.run({ command: name.replace('_', '.'), input });

		return { content: JSON.stringify(res), tool_use_id: id, type: 'tool_result' };
	} catch (err) {
		if (!(err instanceof EmbedEditorError)) {
			throw err;
		}

		return { content: JSON.stringify({ code: err.code, props: err.props }), is_error: true, tool_use_id: id, type: 'tool_result' };
	}
};

Send tools with the request. For each tool_use block in the model’s reply, send await runTool(block) back in the next user message.

An error tells the model what to fix. INVALID_COMMAND names the input’s paths, and each description ends with an example call.
Calls run in order. The commands queue, so the calls of one reply run in the order you pass them to runTool, each after the image is on screen and the user’s hand is off.
Values are the saved state’s. read.state and read.size tell the model what the image holds and in which px, in the units every command takes.
Leave out what the model shouldn’t call. The example filters commands() before it maps it: output.save, output.exportFile and read.screenshot resolve to a File, which doesn’t travel as JSON, and editor.close closes the editor. Filter anything else the same way.
Show the model the photo as an image. Call read.screenshot() yourself and send the PNG it gives, in base64, as an image block in the next user message.

Overview

Entry Points

Commands

Reference

run()

commands()

Give a Model the Editor

​run()

​commands()

​Give a Model the Editor

run()

commands()

Give a Model the Editor