tldraw agent

This starter kit demonstrates how to build an agent that can manipulate the tldraw canvas.

A chat panel on the right side of the screen lets users communicate with the agent, add context, and see chat history.

Environment setup

Create a .dev.vars file in the root directory and add API keys for any model providers you want to use.

ANTHROPIC_API_KEY=your_anthropic_api_key_here
GOOGLE_API_KEY=your_google_api_key_here
OPENAI_API_KEY=your_openai_api_key_here

We recommend using Anthropic for best results. Get your API key from the Anthropic dashboard.

Local development

Install dependencies with yarn or npm install.

Run the development server with yarn dev or npm run dev.

Open http://localhost:5173/ in your browser to see the app.

Agent overview

With its default configuration, the agent can perform the following actions:

• Create, update and delete shapes.
• Draw freehand pen strokes.
• Use higher-level operations on multiple shapes at once: Rotate, resize, align, distribute, stack and reorder shapes.
• Write out its thinking and send messages to the user.
• Keep track of its task by writing and updating a todo list.
• Move its viewport to look at different parts of the canvas.
• Count shapes matching a given expression.
• Schedule further work and reviews to be carried out in follow-up requests.
• Call example external APIs: Looking up country information.

To make decisions on what to do, we send the agent information from various sources:

• The user's message.
• The user's current selection of shapes.
• What the user can currently see on their screen.
• Any additional context that the user has provided, such as specific shapes or a particular position or area on the canvas.
• Actions the user has recently taken.
• A screenshot of the agent's current view of the canvas.
• A simplified format of all shapes within the agent's viewport.
• Information on clusters of shapes outside the agent's viewport.
• The history of the current session, including the user's messages and all the agent's actions.
• Lints identifying potential issues with shapes on the canvas.

Use the agent programmatically

Aside from using the chat panel UI, you can also prompt the agent programmatically.

The simplest way is to call the prompt() method to start an agentic loop. The agent continues until it finishes the task.

// Inside a component wrapped by TldrawAgentAppProvider
const agent = useAgent()
agent.prompt('Draw a cat')

You can specify further details about the request as an AgentInput object:

agent.prompt({
	message: 'Draw a cat in this area',
	bounds: { x: 0, y: 0, w: 300, h: 400 },
})

The TldrawAgent class has additional methods:

• agent.cancel() - Cancel the agent's current task.
• agent.reset() - Reset the agent's chat and memory.
• agent.request(input) - Send a single request to the agent and handle its response without entering into an agentic loop.

Architecture overview

The agent starter is organized into three main areas:

• client/ - React components, agent logic, and utils that run in the browser
• worker/ - Cloudflare Worker that handles model requests and prompt building
• shared/ - Types, schemas, and utilities shared between client and worker

Customize the agent

The agent's behavior is defined in client/modes/AgentModeDefinitions.ts. The AGENT_MODE_DEFINITIONS array contains mode definitions. Each mode has two arrays:

• parts determine what the agent can see.
• actions determine what the agent can do.

Add, edit or remove an entry in either array to change what the agent can see or do in a given mode.

Mode system

The agent uses a mode system to control what parts and actions it has access to at any given time. Modes are defined in client/modes/AgentModeDefinitions.ts.

The default working mode includes all standard capabilities. You can create additional modes with different subsets of parts and actions.

Modes can be transitioned between over the course of a prompt depending on the behavior you desire. Call agent.mode.setMode(modeType) to change modes. To control the lifecycles of different modes, you can optionally implement any desired mode lifecycle hooks in client/modes/AgentModeChart.ts. You have access to:

• onEnter(agent, fromMode) - runs when you enter a mode
• onExit(agent, toMode) - runs when you exit a mode
• onPromptStart(agent, request) - runs when a prompt commences, either because a user has prompted it or because it has entered another step in its agentic loop
• onPromptEnd(agent, request) - runs when a prompt ends
• onPromptCancel(agent, request) - runs when a prompt is canceled

Change what the agent can see

Change what the agent can see by adding, editing or removing a prompt part.

Prompt parts assemble and build the prompt that we give to the model, with each util adding a different piece of information. This includes the user's message, the model name, the system prompt, chat history and more.

This example shows how to let the model see what the current time is.

First, define a prompt part type in shared/schema/PromptPartDefinitions.ts:

export interface TimePart extends BasePromptPart<'time'> {
	time: string
}

Next, create a prompt part util in client/parts/:

export const TimePartUtil = registerPromptPartUtil(
	class TimePartUtil extends PromptPartUtil<TimePart> {
		static override type = 'time' as const

		override getPart(): TimePart {
			return {
				type: 'time',
				time: new Date().toLocaleTimeString(),
			}
		}
	}
)

The getPart method gather any data needed to construct the prompt. It can take (request: AgentRequest, helpers: AgentHelpers) parameters for access to the current request and helper methods.

Then, back in shared/schema/PromptPartDefinition.ts, create the definition for that prompt part.

export const TimePartDefinition: PromptPartDefinition<TimePart> = {
	type: 'time',
	priority: -100,
	buildContent({ time }: TimePart) {
		return [`The user's current time is: ${time}`]
	},
}

The prompt part definition is used by the worker to turn prompt parts into messages sent to the model. Override priority to control what order the part should be added in the messages. Override buildContent to control how the data is turned into a message for the model.

There are other methods available on the PromptPartDefinition interface that you can override for more granular control.

• getModelName - Determine which AI model to use.
• buildMessages - Manually override how prompt messages are constructed from the prompt part.

Enable the prompt part

To enable the prompt part, import its util in client/modes/AgentModeDefinitions.ts and add its type to a mode's parts array. It's important to make sure you import it here and use its type field, instead of using the type string literal. This is to ensure the util properly self-registers.

import { TimePartUtil } from '../parts/TimePartUtil'

// Then in the mode definition:
parts: [
	// ... other parts
	TimePartUtil.type,
]

Change what the agent can do

Change what the agent can do by adding, editing or removing an agent action.

Agent action utils define the actions the agent can perform. Each AgentActionUtil adds a different capability.

This example shows how to allow the agent to clear the screen.

First, define an agent action schema in shared/schema/AgentActionSchemas.ts:

export const ClearAction = z
	// All agent actions must have a _type field
	// The underscore encourages the model to put this field first
	.object({
		_type: z.literal('clear'),
	})
	// A title and description tell the model what the action does
	.meta({
		title: 'Clear',
		description: 'The agent deletes all shapes on the canvas.',
	})

// Infer the action's type
export type ClearAction = z.infer<typeof ClearAction>

Then, create an agent action util in client/actions/:

export const ClearActionUtil = registerActionUtil(
	class ClearActionUtil extends AgentActionUtil<ClearAction> {
		static override type = 'clear' as const

		override applyAction(action: Streaming<ClearAction>) {
			// Don't do anything until the action has finished streaming
			if (!action.complete) return

			// Delete all shapes on the page
			const { editor } = this
			const shapes = editor.getCurrentPageShapes()
			editor.deleteShapes(shapes)
		}
	}
)

The applyAction method executes the action. It can take a second helpers: AgentHelpers parameter for access to helper methods.

Override these methods on AgentActionUtil for more control:

• getInfo - Determine how the action gets displayed in the chat panel UI.
• savesToHistory - Control whether actions get saved to chat history or not.
• sanitizeAction - Sanitize the action before saving it to history and applying it. More details on sanitization below.

Enable the agent action part

To enable the agent action, import its util in client/modes/AgentModeDefinitions.ts and add its type to a mode's actions array.

import { ClearActionUtil } from '../actions/ClearActionUtil'

// Then in the mode definition:
actions: [
	// ... other actions
	ClearActionUtil.type,
]

Change how actions appear in chat history

Configure the icon and description of an action in the chat panel using the getInfo() method.

override getInfo() {
	return {
		icon: 'trash' as const,
		description: 'Cleared the canvas',
	}
}

You can make an action collapsible by adding a summary property.

override getInfo() {
	return {
		summary: 'Cleared the canvas',
		description: 'After much consideration, the agent decided to clear the canvas',
	}
}

To customize an action's appearance via CSS, you can define style for the `agent-action