SWE for AI Orchestrators: Open the Workbench

You see someone give an AI a job, leave it running, and return to working software. Your own sessions involve more supervision: you ask for a change, try the result, describe the failure, and wait for another attempt.

Part of the difference is the setup around the model. When an agent can inspect files, run the program and read a useful test result, it can handle more of that cycle itself. The engineering work includes making those actions possible and giving them a clear purpose.

This lesson opens up that setup. You will run a small program that builds an index of Markdown notes, watch a check expose a fault, and give an AI agent the tools and instructions to repair it. Then you will inspect the change and hand the work to a fresh session.

The starter includes the code, sample notes and tests. You don’t need to write them yourself to begin.

Get the example running

You need an editor, Python 3, and a coding agent that can read and edit local files and run commands. This walkthrough uses VS Code as the editor and Claude Code as an example agent. You can use another coding agent with those capabilities.

An IDE, short for integrated development environment, brings editing, navigation and development tools into one application. VS Code becomes that kind of workbench through its built-in features and extensions. Its job here is to let you see the project and what happens to it. The model is a separate component.

If you need the software, use the official VS Code download, Python downloads and Claude Code setup guide. Complete the agent’s sign-in before starting; account access and model usage depend on your chosen provider.

Download the Agent Workbench Lab ZIP, extract it, and copy its starter folder somewhere you keep practice projects. You will find it inside swe-agent-workbench-lab. Its notes are fictional; the supplied build_index.original.txt preserves the starting code for comparison.

In VS Code, choose File → Open Folder and select starter. The Explorer panel shows:

starter/
  BRIEF.md               What the program should do
  README.md              How to run the exercise
  build_index.py         The program
  build_index.original.txt  Its untouched starting version
  test_index.py          Checks of its behaviour
  notes/
    welcome.md
    research/
      loops.md
      memory.md

A .py file is a Python program. A .md file is a plain-text document that uses Markdown formatting, such as # for a heading. Open BRIEF.md and a sample note by clicking them in Explorer.

Choose View → Terminal. This panel accepts commands and displays their results; a new terminal normally starts in the folder you opened. VS Code’s terminal guide explains the interface.

Run this command by typing it and pressing Enter:

python3 --version

On Windows, use python --version; if that is the command that prints your Python version, use python wherever this lesson says python3. If neither command prints a Python 3 version, finish the Python setup before proceeding.

Now run:

python3 build_index.py

python3 selects the program that executes Python code; build_index.py tells it which file to execute. The output should be:

Wrote INDEX.md

Open the new INDEX.md file. It links to Welcome, although the research folder contains other notes. The command completed successfully, but the result is incomplete. If Python instead reports that it cannot open build_index.py, check that the folder you opened directly contains that file, then open a fresh terminal there.

Give the agent something it can check

Run:

python3 -m unittest -v

This invokes Python’s built-in testing tool. A test runs a small example and compares the result with an expected answer. Here, the checks cover links, nested folders, ordering, an empty folder and preserving the original notes. The -v option prints the individual results. Python’s documentation covers the command.

The starter’s output includes the following (abridged):

test_nested_note_is_linked ... FAIL
...
FAILED (failures=1)

The other checks should pass. The failure says a note inside a subfolder is missing from the generated index, which matches what you just observed.

This creates feedback the agent can use directly. It can run the same command, read the failure, inspect the implementation, make an edit, and run the check again.

Start the agent inside the project

In the same terminal, start your coding agent. For Claude Code, the command is:

claude

The terminal now contains the agent’s conversation interface. If you want to run an ordinary command yourself while it is open, create another terminal using Terminal → New Terminal.

Give the agent this prompt:

Read BRIEF.md and inspect this project. Run the tests before changing anything. Repair the index builder so it includes notes in subfolders. Keep the tests, build_index.original.txt and sample notes unchanged. Run all the tests again and regenerate INDEX.md. If it still fails after three repair attempts, stop and explain what remains unresolved. Finish by telling me what changed and which checks you actually ran.

If your agent requests permission, read the proposed action. For this exercise it needs to read and edit the practice project and run Python there. The request to stop after three attempts is an instruction to the agent; an enforced limit would need a control in the software running it.

As it works, find the entries for reading files, editing build_index.py and running the test command. In Claude Code, Ctrl+O opens the transcript viewer, where you can inspect detailed tool usage. An agent may group these actions differently or ask for confirmation. You are looking for the actual actions and results, rather than expecting a particular conversation transcript.

What is doing what?

The components you have just used have distinct jobs:

Component	Its job in this exercise
Model	Interprets the request and tool results, then proposes actions or an answer.
Agent harness	Runs the surrounding process: supplies context, executes permitted tool calls, and returns their results to the model. Claude Code provides this layer here.
Tools	Read files, apply edits and run the test command.
Workspace	Holds the code, notes and brief that the agent is working with.
Editor	Lets you inspect those files and the resulting changes.

An agent loop is the repeated exchange between the model and its tools. A typical pass in this exercise is:

The model requests a test run
              ↓
The harness runs the command
              ↓
The failed check returns as a tool result
              ↓
The model requests an inspection or edit
              ↓
The harness performs it and returns the result
              ↓
The model can request another test run

The test run supplies evidence that can change the next action. This is why enabling the agent to run checks matters: you no longer have to relay every error yourself. Kiro’s fast feedback loop guide makes this an explicit part of preparing a project for agents.

Inspect the repair

When the agent finishes, open INDEX.md. The supplied sample should now produce:

# Notes index

- [Agent loops](notes/research/loops.md)
- [Persistent memory](notes/research/memory.md)
- [Welcome](notes/welcome.md)

Run the tests yourself in the ordinary terminal. The summary should end in OK, with the nested-note check now passing.

You can also inspect the edit. In VS Code’s Explorer panel, right-click build_index.original.txt and choose Select for Compare, then right-click build_index.py and choose Compare with Selected. These are the documented file-comparison controls. The editor shows a diff, a comparison highlighting removed and added text. Your coding agent may also offer its own change-review panel.

For the reference solution supplied with this lesson, the change is:

-    for path in notes.glob("*.md"):
+    for path in notes.rglob("*.md"):

The original expression searches the immediate folder. The replacement also searches subfolders. You don’t need to memorise those names, but understanding that difference lets you connect the reported fault, the edit and the result.

Your agent might produce another valid repair. Ask it to explain each change, and investigate if it changed the tests or removed sample notes instead of repairing the program.

The reference run was checked at the command line: the starter produced the nested-note failure, the repaired version passed the checks, and its generated links resolved to the sample files. These checks establish those behaviours; they don’t establish that every possible note format is supported.

Give the next session something to remember

Ask the agent:

Write HANDOVER.md with the purpose of this project, the change you made, the commands you ran and their results, and anything still unresolved. Keep it short. Do not invent another task.

Open the file and inspect it. A saved handover is useful because another session can read it; its claims still need to match the work. Close the existing agent session using its exit control (for Claude Code, /exit), then start a new one from the same project folder. Ask:

Read BRIEF.md and HANDOVER.md, inspect build_index.py, and rerun the tests. Tell me what this project does and whether the recorded state matches the current files. Do not change anything.

This is a simple form of persistent memory. The information survives in a file, and you explicitly bring it into a later session’s context. Saving the file does not train the model or guarantee that a future session reads it. Other agent systems automate parts of this process; Hermes, for example, documents saved memory, reusable skills and conversation search. Hermes memory documentation

Where graphs enter

You can draw the development workflow as connected steps:

Read brief → Run checks
                 ├─ Fail → Repair → Run checks again
                 └─ Pass → Review changes → Save handover

This is a workflow graph: the connections describe possible execution paths. In this exercise, the diagram explains a workflow carried out by the existing agent. It isn’t an additional program controlling the agent.

Frameworks such as LangGraph let you implement connected steps, shared state and controlled transitions in software. That becomes a separate practical lesson when we want to enforce a particular sequence, route work to different agents or pause for review.

An Obsidian graph has a different purpose: its points represent notes and its connections represent links between them. It shows relationships in information. Calling both things “graphs” doesn’t make them interchangeable.

Try a change of your own

Add a Markdown note in another subfolder under notes/, give it a heading, and rebuild the index. Check whether the new title appears and its link opens the correct file. This tries an example beyond the supplied fixtures.

Then give the agent a small improvement you care about, such as adding a note count beneath the index heading. Ask it to add a check for the requested behaviour, make the change, run the checks and update the handover. Watch how the files, tool calls and results connect.

The next useful step is Git: a system that records versions of the project, so comparing and recovering changes becomes easier than keeping separate copies. From there, the same project can demonstrate an independent reviewer, controlled workflows and background jobs. Each addition should solve a problem you can now recognise in the working example.