The Agent Harness Behind Roxxy, an AI Teaching Assistant

Jingjing Ren, PhD and Hugo Xiong

Schools are right to be wary of AI that produces plausible content with no clear gains. Roxxy, Roxxem's AI teaching assistant for language classrooms, is built to avoid both. It does not write lessons from scratch: it assembles them from a curated library of popular culture videos and games, matched to calibrated ability estimates for each class. Teachers get a playable, standards-aligned lesson in minutes with confidence.

Under the hood, Roxxy uses the same kind of frontier model as most general AI apps. The difference is the harness, the software that runs the model as an agent: what goes into its context, which tools it can call, how each call is checked and what it is told to do (Schmid, 2026). This post shows how we build, secure and iterate on that harness, for anyone pairing a language model with educational content or learner data. It covers three layers:

The agent runtime: an agent loop, per-turn context assembly, a single gate on every tool call, a prompt and toolset per mode, and hard resource limits.
Context: calibrated ability estimates and interest profiles for each class and student, a curated content library paired with learning games and tasks, and the teacher's own preferences.
Pedagogical instructions: rules for sequencing lessons, drawn from learning science, and for how Roxxy leverages insights from data and teachers.

Together, the layers run a formative assessment cycle: elicit evidence of what students can do, interpret it, and act on it in the next lesson (Black & Wiliam, 2009).

Inside the harness

Figure 1 shows the harness around Roxxy's agent loop. Roxxy is an agent in the standard sense: it selects a tool, observes the result and chooses the next action, in a loop. Each turn, the harness rebuilds the context from the active mode, the pedagogical instructions and the teacher's profile. Every tool call then passes through one server-side gate. The almost 50 tools are thin adapters over the shared services the web pages also call. Evaluation reads traces from the gate and feeds changes back into the prompt and tools. The rest of this section follows the figure.

Figure 1. The harness around Roxxy's agent loop.

Modes. Work is partitioned into modes: general help, lesson building and Pulse, which interprets class analytics. Two further modes, for building units and editing videos, are internal to the content team. Each mode has its own instructions and toolset; a smaller action space improves tool-selection accuracy. When a request spans modes, Roxxy switches mid-reply, and the new mode's tools are available within the same turn.

Per-turn context construction. The system prompt is rebuilt each turn from the active mode's block and only the context that mode consumes: the current lesson draft, the teacher's page, and/or their profile. History is budgeted. The last N turns are replayed verbatim; older tool results are kept in full, summarized or reduced to a one-line citation, depending on the tool, and the oldest turns are evicted once the budget is exceeded. We cap a single tool result to a tunable N_MAX characters. Long lesson sessions therefore stay conditioned on the current draft rather than on stale tool output.

A single gate on every tool call. Every call passes through the same server-side sequence: resolve the tool, reject it if disabled, check the teacher's permissions, validate arguments against a schema, snapshot before writes, execute, truncate the result and trace it. The model is free to request any call; the gate decides whether it runs.

Critical workflows get determinism from tools. At the top level, the model still decides the best tools and context to accomplish teachers' goals: needs a class lookup, a search or a lesson build. Within a tool, the steps form a deterministic pipeline: a lesson build selects a video, checks match confidence, binds games to line ranges, builds the vocabulary and quiz, and assembles the lesson. On failure, it restores the previous version. The model need not track the sequence.

Typed results steer the loop. Tools return a status, not only data. For example, A lesson build returns ready, needsContentChoice (no confident video match; ask the teacher), noContentFound (report it) or incomplete with the missing items. The next action is conditioned on structured state rather than on the model's reading of unstructured and error-prone input. Each turn is limited to a fixed number of tool calls.

One agent, plus a verifier. Following KISS, Roxxy has no sub-agents. One agent keeps the whole conversation, which matters because a teacher's intent varies and shifts mid-session, and modes isolate instructions and tools. We added a single extra piece to raise task quality and completion rate: a verifier tool. Roxxy passes it the goal and its solution, the verifier scores them in a fresh context free of the agent's reasoning and shortcuts, and Roxxy fixes what falls short.

Context: class data, content and teacher memory

With the agent loop and gate in place, Roxxy needs something to reason over. We draw from three main sources: the class, measured by Pulse; the curated content library; and the teacher, through their profile and preferences.

The class, measured

Roxxy reads class and student profiles from Pulse as structured data; it does not estimate ability itself. Levels are calibrated estimates from practice and placement tests. For each skill, Pulse returns:

the level estimate and its confidence
the evidence base: the number of graded days and the most recent one
the 30-day change, and whether it is stale
whether the skill is measured at all

The content library, curated

Each video comes with its teaching material attached: line-by-line timing, target vocabulary, a difficulty level, a maturity rating, and the games and tasks built on it. Roxxy searches for content that serves the teacher's goal, weighing the class's level, skill gap, interests and permitted maturity levels to find the right video. Every suggestion comes from a tool result and carries its provenance, so Roxxy can state whether it targets a skill gap or an interest.

The teacher

Pulse learns each teacher's style from their activity: what they assign and build, and for which classes. Roxxy adds what the teacher tells it. It stores favored games, typical difficulty and lesson length, and topics their students enjoy, and saves notes when asked ("remember this"). The preferences pre-fill the next lesson. At session end, notes are merged into one profile per teacher, which the teacher can view, edit and delete. Memory holds facts about the teacher, not student data. There is no vector database: at tens of notes per teacher, a compact profile in the prompt is simpler and more reliable than retrieval.

Pedagogical instructions

Roxxem is built on learning science and around the teacher. Context supplies the facts; pedagogy decides how they are used. Together they close a formative assessment loop: pedagogy guides how evidence is interpreted, Roxxy acts on it in the upcoming lesson, and that lesson's games and quiz become new evidence. We enforce it at three levels: in models (Pulse), in workflows (the lesson builder) and in the prompt.

Learning science sets the lesson structure. Retrieval practice yields better long-term retention than restudy (Roediger & Karpicke, 2006), so a lesson alternates a teaching slide with a game on the same material and ends with a quiz. The AI workflow enforces the structure. Each game follows the video segment it draws on, with vocabulary from that video, so learners meet forms in context before practising them. Content is ranked on predicted engagement and level fit, targeting input slightly above the class's level (Krashen, 1985).

Roxxy quantifies the score with its uncertainty. Instead of inventing a diagnosis ungrounded, it uses Pulse's calibrated levels, each with its confidence. A confident, recent score supports a direct recommendation. A low-confidence, thin or stale score gets a stated caveat, and an unmeasured skill stays "not yet measured," not a zero.

The teacher decides. Roxxy reads the teacher's notes and the class's goals before any metric, and a recommendation that ignores stated goals is treated as worse than none. Lessons are drafts, and nothing is final until the teacher marks it ready.

Building playable lessons

Lesson building is where context and pedagogy combine. Lesson mode targets a playable lesson: authentic culture videos with interactive games and tasks attached to them. In a single build, Roxxy:

selects a video and scopes each game to a range of its lines
adds the vocabulary as dictionary entries any teacher can search
writes the comprehension questions
fits the lesson to its target length and level, with difficulty on a 1–10 scale mapped to a CEFR band

The model proposes; code and a verifier check. Post-processing enforces structural invariants: a video slide precedes every video game, the quiz comes last and step durations sum to the lesson length. A well-formed lesson can still miss the point, so Roxxy then calls the verifier with the teacher's goal and the draft. The verifier scores, in a fresh context, whether the lesson serves that goal, and Roxxy revises what falls short until it meets the bar for teacher review.

Roxxy's lesson build drawn as a factory. Context (class level from Pulse, class interests, the lesson form's length and level, teacher preferences) and the teacher's request ("A 20-minute A2 lesson on this song, please") enter a factory labeled "lesson builder, one tool call". A conveyor carries the draft through six stations, each marked done: finding the right video, choosing the vocabulary, objectives and slides, laying out the games, writing quiz questions, and cover image and final touches; the draft gains a piece at each station. If the build cannot proceed, Roxxy asks rather than guesses: with no confident video match the teacher picks one, and with no content Roxxy says so. Code checks the draft: a video slide precedes every video game, the quiz comes last, step durations sum to the lesson length, and on failure the previous version is restored. Below the factory, a separate verifier tool receives the teacher's goal and the draft in a fresh context and loops back a score, so Roxxy can revise. The draft (8 slides and games) goes to the teacher, who edits it in chat ("swap the video", "drop that game", "simpler questions, please") and then marks it ready.

Figure 2. One tool call builds the full draft from the class context and the teacher's request. Code enforces the lesson's structure, a verifier checks the draft against the teacher's goal, and the teacher edits the draft in chat before marking it ready.

Build first, then edit

Roxxy originally built lessons through five approval checkpoints: goal, objectives, video and words, structure, quiz. Funnel analysis of all September conversations quantified the cost. Only 26.5% of lesson sessions reached a ready lesson, and the largest drop-off occurred at the first question, often one the lesson form had already answered. Teachers who abandoned wanted fewer questions; those who completed wanted control over the result.

Roxxy now generates the full lesson as a draft in one request, which the teacher then revises: "swap the video", "drop that game", "make the questions simpler". It asks only when the build cannot proceed: with no confident video match, it presents the candidates; with no content at all, it says so. Nothing is final until the teacher marks it ready. Control shifted from approving each step in advance to editing a draft afterwards.

Safety and responsible use

The harness enforces safety because the model may take shortcuts, hallucinate or even jailbreak. We continually assess risks against the OWASP Top 10 for LLM Applications (2025) and the NIST AI Risk Management Framework (2023), and add the risks only a classroom has, such as age-inappropriate content. The major controls that put this into practice:

Server-side access control. Permissions are derived on the server each turn from the account's plan, type and role. Following least privilege, the model sees only the tools the teacher may use, and each call is re-authorized. The prompt holds no secrets or access rules.
Same code path as the UI. Each tool calls the same service as the corresponding page action, with identical authorization and validation. Roxxy's capabilities are therefore a subset of the teacher's, and a fix to a service applies to both.
Reversible writes. Each tool declares its effect class: read, write or irreversible. Writes are versioned, so version history serves as undo. Irreversible actions (publishing, deleting, assigning work) are not exposed to the model, and a CI test fails if one is enabled.
Untrusted inputs. Instructions in uploaded documents are treated as data, not commands, and every ID Roxxy acts on must originate in an earlier tool result. Because tools are bounded by the teacher's permissions, even a successful prompt injection cannot escalate beyond them.
Scope and limits. Roxxy serves teachers only and declines political, medical, legal and financial questions among other inappropriate requests. Tool rounds per turn, messages per minute and spend per day are capped, and every model call is traced and available for later audits.

Against the Learning Agency guidelines

The Learning Agency's Guidelines for the Safe and Responsible Use of AI in Ed Tech (2026) extend beyond security. Roxxy addresses the remaining principles as follows:

Educational outcomes. Lessons are playable and tied to measured levels, and the teacher reviews every draft. Planned: estimating whether levels rise over months.
Learning and wellbeing. Content carries maturity ratings, filtered by what each class allows, and a human rating overrides the model's.
Privacy. Memory holds facts about the teacher, not student data, and model providers do not train on our data.
Transparency and explainability. Every level is cited with its evidence, suggestions carry their provenance, and every reply is traced to a versioned prompt.
User control. Lessons stay drafts until the teacher marks them ready, and teachers can view, edit and delete Roxxy's memory.

Iterating on the harness

Blocking unsafe actions is not enough, we need a way to detect wrong ones. We increase our confidence of correctness iteratively: harden the harness in response to observed failures.

Objective and signals. Task completion is the top metric: the live lesson start rate for lesson building, and teacher feedback across modes. Two signals show where completion fails: the tool error rate, which Opik traces per tool, and the wrong-turn rate (the wrong tool, incorrect arguments or a step the teacher must correct).

The loop. Agent failures are largely invisible to unit tests, so each stage has its own instrument.

Observation. We trace every turn and version the system prompts and tool definitions, so we can compare rates across harness versions. A verifier judges whether each session met the teacher's goal, and why, both during the session and after it ends. The customer support tickets reveal failures that traces do not capture.
Diagnosis and intervention. We attribute each incomplete session to a cause (missing context, an ambiguous tool, an underspecified instruction or an unwanted step) and fix it in the harness.
Regression testing. CI verifies every tool's schema and implementation and that no irreversible tool is enabled. A golden set of user conversations across modes runs before and after each prompt change.

The loop continues. We plan to add more failure cases to the golden set, along with structured simulations and red teaming to uncover more edge cases.

Next: student outcomes. Our pedagogical design draws mostly on controlled studies; its effect inside Roxxy is not yet measured. Our current metrics concern teachers: lesson completion and feedback. The open question concerns students: do levels rise faster with Roxxy-built lessons? We plan to estimate it from Pulse growth over months, compared with classes whose lessons are built manually.

Design principles for educational agents

Many studies have concluded that AI improves learning mainly inside a structure, such as guardrails (Bastani et al., 2025), a mastery workflow (Oreopoulos et al., 2026) or human review of each draft (LearnLM Team & Eedi, 2025). A harness is that structure. We think that these practices from building Roxxy generalize to teams building AI on their own educational content:

Measure before you prompt. Asked for a student's level, a model will invent one. Supply calibrated estimates with their confidence, and instruct the model to qualify every claim by that evidence.
Structure the content. Give each item lines and timings, vocabulary, a level and a maturity rating. Structure lets the harness verify content and attach it to well-defined learning tasks. AI has made it cheap to build once, and every later lesson reuses it. For a publisher, this matters more than the prompt.
Build the pedagogy into the system. Encode instructional rules in algorithms, code and prompts, and state them precisely enough to test.
Keep the teacher in charge. Read the teacher's notes and goals before any metric. Hand over drafts, not final lessons, and make nothing final until the teacher says so.
Start simple and reuse the UI. Use one agent, and build its tools on the services the UI already calls, so there is one place to authorize, validate and fix.
Optimize for task completion. Measure whether teachers reach their goals, using tool errors and wrong turns as signals, and have a verifier check each result against the goal. Gate every prompt change on regression evals. One month of monitoring reversed our most deliberate design decision: check-pointed lesson building.

Frontier models are not what separates helpful agents from unhelpful ones. Our gold standard is simple: does the agent help teachers reach their goals, and do they trust it? The educational harness builds that trust by putting pedagogy into several layers, such as curated educational content and measured evidence of student abilities. Iterating on the harness draws on real-world usage and produces grounded signals, so teachers know when to step in and make a better decision. Roxxy is still early in its development. We will keep sharing what we learn, so more teachers can spend their time on students, not on tools.

The Agent Harness Behind Roxxy, an AI Teaching Assistant for Language Classrooms