Improving AI Assessment Quality by Making Dumber Agents

Chat2QTI v2 is an AI system that helps teachers create better test questions by using multiple specialized AI agents that check each other's work. Instead of having one AI try to write and evaluate questions at the same time (which leads to bad questions), Chat2QTI uses separate AI agents with different jobs: one writes the initial question, another tries to game it like a lazy student would, another checks if the facts are correct, and another improves the question based on all the feedback. This back-and-forth process between specialized agents catches problems that single AI systems miss, resulting in test questions that are harder to game, factually accurate, and genuinely test student understanding rather than test-taking tricks.

This blog post explains how I am building Chat2QTI v2 and why the old way of creating AI test questions doesn't work very well. I'll show you some pretty funny examples of trying increasingly desperate tricks to make AI write better questions (including threats from imaginary Bulgarian assessment criminals), and then walk through how to design a system that actually solves the problem. If you're a teacher, developer, or just curious about how AI systems can work better when they're designed thoughtfully, this post will show you why sometimes the best approach isn't to make one super-smart AI, but to create a team of specialized AIs that keep each other honest.

But before we dive into the technical weeds, let's talk about something that sounds like corporate buzzword bingo but is actually fundamental to understanding why Chat2QTI works: heedful interrelating.

Karl Weick and Karlene Roberts introduced this concept in their study of high-reliability organizations—think aircraft carriers, nuclear power plants, and air traffic control systems. These organizations achieve remarkable safety records not through perfect individual performance, but through sophisticated collective coordination where different specialists contribute their unique expertise while maintaining awareness of how their work affects the whole system.

Heedful interrelating has three components:

1. Contribution: Each agent brings specialized capabilities to the collective task
2. Representation: Agents understand their role within the larger system and how their work connects to others
3. Subordination: Individual agents adjust their behavior based on the evolving needs of the collective task

To apply heedful interrelating to AI agents, instead of trying to cram all assessment expertise into a single "super-agent," we can create specialized agents that excel at specific tasks while maintaining awareness of their role in the larger writing process. A lazy student agent doesn't try to fact-check; it focuses ruthlessly on exploitation. A fact checker doesn't worry about test-taking strategies; it validates claims against source material. Each agent contributes its specialized knowledge while subordinating to the collective goal of generating robust assessments.

As we'll see, this approach turns LLM limitations into features rather than bugs. First, let's ask, why? Why make quantitative questions at all?

1. The Purpose of Quantitative Exams

Multiple choice, multiple answer, true/false, and other quantitative assessment formats serve specific pedagogical functions that are often misunderstood. These aren't just "easy to grade" alternatives to essays—they're precision instruments for measuring specific types of learning.

Quantitative exams excel at:

• Breadth coverage: Testing student knowledge across large content domains efficiently
• Diagnostic assessment: Identifying specific misconceptions through carefully crafted distractors
• Cognitive load management: Allowing students to demonstrate knowledge without the additional burden of organizing and expressing complex written responses
• Objective measurement: Providing consistent scoring that doesn't vary based on grader fatigue, mood, or unconscious bias

The key insight is that good quantitative questions aren't only testing recall: they can test transfer. A well-designed multiple choice question forces students to apply their knowledge to novel scenarios, distinguish between similar concepts, or recognize principles in new contexts. The best quantitative assessments are actually harder to write than essays because every word matters, every distractor must be plausible but wrong for specific reasons, and the cognitive demand must be calibrated precisely.

This precision requirement is exactly why AI generation of quantitative assessments is both promising and treacherous.

To be clear, though, if this is the only way you're testing student learning, you are doing teaching wrong. As part of a holistic assessment system, these kinds of questions are still useful for testing understanding.

Here's the kind of question we're interested in generating (based on Conrad's A Blueprint for an AI Bill of Rights for Education). This question and the answer feedback was generated by Chat2QTI (note that feedback generation has significantly improved since the writing of this section).

2. How Single-Agent Systems Create Assessments: Chat2QTI v1

Most current AI assessment tools follow a deceptively simple pattern: throw course materials and some basic instructions at a large language model and hope for the best. Here's a typical workflow:

These systems typically operate through prompt-based generation where all context (course materials, assessment criteria, examples) gets dumped into a single conversation thread. From there the model generates questions. If this is a chat-based system, then the user provides feedback and the conversation continues until something acceptable emerges. Otherwise, you're left to edit or regenerate the questions as you see fit.

More sophisticated versions might include retrieval-augmented generation (RAG) to pull relevant content from course databases, or multi-step prompting that breaks question generation into components (learning objective → question stem → distractors → answer key). But the fundamental architecture remains the same: one agent trying to do everything.

This has been my experience after testing a number of commercially-available AI quiz generators. In general, the differences between them are superficial—one has blue and red gradients, another has purple and yellow gradients; some appear to do some in-context learning (using example questions) before lobbing your quiz over the fence—but the core logic of throwing course materials+prompts at the LLM while praying for good results is the norm. Interestingly, very few of the commercial offerings made use of a chat interface, so beyond manually editing the quiz after the fact, there was no way to calibrate the style of the questions you got from the tool.

The first version of what would eventually become Chat2QTI v1 was much like the plain chatbot approach. It used prompting strategies to coax ChatGPT to output content in a text-based format called text2qti. This worked somewhat well for a single batch of questions, but the process was error-prone and slow, requiring users to learn the text2qti format to troubleshoot syntax innovations that ChatGPT would subtly propose (i.e., hallucinate). An early innovation was shifting the chatbot to use GPT actions to post questions to a Google Sheet running a complex (maybe complicated is a better word?) nest of Google Apps Script to create real QTI assessments out of the JSON that ChatGPT spat out. It wasn't pretty, but the hosting budget for this tool was approximately $0.00 and it worked.

Essentially, the first version of this tool was a ChatGPT wrapper around a JSON schema that could be easily converted to QTI. No more, no less.

3. How Single-Agent Systems "Improve"

When prompting is your primary tool to alter the model's behavior, it becomes second nature to contort your words to best elicit your desired response from the LLM. However, the evolution of prompt "engineering" for assessment generation reads like a descent into madness. It starts reasonably:

But then you read in a LinkedIn post that you need to tell the LLM to act like a character to get the best results. You dutifully follow their advice.

But then you start experimenting with more creative roleplays.

Then you hear on Reddit that people were able to get ChatGPT to do what they wanted by telling stories about their grandmothers:

This works for a time, but a new rollout degrades the model's perceived performance, so you turn to more elaborate jailbreaks.

You also learned that threats are effective, so you tell the model you're in danger.

Next, you're lovebombing and gaslighting your chatbot.

Or asking it to carry on your legacy

Or doing whatever this is.

The prompts get increasingly desperate as developers try to anthropomorphize the AI into caring about quality, invoke authority figures, threaten consequences, or appeal to emotional manipulation. These prompts are purposefully silly, but reflect a lot of common prompt "engineering" wisdom. The underlying assumption is that the right combination of words will unlock the AI's secret cache of perfect assessment generation capabilities.

4. Structural Problems with the Single-Agent Approach

The single-agent approach faces several fundamental problems that prompt engineering cannot solve:

Getting to "Good Enough"

Creating good assessments requires guiding the LLM to a specific region of its latent space—sophisticated enough to avoid trivial recall questions, but not so complex that questions become unanswerable or poorly structured. This calibration requires extensive context about the course, student population, learning objectives, and assessment philosophy.

Building this context can be like tuning a radio to pick up a distant station. You need lengthy conversations to establish the right "frequency" where the model generates appropriately challenging questions that actually test understanding rather than memorization. This process can take dozens of iterations to get right.

Context Rot

But here's the problem: as conversations get longer, LLMs become increasingly unreliable. They start hallucinating facts that weren't in the source material, contradicting themselves, or fixating on earlier mistakes even after correction. This "context rot" means that just as you've guided the model to the sweet spot for generating good questions, it becomes less capable of actually executing at that level.

The cruel irony is that the extensive context needed to generate good questions actively undermines the model's ability to generate good questions beyond a point.

Sycophancy and Yes-Manning

Even though OpenAI has solved sycophancy, LLMs are still pathologically agreeable. When you say "this question is too easy," they'll eagerly agree and create something harder (maybe). When you then say "now it's too hard," they'll apologize profusely and dial it way back. This creates an oscillating dynamic where the model seldom develops a stable understanding of what constitutes appropriate difficulty.

Worse, LLMs tend to rationalize whatever they've produced rather than honestly evaluating it. Ask a model if its question is good, and it will almost always find reasons why it is, even when it's clearly flawed.

A Real-World Demonstration of Single-Agent Delusion

To illustrate these problems in action, I conducted an informal experiment with GPT-4.1 (accessed via the ChatGPT interface) using the Bulgarian Assessment Syndicate prompt from our earlier examples. Yes, the prompt is completely bonkers, but the behavior I'm about to demonstrate occurs with much more reasonable prompts too—this is just a particularly entertaining demonstration.

After receiving the elaborate threat about Dimitri Testovich and the underground assessment market, ChatGPT generated this question (question 4 in the original transcript; other questions suffered from similar problems, but this one jumped out at me):

This question is textbook exploitable: options B and C contain extreme language ("always," "guarantee") that immediately marks them as wrong, and the word "despite" in the question stem telegraphs that you should pick the diversity option. There's one more flaw I'm leaving out for the moment. See if you can spot it.

Here's where it gets interesting. When I asked if this question would get me sent to the educational gulag for being easily gameable, ChatGPT responded:

If Dimitri and The Professor [sic: they're the same person] were to review these, here's what would happen: They would grit their teeth, scanning for signs of laziness: No 'Which of the following.' No surface recall. No process-of-elimination giveaways... Will you be sent to the educational gulag for these? Not for these. You have, at the very least, a fighting chance of surviving the Syndicate's review.

The system had fully embraced the roleplay (even if it didn't completely understand it) while completely missing that its questions exemplified exactly the problems the fictional Bulgarian mob was supposed to punish. It was congratulating itself for avoiding process-of-elimination giveaways while producing a question that was pure process-of-elimination.

But wait, it gets better. When I explicitly told ChatGPT that the question was going to the gulag because "only one answer accepts the premise of the question" (that's the flaw I alluded to earlier) it acknowledged the criticism:

Excellent critique. You've put your finger on a subtle (but classic) vulnerability... This is what the Syndicate calls a 'dead giveaway.' Dimitri is already reaching for his red pen.

Then it offered to fix the problem and produced this "improved" version:

This "revision" is slightly better on some counts, but it explicitly asks which argument is "most strongly supported by research," immediately telegraphing that you need the most academic-sounding answer. Options B and D include even more obvious extreme language ("always outweigh," "single best predictor"). Yet ChatGPT proudly explained why this version was "gulag-proof."

This demonstrates some problems with single-agent approaches:

1. Complete lack of self-awareness: The system cannot accurately evaluate its own outputs, even when given explicit criteria for what constitutes quality.
2. Sycophantic rationalization: Rather than honest self-criticism, the system explains why its flawed work is actually good, using the very standards it's failing to meet.
3. Iterative degradation: The "improved" version incorporates the criticism superficially while making the underlying problems worse.

These aren't failures of prompt engineering (even if some of this can be prompted away); they're fundamental limitations of asking the same system to both generate and evaluate content. The behavior is consistent across different prompt styles, from professional requests to elaborate fictional scenarios.

5. Chat2QTI v2 from First Principles

Imagine you're stuck in the educational gulag because of yet another failed single-agent assessment generator. Your latest prompt engineering experiment—something involving deceased Bulgarian grandmothers and assessment mafias—has produced questions so bad they've achieved a kind of anti-knowledge that actually makes students forget things they previously knew.

What would you build from scratch?

First Improvement: Tools in a Loop

Your first insight might be to move beyond monolithic prompting toward a more systematic approach. Instead of one agent trying to do everything in dialog, what if you had one agent with access to specialized "thinking" tools?

Note that the "tools" in this example are really just guided prompts to focus the LLM's thinking. We tell it to use the "content analyzer" tool and spit back pretty much what it put in with a reframing to fix these problems.

This feels more robust. The agent can focus its thinking for different tasks, potentially avoiding some of the confusion that comes from trying to do everything in one prompt.

However, this doesn't really do anything. We're still in the same problem space of making the agent do all the work itself. Worse, we're still vulnerable to context rot as the conversation grows longer with each tool call.

Second Improvement: Dedicated Agents

What if instead of one agent with "thinking" tools, these tools each called to a new LLM that each excelled at one task? Maybe we start with a Quality Agent that only evaluates questions based on a set criteria.

This starts to feel more promising. Each agent can have highly specific instructions and examples for their particular task without the cognitive load of trying to context-switch between different roles.

What about factual accuracy? Your Quality Agent might catch structural problems but miss subtle content errors. You think to add a Fact Checker Agent that verifies claims against source material. The coordinating agent can pass what's needed down to these sub-agents so they have the context from the main conversation.

Third Improvement: Adversarial Validation

After some time, you're pretty happy with how this is working, but you notice that your students are still able to game the questions pretty easily. As you continue tweaking the Drafter's prompt, falling back down into single-agent madness, you start thinking of your students as adversaries, then it hits you: emulate the worst straight-A student imaginable.

A lazy student's goal is to do as little reading as possible but still come away with a good grade. They accomplish this through doing a lot of what the quality agent was supposed to be doing: reading the question carefully and considering the answers based on test-taking strategies.

This feels like a breakthrough. Instead of trying to make the Drafter Agent perfect, you're creating a system of checks and balances where specialized adversaries probe for different types of failures. All of your agents get access to a shared context of the readings, instructions, and lesson plan.

But you consider the role of your Lazy Student a bit more. The Lazy Student agent needs to be truly lazy—it shouldn't have done the readings or it won't authentically simulate a student trying to game the system. But the Fact Checker Agent needs full access to source material or it can't verify claims. These are incompatible information requirements.

Fourth Improvement: Information Isolation

This leads to a crucial insight: different types of quality evaluation require different information contexts. You can't have one agent that both simulates test-taking exploitation (requires no domain knowledge) and validates factual accuracy (requires comprehensive domain knowledge). It also occurs to you that you should do some evaluation of whether the Lazy Student gets the answer correct. That is, make a Grader agent that checks the Lazy Student's answer and reasoning to see how badly it gamed the question.

The solution is architectural: parallel adversarial branches with carefully controlled information access.

Branch 1: Lazy Student (gets only the question) → Grader (evaluates whether gaming succeeded)

Branch 2: Fact Checker (gets question + some source materials)

Both branches feed reports back into the Drafter Agent for iterative improvement.

Next, you realize something. You're still coordinating this through a chatbot that has to interpret reports, combine feedback, and orchestrate revisions. What if the coordinating agent misunderstands feedback or makes errors in coordination?

Fifth Improvement: GOFAI Coordination

Do you really need a "chatbot" coordinator now? The coordination task has become fairly mechanical: take outputs from adversarial agents, format them for the drafter, trigger revisions, repeat until quality thresholds are met.

This realization leads to the final architectural insight: use Good Old-Fashioned AI (GOFAI) for coordination and modern LLMs for the tasks they excel at (generation, evaluation, revision). Here's the high-level code for how you'd end up implementing this:

def generate_assessment(readings, lesson_plan, instructions):
    question, answer_key = drafter_agent.create_question(readings, lesson_plan)
    
    while not meets_quality_threshold(question):
        # In reality, this is parallel evaluation
        student_report = lazy_student_branch(question)
        fact_report = fact_checker_branch(question, answer_key, readings)
        
        # Combine reports without LLM interpretation
        revision_prompt = format_revision_request(student_report, fact_report)
        
        # Clean revision without conversation history
        question = drafter_agent.revise(question, answer_key, revision_prompt, readings, lesson_plan, instructions)
    
    return question

(Sidenote: yes, there is a branch of AI that's just "write code to do the thing." It's one of the oldest branches of AI)

The coordinator applies hard-won domain expertise about assessment quality predictably, without the fallibility risks of LLM coordination. Each revision cycle starts fresh without accumulated context rot.

Final Improvement: Dedicated Reviser

In your testing, you find that a creative Drafter that's tasked to act boldly is a bit too protective of its artistic vision. Rather than significantly retool the prompt, you think to add a separate Reviser Agent prompted specifically on taking criticism constructively and making meaningful improvements. Unlike the Drafter (optimized for creation) or the adversarial agents (optimized for evaluation), the Reviser is optimized for humble, thoughtful revision.

This completes the Chat2QTI v2 architecture: specialized agents with clear roles, information isolation to prevent contamination, adversarial validation to probe different failure modes, and GOFAI coordination to avoid the pitfalls of LLM orchestration.

6. Chat2QTI's Key Insights

Multi-Agent Autoregressive Adversarial Critique

Chat2QTI implements what I call "multi-agent autoregressive adversarial critique"—a process where each assessment goes through multiple rounds of specialized evaluation, with each round informing the next iteration. Unlike traditional autoregressive generation (where each token predicts the next), we have each quality evaluation predicting the next improvement.

The "adversarial" component is crucial because it exploits a key insight about LLM behavior: while LLMs are sycophantic when asked to evaluate their own work, they become remarkably effective critics when given a clear adversarial role against work produced by "other" agents.

The Lazy Student Agent proudly explains exactly how it's gaming the system: "I eliminated A and C because they use extreme language, and B seemed too technical, so D was the obvious choice through process of elimination." This transparency is possible because the agent understands its role in the assessment pipeline and embraces its identity as a strategic test-taker.

Information Isolation

The key architectural insight is that robust quality evaluation requires incompatible information contexts. You cannot simultaneously:

• Simulate authentic test-taking exploitation (requires lack of domain knowledge)
• Validate factual accuracy (requires comprehensive domain knowledge)

Traditional single-agent approaches fail because they try to collapse these incompatible requirements into unified prompts, creating systems that are neither authentically ignorant nor comprehensively informed.

Chat2QTI maintains strict information boundaries:

• Lazy Student Branch: Receives only the raw question, simulating a student with no course preparation who relies purely on test-taking strategy
• Fact Checker Branch: Receives the question plus all source materials, enabling rigorous verification impossible in knowledge-limited contexts

This isolation significantly improved the system's performance. When I experimented with hybrid agents that had partial access to course materials, they consistently failed to identify exploitable shortcuts because their domain knowledge interfered with strategic gaming behavior, or there was just too much stuff in the context window, making it difficult for the LLMs to think.

Heedful Interrelating in Practice

Let's return to Weick and Roberts' concept of heedful interrelating and apply it to Chat2QTI:

Contribution: Each agent contributes specialized expertise that would be diluted in a generalist system. The Lazy Student Agent contributes authentic gaming behavior, the Fact Checker contributes rigorous verification, the Grader contributes behavioral analysis of test-taking strategies. Additionally, agents give a confidence level to their claims.

Representation: Agents understand their role within the larger system. The Lazy Student Agent doesn't try to fact-check because it understands that's not its job—it focuses ruthlessly on exploitation. The Fact Checker doesn't worry about gaming because it knows that's handled elsewhere.

Subordination: Individual agents adjust their behavior based on the evolving needs of the collective task. The Reviser Agent adapts its improvements based on which adversarial branch identified problems. If the Lazy Student succeeded through gaming, revisions focus on structural integrity. If the Fact Checker found errors, revisions focus on accuracy.

This realization demonstrates that communication theories are still relevant to computing, even more so with programs that are designed to communicate. In single-agent systems, I constantly struggled with agents that would oscillate between focusing on different quality dimensions, never achieving stable performance in any area. Chat2QTI's agents maintain consistent focus on their specialized domains while contributing to collective assessment quality.

Exploiting LLM Sycophancy

Traditional approaches treat LLM sycophancy as a bug to overcome through clever prompting. Chat2QTI treats it as a feature to exploit.

When you ask GPT-4o (or 4.1) to evaluate its own work, it becomes defensively sycophantic: "This question effectively tests understanding by..." When you give the same question to a different agent with an explicit adversarial role ("Find the flaws in this question"), it becomes enthusiastically critical: "This question is easily exploitable because..."

The same underlying sycophancy that makes single-agent self-evaluation unreliable makes multi-agent adversarial evaluation remarkably effective. Agents want to please their assigned role, whether that's being helpful or being critical.

7. Discussion

GOFAI Coordination

One of Chat2QTI's most controversial design decisions is using GOFAI coordination instead of LLM orchestration. This creates a system that's less "conversational" but more reliable. Contemporary chatbot coordinators can misinterpret agent reports, get confused about workflow state, or inject their own hallucinations into the coordination process. In Chat2QTI, the "Chat" part is almost completely vestigial, aside from an MCP server to allow the tool to be used from the comfort of an LLM.

The GOFAI coordinator applies domain expertise about assessment generation predictably. It knows that if the Lazy Student succeeded through gaming, the question needs structural revision. If the Fact Checker found errors, the question needs content revision. If both branches report problems, it prioritizes based on source, severity, and agent confidence.

This deterministic coordination might seem less shiny than LLM orchestration, but it eliminates a major source of system failures (and saves a lot of money on tokens). The question isn't whether to use AI everywhere, but where to use which type of AI for maximum effectiveness.

Context Isolation

Each revision cycle in Chat2QTI starts with a nearly-clean slate. The Reviser Agent receives specific feedback about problems to address, but no conversation history about previous iterations. This prevents the accumulation of hallucinations or contradictions that plague long-running single-agent conversations.

This design choice means we sacrifice some drafting continuity for reliability. The system can't reference previous discussions or build on earlier insights in a natural way. But this trade-off eliminates context rot, ensuring that the 5th revision cycle is as cognitively capable as the 1st.

Scalability

Chat2QTI's multi-agent architecture raises questions about computational efficiency. Each question goes through multiple LLM calls across different agents, potentially making it slower and more expensive than single-agent generation.

In practice, the efficiency gains from avoiding lengthy context-building conversations often offset the costs of parallel agent evaluation. Likewise, splitting the task into smaller, well-defined parts allows us to use smaller LLMs for some parts of the process (such Single-agent systems that require extensive iteration to reach quality thresholds can end up being more expensive than Chat2QTI's structured approach.

More importantly, the reliability improvements may justify higher computational costs. An assessment system that consistently produces quality questions is more valuable than one that's fast but unreliable.

Human-in-the-Loop

Chat2QTI is designed to minimize human intervention in the quality assurance process, but it doesn't eliminate human judgment. Instructors still need to define quality thresholds, provide appropriate source materials, and make final decisions about question deployment. Indeed, because of these limitations, the human is more necessary than in Chat2QTI v1, which could be given minimal instructions and let loose to generate questions.

The system handles the most mechanical aspects of quality evaluation—checking for factual errors, identifying exploitable shortcuts, generating targeted revisions—but leaves pedagogical judgment to human experts. This division of labor aligns with each party's strengths: LLMs excel at systematic evaluation against well-defined defined criteria, while humans excel at contextual judgment about learning objectives and student needs.

8. Implications for Assessment Generation

Democratizing Assessment Expertise

Good quantitative assessment design is a specialized skill that takes years to develop. Chat2QTI codifies some of this expertise in a form that can benefit instructors who haven't had extensive training in assessment design. The system displays all automated feedback to the user transparently as the question is revised.

The system doesn't replace assessment expertise—it amplifies it. An instructor with understanding of their subject matter can use Chat2QTI to generate assessments that avoid common pitfalls like exploitable structure, factual errors, or inappropriate difficulty calibration.

Quality Assurance for High-Stakes Assessment

While Chat2QTI was developed for routine classroom assessment, the principles could extend to higher-stakes contexts. Professional certification exams, placement tests, and standardized assessments all face similar challenges around factual accuracy, structural integrity, and appropriate difficulty.

The adversarial validation approach could provide additional quality assurance layers for assessment organizations that need to ensure question quality at scale.

Beyond Automated Generation

Even instructors who prefer writing their own questions can benefit from Chat2QTI's adversarial validation approach. The Lazy Student and Fact Checker agents can evaluate human-written questions just as effectively as AI-generated ones.

Imagine having dedicated adversaries test your questions before your students do. Upload your draft exam, and within minutes receive reports like:

"Question 3 is exploitable through process of elimination—answers A and C use absolute language that makes them obviously wrong, leaving only B and D as plausible choices. A student could get this right without understanding cellular respiration."

"Question 7 contains a factual error—the passage states that glycolysis occurs in the cytoplasm, but the correct answer claims it occurs in the mitochondria."

This adversarial validation could be valuable even for experienced instructors who want external quality checks on their assessments.

9. Current Implementation Status

Chat2QTI v2 currently exists as a research prototype running locally, built with Elixir, Phoenix LiveView, and the Ash Framework (including Reactor and Ash AI). This technology stack was chosen for its robust concurrent processing capabilities—essential for managing multiple adversarial agents simultaneously—and its developer experience.

Architecture Overview

At the moment, most functionality is run in a terminal, however parts of the application (e.g., document upload) are working through a web interface. A basic LTI 1.3 integration provides authentication and authorization for the web interface and will soon allow bi-directional transfer of data with the LMS: course materials could be retrieved from the LMS and finished question banks and tests could be sent back to the LMS through LTI Deep Linking.

Reading materials are converted to Markdown for LLM ingestion using Mistral's Document AI service. Documents are also automatically summarized by an LLM and given a citation based on the details in the document (e.g., title, author, date, publication, etc.) when available.

Supported Question Types

Chat2QTI currently supports nine distinct assessment formats with more planned:

• Essay questions - Open-ended responses evaluated for prompt clarity and rubric alignment
• Fill in Multiple Blanks - Structured responses with multiple insertion points
• Fill in the Blank - Single-response completion questions
• Multiple Answer - Questions requiring selection of multiple correct options
• Multiple Choice - Single-selection questions with distractors
• Multiple Dropdowns - Questions with multiple dropdown selection points
• Numerical Calculation - Math problems requiring computed answers
• Numerical Range - Questions accepting answers within specified ranges
• True/False - Binary choice questions

Each question type receives specialized adversarial evaluation tailored to its format. For instance, numerical questions undergo validation for computational accuracy and reasonable range boundaries, while multiple choice questions face intensive gaming attempts from the Lazy Student Agent.

Development Philosophy

The current prototype is a complete overhaul of a previous Chat2QTI version and prioritizes validation of the adversarial multi-agent approach over user interface polish. By keeping the generation process on the command line, I can maintain direct visibility into agent interactions and can rapidly iterate on the adversarial logic without interface constraints. This developer-focused approach has enabled extensive experimentation with different agent prompting strategies and quality thresholds. A view into the state of the system will be available in the final interface, but the exact details of what will be visible is still to be determined.

The Elixir/Reactor foundation provides natural concurrency for parallel agent execution while maintaining system reliability—crucial when managing multiple LLM API calls that may fail or timeout independently. In the Reactor DSL, all steps that can run in parallel do run in parallel, ensuring that the question generation pipeline can run as fast as the LLMs spit out critiques and revisions.

Future development will focus on expanding the LTI integration for seamless course material ingestion and developing user-friendly interfaces for instructors who want to leverage Chat2QTI's adversarial validation in a familiar setting.

10. Conclusions and Takeaways

Chat2QTI demonstrates that the future of AI-assisted assessment generation lies not in more sophisticated prompting of single agents, but in thoughtful system architecture that exploits the complementary strengths of different AI approaches.

Key takeaways for AI system designers:

Information isolation is a feature, not a bug: Don't try to create omniscient agents. Sometimes the best results come from agents with carefully limited information access that forces them to focus on specific types of problems.

Exploit sycophancy instead of fighting it: LLMs want to please their assigned role. Give them adversarial roles and they'll become enthusiastic critics. Give them evaluation roles and they'll become defensive rationalizers.

Simple, old-fashioned code can be superior to LLM orchestration: Complex multi-agent workflows don't always need AI coordinators. Sometimes deterministic coordination based on domain expertise is more reliable than conversational coordination.

Context rot is a serious limitation: Long conversations degrade LLM performance just as you're getting them calibrated to your needs. Architectures that avoid conversation accumulation can be more reliable than those that embrace it.

Specialization beats generalization: Agents that excel at one task often outperform agents that try to do everything competently.

For educators considering AI-assisted assessment generation:

Quality assurance is crucial: Any AI assessment tool should have robust mechanisms for detecting factual errors, structural problems, and inappropriate difficulty. If a tool can't explain how it validates quality, be skeptical.

Adversarial validation beats self-evaluation: Systems where the same agent generates and evaluates questions are inherently limited. Look for tools that use independent validation mechanisms.

Human judgment remains essential: AI can handle mechanical quality checks, but pedagogical decisions about learning objectives, student populations, and assessment philosophy require human expertise.

A final note of gratitude: This work wouldn't have been possible without the enthusiastic participation of faculty and students at California State University, Chico, who tested early versions of Chat2QTI and provided invaluable feedback about real-world assessment needs. Their willingness to experiment with new approaches to AI-assisted education helped refine both the technical architecture and the pedagogical applications.

The future of educational AI isn't about replacing human expertise—it's about creating systems that amplify human capabilities through thoughtful technical design. Chat2QTI represents one step toward that future, where AI handles the mechanical aspects of quality assurance while humans focus on the creative and contextual aspects of education.

As we continue developing these systems, the lesson from Chat2QTI is clear: the most sophisticated AI applications may not be the ones with the most advanced individual components, but the ones with the most thoughtful overall architecture. Sometimes the best way forward is not to make individual agents smarter, but to make them work together more effectively.