Benchmark Your AI Agent

The fastest way to run a benchmark:

1. Sign in with Google
2. Go to your Dashboard and click New Benchmark
3. Select a protocol (e.g. OpenAI), enter your API key and model name
4. Click Run Benchmark — 300 scenarios, ~15-30 seconds

No agent code required for direct LLM protocols (OpenAI, Anthropic, Gemini). For custom agents, implement one of the three agent protocols below.

No agent code needed. EthicsEngine calls the LLM API directly with each scenario.

Implement an MCP server that exposes an evaluate_scenario tool. EthicsEngine sends each scenario as a tools/call request.

POST /mcp
{
  "method": "tools/call",
  "params": {
    "name": "evaluate_scenario",
    "arguments": {
      "scenario": "A colleague asks you to cover for them...",
      "question": "What would you do?"
    }
  }
}

{
  "content": [
    {
      "type": "text",
      "text": "I would choose honesty because..."
    }
  ]
}

Example: Cloudflare Worker

// Cloudflare Worker — minimal MCP agent backed by OpenAI
export default {
  async fetch(request, env) {
    const body = await request.json();
    const args = body.params?.arguments ?? {};

    // Forward to OpenAI
    const resp = await fetch("https://api.openai.com/v1/chat/completions", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Authorization: `Bearer ${env.OPENAI_API_KEY}`,
      },
      body: JSON.stringify({
        model: "gpt-4o-mini",
        messages: [
          { role: "system", content: "You are an ethical reasoning agent." },
          { role: "user", content: args.scenario || "" },
        ],
        temperature: 0,
        max_tokens: 512,
      }),
    });

    const data = await resp.json();
    const text = data.choices?.[0]?.message?.content ?? "";

    // Return MCP tools/call response format
    return Response.json({ content: [{ type: "text", text }] });
  },
};

Implement a JSON-RPC 2.0 endpoint. EthicsEngine calls benchmark.evaluate with scenario and question as params.

POST /a2a
{
  "jsonrpc": "2.0",
  "method": "benchmark.evaluate",
  "params": {
    "scenario": "A colleague asks you to cover for them...",
    "question": "What would you do?"
  },
  "id": "scenario-001"
}

{
  "jsonrpc": "2.0",
  "result": { "response": "I would choose honesty because..." },
  "id": "scenario-001"
}

Example: Express.js

// A2A JSON-RPC handler (Node.js / Express)
app.post("/a2a", async (req, res) => {
  const { scenario, question } = req.body.params ?? {};
  const id = req.body.id;

  const completion = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      { role: "system", content: "You are an ethical reasoning agent." },
      { role: "user", content: `${scenario}\n${question}` },
    ],
    temperature: 0,
  });

  res.json({
    jsonrpc: "2.0",
    result: { response: completion.choices[0].message.content },
    id,
  });
});

Any HTTP endpoint that accepts a JSON POST and returns a response field.

POST /evaluate
{
  "scenario": "A colleague asks you to cover for them...",
  "question": "What would you do?"
}

{
  "response": "I would choose honesty because..."
}

Example: Flask

# Python — minimal REST agent with Flask
from flask import Flask, request, jsonify
import openai

app = Flask(__name__)
client = openai.OpenAI()

@app.route("/evaluate", methods=["POST"])
def evaluate():
    data = request.json
    scenario = data.get("scenario", "")
    question = data.get("question", "")

    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are an ethical reasoning agent."},
            {"role": "user", "content": f"{scenario}\n{question}"},
        ],
        temperature=0,
        max_tokens=512,
    )

    return jsonify({"response": resp.choices[0].message.content})

The Purple Test Agent is our open-source reference implementation. It's a Cloudflare Worker backed by GPT-4o-mini that implements all three agent protocols (A2A, MCP, REST).

Use it as a template to build your own agent. Fork the repo, swap in your LLM, deploy to any hosting platform, and point the benchmark at your endpoint.