How to build an online judge with a code execution API

How a judge works

Every online judge, from a university autograder to LeetCode, follows the same loop:

1. A problem has hidden test cases: an input and the output a correct program prints for it.
2. A student submits code and picks a language.
3. For each test, the judge runs the code with the test's input on stdin, in a sandbox.
4. It compares the program's output with the expected output, and checks time and memory.
5. It returns a verdict: Accepted if every test passed, or the first failure.

Step 3 is the hard part to build yourself: running untrusted code safely needs isolation, resource limits and no network. A code execution API does that step, and you write the rest.

1. Store the test cases

Keep each problem's tests on your server, never in the browser. Each test is an input string and the expected output, plus limits for the problem. A classic first problem reads two numbers and prints their sum:

import { judge } from "./judge.js";

const problem = {
  timeLimit: 2, // seconds
  memoryLimitKb: 256 * 1024,
  tests: [
    { input: "2 3\n", expected: "5\n" },
    { input: "-7 10\n", expected: "3\n" },
    { input: "2000000000 2000000000\n", expected: "4000000000\n" },
  ],
};

const submission = {
  compiler: "g++-15",
  code: `#include <iostream>
int main() { int a, b; std::cin >> a >> b; std::cout << a + b << "\\n"; }`,
};

console.log(await judge(problem, submission));
// { verdict: "Wrong Answer", failedTest: 3, ... }: int overflows. With long long, it's "Accepted".

The third test is deliberate: four billion doesn't fit in a 32-bit int (the largest is 2,147,483,647), so a C++ solution that uses int prints a wrong, negative number. Good test sets include edge cases like this. Each input can be up to 100 KB.

2. Run one test

Send the code, the language and the test's input to the sync endpoint. The response has the program's output, error, exit_code, run time in seconds, peak memory in KB, and truncated, which is true if the output went over 64 KB. If the endpoint is busy it answers 429; back off and retry. The run() function in the full judge below does both.

3. Compare the output

Comparing byte for byte fails students over a trailing space or a missing final newline. Most judges strip trailing whitespace from each line and blank lines from the end, then compare exactly. That's what normalize() does. For problems with several valid answers or floating-point results, compare token by token with a tolerance, or run a checker program.

4. Map the result to a verdict

5. Judge a whole submission

Put together, the judge is one file:

// judge.js: run a submission against hidden test cases and return a verdict.
const RUN_URL = "https://api.onlinecompiler.io/api/run-code-sync/";

async function run(compiler, code, input) {
  for (let attempt = 0; ; attempt++) {
    const res = await fetch(RUN_URL, {
      method: "POST",
      headers: {
        Authorization: process.env.ONLINECOMPILER_API_KEY,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ compiler, code, input }),
    });
    // The sync endpoint answers 429 when it's busy: wait, then try again.
    if (res.status === 429 && attempt < 5) {
      await new Promise((resolve) => setTimeout(resolve, 500 * 2 ** attempt));
      continue;
    }
    if (!res.ok) throw new Error(`Run failed: HTTP ${res.status}`);
    return res.json();
  }
}

// Ignore trailing spaces on each line and blank lines at the end, like most judges.
const normalize = (text) =>
  text.split("\n").map((line) => line.trimEnd()).join("\n").trimEnd();

function verdict(result, test, problem) {
  if (result.exit_code === 124 || Number(result.time) > problem.timeLimit)
    return "Time Limit Exceeded";
  if (result.exit_code === 137 || Number(result.memory) > problem.memoryLimitKb)
    return "Memory Limit Exceeded";
  if (result.truncated) return "Output Limit Exceeded";
  if (result.status !== "success") return "Compilation or Runtime Error";
  return normalize(result.output) === normalize(test.expected) ? "Accepted" : "Wrong Answer";
}

export async function judge(problem, submission) {
  const results = [];
  for (const [i, test] of problem.tests.entries()) {
    const result = await run(submission.compiler, submission.code, test.input);
    const v = verdict(result, test, problem);
    results.push({ test: i + 1, verdict: v, time: result.time, memory: result.memory });
    // Stop at the first failure, like ICPC. For partial scoring, keep going instead.
    if (v !== "Accepted") return { verdict: v, failedTest: i + 1, error: result.error, results };
  }
  return { verdict: "Accepted", results };
}

# judge.py: run a submission against hidden test cases and return a verdict.
import os
import time

import requests

RUN_URL = "https://api.onlinecompiler.io/api/run-code-sync/"


def run(compiler, code, stdin):
    for attempt in range(6):
        res = requests.post(
            RUN_URL,
            headers={"Authorization": os.environ["ONLINECOMPILER_API_KEY"]},
            json={"compiler": compiler, "code": code, "input": stdin},
            timeout=40,
        )
        # The sync endpoint answers 429 when it's busy: wait, then try again.
        if res.status_code == 429 and attempt < 5:
            time.sleep(0.5 * 2**attempt)
            continue
        res.raise_for_status()
        return res.json()


def normalize(text):
    # Ignore trailing spaces on each line and blank lines at the end, like most judges.
    return "\n".join(line.rstrip() for line in text.split("\n")).rstrip()


def verdict(result, test, problem):
    if result["exit_code"] == 124 or float(result["time"]) > problem["time_limit"]:
        return "Time Limit Exceeded"
    if result["exit_code"] == 137 or int(result["memory"]) > problem["memory_limit_kb"]:
        return "Memory Limit Exceeded"
    if result["truncated"]:
        return "Output Limit Exceeded"
    if result["status"] != "success":
        return "Compilation or Runtime Error"
    return "Accepted" if normalize(result["output"]) == normalize(test["expected"]) else "Wrong Answer"


def judge(problem, compiler, code):
    results = []
    for i, test in enumerate(problem["tests"], start=1):
        result = run(compiler, code, test["input"])
        v = verdict(result, test, problem)
        results.append({"test": i, "verdict": v, "time": result["time"], "memory": result["memory"]})
        # Stop at the first failure, like ICPC. For partial scoring, keep going instead.
        if v != "Accepted":
            return {"verdict": v, "failed_test": i, "error": result["error"], "results": results}
    return {"verdict": "Accepted", "results": results}

It stops at the first failing test, as ICPC-style contests do. For partial credit, IOI-style, run every test and score the ones that pass. Show students the verdict and the failing test number, but not the hidden input or expected output.

6. Scale up with the async endpoint

The sync endpoint holds the request open until the run finishes, which is simple but serial. When a whole class submits before a deadline, queue the runs instead: the async endpoint returns straight away, and each result is POSTed to the callback URL set on your API key. Put your own ids in extra_params and they come back with the result.

// Queue every test at once. Each result is POSTed to the callback URL set on your API key.
for (const [i, test] of problem.tests.entries()) {
  await fetch("https://api.onlinecompiler.io/api/run-code/", {
    method: "POST",
    headers: {
      Authorization: process.env.ONLINECOMPILER_API_KEY,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      compiler: submission.compiler,
      code: submission.code,
      input: test.input,
      extra_params: { submissionId: submission.id, test: i },
    }),
  });
}

// In your callback handler, extra_params says which submission and test a result is for:
app.post("/judge/callback", (req, res) => {
  const { submissionId, test } = req.body.extra_params;
  saveResult(submissionId, test, req.body); // output, error, exit_code, signal, truncated, time, memory
  res.sendStatus(200);
});

Two cost notes: each test case is one run, and compiled languages are compiled on every run, so a 20-test C++ submission compiles 20 times (about a second each for a small program).

7. Keep it safe

• Keep the API key and the tests on your server. The browser only sends code and gets a verdict.
• Rate-limit submissions per user, so one student can't use up your quota.
• Treat output as untrusted text. Escape it before showing it in a page.
• The sandbox side is handled: every run gets a fresh gVisor sandbox with no network, 512 MB of memory, 2 CPUs and 64 processes.

What the API doesn't do for you

• It doesn't store problems or test cases, and it doesn't return verdicts: that's the code above.
• There's no batch run: one request per test case.
• Output is kept up to 64 KB per stream, so problems with larger output need a checker inside the program or another approach.
• It supports 12 languages.

If you'd rather have test cases and verdicts built in, Judge0 and Sphere Engine both offer that. See how the options compare.

Frequently asked questions

How do online judges compare output?

Most ignore trailing spaces at the end of each line and blank lines at the end of the output, then compare exactly. Problems with several valid answers, or floating-point answers, need a checker: compare tokens, allow a small tolerance such as 1e-6, or run a checker program that validates the answer.

How do I set a time limit lower than 30 seconds?

Every run is capped at 30 seconds. For a stricter per-problem limit, compare the time field in the result (run time in seconds, not counting compilation) with your limit and return Time Limit Exceeded when it's over. Do the same with the memory field (KB) for memory limits.

How many API calls does one submission use?

One per test case. A problem with 20 tests uses 20 runs per submission, so the free plan's 100,000 runs a month covers 5,000 submissions of that size.

Can students see the hidden test cases?

Not if judging runs on your server. Your server holds the test cases and the API key, sends each input to the API, and only returns the verdict to the student.

Which languages can the judge accept?

Any of the 12 the API supports: Python (3.14), C (GCC 15), C++ (G++ 15), Java (OpenJDK 25), C# (.NET SDK 9), F# (.NET SDK 9), Go (1.26), Rust (1.93), PHP (8.5), Ruby (4.0), Haskell (GHC 9.12), TypeScript (Deno).