A code execution API for AI interviews

Também em portuguêsTambién en español

How it fits together

1. You store each interview question once as a stored problem: a statement, sample tests, hidden tests, and optionally a reference solution and a custom checker.
2. During the interview the candidate writes code in your editor. When they press Run or Submit, or your AI decides to run their code, the AI calls a tool on your backend.
3. Your backend sends the code to the problem with your API key. The API runs it against the tests, each in a fresh sandbox, and returns a verdict for each test and a summary.
4. Your backend hands the AI the summary and the visible tests' results. The hidden tests' inputs and expected outputs never enter the conversation.

1. Store the problem once

POST /api/problems/ creates a problem from a title, a statement, the languages a candidate may use and the tests. Each test is an input and an expected output, and a test is hidden unless you say "hidden": false, so the only tests a candidate (or your AI) can see are the samples you choose. Give a reference_solution and publishing checks it first: POST /api/problems/{problem}/publish/ answers 202, the version is checking while the reference solution runs against every test, and candidates are judged against the version only once it has passed every test and is published. If it fails, the version is rejected; fix the draft and publish again.

# 1. Store the problem. A test is hidden unless you say "hidden": false.
curl -X POST https://api.onlinecompiler.io/api/problems/ \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Sum of two numbers",
    "slug": "interview-sum",
    "statement": "Read two integers on one line and print their sum.",
    "languages": ["python-3.14", "nodejs-24", "openjdk-25"],
    "scoring": "percent",
    "tests": [
      {"name": "sample 1", "input": "1 2\n", "expected_output": "3\n", "hidden": false},
      {"name": "sample 2", "input": "5 7\n", "expected_output": "12\n", "hidden": false},
      {"input": "-4 4\n", "expected_output": "0\n"},
      {"input": "2000000000 2000000000\n", "expected_output": "4000000000\n"}
    ],
    "reference_solution": {"compiler": "python-3.14", "code": "a, b = map(int, input().split())\nprint(a + b)\n"}
  }'

# 2. Publish it. With a reference solution the answer is 202 and the version is "checking"
#    until that solution has passed every test. Read the problem until it is "published".
curl -X POST https://api.onlinecompiler.io/api/problems/interview-sum/publish/ -H "Authorization: YOUR_API_KEY"
curl https://api.onlinecompiler.io/api/problems/interview-sum/ -H "Authorization: YOUR_API_KEY"

For a question with more than one right answer, such as "print any valid path", set checker to custom and give a custom_checker: a program of yours that reads the input, the expected output and the candidate's output and decides, with partial credit if you want it. For "write this function" questions, a driver wraps the candidate's function in a program of yours. Drivers are not shown to candidates, but a candidate's program can read and print its own driver, so keep test data out of drivers. A plan keeps 25 active problems on Free, 250 on Starter and 2,500 on Pro and Business. The stored problems reference has every field and limit.

2. Your backend judges, not your AI's prompt

When the candidate runs or submits, your backend calls POST /api/problems/{problem}/submissions/ with the compiler and the code. mode is samples for a Run button, which judges only the visible tests and counts only them, and all for Submit. The tests, the checker, the scoring and the time limits come from the problem's published version, so the request carries no test data, and the result gives a hidden test only its verdict.

That is what keeps the hidden tests out of the conversation. They are never in your AI's context, its tool calls or its tool results, so there is nothing for the AI to repeat, however the candidate asks. Two rules keep it that way:

• Keep the API key on your backend. Never give it to the AI or to the browser. With your key, GET /api/problems/{problem}/ returns the hidden tests, so it must never be a tool the AI can call.
• Let the AI decide, not supply. The AI's tool call should carry only its decision: Run or Submit. Your backend fills in the problem, the candidate's language, the code from the editor and an attempt id from the interview it is running, so the AI cannot change the code it is judging.

For the statement and the samples the AI may discuss, read GET /api/problems/{problem}/public/: the title, the statement, the languages, any starter code and the visible tests, and for the hidden tests only how many there are.

3. Give the AI only what it may say

This function is the backend side of the AI's tool. It sends the code, waits for the result (for five minutes at most, so a tool call never hangs) and returns two things: the result's summary, and the visible tests' verdicts and output. Wire it to your model's tool calling the way your AI provider does it; nothing in it depends on one.

# judge_for_ai.py: the backend side of your AI interviewer's "run the code" tool.
# It runs on your server with your API key. The AI sees only what judge_for_ai() returns.
import os
import time

import requests

API = "https://api.onlinecompiler.io"
HEADERS = {"Authorization": os.environ["ONLINECOMPILER_API_KEY"]}
WAIT_SECONDS = 300  # how long a tool call waits for the verdicts before it gives up


def judge_for_ai(problem, compiler, code, attempt_id, submit=False):
    """Judge the candidate's code against a stored problem (its id or slug).

    submit=False is the Run button: only the visible tests. submit=True judges every test.
    attempt_id is an opaque id of yours for this one Run or Submit (up to 100
    printable ASCII characters), never a name or an email.
    """
    body = {
        "compiler": compiler,
        "code": code,
        "mode": "all" if submit else "samples",
        "async": True,
        "external_ref": attempt_id,
    }
    # A retry with the same Idempotency-Key gets the same submission back: judged and charged once.
    headers = {**HEADERS, "Idempotency-Key": attempt_id}
    for attempt in range(6):
        res = requests.post(f"{API}/api/problems/{problem}/submissions/", headers=headers, json=body, timeout=30)
        # 429: your plan's limits are busy. Wait, then try again (5 times at most).
        if res.status_code != 429 or attempt == 5:
            break
        time.sleep(0.5 * 2**attempt)
    # 404: not your problem. 409: archived, or no published version yet. 429: still busy.
    # 400: a compiler the problem does not allow, or "samples" on a problem with no visible test.
    res.raise_for_status()
    submission = res.json()["submission"]  # 202: queued (a retry with the same key gets the same submission)
    deadline = time.monotonic() + WAIT_SECONDS
    while True:
        if time.monotonic() > deadline:
            # A tool call must not hang. The submission may still be judged later: look it up,
            # or DELETE it while it is queued.
            raise TimeoutError(f"Submission {submission} was not judged within {WAIT_SECONDS} seconds")
        time.sleep(2)
        res = requests.get(f"{API}/api/submissions/{submission}/", headers=HEADERS, timeout=30)
        res.raise_for_status()  # 404: the submission was deleted
        sub = res.json()
        if sub["status"] in ("finished", "failed"):
            break
    if sub["status"] == "failed":
        # Not judged: a failed submission has an error and no result, so there are no verdicts to give.
        return {"status": "failed", "error": sub["error"]}
    result = sub["result"]
    return {
        "status": "finished",
        # Plain sentences built from the verdicts by fixed rules: sample results, hidden counts.
        "summary": result["summary"],
        # A hidden test carries only its verdict, so it is left out; the summary counts it.
        "visible_tests": [
            {"verdict": test["verdict"], "output": test.get("output")}
            for test in result["tests"]
            if not test["hidden"]
        ],
    }

A Submit to the problem above, from a Java answer that adds two ints, returns this. The AI knows a hidden test failed, not which one or why, so it can ask the candidate about edge cases without giving the test away:

{
  "status": "finished",
  "summary": "2 of 2 sample tests passed. 1 of 2 hidden tests passed.",
  "visible_tests": [
    {"verdict": "accepted", "output": "3\n"},
    {"verdict": "accepted", "output": "12\n"}
  ]
}

The summary is built by fixed rules from the verdicts, with no AI involved. It says how many sample tests passed, what went wrong on the first sample that failed (a Run whose second sample printed 2 gets "1 of 2 sample tests passed. Sample 2 expected 12 but the program printed 2."), and how many hidden tests passed. It never holds a hidden test's input, expected output or output. If you want the AI to help with a compile error or a crash, also pass compile_error and each visible test's error: the API fills them in only when a visible test was judged.

Treat what the program printed as untrusted text. A candidate's program can print anything, including words aimed at your AI, and the summary can quote a short line of its output. Pass the result to the AI as data, not as instructions.

Calling it from an agent framework (MCP)

The MCP server has a judge_problem tool that does the judging step for an agent framework or an MCP client: it takes problem (the id or slug), compiler, code and an optional mode (all, or samples for the visible tests only), judges the code against your stored problem with the key the connection uses, waits for the result and returns it, with the summary, a verdict for each test (a hidden test has its verdict and nothing else), the submission id, the signed receipt and the problem id and version. A problem that is missing, archived or unpublished comes back as an error, never as an invented result.

Keep the decision in your backend all the same. This tool takes the problem and the code from the model's arguments, so a model can pick another problem or change the code it is judging, and it judges for the record only when the model calls it. It suits an agent you trust with those, such as a practice tutor trying a learner's code against the samples. For an interview, let your backend decide when a Run or a Submit is judged, as in step 2, and give the AI the result function from step 3. Because the tool waits instead of queueing, a problem with a custom checker or many tests belongs on the backend path, with async.

4. Long runs and retries

A problem with a custom checker or many tests takes longer to judge, so the function above sends async: true: the answer is 202 with a submission id, and the result is read with GET /api/submissions/{id}/. Instead of polling, send a callback_url and the finished submission is posted to you in a signed webhook, retried if your endpoint does not answer, so your AI can keep the conversation going while the code is judged and pick up the result when it arrives.

Networks drop requests and tool calls get retried. Send an Idempotency-Key header, such as the attempt id, and a request repeated within 24 hours with the same body runs nothing and is not counted again: it answers with the original submission, so one Run or Submit is judged and charged once. Use a new key for each new Run or Submit. A submission that ends failed was not judged and frees its key, so sending it again runs it. The retry rules have the details.

5. A record you can show

Every judged submission gets a signed receipt: a statement, signed with our Ed25519 key, that names the problem, the version and whether all tests or only the samples were judged, with the hash of the candidate's code, a verdict for each test and the score. Anyone can verify it offline against our public keys, and sharing a submission publishes its receipt on a link, without the code, the tests or the output. Keep the receipt with your interview record: if a candidate or a hiring manager asks how a verdict was reached, it shows that this code got these verdicts against this version of the problem, and that the result has not been edited since.

It does not prove who wrote the code, or whether the candidate had help. A receipt for a Run (samples) vouches for no hidden test.

6. Candidate privacy

• A candidate's code can identify them, so treat it as personal data, along with anything you put in external_ref or metadata. Use opaque ids, such as an attempt id, never a name or an email: external_ref is signed into the receipt and is published with it if you share it.
• DELETE /api/submissions/{id}/ deletes a submission with the code, input and output kept for its tests, and its receipt stops being public. A running one answers 409, so delete it once it has finished. Runs already counted stay counted, and a copy of the receipt you saved still verifies.
• Judged submissions are kept for 90 days on Free, 365 days on Starter, 730 days on Pro and 730 days on Business, then deleted.
• Our DPA and sub-processors are published.

7. What it costs

Each judged test is one run against your plan's monthly runs and its per-minute and pending limits: a Run with samples costs one run per visible test, and a Submit one run per test. A custom checker is not charged, and a reference check costs one run per test when you publish. A candidate needs no account of ours: your key submits for them, so the runs are yours.

Say a problem has 2 sample tests and 8 tests in all (the most a Free submission takes), and in one interview the candidate presses Run 5 times and submits 2 times. That is 5 × 2 + 2 × 8 = 26 runs.

The runs a minute matter when many candidates submit at once: 30 runs a minute on Free is at most 3 Submits of 8 tests in one minute, and 120 on Pro is at most 15; the pending and at-once limits can make it fewer. An async submission past that waits in your account's queue (up to 80 submissions on Free) instead of being refused. The Free plan is enough to build and try the whole flow; Pricing has the rest of each plan.

What you build yourself

The API judges code. The interview around it is yours: the AI and its prompts, the editor, the candidate's session and timing, which questions an interview uses, and the hiring decision. There is no proctoring and no plagiarism or cheating detection here; a verdict says what the code did against your tests, not how it was written.

Building a practice site or a contest instead? How to build an online judge uses the same stored problems with a leaderboard, and the coding assessment API page covers receipts in depth.

Frequently asked questions

Does the AI ever see the hidden tests?

Not through judging. A submission's result gives a hidden test only its verdict, and the summary gives only how many hidden tests passed. The hidden tests are readable only with your API key, through GET /api/problems/{problem}/, so keep the key on your backend and never make that call a tool the AI can use. Drivers are not shown to candidates either, but a candidate's program can read and print its own driver on a visible test, so keep test data out of drivers.

Can the AI judge the code itself?

It can talk about the code, but it should not decide whether the code works. Here every verdict comes from running the code against your tests in a sandbox and checking the output with the problem's checker, and the summary is built from those verdicts by fixed rules. No AI takes part in judging.

Which languages can candidates use?

Any of the 46 the API supports, including Python, Java, C++, JavaScript, Go and SQL. A problem's languages field limits a question to the ones you choose; a submission in another language is refused with 400.

What does a judged answer cost?

One run per judged test: a Run on the samples costs one run per visible test, and a Submit one run per test. Checking with a custom checker is not charged. The Free plan has 100,000 runs a month, and paid plans start at $10 a month for 150,000.

Can a candidate's code be deleted?

Yes. DELETE /api/submissions/{id}/ deletes a submission with the code, input and output kept for its tests, and its receipt stops being public. Otherwise judged submissions are deleted after their plan's retention: 90 days on Free, 365 days on Starter, 730 days on Pro and 730 days on Business.

Can I use my own checker for questions with many right answers?

Yes. Set checker to custom and give a custom_checker, plus a reference solution that passes it. The checker reads the input, the expected output and the candidate's output, exits 0 to accept, and can print a credit from 0 to 1 for partial scores. It gets 5 seconds per test and is not charged as runs. If the checker itself fails, the candidate gets checker_error, never a wrong answer, and the summary says only that the problem could not be judged.

Does each candidate need an account?

No. Your API key submits for them, so candidates need no account of ours, and the runs count against your plan. Keep the key on your server.