Stored problems: tests kept on the server

Run tests takes the tests in each request. A stored problem keeps them for you: you create it once, publish a version, and then POST /api/problems/{problem}/submissions/ with a compiler and code. Submissions, queueing, webhooks, Idempotency-Key and signed receipts work as they do for run-tests. Every endpoint takes your API key in the Authorization header, and the OpenAPI spec lists them all. Stored problems are not available through RapidAPI: a request through it answers 403.

Building an AI interviewer? Code execution for AI interviews shows how your backend judges a candidate's code against a stored problem and gives the AI only the summary and the visible tests. To put the problem in a candidate's browser, see problem mode in the widget and sessions.

Endpoints

{problem} is the problem's id (prob_ and 24 characters) or its slug. Another account's problem, and a reference of any other shape, answers 404, so ids cannot be probed. A request body over 4 MB answers 400 in JSON; send a big input alone with the raw upload.

What a write answers. Creating a problem, PATCH, both draft PUTs, archive and restore answer with the problem as GET shows it, except that each test of the draft and the current version has input_bytes, input_sha256, expected_output_bytes and expected_output_sha256 (its size in UTF-8 bytes and its SHA-256 in hex) in place of input and expected_output, so a write never sends your test data back. GET /api/problems/{problem}/ returns the data itself.

How much you can store

An active problem counts toward your plan's problem limit; an archived one does not. Test data (inputs and expected outputs) counts toward a data limit. It is stored once per account and content, so an input that two versions share is counted once. Only data that some test uses counts, in any version and in archived problems too; data no test uses any more (say, the old draft's after you replace it) does not count, and is removed after a while. A change that would pass a limit answers 403 with the limit and a link to upgrade.

The draft

A problem has at most one draft. POST /api/problems/ creates it, and PUT .../draft/ replaces all of it with the same fields (no title). Fields you leave out take their defaults. A draft is not judged against until you publish it.

{
  "title": "Sum of two numbers",
  "slug": "sum",
  "public": true,
  "statement": "Write solve(a, b), which returns the sum of two integers.",
  "languages": ["python-3.14"],
  "scoring": "percent",
  "tests": [
    {"name": "sample", "input": "1 2\n", "expected_output": "3\n", "hidden": false},
    {"input": "5 7\n", "expected_output": "12\n"},
    {"input": "-4 4\n", "expected_output": "0\n"}
  ],
  "starter_code": {"python-3.14": "def solve(a, b):\n    pass\n"},
  "drivers": {"python-3.14": "##CODE##\n\na, b = map(int, input().split())\nprint(solve(a, b))\n"},
  "reference_solution": {"compiler": "python-3.14", "code": "def solve(a, b):\n    return a + b\n"}
}

Big inputs. An input of up to 5 MB does not fit in a JSON body with others. Create the test with a short input, then send the real one as the raw body, with Content-Type: text/plain, to PUT /api/problems/{problem}/draft/tests/{n}/, with a Content-Length header (a chunked upload answers 411). It must be UTF-8 without NUL, and the draft must stay within 20 MB. The problem needs a draft: after publishing, PUT .../draft/ a new one first.

Publishing and the reference check

POST .../publish/ numbers the draft (one more than your highest version) and turns it into a version. A version is draft, checking, published or rejected, and the newest published one is the problem's current version, the one submissions are judged against. Earlier versions stay readable.

• Without a reference solution the version is published at once (200), with a warning that the expected outputs were not checked.
• With one, the version is checking and the answer is 202 with the check's submission id. The reference solution is judged against every test as an async submission in your account's queue, and counts as one run for each test, like any submission. If it is accepted on every test the version is published and becomes current. Otherwise it is rejected and keeps its number; GET /api/problems/{problem}/ shows each test's verdict and time in the version's check, hidden tests included, and a message that says why: the reference solution did not pass every test, the custom checker failed (its checker_log says how), the checker did not compile, or the check could not finish because of an error on our side, which publishing again fixes. A warnings entry names each test whose time limit is less than twice the reference solution's time.
• If a version is rejected, its content becomes the draft again (unless the problem already has one), so fix what the check found with PUT and publish again, without sending the tests again; the rejected version keeps its number and is not reused. The check's own submission, which GET /api/submissions/{id}/ returns by the id in check.submission, has the per-test verdicts and any checker_log.
• Problems and versions are never deleted, only archived, so a rejected version keeps its test data and that data counts toward your plan's test data limit. An archived problem takes no submissions and is not published to: restore it first.
• While a version is checking you cannot change the draft or publish again (409 or 400 with the version's number); deleting the check's submission rejects the version.
• Publishing answers 409 for an archived problem, 400 if the draft has more tests than your plan allows today or if its tests could take longer than one submission may run (60 seconds) in every language it allows, and 429 if your queue has no room for the check.

# No reference solution: published at once (200)
{"problem": "prob_7Hq2Zp4Wn8Ks1Ab6Ct3Dv5Ex", "version": 1, "state": "published",
 "warnings": ["No reference solution: the expected outputs were not checked."]}

# With a reference solution: its check is queued (202)
{"problem": "prob_7Hq2Zp4Wn8Ks1Ab6Ct3Dv5Ex", "version": 2, "state": "checking",
 "check": "sub_3k9Fj2Lq8Rt5Wm1Zn4Bc6DxY"}

Drivers

A driver is a program of yours with ##CODE## where the candidate's code goes, so a candidate can write only a function and your driver reads the input, calls it and prints the answer. A driver without ##CODE## is refused. The first mark is replaced; if only spaces and tabs precede it on its line, every later line of the candidate's code gets that indentation, as with the widget's templates. A submission with no driver for its compiler runs the candidate's code as it is.

In compiler and runtime messages that name the program's file, a line number in the candidate's code is shifted back to count from the first line of that code; a line of the driver keeps its number in the whole program. Drivers are not shown to candidates, but they are not secret. A candidate's program can read its own source, driver included, and print it on a visible test, and a compile error or a visible test's error can quote a driver line, so keep test data and anything sensitive out of drivers. Hidden tests stay secret: a candidate gets only their verdicts.

Checkers

The built-in checkers (trim, exact, tokens, float, rows) work as in run-tests. For answers that are not one fixed text, such as "any valid path", set checker to custom and give a custom_checker program and a reference_solution. The reference solution has to pass your checker before the version can be published.

• Input. For each test that the program ran cleanly, the checker sees the test's input, the expected output and the program's output as read-only files: /judge/input, /judge/expected and /judge/output. The input is also on stdin, except for the SQL compilers (sqlite-3, duckdb-1.5), whose checkers get it only at /judge/input.
• Exit code. 0 accepts the output. 1 or 2 is a wrong answer. Anything else is a checker error.
• Credit. A first line of stdout that is a plain number from 0 to 1 is the test's credit, which counts instead of 1 or 0 under percent and weighted scoring and is ignored by all_or_nothing. More than 64 KB of stdout is a checker error.
• Time. Each test gets 5 seconds, plus the start-up allowance the language has for every run. All the tests of one submission are checked in one checker sandbox, which has 30 seconds in all. A checker sandbox is not charged to your runs.
• Languages. A checker can be written in any supported language. Python, JavaScript (Node), C, C++ and Java are recommended: C, C++ and Java checkers are compiled once, during the reference check, and the build is kept for every later submission, and interpreted languages start fast. Other compiled languages (Rust, Go, Kotlin, C# and so on) compile again for every test inside the checker sandbox's 30 seconds, so on a problem with many tests they can run out of time and give checker errors. A checker that does not compile, or never starts, is a checker error; a compile failure during the reference check rejects the version and puts the compiler's output in the check's message. The exception is Java: a checker that javac cannot build on its own (one that needs the single-file launcher) runs through the launcher instead, compiling for every test, and if it does not compile there either, the launcher exits with 1 in every test, which reads as a wrong answer. The reference check then rejects the version as if the reference solution had failed, and the compiler's output is in the check's checker_log (GET /api/submissions/{id}/), not in its message.
• Catch your own errors. Python, Node, Ruby and Java exit with code 1 on an uncaught exception, which would read as a wrong answer. Catch errors in your checker and exit with 3, as below.

#!/usr/bin/env python3
# A checker for "print any two numbers that add up to the input".
import sys

try:
    wanted = int(open("/judge/input").read())
    a, b = map(int, open("/judge/output").read().split())
except Exception as problem:
    print(problem, file=sys.stderr)  # shows in checker_log, for you only
    sys.exit(3)  # a checker error, not a wrong answer

if a + b == wanted:
    print(1)  # credit for this test, 0 to 1
    sys.exit(0)
sys.exit(1)

When a checker fails

A test the problem or its checker could not judge gets the verdict checker_error, or problem_error when the problem itself is broken (its custom checker does not compile, which the reference check finds before the version is published). It is never the candidate's fault, and the overall verdict is problem_error or checker_error whatever the other tests did; the summary says only that the problem could not be judged. A failure on our side is internal_error, which is not the author's fault either.

• You, the owner, see why: GET /api/submissions/{id}/ returns checker_log with what the checker wrote to stderr and a line for each test whose checker failed (it ran too long, did not start, printed too much, or exited with another code). No other response, webhook or receipt has it.
• You get at most one email a day listing the problems that had checker errors.

The public view

GET /api/problems/{problem}/public returns what a candidate's page needs: the title, version, statement, languages, starter code, the checker's name, scoring, the time limit, the visible tests (the samples), and for the hidden ones only hidden_tests and total_tests. Never a driver, a reference solution, checker code or a check record. A public, active problem with a published version answers anyone who asks by its id, with no key: JSON readable from any origin and cacheable for a minute (the slug does not work there without a key). Anything else needs your key, so a caller without it cannot tell a private problem from a missing one; with it you can preview a private or archived one.

Submitting code

POST /api/problems/{problem}/submissions/ takes compiler and code, an optional mode, and the run-tests options async, callback_url, external_ref and metadata, with the Idempotency-Key header (the same body sent to another problem with the same key is a 409). The tests, checker, scoring and limits come from the problem's current version. The answer is the run-tests response with problem and version added, and a signed receipt.

• mode is all (default) or samples. samples judges only the visible tests, so a candidate can try code before submitting; the receipt says it judged only the samples, and a problem with no visible test refuses it with 400.
• Sync or async. A synchronous request waits for the result. For a problem with a custom checker, send async: true: a synchronous request also waits for the checker's sandbox after the tests. Async submissions, webhooks and lookups work as for run-tests; a queued submission is judged against the version that was current when it was accepted.
• Order. Every visible test runs before any hidden test starts, so a candidate's program never runs on a hidden test while it runs on a visible one, whose output the answer shows. The answer and the receipt still list the tests in the problem's order, and the time a submission may take counts the visible and the hidden tests one group after the other.
• Refusals. 404 for a problem that is not yours or does not exist; 409 for an archived problem or one with no published version yet; 400 for a compiler outside the problem's languages, an unknown mode, code that is too long once the driver is added, or a problem whose tests could take longer than one submission may run on your plan (lower the problem's time limits or use fewer tests).
• What counts. Each judged test is one run against your monthly allowance and your per-minute and pending limits, as in run-tests; samples counts only the visible tests. The custom checker is not counted. A candidate has no account: your key submits for them, so the runs are yours.
• Retention. Judged submissions are kept for the plan's period (Free 90 days, Starter 365 days, Pro 730 days, Business 730 days), as for run-tests. Problems and versions have no delete, only archive, so they and the test data they use are kept while your account exists.

Sessions: letting a candidate's page submit

A candidate's browser must never hold your API key, and must never be trusted to report its own score. A session lets one candidate's widget Submit to one problem without either: your server creates the session with your API key and gives the page only the session's token.

• Create. POST /api/problems/{problem}/sessions/ takes external_ref (required, up to 200 characters: your reference for the candidate or attempt), and optionally metadata (up to 4 KB of JSON), callback_url, expires_in and max_submits. The answer is 201 with the session and its token. The token is shown once and kept only as a hash, so it cannot be read again: create a new session if it is lost. 409 for an archived problem or one with no published version yet.
• Limits. A session lasts expires_in seconds, from 60 up to 24 hours (60 minutes if left out), and takes max_submits Submits, from 1 to 20 (1 if left out). A candidate's Run samples is not a Submit and does not use the session up.
• Pinning. A session is pinned to the version that was current when you created it. Publishing a newer version later does not change what that candidate is judged against.
• The result. A Submit judges every test, in your account's queue, as an async submission of the API key that created the session, with the session's external_ref and metadata. The signed webhook goes to the session's callback_url, or to the key's own webhook if there is none. Record the score only from that webhook or from GET /api/submissions/{id}/, never from what the browser reports. The runs count against your allowance, as for any submission.
• Give-back. Each Submit uses one of the session's Submits when it is accepted. A Submit refused after that, and one that failed on our side, gives it back, so a candidate does not lose an attempt to an error that was not theirs.
• Read and revoke. GET shows submits_used, the expiry and the ids of the session's submissions, never the token. DELETE revokes the session at once: its next Submit is refused, submissions already made are judged as usual, and the session stays readable. A session of another problem or account answers 404.
• Retention. A session and its reference (its external_ref, metadata and callback_url) are kept for your plan's retention after the session expires, the same period as judged submissions above, and then deleted; reading it after that answers 404. Its submissions are not deleted with it: they keep their own copy of the reference and follow their own retention.

A widget opens only problems of the widget key's own account, and a widget key is refused on this API. A candidate sees the problem's public view and, for the hidden tests, verdicts only.

Receipts

A stored problem's receipt has a problem block in place of a top-level tests_sha256, and it is covered by the signature. mode is all or samples, so a receipt for a samples run says it judged only the samples. settings.time_limit is the version's default; a test's own limit is pinned by the signed problem id and version.

"problem": {
  "id": "prob_7Hq2Zp4Wn8Ks1Ab6Ct3Dv5Ex",
  "version": 1,
  "mode": "all",
  "tests_sha256": "9b1f..."
}

Example: create, publish, submit

import time
import requests

API = "https://api.onlinecompiler.io"
H = {"Authorization": "YOUR_API_KEY"}

draft = {
    "title": "Sum of two numbers",
    "slug": "sum",
    "languages": ["python-3.14"],
    "tests": [
        {"name": "sample", "input": "1 2\n", "expected_output": "3\n", "hidden": False},
        {"input": "5 7\n", "expected_output": "12\n"},
    ],
    "reference_solution": {"compiler": "python-3.14", "code": "a, b = map(int, input().split())\nprint(a + b)"},
}
created = requests.post(f"{API}/api/problems/", json=draft, headers=H)
created.raise_for_status()

res = requests.post(f"{API}/api/problems/sum/publish/", headers=H)
res.raise_for_status()
number = res.json()["version"]
while True:  # the reference check runs as a queued submission
    problem = requests.get(f"{API}/api/problems/sum/", headers=H).json()
    version = next(v for v in problem["versions"] if v["version"] == number)
    if version["state"] != "checking":
        break
    time.sleep(5)
if version["state"] == "rejected":
    # Fix the draft and publish again; don't submit against a rejected version.
    raise SystemExit(f"Version {number} was rejected: {version['check']['message']}")

code = "a, b = map(int, input().split())\nprint(a + b)"
res = requests.post(f"{API}/api/problems/sum/submissions/", headers=H,
                    json={"compiler": "python-3.14", "code": code, "async": True})
res.raise_for_status()
while True:
    sub = requests.get(f"{API}/api/submissions/{res.json()['submission']}/", headers=H).json()
    if sub["status"] in ("finished", "failed"):
        break
    time.sleep(2)
if sub["status"] == "failed":
    raise SystemExit(f"Not judged: {sub['error']}")  # a failed submission has an error and no result
print(sub["result"]["verdict"], sub["result"]["score"])

const API = 'https://api.onlinecompiler.io';
const headers = { Authorization: 'YOUR_API_KEY', 'Content-Type': 'application/json' };
const call = async (path, method = 'GET', body) => {
  const res = await fetch(API + path, { method, headers, body: body && JSON.stringify(body) });
  if (!res.ok) throw new Error(res.status + ' ' + (await res.text()));
  return res.json();
};
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

const code = 'a, b = map(int, input().split())\nprint(a + b)';
await call('/api/problems/', 'POST', {
  title: 'Sum of two numbers',
  slug: 'sum',
  languages: ['python-3.14'],
  tests: [
    { name: 'sample', input: '1 2\n', expected_output: '3\n', hidden: false },
    { input: '5 7\n', expected_output: '12\n' }
  ],
  reference_solution: { compiler: 'python-3.14', code }
});

const published = await call('/api/problems/sum/publish/', 'POST');
let version = published;
while (version.state === 'checking') {
  await sleep(5000); // the reference check runs as a queued submission
  const problem = await call('/api/problems/sum/');
  version = problem.versions.find((v) => v.version === published.version);
}
if (version.state === 'rejected') {
  // Fix the draft and publish again; don't submit against a rejected version.
  throw new Error('Version ' + version.version + ' was rejected: ' + version.check.message);
}

const sub = await call('/api/problems/sum/submissions/', 'POST', { compiler: 'python-3.14', code, async: true });
let result;
do {
  await sleep(2000);
  result = await call('/api/submissions/' + sub.submission + '/');
} while (result.status === 'queued' || result.status === 'running');
if (result.status === 'failed') {
  throw new Error('Not judged: ' + result.error); // a failed submission has an error and no result
}
console.log(result.result.verdict, result.result.score);