Run tests: judge code against test cases

Run one program against a list of tests and get a verdict for each, an overall verdict and a score, in one request. It authenticates like the sync endpoint and always waits for the result. Each test runs in its own fresh sandbox, and each test counts as one run against your monthly allowance and your per-minute and pending limits. Each test takes one of your account's pending slots when the request arrives and frees it when that test finishes, so on Free one 8-test request starts out using all of them, and your other runs (the widget's included) wait or get 429 until its tests start finishing.

Limits for one request

The last column is for a language with the default round overhead of 3 seconds and no start-up allowance, such as Python or C++. Languages that compile slowly fit fewer tests (see below).

• The number of tests is capped by your plan's pending limit. Tests run in the order you list them, at most as many at once as the table shows, and never more than your plan's runs at once.
• One request has a 60-second budget. Tests run in rounds (tests divided by tests at once, rounded up), and a round takes up to time_limit + start-up allowance + the language's round overhead for compiling and starting. All the rounds must fit in 60 seconds, and the last must be able to start within 45. A request over budget gets 400 with the largest time_limit that would fit.
• The round overhead is 3 seconds, except for languages that compile slowly: Crystal (13 s), Kotlin (6 s), Nim (5.5 s), Zig (4.5 s), Scala (4 s), C# (4 s) and Visual Basic (4 s). Each test compiles the program again, so these fit fewer tests; at the default time_limit, a Crystal request holds at most 12 tests on Pro.
• C, C++ and Java compile once per request when there are more tests than run at once: one sandbox compiles the program, without any test input, then each test runs the build in its own fresh sandbox. If a C or C++ compile fails, the first test gets the verdict and counts as one run, and the rest are skipped; a Java program that fails to compile runs as before, compiling in every test. A request whose tests all run at once compiles in every test, which is faster for so few tests.
• No test starts after 45 seconds from the request's arrival. Tests that never started are skipped, are not counted as runs and leave no history. When the cut-off skipped tests and none failed, the overall verdict is incomplete. The budget leaves time for every round, so this usually means tests waited for a sandbox, held by other users or by your account's other runs: retry the request. A retry runs, and counts, every test again.
• A compilation error stops the remaining tests, because they would fail the same way. With stop_on_failure, the first test that is not accepted does too.
• Inputs and expected outputs together may hold at most 1 MB.
• A bad request answers 400 with {"error": "..."} and starts nothing. Account limits and a busy host answer 429, and nothing runs.

Verdicts

Each test gets one verdict, decided in the order of this table (accepted and wrong_answer last). The last column is the Judge0 status id it maps to.

Checkers and scoring

Response (200)

{
  "verdict": "runtime_error",
  "score": 33.33,
  "passed": 1,
  "total": 2,
  "time_limit": 2,
  "compile_error": null,
  "tests": [
    {"verdict": "accepted", "hidden": false, "time": "0.0213", "memory": "9000",
     "exit_code": 0, "signal": null, "output": "3
", "error": "", "truncated": false},
    {"verdict": "runtime_error", "hidden": true}
  ]
}

Inputs and expected outputs are not sent back. output and error are cut to the first 4096 characters, and truncated is true when either was cut. time and memory are strings, as on the sync endpoint. The response above is for the examples below: the hidden test puts the two numbers on separate lines, which the program does not read, so it ends in a runtime_error and the score is 1 of 3 weight points.

Start-up allowance for Java, Groovy and F#

These three compile as they start, inside the timed run, so even a hello-world uses part of a small time_limit. They get extra seconds on top of the limit: Java (1 s), Groovy (1.5 s) and F# (2.5 s). The run is stopped at time_limit plus the allowance, and a test is time_limit_exceeded only past that. time is still reported as measured, start-up included, and the response's time_limit is the one you asked for. The allowance applies wherever a time limit is honoured: here, the native time_limit on the run endpoints, and the Judge0 and Piston layers. In run-tests requests that compile once, Java compiles before the timed run, so its time is lower there; the allowance still applies everywhere for now. We will remove it once compiling always happens before the timed run.

Examples

curl -X POST https://api.onlinecompiler.io/api/run-tests/ \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "compiler": "python-3.14",
    "code": "a, b = map(int, input().split())\nprint(a + b)",
    "tests": [
      {"input": "1 2\n", "expected_output": "3\n"},
      {"input": "5\n7\n", "expected_output": "12\n", "hidden": true, "weight": 2}
    ],
    "scoring": "weighted"
  }'

const res = await fetch('https://api.onlinecompiler.io/api/run-tests/', {
  method: 'POST',
  headers: { Authorization: 'YOUR_API_KEY', 'Content-Type': 'application/json' },
  body: JSON.stringify({
    compiler: 'python-3.14',
    code: 'a, b = map(int, input().split())\nprint(a + b)',
    tests: [
      { input: '1 2\n', expected_output: '3\n' },
      { input: '5\n7\n', expected_output: '12\n', hidden: true, weight: 2 }
    ],
    scoring: 'weighted'
  })
});
const result = await res.json();
console.log(result.verdict, result.score, result.passed + '/' + result.total);