Autograder API: grade programming assignments against test cases

Grade a submission in one request

The assignment: print the largest of a line of numbers, or empty when there are none. Tests are worth 2, 3 and 5 points, and the 5-point test is hidden. Get an API key by signing in, then send the student's code with the tests:

curl -X POST https://api.onlinecompiler.io/api/run-tests/ \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "compiler": "python-3.14",
    "code": "nums = list(map(int, input().split()))\nprint(max(nums))",
    "tests": [
      {"input": "3 9 4\n", "expected_output": "9\n", "weight": 2},
      {"input": "-5 -2 -8\n", "expected_output": "-2\n", "weight": 3},
      {"input": "\n", "expected_output": "empty\n", "weight": 5, "hidden": true}
    ],
    "scoring": "weighted"
  }'

The submission passes both visible tests and crashes on the hidden empty line, so it earns 5 of 10 points. Each test ran in its own fresh sandbox:

{
  "verdict": "runtime_error",
  "score": 50.0,
  "passed": 2,
  "total": 3,
  "time_limit": 2,
  "compile_error": null,
  "tests": [
    {"verdict": "accepted", "hidden": false, "time": "0.0204", "memory": "9000",
     "exit_code": 0, "signal": null, "output": "9\n", "error": "", "truncated": false},
    {"verdict": "accepted", "hidden": false, "time": "0.0198", "memory": "9000",
     "exit_code": 0, "signal": null, "output": "-2\n", "error": "", "truncated": false},
    {"verdict": "runtime_error", "hidden": true}
  ]
}

score is the share of points earned. To record points rather than a percentage, multiply it by the assignment's total: 50 × 10 / 100 = 5.

Points, partial credit and hidden tests

• Points: each test's weight is its points with "scoring": "weighted". percent counts every test the same, and all_or_nothing, the default, stops at the first failure.
• Hidden tests: return only their verdict, so students can't print the hidden inputs and hard-code the answers.
• Comparing output: trim, the default, ignores trailing spaces and blank lines; tokens ignores all spacing and line breaks; float accepts numbers within a tolerance; exact compares byte for byte.
• Verdicts students understand: accepted, wrong_answer, time_limit_exceeded, memory_limit_exceeded, runtime_error and compilation_error, with the program's output and errors for visible tests.
• Time limits: 2 seconds per test by default and up to 30, so a slow solution fails a large test the way it would on a judge.

The full list of fields, checkers and verdicts is in the run tests API reference.

Grade a folder of submissions into a gradebook

Put the tests in tests.json (the same list as "tests" above) and one file per student in submissions/. This script grades every file and writes grades.csv:

import csv, json, pathlib, time
import requests  # pip install requests

API = "https://api.onlinecompiler.io/api/run-tests/"
KEY = "YOUR_API_KEY"
tests = json.loads(pathlib.Path("tests.json").read_text())

with open("grades.csv", "w", newline="") as f:
    grades = csv.writer(f)
    grades.writerow(["student", "verdict", "passed", "score"])
    for path in sorted(pathlib.Path("submissions").glob("*.py")):
        body = {"compiler": "python-3.14", "code": path.read_text(),
                "tests": tests, "scoring": "weighted"}
        while True:
            res = requests.post(API, json=body, headers={"Authorization": KEY}, timeout=90)
            if res.status_code != 429:
                break
            time.sleep(15)  # over the per-minute or pending limit: wait, then retry
        res.raise_for_status()
        result = res.json()
        grades.writerow([path.stem, result["verdict"],
                         f"{result['passed']}/{result['total']}", result["score"]])

Each test counts towards your plan's submissions a minute, so the plan sets the pace: 300 submissions of 10 tests take at least 100 minutes on Free, 50 on Starter and 25 on Pro. The script waits and retries when it gets HTTP 429.

Moving from GitHub Classroom autograding

GitHub decommissioned GitHub Classroom on 28 August 2026, and tests defined outside repositories were deleted with it. If your course used its input/output tests, each one maps onto a test here:

What it doesn't do

• Rosters, repositories and deadlines: your course site or LMS keeps them.
• Grade passback: nothing posts to Canvas, Moodle or Blackboard; your code sends the score.
• Plagiarism checks: compare submissions with a tool such as MOSS.
• Packages and the internet: each run has the language's standard library, 512 MB of memory and no network. The security page has the details.

Building something closer to LeetCode, with problems and a submission queue? See how to build an online judge.

Frequently asked questions

What is an autograder API?

An HTTP API that runs a student's program against an assignment's test cases and returns the results, so your course site, LMS integration or grading script never runs student code itself. OnlineCompiler.io's returns a verdict for each test, an overall verdict and a score from 0 to 100, in 45 languages.

How do I give partial credit?

Give each test a weight equal to its points and send "scoring": "weighted". The score is the share of points earned, from 0 to 100, so a student who passes tests worth 5 of 10 points scores 50. "percent" counts every test the same, and "all_or_nothing" (the default) gives 100 or 0.

Can students see the hidden tests?

No. A hidden test comes back with only its verdict: no output, error, time or memory a student could read the input from. Compiler errors are returned only when at least one test is visible.

How much does autograding a course cost?

Each test is one run. 300 students making 30 graded submissions each against 10 tests, plus 50 other runs each, is 105,000 runs a month: Starter, at $10 a month. Smaller classes fit in the free 100,000 runs. The estimator on the pricing page works it out for your numbers.

Does it replace GitHub Classroom autograding?

It replaces the input/output tests: the program, its input, the expected output, a timeout and points. It does not replace rosters, assignment repositories or deadlines, which your course site or LMS keeps.

Can it run unit tests, such as pytest or JUnit?

Not as frameworks: each test compares what the program prints with the expected output, and runs have no package installs. To test a function, append a short driver to the submission that calls it and prints the result, and compare that.