pytest, Jest and JUnit Autograder API: Grade Whole Projects

Grade a Flask app with pytest

The task: a small Flask API that stores items. The student writes app.py; your tests sit beside it in tests/test_app.py and use Flask's own test client.

app.py (the student's)

from flask import Flask, jsonify, request

app = Flask(__name__)
items = []


@app.post("/items")
def add_item():
    name = request.get_json().get("name", "").strip()
    if not name:
        return jsonify(error="name is required"), 400
    items.append(name)
    return jsonify(items=items), 201


@app.get("/items")
def list_items():
    return jsonify(items=items)

tests/test_app.py (yours)

from app import app

client = app.test_client()


def test_add_item():
    r = client.post("/items", json={"name": "milk"})
    assert r.status_code == 201
    assert r.get_json()["items"] == ["milk"]


def test_list_items():
    assert "milk" in client.get("/items").get_json()["items"]


def test_empty_name_is_refused():
    r = client.post("/items", json={"name": "  "})
    assert r.status_code == 400

Send both files to the python-3.14-tests pack with the pytest preset. With the Python client (pip install onlinecompiler):

grade.py

from pathlib import Path
from onlinecompiler import OnlineCompiler

client = OnlineCompiler()  # reads ONLINECOMPILER_API_KEY
names = ["app.py", "tests/test_app.py"]
result = client.run_suite(
    "python-3.14-tests",
    [{"name": name, "content": Path(name).read_text()} for name in names],
    preset="pytest",
)
print(result["summary"])  # 3 of 3 tests passed.
for test in result["tests"]:
    print(test["status"], test["name"])

That prints 3 of 3 tests passed. and each test with passed. pytest found the tests its usual way, ran them in one process, and its report came back parsed: no output to compare and nothing to scrape.

Grade an Express API with Jest and Supertest

The task: a signup endpoint that refuses invalid and repeated emails. This student forgot the repeated-email check.

app.js (the student's)

const express = require('express');

const app = express();
app.use(express.json());

const users = new Map();

app.post('/signup', (req, res) => {
  const { email } = req.body;
  if (!email || !email.includes('@')) return res.status(400).json({ error: 'invalid email' });
  users.set(email, { email });
  res.status(201).json({ email });
});

module.exports = app;

app.test.js (yours)

const request = require('supertest');
const app = require('./app');

describe('POST /signup', () => {
  test('creates a user', async () => {
    const res = await request(app).post('/signup').send({ email: 'ada@example.com' });
    expect(res.status).toBe(201);
  });

  test('refuses a second signup with the same email', async () => {
    const res = await request(app).post('/signup').send({ email: 'ada@example.com' });
    expect(res.status).toBe(409);
  });

  test('refuses an invalid email', async () => {
    const res = await request(app).post('/signup').send({ email: 'ada' });
    expect(res.status).toBe(400);
  });
});

Supertest starts the app on the sandbox's own 127.0.0.1 and sends it real HTTP requests. Express, Supertest and Jest are already installed in nodejs-24-tests, so you send no package.json and no node_modules. With the Node.js client (npm install onlinecompiler):

grade.mjs

import { readFileSync } from 'node:fs';
import { OnlineCompiler } from 'onlinecompiler';

const client = new OnlineCompiler(); // reads ONLINECOMPILER_API_KEY
const files = ['app.js', 'app.test.js'].map((name) => ({ name, content: readFileSync(name, 'utf8') }));
const result = await client.runSuite({ compiler: 'nodejs-24-tests', suite: { preset: 'jest' }, files });

console.log(result.summary, result.score); // 2 of 3 tests passed. 66.67
for (const test of result.tests) {
  console.log(test.status, test.name, test.message ?? '');
}

The answer, shortened:

Response

{
  "verdict": "failed",
  "gradable": true,
  "passed": 2, "failed": 1, "errors": 0, "skipped": 0, "total": 3,
  "score": 66.67,
  "summary": "2 of 3 tests passed.",
  "tests": [
    {"name": "creates a user", "classname": "POST /signup", "file": "app.test.js",
     "status": "passed", "time": 0.041, "message": null},
    {"name": "refuses a second signup with the same email", "classname": "POST /signup",
     "file": "app.test.js", "status": "failed", "time": 0.008,
     "message": "Error: expect(received).toBe(expected) // Object.is equality\n\nExpected: 409\nReceived: 201\n    at Object.toBe (app.test.js:12:24) ..."},
    {"name": "refuses an invalid email", "classname": "POST /signup", "file": "app.test.js",
     "status": "passed", "time": 0.004, "message": null}
  ],
  "run": {"exit_code": 1, "time": 1.42, "...": "..."},
  "compiler": "nodejs-24-tests",
  "preset": "jest",
  "pack": {"version": "1.1"},
  "time_limit": 20,
  "submission": "sub_...",
  "receipt": {"...": "..."}
}

The failed test carries Jest's own message, Expected: 409 and Received: 201, which tells the student exactly what to fix. classname holds the describe blocks, and score is the share of tests passed. Mocha and node:test work the same way with the mocha and node-test presets.

Grade an app that uses PostgreSQL

Many web tasks need a real database. Add "services": ["postgresql"] to the suite and a fresh PostgreSQL 18.6 server starts in the same sandbox before the tests, with DATABASE_URL set. Your .sql files under db/ run first, in name order, so every student starts from the same data. This is on Pro and above, and the exam tiers, with a memory_limit of at least 1,024 MB, since the database's memory counts in the run's.

The task: a Flask API over a books table. Your seed creates the table and two books:

db/01-books.sql (yours)

CREATE TABLE books (
    id serial PRIMARY KEY,
    title text NOT NULL,
    year integer NOT NULL
);
INSERT INTO books (title, year) VALUES ('Dune', 1965), ('Neuromancer', 1984);

The student maps it with Flask-SQLAlchemy, reading the database from DATABASE_URL:

app.py (the student's)

import os

from flask import Flask, jsonify, request
from flask_sqlalchemy import SQLAlchemy

app = Flask(__name__)
app.config["SQLALCHEMY_DATABASE_URI"] = os.environ["DATABASE_URL"]
db = SQLAlchemy(app)


class Book(db.Model):
    __tablename__ = "books"
    id = db.Column(db.Integer, primary_key=True)
    title = db.Column(db.Text, nullable=False)
    year = db.Column(db.Integer, nullable=False)


@app.get("/books")
def list_books():
    books = db.session.scalars(db.select(Book).order_by(Book.year)).all()
    return jsonify([{"title": b.title, "year": b.year} for b in books])


@app.post("/books")
def add_book():
    data = request.get_json()
    book = Book(title=data["title"], year=data["year"])
    db.session.add(book)
    db.session.commit()
    return jsonify(id=book.id), 201

tests/test_app.py (yours)

from app import app

client = app.test_client()


def test_lists_the_seeded_books_by_year():
    titles = [b["title"] for b in client.get("/books").get_json()]
    assert titles == ["Dune", "Neuromancer"]


def test_adds_a_book():
    r = client.post("/books", json={"title": "Hyperion", "year": 1989})
    assert r.status_code == 201
    assert "Hyperion" in [b["title"] for b in client.get("/books").get_json()]


def test_refuses_a_book_without_a_year():
    r = client.post("/books", json={"title": "Untitled"})
    assert r.status_code == 400

grade.py

from pathlib import Path
from onlinecompiler import OnlineCompiler

client = OnlineCompiler()  # reads ONLINECOMPILER_API_KEY
names = ["db/01-books.sql", "app.py", "tests/test_app.py"]
result = client.run_suite(
    "python-3.14-tests",
    [{"name": name, "content": Path(name).read_text()} for name in names],
    preset="pytest",
    services=["postgresql"],
    memory_limit=1024,  # the database's memory counts in the run's
)
print(result["summary"])  # 2 of 3 tests passed.

The student never checked that a year was sent, so the third test got a 500 where it expected a 400. The answer, shortened:

Response

{
  "verdict": "failed",
  "passed": 2, "failed": 1, "total": 3,
  "score": 66.67,
  "tests": [
    {"name": "test_lists_the_seeded_books_by_year", "status": "passed", "time": 0.024, "message": null},
    {"name": "test_adds_a_book", "status": "passed", "time": 0.008, "message": null},
    {"name": "test_refuses_a_book_without_a_year", "status": "failed", "time": 0.002,
     "message": "assert 500 == 400\n +  where 500 = <WrapperTestResponse streamed [500 INTERNAL SERVER ERROR]>.status_code"}
  ],
  "run": {"exit_code": 1, "time": 1.7149, "output": "", "error": "..F    [100%]\n=== FAILURES ===\n..."},
  "pack": {"version": "1.1"},
  "summary": "2 of 3 tests passed."
}

The whole run, server start and seed included, took under two seconds, and it is still one run. On nodejs-24-tests, pg reads the same settings from the environment, so an Express app needs no configuration. With the same seed and three Supertest tests in Jest, this one passed 3 of 3:

app.js (the student's)

const express = require('express');
const { Pool } = require('pg');

const pool = new Pool(); // reads PGHOST, PGPORT, PGUSER and PGDATABASE
const app = express();
app.use(express.json());

app.get('/books', async (req, res) => {
  const { rows } = await pool.query('SELECT title, year FROM books ORDER BY year');
  res.json(rows);
});

app.post('/books', async (req, res) => {
  const { title, year } = req.body;
  if (!title || !Number.isInteger(year)) return res.status(400).json({ error: 'title and year are required' });
  const { rows } = await pool.query('INSERT INTO books (title, year) VALUES ($1, $2) RETURNING id', [title, year]);
  res.status(201).json(rows[0]);
});

module.exports = { app, pool };

A mistake in a seed stops the run before any test, with PostgreSQL's own message, so you see it at once:

Response (shortened)

{
  "verdict": "no_report",
  "gradable": false,
  "tests": [],
  "run": {"exit_code": 1, "time": 0.3604, "output": "",
          "error": "Seeding db/01-books.sql failed:\npsql:db/01-books.sql:7: ERROR:  relation \"book\" does not exist\n..."},
  "summary": "No test report was written, or it named no tests."
}

The server allows 8 connections at once and keeps its data in memory, and nothing is left after the run. Jest, Mocha and node:test end the run once their tests are done, so a pool left open doesn't hold it. The run-tests reference has every setting and error. To grade SQL questions query by query, see the guide to auto-grading PostgreSQL exercises.

Grade Java with JUnit, and C# with xUnit or NUnit

Send the .java files to openjdk-25-tests with the junit preset. It compiles them all in one go and runs every JUnit test it finds, JUnit 5 or JUnit 4, whatever the class is called; AssertJ and Mockito are included. The Java autograder page has a fuller example and a script that grades a whole class.

Solution.java (the student's)

public class Solution {
    public static int add(int a, int b) {
        return a + b;
    }
}

SolutionTest.java (yours)

import org.junit.jupiter.api.Test;

import static org.assertj.core.api.Assertions.assertThat;

class SolutionTest {
    @Test
    void adds() {
        assertThat(Solution.add(2, 3)).isEqualTo(5);
    }

    @Test
    void fails() {
        assertThat(Solution.add(2, 2)).isEqualTo(5);
    }
}

Terminal

curl -X POST https://api.onlinecompiler.io/api/run-tests/ \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "compiler": "openjdk-25-tests",
    "suite": {"preset": "junit"},
    "files": [
      {"name": "Solution.java", "content": "..."},
      {"name": "SolutionTest.java", "content": "..."}
    ]
  }'

Response (shortened)

{
  "verdict": "failed",
  "passed": 1, "failed": 1, "errors": 0, "skipped": 0, "total": 2,
  "score": 50.0,
  "summary": "1 of 2 tests passed.",
  "tests": [
    {"name": "adds()", "classname": "SolutionTest", "status": "passed", "time": 0.048, "message": null},
    {"name": "fails()", "classname": "SolutionTest", "status": "failed", "time": 0.014,
     "message": "\nexpected: 5\n but was: 4"}
  ],
  "compiler": "openjdk-25-tests",
  "preset": "junit",
  "pack": {"version": "1.0"},
  "...": "..."
}

The failed test carries AssertJ's message: expected 5, but was 4. Import what your tests use by name, as above, rather than import org.junit.jupiter.api.*;: a student's own class named Assertions would otherwise take the place of the real one. When the code doesn't compile, nothing runs and the answer says so, with javac's message:

Response (shortened)

{
  "verdict": "compilation_error",
  "total": 0,
  "tests": [],
  "run": {"exit_code": 65, "error": "Solution.java:3: error: ';' expected\n        return a + b\n                    ^\n1 error\n", "...": "..."},
  "summary": "The run stopped: the code did not compile.",
  "...": "..."
}

C# works the same way on dotnet-csharp-10-tests, with the xunit preset (the default) or nunit. Test files written for a dotnet new xunit3 or dotnet new nunit project compile as they are, and a Main in the student's code is ignored. Two things differ from the other packs: xUnit and NUnit have no per-test time limit, so a test that hangs runs until the suite's time_limit; and code that ends the process (System.exit in Java, Environment.Exit in C#) leaves no report, so the answer is no_report.

Points per test, with your own command

A preset scores by the share of tests passed. For points, or for checks a framework doesn't express, send a command instead. It runs with bash where the files are and must write $OUTPUT_DIR/results.json in Gradescope's format (each test with a name, and a status or a score and max_score) or $OUTPUT_DIR/report.xml as JUnit XML, which pytest writes with --junitxml. This grader gives 2, 3 and 5 points:

grade.py (yours)

import json
import os
import subprocess

# (test name, points, a check that raises if the student's code is wrong)
CHECKS = [
    ("add works", 2, "from calculator import add; assert add(2, 3) == 5"),
    ("add handles negatives", 3, "from calculator import add; assert add(-1, -1) == -2"),
    ("divide by zero returns None", 5, "from calculator import divide; assert divide(1, 0) is None"),
]

tests = []
for name, points, check in CHECKS:
    run = subprocess.run(["python", "-c", check], capture_output=True, text=True, timeout=10)
    passed = run.returncode == 0
    tests.append({"name": name, "score": points if passed else 0, "max_score": points,
                  "output": "" if passed else run.stderr.strip().splitlines()[-1]})

with open(os.path.join(os.environ["OUTPUT_DIR"], "results.json"), "w") as f:
    json.dump({"tests": tests}, f)

Terminal

curl -X POST https://api.onlinecompiler.io/api/run-tests/ \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "compiler": "python-3.14-tests",
    "suite": {"command": "python grade.py"},
    "files": [
      {"name": "calculator.py", "content": "..."},
      {"name": "grade.py", "content": "..."}
    ]
  }'

Response (shortened)

{
  "verdict": "failed",
  "passed": 2, "failed": 1, "total": 3,
  "score": 50.0,
  "tests": [
    {"name": "add works", "status": "passed", "score": 2.0, "max_score": 2.0, "message": null},
    {"name": "add handles negatives", "status": "passed", "score": 3.0, "max_score": 3.0, "message": null},
    {"name": "divide by zero returns None", "status": "failed", "score": 0.0, "max_score": 5.0,
     "message": "ZeroDivisionError: division by zero"}
  ],
  "preset": null,
  "...": "..."
}

The student's divide raises on zero, so they earn 5 of 10 points and score is 50. A grader that already writes Gradescope's results.json needs only its paths changed: the files are in the working directory and the report goes to $OUTPUT_DIR.

What students see, and what stays private

Your tests stay yours. The test files go from your server to ours with each submission. A student sees what you choose to show from the answer: names and statuses during an exam, failure messages while they practise.
A fresh sandbox for every submission. Nothing one student's code writes is left for the next.
No way out. A suite has its own 127.0.0.1 for servers like the one above, and nothing else: no internet, no other machines. See the security page.
A result you can defend. Each answer has an Ed25519-signed receipt with the SHA-256 of the files, the preset or the command's hash, and the pack's version, so a disputed grade can be checked later. See signed results.

What's in the test packs

compiler	Presets	Includes
`python-3.14-tests`	pytest	pytest (which also runs unittest tests), Flask, Flask-SQLAlchemy, SQLAlchemy, FastAPI with httpx, PyJWT and psycopg, beside the data-science stack of python-3.14
`nodejs-24-tests`	jest, mocha, node-test	Jest, Mocha with Chai, node:test, Supertest, Express, jsonwebtoken, bcrypt and pg
`openjdk-25-tests`	junit	JUnit Jupiter (JUnit 5 and 6) and JUnit 4 through the JUnit Platform console launcher, with AssertJ and Mockito
`dotnet-csharp-10-tests`	xunit, nunit	xUnit v3 and NUnit 4 (through NUnitLite), compiled with the C# compiler and run in one process

Packages can't be installed during a run, so a pack holds what courses and hiring tasks use most. A pack's version changes only when its frameworks or libraries do, and every answer says which version graded it. If your tasks need a library that isn't there, tell us which.

Limits

Time: time_limit covers the whole suite, 20 seconds by default and up to 30. pytest, Jest, Mocha, node:test and JUnit also stop any one test after 10 seconds; xUnit and NUnit have no per-test limit.
Files: up to 20 files and 1 MB, in folders if you like.
Tests: up to 1,000 read from a report of up to 2 MB.
Memory: 512 MB, and up to 2,048 MB with memory_limit on Pro and above.
PostgreSQL: on python-3.14-tests and nodejs-24-tests, for Pro and above and the exam tiers, with memory_limit 1,024 or more; 8 connections at once, and its start and seeds count in time_limit.
Synchronous only: the request ends within 60 seconds. When no sandbox is free in time the answer is 429, nothing counts, and you send it again.

Every field, verdict and error is in the run-tests reference.

What it costs

A suite is one run, however many tests it holds. Grading 300 students with a 10-test suite uses 300 runs, where input-and-output grading would use 3,000. The Free plan has 100,000 runs a month; paid plans add more runs, more runs at once and more memory (pricing).

Moving from GitHub Classroom or Gradescope

GitHub Classroom's Python tests ran a setup command and then pytest; here pytest is already installed, so send the repository's files and the pytest preset. Java tests that Classroom ran through Maven or Gradle run here with the junit preset, with no build step: send the .java files. Classroom's input and output tests move to the autograder API, as the GitHub Classroom guide shows. A Gradescope grader that writes results.json runs as a command once its paths point to the working directory and $OUTPUT_DIR.

Try it

Sign in for an API key, then run the examples above. The reference has the curl version, and the Python and Node.js clients take suites from version 0.8.0, and PostgreSQL from 0.9.0.

Frequently asked questions

Which languages can a suite grade?

Python with pytest (which also runs unittest tests) on python-3.14-tests, JavaScript on Node.js 24 with Jest, Mocha or node:test on nodejs-24-tests, Java with JUnit (and AssertJ and Mockito) on openjdk-25-tests, and C# with xUnit or NUnit on dotnet-csharp-10-tests. For C, C++ and the other languages, grade with input and output tests through the autograder API, with a short driver that calls the student's functions.

Does a suite cost one run per test?

No. A whole suite counts as one run against your plan, however many tests it holds, so a class of 300 graded by a 10-test suite uses 300 runs. The Free plan has 100,000 runs a month.

Can the student code reach the internet or install packages?

No. The sandbox has its own 127.0.0.1, so a test can start a server and call it, and nothing else: no internet, no other machines, no package installs. The test packs come with the libraries listed above.

Can the tests use a real database?

Yes, PostgreSQL 18.6 on Pro and above and the exam tiers. Add "services": ["postgresql"] to the suite on python-3.14-tests or nodejs-24-tests, send a memory_limit of 1024 MB or more, and a fresh server starts in the sandbox before the tests, seeded from your .sql files under db/, with DATABASE_URL set for SQLAlchemy, psycopg or pg. It is gone when the run ends. SQLite, a file, works on every plan through Python's sqlite3.

Can students see the tests?

Only if you show them. The test files travel from your server to ours with each submission, and the answer names every test with its status and failure message. Show students the names and statuses during an exam, and the messages too while they practise.

Can I grade with points per test?

Yes. Run your own command that writes results.json in Gradescope's format, with a score and max_score for each test; the overall score is their total over the maximum. With a preset, the score is the percentage of tests that passed.

Can a suite run asynchronously, with a webhook?

Not yet. A suite runs synchronously and its request ends within 60 seconds. Run several at once from your own workers, within your plan's runs at once; a 429 answer counted nothing, so send it again.