Java autograder: grade JUnit tests through an API

Grade one submission

The assignment: a BankAccount class whose deposit refuses amounts that aren't positive and whose withdraw refuses more than the balance. This student forgot the second rule.

public class BankAccount {
    private int balance;

    public BankAccount(int opening) {
        balance = opening;
    }

    public int getBalance() {
        return balance;
    }

    public void deposit(int amount) {
        if (amount <= 0) {
            throw new IllegalArgumentException("Deposit must be positive");
        }
        balance += amount;
    }

    public void withdraw(int amount) {
        balance -= amount;
    }
}

Your tests use JUnit Jupiter with AssertJ, including a parameterized test that checks three deposits:

import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;

import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;

class BankAccountTest {
    @Test
    void startsWithTheOpeningBalance() {
        assertThat(new BankAccount(100).getBalance()).isEqualTo(100);
    }

    @ParameterizedTest
    @CsvSource({"100, 50, 150", "0, 1, 1", "10, 990, 1000"})
    void depositAddsToTheBalance(int opening, int amount, int expected) {
        BankAccount account = new BankAccount(opening);
        account.deposit(amount);
        assertThat(account.getBalance()).isEqualTo(expected);
    }

    @Test
    void refusesANegativeDeposit() {
        assertThatThrownBy(() -> new BankAccount(10).deposit(-5))
            .isInstanceOf(IllegalArgumentException.class);
    }

    @Test
    void refusesToWithdrawMoreThanTheBalance() {
        assertThatThrownBy(() -> new BankAccount(10).withdraw(50))
            .isInstanceOf(IllegalArgumentException.class)
            .hasMessageContaining("Insufficient funds");
    }
}

Send both files to the openjdk-25-tests pack with the junit preset:

curl -X POST https://api.onlinecompiler.io/api/run-tests/ \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "compiler": "openjdk-25-tests",
    "suite": {"preset": "junit"},
    "files": [
      {"name": "BankAccount.java", "content": "..."},
      {"name": "BankAccountTest.java", "content": "..."}
    ]
  }'

The answer, shortened:

{
  "verdict": "failed",
  "gradable": true,
  "passed": 5, "failed": 1, "errors": 0, "skipped": 0, "total": 6,
  "score": 83.33,
  "summary": "5 of 6 tests passed.",
  "tests": [
    {"name": "depositAddsToTheBalance(int, int, int)[2]", "classname": "BankAccountTest", "status": "passed", "message": null},
    {"name": "depositAddsToTheBalance(int, int, int)[3]", "classname": "BankAccountTest", "status": "passed", "message": null},
    {"name": "refusesANegativeDeposit()", "classname": "BankAccountTest", "status": "passed", "message": null},
    {"name": "startsWithTheOpeningBalance()", "classname": "BankAccountTest", "status": "passed", "message": null},
    {"name": "depositAddsToTheBalance(int, int, int)[1]", "classname": "BankAccountTest", "status": "passed", "message": null},
    {"name": "refusesToWithdrawMoreThanTheBalance()", "classname": "BankAccountTest", "status": "failed",
     "message": "\nExpecting code to raise a throwable."}
  ],
  "run": {"exit_code": 1, "time": 2.47, "...": "..."},
  "compiler": "openjdk-25-tests",
  "preset": "junit",
  "pack": {"version": "1.0"},
  "submission": "sub_...",
  "receipt": {"...": "..."}
}

Each case of the parameterized test counts as its own test, so there are six. The failed one carries AssertJ's message, Expecting code to raise a throwable, which tells the student that withdraw should have thrown. score is the share of tests passed: 5 of 6 is 83.33.

When the code doesn't compile

Nothing runs, and the answer says so with javac's own message. Here the student left out a semicolon on line 2:

{
  "verdict": "compilation_error",
  "gradable": true,
  "passed": 0, "total": 0,
  "tests": [],
  "run": {
    "exit_code": 65,
    "error": "BankAccount.java:2: error: ';' expected\n    private int balance\n                       ^\n1 error\n",
    "...": "..."
  },
  "summary": "The run stopped: the code did not compile.",
  "...": "..."
}

Grade a whole class into a gradebook

Put your test files in tests/ and each student's files in submissions/<name>/. This script grades every student with the Python client (pip install onlinecompiler) and writes grades.csv:

import csv
from pathlib import Path
from onlinecompiler import OnlineCompiler

client = OnlineCompiler()  # reads ONLINECOMPILER_API_KEY
tests = [{"name": p.name, "content": p.read_text()} for p in Path("tests").glob("*.java")]

with open("grades.csv", "w", newline="") as f:
    out = csv.writer(f)
    out.writerow(["student", "verdict", "passed", "total", "score", "failed tests"])
    for folder in sorted(Path("submissions").iterdir()):
        files = [{"name": p.name, "content": p.read_text()} for p in folder.glob("*.java")]
        r = client.run_suite("openjdk-25-tests", files + tests, preset="junit")
        failed = [t["name"] for t in r["tests"] if t["status"] != "passed"]
        out.writerow([folder.name, r["verdict"], r["passed"], r["total"], r["score"], "; ".join(failed)])
        print(folder.name, r["verdict"], f"{r['passed']}/{r['total']}")

student,verdict,passed,total,score,failed tests
ada,accepted,6,6,100.0,
alan,failed,5,6,83.33,refusesToWithdrawMoreThanTheBalance()

Each student is one run, however many tests your suite holds. The Node.js client does the same with client.runSuite.

Points per test

The answer names every test, so weighting them is a lookup on your side. With the response above as r:

points = {"refusesToWithdrawMoreThanTheBalance()": 3, "refusesANegativeDeposit()": 2}  # 1 if not named
earned = sum(points.get(t["name"], 1) for t in r["tests"] if t["status"] == "passed")
possible = sum(points.get(t["name"], 1) for t in r["tests"])
print(f"{earned} of {possible} points")  # 6 of 9 points

What the Java pack includes, and its rules

• Libraries: JUnit Jupiter (JUnit 5 and 6) and JUnit 4 through the JUnit Platform console launcher, with AssertJ and Mockito. Exact versions are in the compiler docs.
• Tests: written with JUnit Jupiter (org.junit.jupiter.api) or JUnit 4 (org.junit.Test), in classes of any name, both kinds in one suite if you like. A test is named like fails(); a JUnit 4 parameterized one like adds[0].
• Compiling: every .java file sent is compiled together, so helper classes and several student files work.
• Time: a Jupiter test stops after 10 seconds when it waits (a sleep, a lock), and a JUnit 4 test after its own @Test(timeout = ...); the whole suite has 20 seconds by default and up to 30.
• Memory: 512 MB, and up to 2,048 MB on Pro and above.
• Imports: import what your tests use by name (import static org.assertj.core.api.Assertions.assertThat;) rather than with *, so a student's own class named Assertions can't take the place of the real one.
• Ending the program: student code that calls System.exit stops the JVM before the report is written, and the answer is no_report.

Every field, verdict and limit is in the run-tests reference.

C# with xUnit or NUnit

The C# pack, dotnet-csharp-10-tests, works the same way with the xunit preset (the default) or nunit. Test files written for a dotnet new xunit3 or dotnet new nunit project compile as they are:

namespace Demo;
using Assert = Xunit.Assert;

public class CalcTests
{
    [Fact] public void Adds() => Assert.Equal(5, Calc.Add(2, 3));
    [Fact] public void Fails() => Assert.Equal(5, Calc.Add(2, 2));
}

{
  "verdict": "failed",
  "passed": 1, "total": 2,
  "tests": [
    {"name": "Demo.CalcTests.Adds", "classname": "Demo.CalcTests", "status": "passed", "message": null},
    {"name": "Demo.CalcTests.Fails", "classname": "Demo.CalcTests", "status": "failed",
     "message": "Assert.Equal() Failure: Values differ\nExpected: 5\nActual:   4"}
  ],
  "compiler": "dotnet-csharp-10-tests",
  "preset": "xunit",
  "...": "..."
}

xUnit and NUnit tests have no per-test time limit, so a test that hangs runs until the suite's time limit. Put the tests in a namespace with using Assert = Xunit.Assert; inside it, for the same reason as the Java imports above.

What students see, and what stays private

• Your test files travel with each submission and never appear in the answer, only test names, statuses and messages.
• Every submission gets a fresh sandbox with no network, so nothing one student's code does reaches the next.
• Each answer has an Ed25519-signed receipt with the hash of the files and the pack's version, so a disputed grade can be checked later.

Moving from GitHub Classroom or Gradescope

GitHub Classroom ran Java tests through Maven or Gradle in GitHub Actions. Here there is no build file: send the .java files and the junit preset, as the GitHub Classroom guide describes for its other test types. A Gradescope autograder is a Docker image with a run_autograder script; here the image is ours and stays current, and your test files are all you send.

What it costs

A suite is one run, however many tests it holds: grading 300 students uses 300 runs. The Free plan has 100,000 runs a month; paid plans add runs, runs at once and memory (pricing). Schools can apply for free Pro through the Founding Institutions program. A suite request answers within 60 seconds.

Try it

Sign in for an API key and send the example above. Python, JavaScript and C# suites are on the test-suite autograder page, input-and-output grading for every language is on the autograder API page, and the AP CSA Java page covers running Java in class.

Frequently asked questions

How do I autograde Java assignments with JUnit?

Send the student's .java files and your JUnit test files to POST /api/run-tests/ with "compiler": "openjdk-25-tests" and "suite": {"preset": "junit"}. The answer lists every test by name as passed, failed, error or skipped, with the assertion message, a verdict and a score. Nothing to install: no Gradle, Maven or grader image.

Which JUnit versions does it support?

Both JUnit Jupiter, the API of JUnit 5 and 6 (org.junit.jupiter.api), and JUnit 4 (org.junit.Test), even in the same suite, with parameterized tests, AssertJ and Mockito included. JUnit 4's @Before, @Test(expected = ...) and @Ignore work as usual.

Do I need Maven or Gradle?

No. Every .java file you send is compiled with javac in one go and the tests run in one JVM. The libraries tests usually need (JUnit, AssertJ, Mockito) are already in the pack; packages cannot be downloaded during a run.

What happens when a student's code does not compile?

Nothing runs, and the answer is compilation_error with javac's own message in run.error and no tests, so the student sees the line to fix. It still counts as one run.

Can students see the tests?

Only if you show them. Your test files go from your server to ours with each submission and come back only as names, statuses and messages. Show names and statuses during an exam, and the messages while students practise.

Is there a free Java autograder?

Yes. The Free plan has 100,000 runs a month with commercial use included and no card, and a whole suite counts as one run. Schools and teachers can also apply for free Pro through the Founding Institutions and free Pro programs.

Does it work for AP Computer Science A?

Yes. AP CSA code is ordinary Java: write JUnit tests for the methods the free-response questions ask for, and grade each student's class with them. The AP CSA Java page shows the course side.