onlinecompiler.io

How to auto-grade Rust exercises with test cases

Send a student's Rust program and your test cases in one request. Each test runs the program in a fresh sandbox with the test's input, compares what it prints with the output you expect, and returns a verdict. This guide grades one exercise and five versions of a solution, each sent to the API, so you can see what a borrow-checker error, an integer overflow, an unwrap on a missing line and an unflushed BufWriter each get.

How does automatic grading of Rust work?

A grading request sends the program once with a list of tests. Each test is an input and the output you expect. The API compiles the program with rustc 1.93 (2021 edition, overflow checks on) and runs it once per test in a new sandbox with no network, so a test can't see what an earlier one did. It then returns:

  • a verdict for each test: accepted, wrong_answer, runtime_error, time_limit_exceeded, memory_limit_exceeded or compilation_error;
  • an overall verdict, a score and a one-line summary;
  • the compiler's message when the program does not compile;
  • the output, standard error and exit code of every test you did not mark hidden.

The student's program reads standard input and writes standard output, as on Codeforces or in a competitive programming course. Nothing has to be installed: no Rust toolchain on the grading machine, no Docker.

The exercise and a correct solution

The exercise
The first number is n. Then come n integers, each between -1000000 and 1000000.
Print the sum of their squares.

The largest square is 1012, so the answer needs an i64. A solution that reads the whole input at once and splits it on whitespace doesn't care how the numbers are spread over lines:

solution.rs
use std::io::{self, Read};

fn main() {
    let mut input = String::new();
    io::stdin().read_to_string(&mut input).unwrap();
    let mut numbers = input.split_ascii_whitespace().map(|t| t.parse::<i64>().unwrap());
    let n = numbers.next().unwrap() as usize;
    let total: i64 = numbers.take(n).map(|x| x * x).sum();
    println!("{total}");
}

How do I grade a solution?

Get an API key (the Free plan has 100,000 runs a month; each test counts as one run), then send the program and the tests to POST /api/run-tests/. This script does it for one file. The first test is the sample students see; the other three are hidden and check negative numbers, values that need an i64, and an input with no numbers at all:

grade.py
# grade.py: grades one Rust solution with run-tests. Usage: python grade.py solution.rs
import sys
from pathlib import Path

import requests

TESTS = [
    {"input": "3\n1 2 3\n", "expected_output": "14\n"},  # the sample students see
    {"input": "2\n-4 5\n", "expected_output": "41\n", "hidden": True},
    {"input": "2\n1000000 -1000000\n", "expected_output": "2000000000000\n", "hidden": True},
    {"input": "0\n", "expected_output": "0\n", "hidden": True},
]

response = requests.post(
    "https://api.onlinecompiler.io/api/run-tests/",
    headers={"Authorization": "YOUR_API_KEY"},
    json={"compiler": "rust-1.93", "code": Path(sys.argv[1]).read_text(), "tests": TESTS},
    timeout=120,
)
response.raise_for_status()
result = response.json()
print(result["verdict"], result["passed"], "of", result["total"])
print(result["summary"])
if result["compile_error"]:
    print(result["compile_error"])

For the solution above, it prints:

python grade.py solution.rs
accepted 4 of 4
1 of 1 sample test passed. 3 of 3 hidden tests passed.

The whole response, shortened (the signed receipt and the timings are left out). The hidden tests show only their verdicts:

Response (200)
{
  "verdict": "accepted",
  "score": 100,
  "passed": 4,
  "total": 4,
  "compile_error": null,
  "tests": [
    {"verdict": "accepted", "hidden": false, "output": "14\n", "error": "", "exit_code": 0},
    {"verdict": "accepted", "hidden": true},
    {"verdict": "accepted", "hidden": true},
    {"verdict": "accepted", "hidden": true}
  ],
  "summary": "1 of 1 sample test passed. 3 of 3 hidden tests passed.",
  "submission": "sub_twv8e5gBaPUrPwe4A9npSDDt"
}

What does each mistake get?

Four wrong solutions, each sent with the same script and the same tests:

The mistakeVerdictWhat the student sees
i32 instead of i64runtime_error, 3 of 4The sample passes; hidden test 3 fails.
Reads the numbers from a second line that isn't thereruntime_error, 3 of 4The sample passes; hidden test 4 (n is 0) fails.
Uses a Vec after moving itcompilation_error, 0 of 4rustc's error E0382 with its fix; no test runs.
Ends with std::process::exit(0) after writing to a BufWriterwrong_answer, 0 of 4"Sample 1 expected 14 but the program printed nothing."

An i32 that overflows

The same program with i32 in place of i64:

solution.rs
use std::io::{self, Read};

fn main() {
    let mut input = String::new();
    io::stdin().read_to_string(&mut input).unwrap();
    let mut numbers = input.split_ascii_whitespace().map(|t| t.parse::<i32>().unwrap());
    let n = numbers.next().unwrap() as usize;
    let total: i32 = numbers.take(n).map(|x| x * x).sum();
    println!("{total}");
}
python grade.py solution.rs
runtime_error 3 of 4
1 of 1 sample test passed. 2 of 3 hidden tests passed.

The small tests pass, and test 3 (1000000 squared) stops with a panic. Run on its own with that input, the program's standard error is:

Standard error (exit code 101)
thread 'main' (24) panicked at code.rs:8:46:
attempt to multiply with overflow
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

This is the overflow check at work. Built with cargo build --release, which turns the check off, the same program would wrap around and print a wrong number without any error, and the verdict would be wrong_answer. Here the student gets a panic that names the operation. A panicking Rust program exits with code 101, and any non-zero exit code is a runtime_error.

An unwrap on a line that isn't there

This version reads n from the first line and the numbers from the second, which works for the sample. When n is 0 there is no second line, so lines.next() returns None:

solution.rs
use std::io::{self, BufRead};

fn main() {
    let stdin = io::stdin();
    let mut lines = stdin.lock().lines();
    let _n: usize = lines.next().unwrap().unwrap().trim().parse().unwrap();
    let total: i64 = lines
        .next()
        .unwrap()
        .unwrap()
        .split_whitespace()
        .map(|t| t.parse::<i64>().unwrap())
        .map(|x| x * x)
        .sum();
    println!("{total}");
}
Standard error with input 0 (exit code 101)
thread 'main' (26) panicked at code.rs:9:10:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The grading result is the same as for the overflow: runtime_error, 3 of 4, with test 4 the one that failed. Reading the whole input with read_to_string and splitting it on whitespace, as the correct solution does, avoids this whole class of mistake. That habit is worth teaching before the first graded exercise.

A borrow-checker error

solution.rs
use std::io::{self, Read};

fn main() {
    let mut input = String::new();
    io::stdin().read_to_string(&mut input).unwrap();
    let numbers: Vec<i64> = input.split_ascii_whitespace().map(|t| t.parse().unwrap()).collect();
    let rest = numbers;
    let n = numbers[0] as usize;
    let total: i64 = rest[1..=n].iter().map(|x| x * x).sum();
    println!("{total}");
}

The program never runs. The verdict is compilation_error for every test, and compile_error in the response holds rustc's message, including its suggested fix:

python grade.py solution.rs
compilation_error 0 of 4
The code did not compile, so no tests ran.
error[E0382]: borrow of moved value: `numbers`
 --> code.rs:8:13
  |
6 |     let numbers: Vec<i64> = input.split_ascii_whitespace().map(|t| t.parse().unwrap()).collect();
  |         ------- move occurs because `numbers` has type `Vec<i64>`, which does not implement the `Copy` trait
7 |     let rest = numbers;
  |                ------- value moved here
8 |     let n = numbers[0] as usize;
  |             ^^^^^^^ value borrowed here after move
  |
help: consider cloning the value if the performance cost is acceptable
  |
7 |     let rest = numbers.clone();
  |                       ++++++++

error: aborting due to 1 previous error

For more information about this error, try `rustc --explain E0382`.

Compiler warnings, such as an unused variable, don't change the verdict: the program still runs, and the warning appears in the error field of each test that isn't hidden. Only an error stops the program from running.

A BufWriter that is never flushed

Writing through a BufWriter is the usual advice for programs that print a lot. But std::process::exit ends the process without running destructors, so the buffer is never flushed and the answer is lost:

solution.rs
use std::io::{self, BufWriter, Read, Write};

fn main() {
    let mut input = String::new();
    io::stdin().read_to_string(&mut input).unwrap();
    let mut numbers = input.split_ascii_whitespace().map(|t| t.parse::<i64>().unwrap());
    let n = numbers.next().unwrap() as usize;
    let total: i64 = numbers.take(n).map(|x| x * x).sum();
    let mut out = BufWriter::new(io::stdout().lock());
    writeln!(out, "{total}").unwrap();
    std::process::exit(0);
}
python grade.py solution.rs
wrong_answer 0 of 4
0 of 1 sample test passed. Sample 1 expected 14 but the program printed nothing. 0 of 3 hidden tests passed.

The exit code is 0, so this is a wrong_answer, not a runtime error, even though the code that works out the answer is right. Returning from main normally, or calling out.flush() before exit, fixes it.

How do I grade a whole class?

Run the script once per submission, or store the exercise once as a problem with its tests and a reference solution, then send each student's code to it with your own external_ref (a student id). A stored problem keeps the hidden tests on our side, so they never appear in your grading script, and every submission gets a signed receipt. The GitHub Classroom guide shows grading every push to a student's repository, and the widget puts the same problem in a page where students write and submit code in the browser.

The full request and response fields are in the run-tests docs. To try a program first, the Rust online compiler runs it in your browser with the same compiler.

Frequently asked questions

Which Rust version and edition do the programs use?

Rust 1.93, the 2021 edition. The program is saved as code.rs and built with rustc --edition 2021 -C opt-level=2 -C overflow-checks=on, so it is optimized and still panics on integer overflow, as a debug build or cargo test would. Let chains (if let ... && ...) need the 2024 edition and do not compile.

Can students use crates such as rand or regex?

No. There is no Cargo, and the sandbox has no network, so only the standard library is available. A use line for a crate fails to compile, and the verdict is compilation_error. If a test needs random data, generate it yourself and put it in the test input.

Can I grade with #[test] functions and cargo test?

Not for Rust. The program is built as a binary, not with --test, so #[test] functions in it never run. Write each check as an input and the output you expect, as on this page. Test suites that run a test framework inside the sandbox exist for Python, JavaScript, Java and C#.

How long may each test run, and with how much memory?

2 seconds per test unless the request sets time_limit, up to 30, and 512 MB of memory. A program still running at the limit gets time_limit_exceeded for that test.

Does a panic message give away a hidden test?

No. A hidden test returns only its verdict, so a student sees runtime_error on test 3 but neither its input nor the panic message. The sample test, which is not hidden, returns its output, standard error and exit code. Make the sample show the format, and keep the cases that test the real work hidden.

How is the score worked out?

The default scoring is all_or_nothing: 100 when every test passes, 0 otherwise. That is why the i32 version, which passed 3 of 4 tests, scored 0. Send "scoring": "percent" for the share of tests passed (75 for that program), or "weighted" with a weight on each test.