Jobe "runguard: cannot start sh: Resource temporarily unavailable" and "Unable to fork": Fixes

What you see

The first problem shows up in each job's output. Jobe's runguard could not start the shell that runs the program (jobe#60):

Job output

/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable
Try `/var/www/html/jobe/application/libraries/../../runguard/runguard --help' for more information.

testsubmit.py

{'run_id': None, 'outcome': 11, 'cmpinfo': "/var/www/html/jobe/application/libraries/../../runguard/runguard: cannot start `sh': Resource temporarily unavailable ..."}
Valid Python3
Jobe result: Compile error

C, C++, Python and PHP report it as a compile error (outcome 11); Octave and Node.js as a runtime error. On Jobe 2 the path reads /var/www/html/jobe/app/Libraries/../../runguard/runguard, with the same message.

The second problem happens before any job runs. PHP inside the container cannot start any process at all, so Jobe cannot even ask the compilers for their versions (jobeinabox#16):

Jobe's PHP warning

A PHP Error was encountered
Severity: Warning
Message: exec(): Unable to fork [gcc --version 2>&1]
Filename: libraries/LanguageTask.php

testsubmit.py

39 tests, 0 passed, 0 failed, 39 exceptions
RuntimeError: can't start new thread

The same warning repeats for java -version, python3 --version and every other language, and Moodle shows "An unexpected error occurred. The sandbox may be down. Try again shortly."

"cannot start `sh'": a Jobe user is out of processes

runguard limits how many processes each Jobe user may have, and Linux counts every process with that user's numeric id, wherever it runs. When the count is already at the limit, starting the shell fails with "Resource temporarily unavailable". The reports on Jobe's tracker had three causes.

1. Host processes share the Jobe users' ids (the usual cause in Docker)

Unless Docker remaps user ids, a container and its host share one id space. In jobeinabox#15 the host's GitLab ran its nginx workers as id 998, which was also jobe00's id inside the container, so jobe00 never had a free process slot. Jobe hands out jobe00 first, so the low-numbered users fail most. The signature, from jobe#60:

Java tests pass, because Jobe gives Java a much higher process limit.
Of ten C tests sent at once, some pass and some fail.
Setting {"numprocs": 500} in a question's Parameters makes it work.
Recreating the container does not help.

Check from the host, not from inside the container:

Terminal

# Inside the container: the UIDs of the Jobe users
docker exec <container> grep jobe /etc/passwd

# On the host (not in the container): anything running under those UIDs
ps l -u "<uid1>,<uid2>,<uid3>"

The fix landed in June 2023. Jobe's installer gained a --max_uid option, and the jobeinabox image now installs its users at id 500 and below, away from the system ids that services such as nginx use. Any image built since then has it; a user on jobe#60 confirmed in July 2024 that the current image runs clean with no numprocs override. So pull or rebuild the image:

Terminal

docker pull trampgeek/jobeinabox:latest
docker stop jobe && docker rm jobe
docker run -d -p 4000:80 --name jobe trampgeek/jobeinabox:latest

If you rerun Jobe's installer inside the container, for example after changing the number of Jobe users, pass --max_uid=500 as the image's Dockerfile does. Without it, the installer picks system ids counting down from 999, which can bring the clash back. Raising numprocs in your questions also hides the problem, but Jobe's maintainer offered that only as a diagnostic.

2. LXC containers on Proxmox

In jobe#36, Jobe in an Ubuntu LXC container on Proxmox failed with the same message. Raising nproc to unlimited in /etc/security/limits.conf changed nothing. Installing Jobe in a full virtual machine on the same Proxmox host worked. The root cause was never found; Jobe's maintainer suspected the container's privileges.

3. CentOS or RHEL Docker hosts

Jobe's maintainer said earlier forum reports of this message all involved CentOS or RHEL hosts, and suggested an Ubuntu host for Docker (jobe#60). There is no detailed diagnosis on GitHub.

Same limit, different message: Scala, Haskell and other heavy runtimes

Runtimes that start many threads hit the same per-job limits with their own messages, such as pthread_create failed (EAGAIN) from the JVM or ghc: failed to create OS thread: Cannot allocate memory. Those need a higher memory limit and process limit in the question's prototype; see Scala, Haskell and other languages in CodeRunner.

"exec(): Unable to fork": the host or Docker is too old for the image

When PHP in the container cannot start even gcc --version, the limit comes from outside Jobe. In all three reports the fix was a newer host operating system or Docker engine:

jobeinabox#14: a Debian 10 host. Upgrading the host to Debian 11 made all 39 tests pass.
jobeinabox#16: an AWS host; reinstalling containerd and Docker fixed it.
jobe#67: Jobe's maintainer reproduced the errors on an Ubuntu 16.04 server, and fixed them by upgrading it to 22.04 and reinstalling docker-ce.

All three came after jobeinabox moved to a newer Ubuntu base, and the image has moved again since (Ubuntu 26.04 in October 2026). A likely reason, not confirmed in those threads: newer C libraries create threads with the clone3 system call, which Docker's default security profile blocked until Docker 20.10.10. Upgrade the host and Docker first. The maintainer notes that the distribution's docker.io package is an old Docker; on a host that runs other services, install docker-ce. Then test a clean container, as root on the host:

Terminal

docker run -d -p 5000:80 --name jobetester trampgeek/jobeinabox:latest
docker cp jobetester:/var/www/html/jobe/testsubmit.py .
python3 testsubmit.py --host localhost --port 5000

The maintainer also asked about SELinux in these threads, so if the host enforces it, check its audit log for denials.

Which one do you have?

Symptom	Likely cause	Fix
cannot start `sh'; Java passes, some C passes	Host processes share the Jobe users' ids	Pull or rebuild jobeinabox (June 2023 or later)
cannot start `sh' in an LXC container	Container privileges (unconfirmed)	Run Jobe in a full virtual machine
cannot start `sh' on CentOS or RHEL	Host OS (forum reports)	An Ubuntu host
`exec(): Unable to fork`, every test fails	Host OS or Docker too old for the image	Upgrade the host and Docker (`docker-ce`)
`pthread_create failed`, `failed to create OS thread`	Memory or process limit too low for the runtime	Raise MemLimit and numprocs in the prototype

Or skip running Jobe

If you run Jobe only so that CodeRunner can grade code, a hosted Jobe-compatible server removes the host, the container and its limits from your list. In Moodle, open Site administration > Plugins > Question types > CodeRunner, set Jobe server to https://api.onlinecompiler.io/api and Jobe API key to an OnlineCompiler.io API key. Each run gets a fresh gVisor sandbox with no network access, and the free plan includes 100,000 runs a month.

It is not a drop-in copy of Jobe: the built-in multilanguage question type does not work, and memory and process limits are fixed rather than set per question. The Jobe-compatible API page lists every difference, and CodeRunner without a Jobe server walks through the setup.

Frequently asked questions

What does "runguard: cannot start `sh': Resource temporarily unavailable" mean?

Jobe runs every job through runguard, which caps how many processes the Jobe user (jobe00, jobe01 and so on) may have. The operating system refused to start a new process for that user because the cap was already used up. Either something else is running under the same user id, or the job's limit is too low for what it runs.

Why do Java tests pass while C and Python fail?

Jobe gives Java a much higher process limit (at least 256) because the JVM starts many threads. If only the other languages fail, and setting {"numprocs": 500} in a question's Parameters makes them pass, look for host processes sharing the Jobe users' ids, as described above.

Is "Job could not be run due to server overload" the same error?

No. That message means every Jobe user was busy for longer than jobe_wait_timeout (10 seconds by default), typically when a class submits at once. Raise jobe_max_users in app/Config/Jobe.php (Jobe's README suggests about one per core) and rerun the installer with --purge.

I raised nproc in /etc/security/limits.conf. Why did nothing change?

runguard sets the limit for each job itself, so the system-wide setting does not reach it. In the one thread where someone tried it (an LXC container on Proxmox), it made no difference; moving Jobe to a full virtual machine did.

Can I avoid running Jobe at all?

Yes. CodeRunner can use a hosted Jobe-compatible server. Set its Jobe server to https://api.onlinecompiler.io/api and its Jobe API key to an OnlineCompiler.io API key. The free plan includes 100,000 runs a month.