How scaling works

The capacity cap, the scaling loop, job density, and how kinhin recovers from restarts.

← Home · Why this exists: Why kinhin

Capacity

The effective concurrency cap is min(max_runners, (cores − reserve.cpu) / vm.cpu, (RAM − reserve.memory_gb) / vm.memory_gb, free_disk / disk_per_vm) — disk_per_vm is your disk_gb when set, else a runtime default (50 GB for Tart VMs, 8 GB for Linux container slots). The scaler re-checks free disk before every spawn and holds new VMs when the disk would dip below reserve.disk_gb plus one instance's footprint — a queue burst can never fill your disk. kinhin doctor prints the cap and the binding constraint, and warns under the same rule before the fleet reaches it: WARNING: disk pressure (the guard itself is the diskPressure field of doctor --json), stating how much the images hold and what kinhin image prune would reclaim. Not sure how big a Mac you need? See Hardware requirements & sizing.

When several runtimes are configured, this budget is partitioned across their pools by the runtimes.budget weights — for CPU cores and for RAM separately, so a light-on-cores yet fat-in-memory runtime is bounded by its RAM share too (see Runtimes). The default pool gets whatever share the others leave.

With named container pools of different sizes, the Apple container cap above is counted with the smallest pool's CPU and memory, an upper bound, and each runner is admitted by its own pool's cpus and memory_gb against what the running containers leave of Apple container's CPU and RAM, and by its pool's max:. One pool, or none, keeps exactly the numbers above.

The scaling loop

Every poll_interval seconds the engine:

1. Cleans up any failed instances from the previous pass.
2. Asks each forge for queued jobs on its scope and filters them by labels.
3. Routes each job to a runtime — workflow route → repo route → runs-on: label → default — and buckets the demand per pool.
4. Computes the desired instance count for the whole fleet: desired = min(max(ceil(queue / runners_per_vm), min_runners), cap), where queue is the queued jobs of every pool together and cap is min(max_runners, capacity cap). Each new instance goes to the pool with the most unmet demand that is still below its pool_cap: that runtime's slice of the host budget (budget weights, applied to cores and RAM alike) when multiple runtimes are configured, or the full capacity cap for a single-pool fleet. Instances min_runners asks for beyond the queue go to runtimes.default.
5. Spawns those instances (linked clone → boot → SSH → register runners_per_vm runners → run.sh) — Tart VMs for macOS pools and for Tart Linux VMs, apple containers or Docker slots for the container Linux pools (see Choosing a Linux runtime), and a runner process on the Mac itself for host routes. Then it tops up each pool and image to its min_idle floor with the room left.
6. Deletes idle instances beyond desired (scale to zero by default), never one a pool's queued jobs or a min_idle floor still needs.

When starts keep failing.

If a runtime's instances fail to register twice in a row (a bad image, a runner that exits at once), kinhin stops respawning them in a loop: that runtime waits 30 s, then 1 min, 2 min … up to 10 min before the next attempt, other runtimes keep going, and the status line shows `runner starts failing on

` with the runner's own output. One successful registration clears it. A job

cancelled on GitHub before a runner picks it up is not a failure: the idle instance is simply removed (cancelled remotely in the log).

Poll cadence

poll_interval is the normal cadence. Two things stretch it:

• Idle. After 5 minutes with no queued jobs and no instances, kinhin polls every 60s (or poll_interval, if longer). The first new job restores the normal cadence. The 5-minute grace keeps multi-stage workflows fast, because the fleet is briefly empty between needs: stages.
• API budget. kinhin reads GitHub's rate-limit headers. Below 25% of the hourly budget it polls half as often, and below 10% a quarter as often. When the budget is exhausted it waits for the reset (at most 15 minutes). The status line notes any slowdown, for example queue age: 12s old · checking every 60s (idle).

GitHub reads are conditional, so a poll that finds nothing changed gets a 304, which doesn't count against the rate limit.

min_runners: a fleet-wide floor, not an idle one

min_runners: N keeps at least N instances in the whole fleet. It does not keep idle runners:

• It counts busy instances. While a job runs on the only instance, the floor is met, so no idle runner is registered beside it, and none is listed until the job ends and a new instance registers.
• It fills runtimes.default. The instances it adds beyond the queue all go to the default runtime, with the base image: on Tart, so a mixed fleet never gets an idle runner for its other pools or for a named image's labels from it.
• It is one count, not one per pool: the queue of every pool counts toward it.

To keep a runner listed as idle for each label set the fleet serves, use min_idle (below). min_runners still works as before and can be combined with it: an instance it starts counts toward the default pool's min_idle floor.

Idle runners per pool and image

min_idle: N keeps at least N runners registered and idle for one pool or one Tart image, so the forge's runner list always shows a free runner for its labels, even before any job is queued. It is set per pool and per image:

min_runners: 0
image_min_idle: 1        # the top-level image:
images:
  - name: macos-27
    labels: [macos]
    min_idle: 1
  - name: macos-27-xcode
    labels: [macos, xcode]
    min_idle: 1
container:               # the Apple container pool
  image: kinhin-ci-linux
  min_idle: 1            # with pools:, the default for each pool
  # pools:               # named container pools: each keeps its own floor
  #   big: { labels: [big], min_idle: 1 }
docker:                  # the Docker pool
  image: kinhin-ci-linux
  min_idle: 1

Each key defaults to 0 (off). How the floor is kept:

• What counts. An instance of that pool or image that is starting (clone, boot, registration) or registered and waiting for a job. A busy runner does not count: when an idle runner picks up a job, a replacement starts on the next pass. (On a GitHub scale set, whose queue counts running jobs too, a busy runner covers its own job and nothing more.) Warm VMs are not registered, so they do not count.
• The queue comes first. Queued jobs, and min_runners, get their instances first; floors are topped up with the room left under max_runners, the capacity cap and each pool's pool_cap. When room is short, the floors take turns, one instance each. A runtime backing off after failed starts gets no top-up until it recovers, and a pool that is not configured (container: without a runtimes: section, for example, makes the fleet a single container pool) ignores its floor.
• A queued job does not fill the floor. With min_idle: 1 and one job queued for that pool, kinhin starts two instances: one for the job and one to stay idle.
• Warm VMs. A Tart top-up claims a warm VM of its image when there is one, so registering it is all that is left. Warm VMs of other images are evicted when the slot is needed, as for a job.
• Unprepared images. A named image that was never prepared gets no idle runner. Its min_idle is skipped, with a log line, until it exists. A named container pool whose image is not built is left out entirely: no idle runner and no runner for its queued jobs, which wait for the image rather than land in another pool.
• Named container pools. Each pool keeps its own floor, and an idle runner of one pool is no runner for another pool's jobs (their labels differ), as with Tart images. A top-up only starts where the pool fits under its max: and the CPU and RAM left.
• Scale-down never removes a runner a floor needs.
• min_idle counts instances. With runners_per_vm above 1, an idle instance holds that many idle runners.
• Status. desired in kinhin status still counts the queue and min_runners only; instances kept for min_idle come on top of it, and the log says keeping idle runners (min_idle): starting … when it starts one.

Idle runners take slots under max_runners, so the loader checks that every floor fits: each min_idle (and Docker's within docker.max_slots, a container pool's within its max:), each image's min_idle plus its warm_pool, and all of them together (every container pool's counted) must fit in max_runners.

Tart: two macOS VMs

Apple's limit of two macOS VMs per Mac covers idle runners too, so the min_idle values of macOS Tart images (the top-level image and every images: entry, unless their base_image is a curated Linux image) must add up to 2 or less; the loader rejects more. On most Macs runtimes.budget gives Tart a pool_cap of 2 as well.

That limit is the trade-off. With one idle runner for macos-27 and one for macos-27-xcode, both macOS slots are taken by idle runners:

• A job for either image starts at once on its idle runner. Its replacement waits for a free slot, so that image has no idle runner listed until a job ends.
• A job for a third Tart image (or the base image) waits until a job ends and frees a slot: kinhin does not retire a floor's idle runner to make room for another image's job, the way it evicts a warm VM. An idle runner that no floor needs can be scaled down to make room.

With min_idle on any Tart image, Tart's queued demand is counted per image: an idle runner of one image is not counted as serving another image's queued job, since its labels differ and the forge cannot hand the job to it.

Warm pool

warm_pool: N keeps N Tart VMs cloned, booted and SSH-ready but not registered with any forge. When a job arrives, the scaler claims a warm VM, binds it to the account that has the demand, and runs only the registration step, so the job skips tart clone and boot. With the default 0 the feature is off.

Each Tart image has its own pool. The top-level warm_pool covers image:, and warm_pool on an images: entry covers that image, so an Xcode image can stay warm while the base stays cold, or the other way round.

• Warm VMs take slots under max_runners and count against the capacity cap and Tart's pool_cap, including Apple's limit of two macOS VMs per Mac. They are evicted when a job or a min_idle top-up needs the slot, so a warm VM never delays work that a cold start could serve.
• The pool tops up only in a pass that starts nothing else: no queued job waiting for an instance, min_runners met and every min_idle floor met. In ephemeral mode the replacement for a finished VM is started on the next pass, so the poll interval applies.
• A job claims a warm VM of the image it runs on. Jobs routed to a per-project image (projects:), and the other runtimes, start cold.
• A named image that was never prepared gets no warm VMs. Its warm_pool is skipped, with a log line, until it exists. Kinhin reads tart list at most once a minute, so an image prepared while the fleet runs starts its warm pool within a minute.
• The base warm_pool and every entry's warm_pool together must fit in max_runners.
• A warm VM older than warm_max_age_minutes (default 60) is recycled, and so is one whose pool is turned off or whose images: entry was removed. After a restart, warm VMs are re-checked rather than trusted.

Warm VMs are not registered, so the forge does not list them. To have a registered idle runner listed for an image, use min_idle.

Healing

The scaling loop recovers from most failures by itself: the spawn backoff above, teardown retries, forge re-attach backoff, and an orphan sweep for instances kinhin started but has no record of (see Restarts). The healing agent runs in its own loop beside it and covers what those do not, without ever delaying a scaling pass:

• Warm VMs. A warm VM waits with nothing driving it, so no liveness watch sees it die. The engine's orphan sweep (every 2 minutes, healing on or off) lists each runtime once and removes the record of a warm VM deleted or stopped outside kinhin, so it stops holding a slot and the pool tops up again; the healing journal notes each removal.
• Disk pressure. When the disk guard holds spawns, the agent prunes the toolchain cache (with auto_prune on) and then asks before pruning unused images.
• Spawn backoff. The backoff keeps a failing runtime from respawning in a loop; the agent looks at why. A stopped Apple container system is started on its own; a missing runtime or a broken tart waits for your approval.
• Detached accounts. An account that dropped off for a minute is re-attached at once instead of waiting out its backoff; a rejected token is shown to you.

With healing.enabled: false all of the engine's own retries still run. kinhin heal shows what the agent did.

Restarts

When the daemon or app restarts, kinhin reads its saved state (fleet.json in the state directory) and checks what is still running on each runtime before it starts anything new.

• Intact fleet.json. Records are reloaded. A runner that was registered (idle or busy) is watched, not registered again, so its job is not disturbed. The exception is an idle classic (non-ephemeral) runner in an Apple container or Docker container: the agent dies with a stopped container, so when the forge lists runners and has none for it, it is registered again. A warm VM returns to the warm pool.

• Unreadable fleet.json. kinhin moves the file aside to fleet.json.corrupt-<timestamp> (it is never overwritten), and the status alert and kinhin status name the kept copy. The engine starts with no VM records, so job and repo attribution is lost. It then re-adopts each VM it finds and checks that VM's runner with the forge, asking each account once:

  • a runner the forge still has (busy or idle) is watched, not registered again, so a running job is not disturbed;
  • a VM with no runner on the forge is registered as normal;
  • a VM whose forge cannot be asked is held and not registered until it can be (shown by the adoption status alert).

  Beta 2 registered every re-adopted VM again, which could fail a running job (see the known issue in GitHub forge notes). The fix shipped in beta 3; on beta 2, restart only when no jobs are running.

• Unlisted runtime. If a runtime can't be listed (Tart wedged, a daemon down), its records are kept, and nothing is removed on a failed listing.

• Orphan sweep. Only an instance with no record, under a name kinhin generates, is removed, and only after it stays unclaimed for ten minutes (orphanGrace). An instance the restart re-adopted has a record, so the sweep does not remove it.

• Warm VMs after an unreadable fleet.json. The pool flag is lost, so a re-adopted warm VM comes back as an ordinary runner. It counts against max_runners until scale-down removes it.

• Forges with no runner list. Some servers (Gitea before 1.24, Forgejo, Bitbucket) cannot say which runner belongs to which VM. With nothing running in the account's scope, a re-adopted VM is registered. With a job running, it is watched as busy, and it is removed once the scope goes quiet.

• Partly registered VMs. With runners_per_vm above 1, a VM can have only some of its runners on the forge. If none of them is busy, the set is registered again. If one is busy, the VM is watched. The missing runner never appears, so the VM is removed only at the watch deadline (up to 6 hours). This is a known limit.

• Accounts without a forge. A VM whose account is still configured but has no forge attached is held, with the adoption status alert, and removed after one hour. A VM whose account was removed from the config is released at once.

Pausing and auto-resume

A paused fleet doesn't spawn new instances. With auto_resume: true it keeps watching the queue while paused and resumes itself once the queue is empty. This is what makes a deliberate pause safe, for example the Shortcuts "Delete VM" action pauses first so autoscaling can't recreate what you just deleted. If every forge fetch fails while paused, the engine stays paused rather than treat silence as "nothing to do".

Every status surface (menu bar, Fleet tab, kinhin status, Siri) shows how old the queue numbers are, so a paused snapshot never looks fresh during an outage. kinhin status --json exposes paused, autoResume, pollInterval and queueAgeSeconds.

Job density

runners_per_vm is the density knob: with the default 1, each job gets its own VM (most isolation); with 4, one VM serves four concurrent jobs before the scaler spawns another. Small jobs on identical macOS stacks are the sweet spot — set it high enough that vm.cpu/vm.memory_gb cover the sum of the jobs you expect on one VM.

With a GitHub scale set (scale_set:), keep runners_per_vm: 1: GitHub does not hand a second job to another idle runner in the same VM, and the scaler counts those slots as capacity, so the job waits for the first one to finish (details).

Running on multiple Macs

Each Mac runs its own engine and polls the forge independently; kinhin does not coordinate between machines (the daemon lock only guards against two daemons on the same Mac). If two Macs serve the same repo with the same labels, both see a queued job and both spawn a runner for it. The forge hands the job to exactly one of them, so it never runs twice, but the other Mac boots a VM it didn't need.

To keep demand from overlapping, give each Mac its own label:

# Mac A
labels: ["mac-a"]

# Mac B
labels: ["mac-b"]

A workflow then picks the Mac it wants:

runs-on: [self-hosted, macos, arm64, mac-a] # only Mac A spawns for this job

A job counts as demand only when it requests every configured label, so the other Mac ignores it. To fan one job across both Macs, use a matrix:

strategy:
  matrix:
    mac: [mac-a, mac-b]
runs-on: [self-hosted, macos, arm64, "${{ matrix.mac }}"]

Things to know:

• A job with neither label matches no Mac and stays queued.
• A shared pool (the same labels on both Macs) still works, but a burst can make both Macs spawn. Lower max_runners on the weaker Mac to limit the waste.

Requeue and skip

From the Fleet tab, the Requeue and Skip buttons in the Requeue / skip queued section cancel or drop the queue's matching jobs. Org-scope forges re-queue/skip the fleet-wide queue; repo-scope forges act on that one forge. Both are single-command actions behind the same pending-state and AlertKit failure handling as Start/Stop.

See Requeue and skip queued jobs for the full behavior.