# How scaling works

The capacity cap, the scaling loop, job density, and how kinhin recovers from restarts.

[← Back to README](https://kinhin.dev/v0.0.1-beta-7/index.md) · Why this exists: [Why kinhin](https://kinhin.dev/v0.0.1-beta-7/guide/why-kinhin.md)

## Capacity

The effective concurrency cap is
`min(max_runners, (cores − reserve.cpu) / vm.cpu, (RAM − reserve.memory_gb) / vm.memory_gb, free_disk / disk_per_vm)` —
`disk_per_vm` is your `disk_gb` when set, else a runtime default (50 GB for Tart VMs, 8 GB for
Linux container slots). The scaler re-checks free disk before every spawn and holds new VMs when
the disk would dip below `reserve.disk_gb` plus one instance's footprint — a queue burst can
never fill your disk. `kinhin doctor` prints the cap and the binding constraint, and warns under
the same rule before the fleet reaches it: `WARNING: disk pressure` (the guard itself is the
`diskPressure` field of `doctor --json`), stating how much the images hold and what
`kinhin image prune` would reclaim.
Not sure how big a Mac you need? See [Hardware requirements & sizing](https://kinhin.dev/v0.0.1-beta-7/guide/hardware.md).

When several runtimes are configured, this budget is partitioned across their pools by the
`runtimes.budget` weights — for CPU cores and for RAM separately, so a light-on-cores yet fat-in-memory runtime is
bounded by its RAM share too (see [Runtimes](https://kinhin.dev/v0.0.1-beta-7/guide/runtimes.md#per-repo-and-per-workflow-routing)). The default pool gets
whatever share the others leave.

With [named container pools](https://kinhin.dev/v0.0.1-beta-7/guide/runtimes.md#several-container-pools) of different sizes, the Apple container cap above is
counted with the smallest pool's CPU and memory, an upper bound, and each runner is admitted by its own pool's
`cpus` and `memory_gb` against what the running containers leave of Apple container's CPU and RAM, and by its pool's
`max:`. One pool, or none, keeps exactly the numbers above.

## The scaling loop

Every `poll_interval` seconds the engine:

1. Cleans up any failed instances from the previous pass.
2. Asks each forge for queued jobs on its scope and filters them by `labels`.
3. **Routes** each job to a runtime — workflow route → repo route → `runs-on:` label → default —
   and buckets the demand per pool.
4. Computes the desired instance count **for the whole fleet**:
   `desired = min(max(ceil(queue / runners_per_vm), min_runners), cap)`, where `queue` is the queued jobs of every
   pool together and `cap` is `min(max_runners, capacity cap)`. Each new instance goes to the pool with the most unmet
   demand that is still below its `pool_cap`: that runtime's slice of the host budget (`budget` weights, applied to
   cores and RAM alike) when multiple runtimes are configured, or the full [capacity cap](#capacity) for a single-pool
   fleet. Instances `min_runners` asks for beyond the queue go to `runtimes.default`.
5. Spawns those instances (linked clone → boot → SSH → register `runners_per_vm`
   runners → `run.sh`) — Tart VMs for macOS pools and for Tart Linux VMs, apple containers or
   Docker slots for the container Linux pools (see [Choosing a Linux runtime](https://kinhin.dev/v0.0.1-beta-7/guide/runtimes.md#choosing-a-linux-runtime)),
   and a runner process on the Mac itself for host routes. Then it tops up each pool and image to its
   [`min_idle`](#idle-runners-per-pool-and-image) floor with the room left.
6. Deletes idle instances beyond `desired` (scale to zero by default), never one a pool's queued jobs or a `min_idle`
   floor still needs.

When starts keep failing.

If a runtime's instances fail to register twice in a row (a bad image, a runner
that exits at once), kinhin stops respawning them in a loop: that runtime waits 30 s, then 1 min, 2 min … up to
10 min before the next attempt, other runtimes keep going, and the status line shows `runner starts failing on
<reason> … retrying in Ns: </reason>` with the runner's own output. One successful registration clears it. A job
cancelled on GitHub before a runner picks it up is not a failure: the idle instance is simply removed
(`cancelled remotely` in the log).

### Poll cadence

`poll_interval` is the normal cadence. Two things stretch it:

- **Idle.** After 5 minutes with no queued jobs and no instances, kinhin polls every 60s (or `poll_interval`, if
  longer). The first new job restores the normal cadence. The 5-minute grace keeps multi-stage workflows fast, because
  the fleet is briefly empty between `needs:` stages.
- **API budget.** kinhin reads GitHub's rate-limit headers. Below 25% of the hourly budget it polls half as often, and
  below 10% a quarter as often. When the budget is exhausted it waits for the reset (at most 15 minutes). The status
  line notes any slowdown, for example `queue age: 12s old · checking every 60s (idle)`.

GitHub reads are conditional, so a poll that finds nothing changed gets a 304, which doesn't count against the rate
limit.

### `min_runners`: a fleet-wide floor, not an idle one

`min_runners: N` keeps at least `N` instances in the whole fleet. It does not keep idle runners:

- **It counts busy instances.** While a job runs on the only instance, the floor is met, so no idle runner is
  registered beside it, and none is listed until the job ends and a new instance registers.
- **It fills `runtimes.default`.** The instances it adds beyond the queue all go to the default runtime, with the base
  `image:` on Tart, so a mixed fleet never gets an idle runner for its other pools or for a named image's labels
  from it.
- **It is one count**, not one per pool: the queue of every pool counts toward it.

To keep a runner listed as idle for each label set the fleet serves, use `min_idle` (below). `min_runners` still works
as before and can be combined with it: an instance it starts counts toward the default pool's `min_idle` floor.

## Idle runners per pool and image

`min_idle: N` keeps at least `N` runners **registered and idle** for one pool or one Tart image, so the forge's runner
list always shows a free runner for its labels, even before any job is queued. It is set per pool and per image:

```yaml
min_runners: 0
image_min_idle: 1        # the top-level image:
images:
  - name: macos-27
    labels: [macos]
    min_idle: 1
  - name: macos-27-xcode
    labels: [macos, xcode]
    min_idle: 1
container:               # the Apple container pool
  image: kinhin-ci-linux
  min_idle: 1            # with pools:, the default for each pool
  # pools:               # named container pools: each keeps its own floor
  #   big: { labels: [big], min_idle: 1 }
docker:                  # the Docker pool
  image: kinhin-ci-linux
  min_idle: 1
```

Each key defaults to `0` (off). Every one of them applies on a config reload without restarting the engine: edit a
floor in the app or in `config.yaml`, reload, and the next scaling pass tops up (or stops topping up) to the new value,
keeping every running instance. (Adding or removing a pool or runtime, or changing a pool's `max:`, still rebuilds the
engine.) How the floor is kept:

- **What counts.** An instance of that pool or image that is starting (clone, boot, registration) or registered and
  waiting for a job. A **busy runner does not count**: when an idle runner picks up a job, a replacement starts on the
  next pass. (On a [GitHub scale set](https://kinhin.dev/v0.0.1-beta-7/guide/forges/github.md#scale-set-mode-in-kinhin), whose queue counts running jobs too,
  a busy runner covers its own job and nothing more.) Warm VMs are not registered, so they do not count.
- **The queue comes first.** Queued jobs, and `min_runners`, get their instances first; floors are topped up with the
  room left under `max_runners`, the capacity cap and each pool's `pool_cap`. When room is short, the floors take
  turns, one instance each. A runtime backing off after failed starts gets no top-up until it recovers, and a pool
  that is not configured (`container:` without a `runtimes:` section, for example, makes the fleet a single container
  pool) ignores its floor.
- **A queued job does not fill the floor.** With `min_idle: 1` and one job queued for that pool, kinhin starts two
  instances: one for the job and one to stay idle.
- **Warm VMs.** A Tart top-up claims a warm VM of its image when there is one, so registering it is all that is left.
  Warm VMs of other images are evicted when the slot is needed, as for a job.
- **Unprepared images.** A named image that was never prepared gets no idle runner. Its `min_idle` is skipped, with a
  log line, until it exists. A named container pool whose image is not built is left out entirely: no idle runner and
  no runner for its queued jobs, which wait for the image rather than land in another pool.
- **Named container pools.** Each pool keeps its own floor, and an idle runner of one pool is no runner for another
  pool's jobs (their labels differ), as with Tart images. A top-up only starts where the pool fits under its `max:`
  and the CPU and RAM left.
- **Scale-down** never removes a runner a floor needs.
- **`min_idle` counts instances.** With `runners_per_vm` above 1, an idle instance holds that many idle runners.
- **Status.** `desired` in `kinhin status` still counts the queue and `min_runners` only; instances kept for
  `min_idle` come on top of it, and the log says `keeping idle runners (min_idle): starting …` when it starts one.

Idle runners take slots under `max_runners`, so the loader checks that every floor fits: each `min_idle` (and Docker's
within `docker.max_slots`, a container pool's within its `max:`), each image's `min_idle` plus its `warm_pool`, and all
of them together (every container pool's counted) must fit in `max_runners`.

### Tart: two macOS VMs

Apple's limit of two macOS VMs per Mac covers idle runners too, so the `min_idle` values of macOS Tart images (the
top-level image and every `images:` entry, unless their `base_image` is a curated Linux image) must add up to 2 or
less; the loader rejects more. On most Macs `runtimes.budget` gives Tart a `pool_cap` of 2 as well.

That limit is the trade-off. With one idle runner for `macos-27` and one for `macos-27-xcode`, both macOS slots are
taken by idle runners:

- A job for either image starts at once on its idle runner. Its replacement waits for a free slot, so that image has no
  idle runner listed until a job ends.
- A job for a **third** Tart image (or the base image) waits until a job ends and frees a slot: kinhin **does not
  retire a floor's idle runner** to make room for another image's job, the way it evicts a warm VM. An idle runner
  that no floor needs can be scaled down to make room.

With `min_idle` on any Tart image, Tart's queued demand is counted per image: an idle runner of one image is not
counted as serving another image's queued job, since its labels differ and the forge cannot hand the job to it.

## Warm pool

`warm_pool: N` keeps `N` Tart VMs cloned, booted and SSH-ready but **not registered** with any forge. When a job
arrives, the scaler claims a warm VM, binds it to the account that has the demand, and runs only the registration step,
so the job skips `tart clone` and boot. With the default `0` the feature is off.

Each Tart image has its own pool. The top-level `warm_pool` covers `image:`, and `warm_pool` on an
[`images:` entry](https://kinhin.dev/v0.0.1-beta-7/guide/runtimes.md#multiple-macos-images) covers that image, so an Xcode image can stay warm while the base
stays cold, or the other way round.

- Warm VMs take slots under `max_runners` and count against the capacity cap and Tart's `pool_cap`, including Apple's
  limit of two macOS VMs per Mac. They are evicted when a job or a `min_idle` top-up needs the slot, so a warm VM never
  delays work that a cold start could serve.
- The pool tops up only in a pass that starts nothing else: no queued job waiting for an instance, `min_runners` met
  and every `min_idle` floor met. In ephemeral mode the replacement for a finished VM is started on the next pass, so
  the poll interval applies.
- A job claims a warm VM of the image it runs on. Jobs routed to a per-project image (`projects:`), and the other
  runtimes, start cold.
- A named image that was never prepared gets no warm VMs. Its `warm_pool` is skipped, with a log line, until it exists. Kinhin reads `tart list` at most once a minute, so an
  image prepared while the fleet runs starts its warm pool within a minute.
- The base `warm_pool` and every entry's `warm_pool` together must fit in `max_runners`.
- A warm VM older than `warm_max_age_minutes` (default 60) is recycled, and so is one whose pool is turned off or whose
  `images:` entry was removed. After a restart, warm VMs are re-checked rather
  than trusted.

Warm VMs are not registered, so the forge does not list them. To have a registered idle runner listed for an image,
use [`min_idle`](#idle-runners-per-pool-and-image).

## Healing

The scaling loop recovers from most failures by itself: the spawn backoff above, teardown retries, forge re-attach
backoff, and an orphan sweep for instances kinhin started but has no record of (see [Restarts](#restarts)). The [healing agent](https://kinhin.dev/v0.0.1-beta-7/guide/healing.md) runs in its
own loop beside it and covers what those do not, without ever delaying a scaling pass:

- **Warm VMs.** A warm VM waits with nothing driving it, so no liveness watch sees it die. The engine's orphan sweep
  (every 2 minutes, healing on or off) lists each runtime once and removes the record of a warm VM deleted or stopped
  outside kinhin, so it stops holding a slot and the pool tops up again; the healing journal notes each removal.
- **Disk pressure.** When the disk guard holds spawns, the agent prunes the toolchain cache (with `auto_prune` on) and
  then asks before pruning unused images.
- **Spawn backoff.** The backoff keeps a failing runtime from respawning in a loop; the agent looks at why. A stopped
  Apple container system is started on its own; a missing runtime or a broken tart waits for your approval.
- **Detached accounts.** An account that dropped off for a minute is re-attached at once instead of waiting out its
  backoff; a rejected token is shown to you.

With `healing.enabled: false` all of the engine's own retries still run. `kinhin heal` shows what the agent did.

## Restarts

When the daemon or app restarts, kinhin reads its saved state (`fleet.json` in the state directory) and checks what is still
running on each runtime before it starts anything new.

- **Intact `fleet.json`.** Records are reloaded. A runner that was registered (idle or busy) is watched, not registered again,
  so its job is not disturbed. The exception is an idle classic (non-ephemeral) runner in an Apple container or Docker
  container: the agent dies with a stopped container, so when the forge lists runners and has none for it, it is registered
  again. A warm VM returns to the warm pool.
- **Unreadable `fleet.json`.** kinhin moves the file aside to `fleet.json.corrupt-<timestamp>` (it is never overwritten), and
  the status alert and `kinhin status` name the kept copy. The engine starts with no VM records, so job and repo attribution
  is lost. It then re-adopts each VM it finds and checks that VM's runner with the forge, asking each account once:
  - a runner the forge still has (busy or idle) is **watched, not registered again**, so a running job is not disturbed;
  - a VM with no runner on the forge is registered as normal;
  - a VM whose forge cannot be asked is **held** and not registered until it can be (shown by the `adoption` status alert).

  Beta 2 registered every re-adopted VM again, which could fail a running job (see the known issue in
  [GitHub forge notes](https://kinhin.dev/v0.0.1-beta-7/guide/forges/github.md)). The fix shipped in beta 3; on beta 2, restart only when no jobs are running.
- **Unlisted runtime.** If a runtime can't be listed (Tart wedged, a daemon down), its records are kept, and nothing is removed
  on a failed listing.
- **Orphan sweep.** Only an instance with no record, under a name kinhin generates, is removed, and only after it stays
  unclaimed for ten minutes (`orphanGrace`). An instance the restart re-adopted has a record, so the sweep does not remove it.
- **Warm VMs after an unreadable `fleet.json`.** The pool flag is lost, so a re-adopted warm VM comes back as an ordinary runner.
  It counts against `max_runners` until scale-down removes it.
- **Forges with no runner list.** Some servers (Gitea before 1.24, Forgejo, Bitbucket) cannot say which runner belongs to which
  VM. With nothing running in the account's scope, a re-adopted VM is registered. With a job running, it is watched as busy, and
  it is removed once the scope goes quiet.
- **Partly registered VMs.** With `runners_per_vm` above 1, a VM can have only some of its runners on the forge. If none of them
  is busy, the set is registered again. If one is busy, the VM is watched. The missing runner never appears, so the VM is removed
  only at the watch deadline (up to 6 hours). This is a known limit.
- **Accounts without a forge.** A VM whose account is still configured but has no forge attached is held, with the `adoption`
  status alert, and removed after one hour. A VM whose account was removed from the config is released at once.

## Pausing and auto-resume

A paused fleet doesn't spawn new instances. With `auto_resume: true` it keeps watching the queue while paused and
resumes itself once the queue is empty. This is what makes a deliberate pause safe, for example the Shortcuts "Delete
VM" action pauses first so autoscaling can't recreate what you just deleted. If every forge fetch fails while paused,
the engine stays paused rather than treat silence as "nothing to do".

Every status surface (menu bar, Fleet tab, `kinhin status`, Siri) shows how old the queue numbers are, so a paused
snapshot never looks fresh during an outage. `kinhin status --json` exposes `paused`, `autoResume`, `pollInterval` and
`queueAgeSeconds`.

## Job density

`runners_per_vm` is the density knob: with the default `1`, each job gets its own VM (most
isolation); with `4`, one VM serves four concurrent jobs before the scaler spawns another. Small
jobs on identical macOS stacks are the sweet spot — set it high enough that `vm.cpu`/`vm.memory_gb`
cover the _sum_ of the jobs you expect on one VM.

With a GitHub scale set (`scale_set:`), keep `runners_per_vm: 1`: GitHub does not hand a second job to another idle
runner in the same VM, and the scaler counts those slots as capacity, so the job waits for the first one to finish
([details](https://kinhin.dev/v0.0.1-beta-7/guide/forges/github.md#a-second-job-stays-queued-while-the-vm-has-idle-runners-runners-per-vm-1)).

## Running on multiple Macs

Each Mac runs its own engine and polls the forge independently; kinhin does not coordinate between
machines (the daemon lock only guards against two daemons on the _same_ Mac). If two Macs serve the
same repo with the same labels, both see a queued job and both spawn a runner for it. The forge
hands the job to exactly one of them, so it never runs twice, but the other Mac boots a VM it didn't
need.

To keep demand from overlapping, give each Mac its own label:

```yaml
# Mac A
labels: ["mac-a"]
```

```yaml
# Mac B
labels: ["mac-b"]
```

A workflow then picks the Mac it wants:

```yaml
runs-on: [self-hosted, macos, arm64, mac-a] # only Mac A spawns for this job
```

A job counts as demand only when it requests _every_ configured label, so the other Mac ignores it.
To fan one job across both Macs, use a matrix:

```yaml
strategy:
  matrix:
    mac: [mac-a, mac-b]
runs-on: [self-hosted, macos, arm64, "${{ matrix.mac }}"]
```

Things to know:

- A job with neither label matches no Mac and stays queued.
- A shared pool (the same labels on both Macs) still works, but a burst can make both Macs spawn.
  Lower `max_runners` on the weaker Mac to limit the waste.

### Requeue and skip

From the Fleet tab, the **Requeue** and **Skip** buttons in the _Requeue / skip queued_
section cancel or drop the queue's matching jobs. Org-scope forges re-queue/skip the
fleet-wide queue; repo-scope forges act on that one forge. Both are single-command
actions behind the same pending-state and AlertKit failure handling as Start/Stop.

See [Requeue and skip queued jobs](https://kinhin.dev/v0.0.1-beta-7/guide/requeue-skip.md) for the full behavior.
