I did a talk on melting a CI snowflake

I presented at Melbourne CocoaHeads on how my team at REA Group moved iOS CI from a pile of laptops and Mac minis (one of which was on display in our tech hub, another with two post-it notes asking cleaners not to unplug it) into the same Buildkite-based infrastructure the rest of the org was already using.

The constraint that makes iOS CI special is that Xcode only runs on macOS, so we couldn’t just use the same Docker-on-Linux setup the rest of REA was using. We had three goals: let each iOS project pick its own Xcode version, stop projects from caring about the state of the host, and get the time-to-new-Xcode-version under a week.

We did briefly consider buying a few Mac minis for the office, but the more we listed out what that implied — networking, power, physical security, what happens if we run out of capacity — the less it looked like something we wanted to own.

The shape of the solution:

MacStadium for rack-mounted Mac minis (eight of them) with a peer firewall back into REA’s network.
Anka to virtualise macOS so every build runs in a fresh, ephemeral VM cloned from a suspended reference image. Persistent CI machines accumulate state — stale DerivedData, drifting simulator runtimes, mystery keychain entries — and we wanted that problem out of the picture.
Ansible to keep the metal hosts uniform (Homebrew, the Buildkite agent, Anka and its licence, launch-agent plists, the bootstrap hook, AWS CLI, pre-pulled VM images) and to build the VM images in layers: a base image with the macOS install and CI tweaks like disabling Spotlight, a tools image with rbenv / firebase-cli / aws-cli / jq, and an Xcode image with the chosen Xcode plus pre-warmed simulators.
Buildkite agents on the metal, three per host. Each agent’s bootstrap hook clones the chosen VM, installs certs, hands the job into the VM’s own Buildkite agent, then destroys the VM when the job finishes.

This setup got our time-to-new-Xcode-version down from “weeks” to comfortably under a week, and lets us meet Apple’s 24-hour turnaround when we have to. Teams pick their own Xcode and tool chain via an env var in their pipeline.