CI: resolve or create the iOS Simulator instead of hard-coding iPhone 17 (#96) #27

Merged

backmeupplz merged 8 commits from ci/96-ios-simulator-fallback into main

2026-10-13 03:27:45 +00:00

backmeupplz commented

2026-10-13 00:37:45 +00:00

(Migrated from github.com)

Closes agentboard #96 (https://agentboard.icefish-betta.ts.net/#96).

Problem

Some macos-26 runners fail iOS jobs about 2 minutes in, before any test runs, with Unable to find a device matching the provided destination specifier.

CI on this PR showed what actually happens on such a runner (job 115074102662). Every stock runtime and device is present, and simctl boots the stock iPhone 17 (iOS 26.5) without trouble. But xcodebuild's device discovery lists no simulators within the default destination timeout, even after a CoreSimulatorService restart. Locally, -destination-timeout N makes xcodebuild wait N seconds and then fail with exactly this error (exit 70). The original failures came about 60 s in, which fits slow discovery hitting the timeout.

Change

New Resolve iOS Simulator step runs ios/ci_simulator.py. The script:
- Polls xcrun simctl list -j for up to 3 minutes and restarts a stuck CoreSimulatorService once. If no usable iOS runtime ever appears, it runs xcodebuild -downloadPlatform iOS once, streamed to the log.
- Picks the newest available iOS runtime between the app's deployment target (read from project.pbxproj, currently 26.1) and the active iphonesimulator SDK. The upper bound matches the old OS=latest.
- Reuses an existing iPhone 17 on that runtime or creates one with xcrun simctl create. If the runtime doesn't support the iPhone 17, it uses a device type with the same screen size (1206×2622 @3x, read from the device type's capabilities.plist, e.g. iPhone 17 Pro or 16 Pro), so the geometry assertions still hold. It never uses a different screen size and fails with ::error:: instead.
- Boots the simulator (simctl bootstatus -b, capped at 300 s). If the create or boot fails, it restarts CoreSimulatorService and resolves again once.
- Logs the device, runtime and build, whether it reused or created the simulator and why, and the UDID as a ::notice::.
Tests are split into two steps. build-for-testing runs against generic/platform=iOS Simulator (ARCHS=arm64, no device discovery). Then test-without-building runs on id=$SIMULATOR_ID with -destination-timeout 300, so xcodebuild looks for the already-booted device minutes later and waits for it instead of failing at the default timeout. Both steps write result bundles, and both are uploaded on failure.
Timeouts: the job limit goes from 30 to 50 minutes, which covers the download fallback plus the destination wait. The resolve step is capped at 25 minutes.
No test is skipped and the matrix is unchanged.

Tests

python3 -m unittest ios/ci_simulator_test.py: 9 selection tests covering reuse, creation, unavailable devices and runtimes, the SDK and deployment-target bounds, other platforms, the same-screen fallback and refusing a different screen size. They also run in CI before resolving.
Local end-to-end run on Xcode 27 with the reuse path disabled: the script created and booted ABS+ CI iPhone 17 and printed its UDID. Then a generic arm64 build-for-testing followed by test-without-building on that UDID ran UITests/Accessibility with 3 tests and 0 failures. I deleted the simulator afterwards.
Retry and failure paths were exercised with stubs: one failed create recovered, and two failures ended with ::error:: and exit 1.
On CI with the earlier head b44ccc2, two healthy runners reused the stock iPhone 17 on iOS 26.5, and xcodebuild then found the pre-booted device in under a second.
Not yet proven live: that -destination-timeout 300, applied after the build, is enough on a runner whose discovery is slow. The download fallback also isn't exercised live.

Closes agentboard #96 (https://agentboard.icefish-betta.ts.net/#96). ## Problem Some `macos-26` runners fail iOS jobs about 2 minutes in, before any test runs, with `Unable to find a device matching the provided destination specifier`. CI on this PR showed what actually happens on such a runner (job 115074102662). Every stock runtime and device is present, and `simctl` boots the stock iPhone 17 (iOS 26.5) without trouble. But xcodebuild's device discovery lists no simulators within the default destination timeout, even after a CoreSimulatorService restart. Locally, `-destination-timeout N` makes xcodebuild wait N seconds and then fail with exactly this error (exit 70). The original failures came about 60 s in, which fits slow discovery hitting the timeout. ## Change - **New `Resolve iOS Simulator` step** runs `ios/ci_simulator.py`. The script: - Polls `xcrun simctl list -j` for up to 3 minutes and restarts a stuck CoreSimulatorService once. If no usable iOS runtime ever appears, it runs `xcodebuild -downloadPlatform iOS` once, streamed to the log. - Picks the newest available iOS runtime between the app's deployment target (read from `project.pbxproj`, currently 26.1) and the active iphonesimulator SDK. The upper bound matches the old `OS=latest`. - Reuses an existing iPhone 17 on that runtime or creates one with `xcrun simctl create`. If the runtime doesn't support the iPhone 17, it uses a device type with the same screen size (1206×2622 @3x, read from the device type's `capabilities.plist`, e.g. iPhone 17 Pro or 16 Pro), so the geometry assertions still hold. It never uses a different screen size and fails with `::error::` instead. - Boots the simulator (`simctl bootstatus -b`, capped at 300 s). If the create or boot fails, it restarts CoreSimulatorService and resolves again once. - Logs the device, runtime and build, whether it reused or created the simulator and why, and the UDID as a `::notice::`. - **Tests are split into two steps.** `build-for-testing` runs against `generic/platform=iOS Simulator` (`ARCHS=arm64`, no device discovery). Then `test-without-building` runs on `id=$SIMULATOR_ID` with `-destination-timeout 300`, so xcodebuild looks for the already-booted device minutes later and waits for it instead of failing at the default timeout. Both steps write result bundles, and both are uploaded on failure. - **Timeouts:** the job limit goes from 30 to 50 minutes, which covers the download fallback plus the destination wait. The resolve step is capped at 25 minutes. - No test is skipped and the matrix is unchanged. ## Tests - `python3 -m unittest ios/ci_simulator_test.py`: 9 selection tests covering reuse, creation, unavailable devices and runtimes, the SDK and deployment-target bounds, other platforms, the same-screen fallback and refusing a different screen size. They also run in CI before resolving. - Local end-to-end run on Xcode 27 with the reuse path disabled: the script created and booted `ABS+ CI iPhone 17` and printed its UDID. Then a generic arm64 `build-for-testing` followed by `test-without-building` on that UDID ran `UITests/Accessibility` with 3 tests and 0 failures. I deleted the simulator afterwards. - Retry and failure paths were exercised with stubs: one failed create recovered, and two failures ended with `::error::` and exit 1. - On CI with the earlier head b44ccc2, two healthy runners reused the stock iPhone 17 on iOS 26.5, and xcodebuild then found the pre-booted device in under a second. - **Not yet proven live:** that `-destination-timeout 300`, applied after the build, is enough on a runner whose discovery is slow. The download fallback also isn't exercised live.