iOS & Mobile
RallySnap: Computer Vision & Custom Camera System
Tennis app that records a session, recognises each shot as it happens, and saves it as its own highlight clip. A player finishes a match with the reel already cut.
Highlights are cut during play rather than after it. A shot clears 0.85 confidence and a three second cooldown before the app reaches back into a ten second buffer and writes it out.
- Swift
- SwiftUI
- AVFoundation
- CoreML
- Vision
- Combine
- WatchConnectivity
Problem
Club players cannot reliably rate their own tennis. Doing it honestly needs an expert watching, which rarely happens, or filming yourself and comparing against better players, which takes more effort than most people will spend. So the skill levels people claim in a community drift away from their real ones in both directions, and matchmaking produces lopsided games. The players who do film themselves hit a second wall, because turning an hour of footage into anything worth posting means selecting, cutting, and editing it by hand.
Solution
Record the match and let the app find the highlights. A body pose action classifier trained on tennis strokes watches the player as they play, and when it recognises a shot confidently enough it reaches backwards into a rolling buffer and writes those seconds out as a clip. A session ends with the highlights already cut and waiting in a gallery, which is the part that was costing people an evening.
Architecture
Capture, classification, and clip writing sit behind Combine publishers as separate services, so the camera never has to know what a highlight is and the classifier never has to know how frames were captured.
- A tennis-specific action classifier was trained rather than a general model, reading 18 body keypoints per frame over a rolling 120-frame window.
- Detections clear a 0.85 confidence bar and a three second cooldown, which is what stops a single rally producing a burst of near-identical clips.
- Ten seconds of frames stay in a ring buffer at all times, so a clip can be written backwards from the moment a shot is recognised rather than forwards from a button press.
- Device discovery walks triple, dual-wide, dual, then single wide-angle cameras, taking the richest array the handset actually has.
- A neutral zoom factor is derived per device, 2.0 where the true 1x is an ultra-wide and 1.0 otherwise, and every requested factor is scaled through it.
- Zoom transitions call ramp(toVideoZoomFactor:withRate:), so the lens travels at a fixed rate rather than jumping to the new value.
Key features
- A zoom control that frames the same shot across different iPhone camera hardware
- Hardware zoom ramping so focal changes glide during a rally instead of stepping
- On-device action classification that writes each detected shot out of a rolling buffer as its own highlight clip
- Gallery grouping highlights by session and date, with albums, favourites, and per-clip playback
- Apple Watch companion for starting a session and marking a highlight by hand from the wrist
- Rule-of-thirds grid overlay and custom SwiftUI camera controls
- Orientation-aware recording that keeps clips the right way up
Direct contributions
- Built the camera service the whole app records through, covering capture session setup, device selection, and the zoom pipeline.
- Worked out the neutral zoom normalisation, so one control frames the same shot whichever camera array the phone has.
- Wrote the camera controls overlay and the in-camera tutorial pages.
- Worked in a team of five, with the camera as my area.
Implementation
Making 1x mean the same thing on every iPhone
Where the camera array includes an ultra-wide, the hardware's 1x is that ultra-wide, so the shot a player thinks of as normal actually sits at 2x. The neutral factor absorbs that difference before anything else touches the zoom, the front camera gets its own mapping because its scale does not match the rear, and ramp moves the lens at a fixed rate so a mid-rally zoom glides rather than snapping.
private func getBestCamera(for position: AVCaptureDevice.Position) -> AVCaptureDevice? {
let deviceTypes: [AVCaptureDevice.DeviceType] = [
.builtInTripleCamera,
.builtInDualWideCamera,
.builtInDualCamera,
.builtInWideAngleCamera
]
let discoverySession = AVCaptureDevice.DiscoverySession(
deviceTypes: deviceTypes, mediaType: .video, position: position
)
return discoverySession.devices.first
}
private func updateNeutralZoom(for device: AVCaptureDevice) {
if device.deviceType == .builtInDualWideCamera || device.deviceType == .builtInTripleCamera {
neutralZoomFactor = 2.0
} else {
neutralZoomFactor = 1.0
}
}
func setZoom(factor: CGFloat) {
guard let device = videoDeviceInput?.device else { return }
do {
try device.lockForConfiguration()
let targetAVZoom: CGFloat
if isFrontCamera {
if factor == 0.5 {
targetAVZoom = 1.0
} else if factor == 1.0 {
targetAVZoom = 1.3
} else {
targetAVZoom = factor * 1.3
}
} else {
targetAVZoom = factor * neutralZoomFactor
}
let minZoom = device.minAvailableVideoZoomFactor
let maxZoom = min(device.maxAvailableVideoZoomFactor, 5.0 * neutralZoomFactor)
let clampedFactor = max(minZoom, min(targetAVZoom, maxZoom))
device.ramp(toVideoZoomFactor: clampedFactor, withRate: 4.0)
device.unlockForConfiguration()
DispatchQueue.main.async {
self.currentZoom = factor
}
} catch {
print("Failed to lock device for zoom: \(error)")
}
}