1. Targets before tests
"Make it fast" is not a pass criterion. We agree, with product and engineering, a short list of numbers written down before any test runs: for example a cold-start budget on a named low-end device, a p95 and p99 latency per critical API, an error-rate ceiling, a peak concurrent-session target, and a battery or data budget for a defined background activity. These come from your users, your business forecast and your contracts, not from us; where nothing exists we propose targets and have you confirm them. The numbers in this article's code samples are placeholders for that reason.
2. A workload model built around the app session
The most useful modelling shift on a mobile product is to make the app session, not the individual API call, the unit of load. One launch might fire a burst of parallel requests, then a sync, then a feed refresh, then idle for a while, then a push-triggered refresh. We derive that shape from the app's own code and from analytics or production logs where they exist, and we capture it as a scenario. This is where mobile and cloud testing meet: the app's behavior (parallel calls on launch, a sync on every foreground, aggressive retries) is the backend's load profile.
We also model the awkward moments that dominate real incidents: a new release prompting every user to update and reopen the app, a marketing push notification that wakes a large share of devices together, and the reconnect wave after a backend outage when every client retries at once.
3. A production-shaped environment
Numbers from a tiny staging stack with an empty database are not evidence. We test against infrastructure built from the same infrastructure-as-code as production, with realistic data volume and distribution (synthetic, never real personal data), the same instance classes, caches and rate limits, and faithful stubs for third-party dependencies that reproduce their latency. On the device side we pick a matrix from your actual install base: at least one low-end and one mid-range Android device, an older supported iPhone and a current one, across the OS versions you support. The slowest supported device sets the budget, because that is where a regression shows first.
4. Instrument both sides first
A load test that reports only "p99 was too high" tells you nothing about why. Before testing, we make sure a request can be followed from the phone to the database: distributed tracing with OpenTelemetry propagated from the client, RED metrics (rate, errors, duration) per endpoint, and USE metrics (utilization, saturation, errors) for databases, queues, pools and hosts. On the app, we add startup and screen-render signposts, and crash and ANR reporting. This costs a day or two and saves weeks of guessing.
5. Baseline, then change one thing at a time
The first run is deliberately unflattering: a recorded baseline of the current behavior. Every later claim ("the new index cut the dashboard query time") is compared with it, run under identical conditions, with enough repetitions that normal variation is separated from real change. Device measurements are repeated many times for the same reason; a single launch time is an anecdote.
6. Run the right tests, and read them properly
The test types are covered below, for the cloud and for the device. Two habits matter more than any tool: we read percentiles rather than averages, and we check that the load generator and the network path were not themselves the bottleneck.
7. Fix, retest, document
Each finding is written up with the evidence (trace, flame graph, query plan, profiler capture), the probable cause, a recommended fix and an expected effect. After the fix we re-run the same scenario and record the before-and-after. A finding is closed by a re-measurement, not by a code review saying it looks right.
8. Keep it from regressing
Performance decays through ordinary changes. The scenarios we build become part of your pipeline: API thresholds checked on a schedule or on release candidates, and startup and frame-time benchmarks that fail a build when they cross the budget. The suites are handed over with documentation, so your team owns them after we leave.
Testing the cloud side
For the backend, we use load generators that let scenarios live in code and run in a pipeline (k6 is our usual choice; Locust, Gatling and JMeter are alternatives where a team already uses them), with the system under test observed through its own telemetry. The six test types we apply are described in our cloud testing note; the table below is what we specifically look at once they run.
| Area | What we check | Typical symptom when it fails |
|---|
| API latency | p95/p99 per endpoint against the SLO, split by payload size and tenant | Fine on average, slow for large accounts |
| Database | Slow-query log, query plans for the top queries, missing or unused indexes, N+1 access patterns, lock contention, replica lag | Latency climbs steeply once the table reaches realistic size |
| Connection pools and threads | Pool sizes against database limits, thread or worker saturation, queueing time | Latency cliff and timeouts while CPU still looks idle |
| Pagination and payloads | Unbounded list endpoints, over-fetching, response size and compression | Mobile screens that load slowly on cellular and burn data |
| Caching | Hit rate under a realistic key distribution, stampede behavior on expiry | Backend load spikes whenever a hot key expires |
| Autoscaling | Time from load increase to new capacity serving traffic, scale-down behavior, serverless cold starts | Errors during the first minutes of a surge |
| Queues and async work | Backlog growth and drain time, consumer idempotency, dead-letter handling | Notifications or syncs arriving long after the event |
| Push and fan-out | Send throughput, provider rate limits, the wave of API calls when many devices open the app together | A campaign push that doubles as a self-inflicted load test |
| Retry and backoff | Client retries with exponential backoff and jitter, server-side rate limiting and load shedding, circuit breakers | Retry storms that turn a brief blip into a long outage |
| Cost | Cost per thousand sessions at each load step, and whether capacity scales back down | A system that scales fine and costs too much to run |
A trimmed example of the app-session scenario in k6 follows. Each iteration plays one session, arrivals are paced at a set rate rather than by how fast the system answers (the open model, which avoids hiding overload), and the pass/fail thresholds are encoded so it can run in a pipeline. The thresholds and rates are placeholders.
import http from 'k6/http';
import { check, sleep } from 'k6';
export const options = {
scenarios: {
app_sessions: {
executor: 'ramping-arrival-rate', // sessions per second, independent of response time
startRate: 5, timeUnit: '1s',
preAllocatedVUs: 100, maxVUs: 500,
stages: [
{ target: 50, duration: '5m' }, // ramp to expected peak
{ target: 50, duration: '20m' }, // hold
{ target: 0, duration: '2m' },
],
},
},
thresholds: { // placeholders: set from your SLOs
http_req_failed: ['rate<0.01'],
'http_req_duration{endpoint:launch}': ['p(95)<400', 'p(99)<900'],
'http_req_duration{endpoint:sync}': ['p(99)<1500'],
},
};
export default function () {
const auth = { headers: { Authorization: `Bearer ${__ENV.TEST_TOKEN}` } };
// 1. Launch: the app fires its startup calls in parallel
const launch = http.batch([
['GET', `${__ENV.BASE}/v1/profile`, null, { ...auth, tags: { endpoint: 'launch' } }],
['GET', `${__ENV.BASE}/v1/feed?limit=20`, null, { ...auth, tags: { endpoint: 'launch' } }],
]);
check(launch[0], { 'profile ok': (r) => r.status === 200 });
// 2. Sync after the user does something
const sync = http.post(`${__ENV.BASE}/v1/sync`, JSON.stringify({ since: 0 }),
{ ...auth, tags: { endpoint: 'sync' } });
check(sync, { 'sync ok': (r) => r.status === 200 });
sleep(Math.random() * 20 + 5); // think time before the next action
}
If the product is a connected device with a companion app, the same approach extends to the device fleet: simulated devices speaking the real protocol with real certificates, reconnect storms and staged firmware rollouts, as described in the cloud testing note and in our piece on BLE sync protocol design, which covers the phone-to-device link that sits between the two.
Testing the mobile side
Mobile performance has to be measured on real hardware. Emulators and simulators run on a desktop CPU with a very different memory, thermal and graphics profile, so they are fine for functional work and misleading for performance. We use real devices on a bench or a device cloud (Firebase Test Lab, AWS Device Farm and BrowserStack are the usual options), and always build the app in a release-like configuration; a debug build can be several times slower and tells you nothing about what users get.
| Area | What we check | How |
|---|
| Startup | Cold, warm and hot start; time to first usable screen; work done before first frame | Android: Macrobenchmark, Perfetto. iOS: Instruments, XCTest launch metrics |
| Rendering | Dropped and slow frames on key flows (lists, transitions, maps, charts); main-thread work | Android: Macrobenchmark frame timing, Perfetto. iOS: Instruments. Flutter: DevTools in profile mode |
| Memory | Peak and steady-state footprint, leaks after repeated navigation, large images, behavior under system memory pressure | Android Studio profiler, LeakCanary. Xcode Allocations and Leaks |
| Battery and background | Energy cost of a defined session, wake-ups, background sync cadence, location and Bluetooth usage | Android: Perfetto and system battery stats. iOS: Instruments Energy Log |
| Network use | Requests per screen, payload sizes, duplicate calls, caching headers, data used by a typical session | Proxy capture (mitmproxy, Charles), client-side tracing |
| Poor connectivity | Behavior on slow, high-latency, lossy and dropping connections; timeouts and retries; offline and reconnect handling | Network shaping (see below); device network-condition tools |
| App size and install | Download size, install time, first-run work, upgrade from the previous version with existing data | Store tooling, scripted install and upgrade runs |
| Field signals | Real-world launch times, ANRs, crash and hang rates by device and OS | Android vitals, Xcode Organizer and MetricKit, crash reporting |
As a flavor of how the startup budget is enforced, here is a trimmed Android Macrobenchmark test. It runs repeated cold starts on a real device and records the timings, and a CI job compares the result with the agreed budget:
@RunWith(AndroidJUnit4::class)
class StartupBenchmark {
@get:Rule
val rule = MacrobenchmarkRule()
@Test
fun coldStartup() = rule.measureRepeated(
packageName = "com.example.app",
metrics = listOf(StartupTimingMetric()),
iterations = 10,
startupMode = StartupMode.COLD,
compilationMode = CompilationMode.Partial(),
) {
pressHome()
startActivityAndWait()
}
}
Network conditions are part of the test
Most app testing happens on office Wi-Fi, and most user complaints do not. We run the critical flows with the network shaped: added latency, reduced bandwidth, packet loss, and mid-request disconnects. On a controlled access point this can be done with Linux traffic control (tc netem), and iOS and Android both have developer tooling for throttling. What we look for is not just slowness but behavior: does the user see a sensible state, are requests retried with backoff, are writes safe to repeat, is a half-finished sync recoverable, and does the app recover cleanly when the connection returns? For products that must work offline, this connects directly to offline-first design.
Putting the two together
The tests that expose the most are the ones that run both sides at once: realistic backend load while a real device runs the critical flows. A screen that renders smoothly against an idle backend may stutter when the API is at peak load, and that gap is the user's actual experience. We run it in three standard ways:
- Loaded-backend device runs. Backend at expected peak while a bench device repeats key journeys and records launch, transition and frame metrics.
- Degradation drills. Slow or fail a dependency, or shape the network, and observe both the backend's reaction (queues, retries, shedding) and what the app shows the user.
- Release-event simulations. The update-and-reopen wave, a push-driven burst, and the reconnect storm after an outage, since these are the events that become incidents.
An illustrative walkthrough
This section describes our method as a composite of how engagements of this kind are run. It is not a report on one named client, and it deliberately contains no client data or measured figures.
Take a connected-device product: hardware that syncs to a companion phone app, with a cloud platform behind it that stores history, runs analytics and sends notifications. The goal is to be confident before launch. We would proceed like this:
- Week one: targets and instrumentation. Agree budgets with the product owner (startup on the lowest supported phone, sync time for a typical backlog, API percentiles, a launch-day concurrency forecast). Add tracing from app to database, and release-mode benchmark builds.
- Baseline. Run the session scenario against a production-shaped stack seeded with synthetic history of realistic size, and record device timings on the benchmark phones. This is the reference every fix is judged against.
- Find the cliffs. Step the load up and watch where latency bends. The usual culprits are the ones in the tables above: a history query that scans instead of seeking, a pool exhausted while CPU is idle, an endpoint that returns far more than the screen shows, a cache that stampedes on expiry.
- Device findings. Profile the cold start and the main flows. Typical findings are work done on the main thread before first frame, images decoded at full size, and background sync that wakes the radio too often.
- Break it on purpose. Spike the load, shape the network, kill a dependency, and then replay the reconnect storm. Confirm the app retries with backoff and jitter, the server sheds load rather than collapsing, and recovery time is measured rather than assumed.
- Fix, re-measure, hand over. Each fix is re-tested against the baseline, and the scenarios and benchmarks are left running in CI with budgets, plus a runbook for investigating a regression.
If you want to see how we approach full products rather than isolated tests, our proof of work pages show the platforms, apps and devices we have built and the engineering behind them.
How to judge any performance-testing claim
Whoever does your testing, these questions separate real evidence from decoration. We are happy to be held to all of them:
- What was the target, and who agreed it before the test ran?
- Was the environment production-shaped, with realistic data volume?
- Was the workload modelled on real usage, and was the arrival model open or closed?
- Are results reported as percentiles, with the load generator shown not to be the bottleneck?
- Were device numbers taken on real, low-end hardware in a release build?
- Was the weak-network behavior tested, not just the strong-network speed?
- Is there a baseline, and was each fix proven by a re-measurement?
- Will the tests still be running, with budgets, after the engagement ends?
