Cloud Testing in Depth: Penetration, Stress and Security Testing for Cloud Platforms

Unit tests tell you a function is correct. They say nothing about whether an IAM role is over-privileged, whether the database falls over at three times normal traffic, whether one tenant can read another's data, or whether the backups actually restore. Cloud systems fail at the seams between components, and those seams are only exercised by testing the deployed system as a system.

Rivix Systems builds cloud and device platforms end to end, so we test them the same way: as part of the engineering, not as a gate bolted on at the end. This note lays out the testing disciplines we apply to a cloud platform, the techniques and tooling behind each, and where the limits are. It is a description of method, not a list of past engagements.

Six questions, six kinds of testing

Most confusion about "cloud testing" comes from treating it as one activity. It is better to start from the question you need answered:

Question	Discipline	Typical output
Can an attacker get in, and how far can they go?	Penetration testing	Exploit chains, ranked findings, retest
Does it hold its targets at peak traffic?	Load testing	Latency percentiles, error rate, saturation points
Where does it break, and how does it fail?	Stress, spike and soak testing	Breaking point, failure mode, recovery time
Are known weaknesses creeping into every change?	Security testing in CI/CD	Pipeline gates: SAST, SCA, secrets, IaC, image and DAST scans
Does it survive component failure?	Resilience and chaos testing	Verified failover, RTO and RPO measured by restoring
Do the pieces still agree with each other?	Integration, contract and data testing	Contract verification, migration and isolation checks

The rest of this note works through them in the order we usually apply them: security first, because it changes what you build; then scale; then failure.

Rules of engagement come before tools

Offensive and high-volume testing against a cloud environment is only appropriate with a written scope and authorization from whoever owns the system. Before any test we agree in writing on:

• Scope. Which accounts, subscriptions or projects, which environments, which endpoints. Third-party SaaS and shared provider infrastructure are out of scope unless their owners have agreed.
• Provider policy. AWS, Azure and Google Cloud each publish a testing policy, and they generally permit customers to security-test their own resources while restricting denial-of-service simulation and anything touching shared infrastructure. The policies change, so we read the current version for each engagement rather than relying on memory.
• Environment. Whenever possible, a production-equivalent staging environment built from the same infrastructure-as-code, not production. Where production must be tested, we agree on windows, rate limits and a stop condition.
• Data handling. No real personal data in test environments; synthetic data at realistic volume and shape instead.
• Communication. A named contact on each side, an emergency stop, and a plan for what happens if we find something critical mid-test — it is reported immediately, not saved for the final report.

The cloud attack surface we test

Cloud penetration testing is only partly about servers. Much of the attack surface is configuration and identity.

Identity and access management

IAM is the most productive area in almost every cloud assessment. We enumerate roles, policies and trust relationships and look for wildcard actions and resources, roles that can be assumed from unexpected principals, cross-account trust that is broader than intended, and privilege-escalation paths — permissions that look harmless individually but chain into administrative access (for example, the ability to attach a more powerful policy, pass a privileged role to a compute service, or modify a function that runs with a broader role). The finding is the path, not the individual permission.

Metadata service and SSRF

A server-side request forgery bug in an application becomes far more serious if it can reach the instance metadata endpoint and read temporary credentials for the attached role. We test whether that path is open, whether the metadata service enforces session-token requests (IMDSv2 on AWS), and whether the role attached to the workload is scoped to what the workload needs. Fixing the SSRF and restricting the role are both required; either one alone leaves a gap.

Storage, secrets and exposed services

Publicly readable or writable storage, snapshots and images shared more widely than intended, management ports and dashboards reachable from the internet, and secrets in source repositories, container images, build logs and environment dumps. Leaked secrets are usually the cheapest way in, so we search history as well as the current tree.

APIs and application logic

For the API layer we follow the OWASP API Security Top 10: broken object-level and function-level authorization, broken authentication, excessive data exposure, mass assignment, unrestricted resource consumption and injection. Authorization flaws are the ones scanners miss, because they require understanding what each user is supposed to be able to see. We test them by acting as two different users, or two tenants, and attempting to cross the line. Design choices that make this easier to get right are covered in API design for device-to-cloud systems.

Containers and Kubernetes

Privileged containers and host mounts, over-broad service-account tokens and RBAC bindings, exposed kubelet or dashboard endpoints, missing network policies that leave every pod able to talk to every other pod, and secrets stored where any workload in the namespace can read them. We benchmark against CIS guidance and then try to use what we find.

Serverless and event-driven code

Functions inherit their risk from their execution role and their event source. We test event injection through queues, storage notifications and HTTP triggers, functions sharing one over-broad role, and secrets passed as plain environment variables.

Multi-tenant isolation

In a SaaS platform, the worst finding is one tenant reaching another's data. We test isolation at every layer where it is enforced — API authorization, database row or schema separation, object-storage prefixes, cache keys, background-job queues and logs — because a single missed layer is enough. See our note on multi-tenant SaaS isolation for the design side.

CI/CD pipeline

The build system usually holds credentials that can deploy to production, which makes it a target in its own right. We review who can modify pipeline definitions, whether pull-request builds from untrusted branches can reach secrets, how cloud credentials are issued to runners (short-lived federated identity rather than long-lived keys), and whether artifacts are signed and verified before deploy.

Tooling, and its limits

Tools speed up enumeration; they do not replace judgment. Our usual kit includes Prowler and ScoutSuite for configuration auditing, Pacu and CloudFox for AWS enumeration and privilege-escalation analysis, kube-bench and Trivy for cluster and image checks, Burp Suite, OWASP ZAP, ffuf and nuclei for the web and API layer, and nmap for the network surface. A scanner will report a hundred misconfigurations; a tester decides which three of them chain into something that matters.

Severity and reporting

Every finding is written up with reproduction steps, the evidence, the affected resources, a severity rating (we use CVSS as a starting point and adjust for real exploitability and business impact) and a specific fix — the IAM policy change, the configuration flag, the code path — not just a category. The report leads with the attack narrative for the most serious chain, then lists the rest. We retest after fixes; a finding is closed when we have failed to exploit it again, not when a ticket is marked done.

One boundary worth stating: some compliance frameworks and enterprise customers require a penetration test from an independent, qualified third party. Where that applies, we recommend engaging one, and we do the engineering groundwork — architecture, hardening, internal testing and evidence — that makes that assessment faster and cleaner. Internal testing and independent attestation do different jobs.

Load and stress testing

Performance testing is really a family, and the members answer different questions:

Test	What it does	What it tells you
Load	Expected and peak traffic, held steady	Whether you meet latency and error-rate targets
Stress	Ramps beyond capacity until something gives	The breaking point, and whether failure is graceful
Spike	A sudden multiple of normal load	Whether autoscaling and queues absorb a surge in time
Soak	Moderate load for many hours	Memory leaks, connection exhaustion, log and disk growth
Capacity	Stepwise increase while measuring cost and headroom	How many units of traffic each unit of infrastructure carries
Recovery	Overload, then return to normal	How long until latency and backlogs return to baseline

What makes a load test believable

• Start from a service-level objective. "Handle load" is not a pass criterion. "p99 latency under 900 ms and error rate under 1% at 800 requests per second" is.
• Model the real workload. The mix of reads and writes, payload sizes, hot keys, session behavior and tenant distribution decide how a system behaves. A uniform synthetic load will flatter your caches and your database.
• Use an open workload model. Real users do not wait politely for the previous response. A closed model, where each virtual user sends the next request only after the last returns, slows down when your system slows down and hides the overload it is supposed to reveal (the problem known as coordinated omission). Arrival-rate executors fix this.
• Read percentiles, not averages. An average can look healthy while one request in a hundred takes ten seconds.
• Test at production shape. Realistic data volume in the database, production-equivalent instance types, the same network path, the same rate limits and the same downstream dependencies (or faithful stubs that reproduce their latency).
• Generate load from outside the system under test, on enough capacity that the load generator itself is not the bottleneck.

A shortened example of an arrival-rate scenario in k6, with pass/fail thresholds encoded so it can run in a pipeline:

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    ingest: {
      executor: 'ramping-arrival-rate',
      startRate: 50, timeUnit: '1s',
      preAllocatedVUs: 200, maxVUs: 1000,
      stages: [
        { target: 200, duration: '5m' },   // ramp to expected peak
        { target: 200, duration: '20m' },  // hold
        { target: 800, duration: '2m' },   // spike
      ],
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(95)<400', 'p(99)<900'],
  },
};

export default function () {
  const res = http.post(`${__ENV.BASE_URL}/v1/telemetry`, JSON.stringify(payload()),
    { headers: { 'Content-Type': 'application/json' } });
  check(res, { accepted: (r) => r.status === 202 });
}

Finding the bottleneck, not just the number

A load test that ends with "it fell over at 900 requests per second" is half a result. While it runs we watch utilization, saturation and errors for every resource (the USE method): CPU and memory on compute, connection-pool and lock waits and replication lag on the database, queue depth and consumer lag on messaging, throttling responses from managed services, and the autoscaler's own reaction time. Then we change one variable and run again. The usual culprits are unglamorous: an undersized connection pool, a missing index, a synchronous call to a slow dependency, a rate limit on a managed service nobody remembered, or an autoscaling policy that reacts a minute after the spike has already done its damage.

Stress testing is about the failure mode

When you push past capacity, what you want to see is load shedding: fast, explicit rejections (429 or 503 with a retry hint), queues that are bounded, and recovery within a known time after load drops. What you do not want is the cascading version — timeouts that trigger retries that multiply the load, threads pinned waiting on a dead dependency, and a system that stays broken after the traffic has gone. We check timeouts, retry budgets with jitter and backoff, circuit breakers and bulkheads explicitly, by making a dependency slow or unavailable during the run.

Note the provider policy point again: stress testing your own workload is normal practice, but tests that resemble a denial-of-service attack against the provider's network or shared services are restricted. We design stress tests to target your application and its provisioned resources, at rates within the current policy.

Testing an IoT cloud backend

A device backend has load patterns that web-application tooling does not reproduce by default, and this is where hardware-to-cloud experience matters.

• Persistent connections at scale. Hundreds of thousands of long-lived MQTT sessions stress the broker's connection, subscription and memory limits differently from request/response traffic. Fleet simulators (for example emqtt-bench, or purpose-built simulators that speak your real protocol and use real per-device certificates) are needed to get the shape right.
• Reconnect storms. After a broker or network outage every device reconnects at once. If firmware retries on a fixed interval, the recovery itself becomes a spike that knocks the system over again. We test recovery with the retry logic the firmware actually ships, and confirm exponential backoff with jitter is really implemented on the device.
• Staged OTA rollouts. A firmware release makes a whole cohort download a large image at once. We test the CDN and signing path under a realistic rollout curve, the pause-and-rollback controls, and what happens to devices that lose power mid-download. See OTA firmware updates.
• Device identity and isolation. Every device credential should reach only its own topics and records. We test that a device certificate cannot subscribe to another device's topic or wildcard, cannot publish to a command channel, and is rejected after revocation. See device identity and provisioning security.
• Shadow and state conflicts. Devices that come back online after days offline, with stale state, racing against commands queued in the cloud. See device shadows and digital twins.
• Certificate and clock events. Expiry dates and clock drift produce fleet-wide failures at a single moment. We test rotation before the date arrives, not after.

Scale considerations for the platform itself are covered in scaling a device fleet; testing is how you find out those designs actually hold.

Resilience, chaos and disaster-recovery testing

Redundancy that has never been exercised is a hypothesis. We test the claims your architecture makes:

• Dependency failure. Inject latency, errors and outright unavailability into databases, queues and third-party APIs (Toxiproxy for local and staging, provider fault-injection services or Chaos Mesh and LitmusChaos for cloud and Kubernetes) and confirm timeouts, retries and fallbacks behave as designed.
• Infrastructure failure. Terminate instances and nodes, drain a zone, fail over a database, and measure how long recovery takes and what users see meanwhile.
• Backup and restore. A backup is only proven when it is restored. We restore into a clean environment, check the data, and time it against the stated recovery time and recovery point objectives (RTO and RPO). The numbers in a disaster-recovery plan should be measurements, not estimates.
• Game days. Scheduled exercises where the team responds to a staged failure using the real runbooks and dashboards. They reveal missing alerts, stale runbooks and unclear ownership, which are more often the cause of long outages than the technical fault itself.

Start small and in staging, with a hypothesis ("if the primary database fails over, error rate stays below 2% for under a minute"), a defined blast radius and a stop condition. Chaos testing is controlled experimentation, not random destruction.

The rest of the test surface

• Contract testing. When services or a mobile app and a backend evolve independently, consumer-driven contracts (for example with Pact) catch breaking API changes before integration, without standing up the whole system.
• Data and migration testing. Schema migrations rehearsed against production-sized data, with rollback proven; idempotency and ordering checks on event consumers, including duplicate and out-of-order delivery.
• Infrastructure-as-code testing. Plan review and policy checks on every change, ephemeral environments created from the same code as production, and periodic drift detection between code and live state.
• Observability testing. Deliberately causing a failure to confirm the right alert fires, reaches the right person and links to a useful dashboard. An alert that has never fired is untested.
• Cost testing. Load tests double as cost models: cost per thousand requests or per device per month at each scale step, and checks that autoscaling scales back down.
• Compliance evidence. Many controls in frameworks such as SOC 2 and ISO 27001 are demonstrated by test records: access reviews, restore tests, vulnerability scans and pen-test reports. We keep the artifacts as a by-product of testing rather than assembling them before an audit. Certification itself is a separate process performed by an independent assessor.

Common questions

What is the difference between load testing and stress testing?

Load testing checks that a system meets its latency and error targets at expected and peak traffic. Stress testing pushes past that ceiling on purpose to find the breaking point, learn how the system fails and confirm it recovers. Spike and soak tests are variations that test sudden surges and long-duration stability.

Do I need permission to penetration test my own cloud environment?

You need written authorization from the system owner and a defined scope, and you must follow the cloud provider's current testing policy. Providers generally allow customers to test their own resources within limits but restrict denial-of-service simulation and testing of shared infrastructure, so check the policy before any test.

What are the most common cloud security findings?

Over-permissive IAM roles and privilege-escalation paths, publicly exposed storage or services, secrets committed to code or CI logs, server-side request forgery reaching the instance metadata service, broken object-level authorization in APIs, and weak tenant isolation in multi-tenant systems.

Can security testing be automated in CI/CD?

A large part of it can: static analysis, dependency and container image scanning, secrets detection, infrastructure-as-code policy checks and baseline dynamic scans all run in pipelines. Automation catches known patterns cheaply; manual penetration testing is still needed for business-logic flaws and chained attack paths.

How is testing an IoT cloud backend different?

Traffic comes from devices, not browsers: persistent MQTT connections, bursty reconnect storms after an outage, staged OTA rollouts and certificate expiry events. Tests need fleet simulators that reproduce those patterns, plus checks that each device identity can only reach its own topics and data.