Development, Testing, and DevSecOps Delivery

Introduction

Serverless applications and microservices are assembled from small, independently deployable services. This architecture can improve scalability and team autonomy, but it also increases the number of interfaces, deployment units, permissions, observability requirements, and possible failure points. A change to one function may affect an event schema, an API consumer, a queue, a database, and an infrastructure policy.

Professional delivery therefore requires more than writing code. Developers must reproduce relevant cloud behaviour locally, test services at several levels, protect the software supply chain, automate quality checks, and release changes through controlled deployment strategies. These activities are commonly brought together under DevSecOps: the integration of development, security, and operations throughout the software delivery lifecycle.

This chapter presents a practical foundation for developing, testing, and delivering serverless and microservice-based systems. It focuses on disciplined engineering rather than dependence on a particular cloud provider.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the role and limitations of local emulation for serverless systems.
2. Select appropriate unit, contract, integration, performance, and failure tests.
3. Design testable functions and microservices using dependency isolation.
4. Apply version-control and CI/CD practices to distributed applications.
5. Integrate security checks into the development and delivery pipeline.
6. Compare deployment strategies, including rolling, blue–green, canary, and feature-flag releases.
7. Interpret test and operational evidence to make responsible release decisions.

1. Delivery Characteristics of Serverless and Microservice Systems

A traditional monolithic application may be built and tested as one deployable artefact. A microservice system instead contains multiple services, each with its own code, configuration, dependencies, data access, and deployment lifecycle. A serverless system adds managed runtimes, event sources, queues, gateways, identity policies, and provider-specific behaviour.

Several characteristics affect delivery:

• Distributed execution: A request may cross several processes, networks, queues, or regions.
• Asynchronous behaviour: A producer may complete before a consumer processes its event.
• Ephemeral compute: Functions can start, stop, scale, and run concurrently.
• Managed dependencies: The cloud provider controls parts of the runtime, networking, and service availability.
• Independent change: A service can be deployed separately, but only if its contracts remain compatible.
• Configuration sensitivity: Environment variables, secrets, permissions, timeouts, and event-source settings influence behaviour as much as source code.

A useful delivery principle is that every change should be repeatable, observable, reversible, and appropriately tested. Repeatability comes from versioned source and infrastructure. Observability comes from logs, metrics, and traces. Reversibility comes from immutable artefacts and rollback or forward-fix procedures. Appropriate testing means matching the test type to the risk rather than attempting to test everything through slow end-to-end scenarios.

2. Local Development and Emulation

Local development reduces feedback time and cloud usage. A developer may run a function directly, use a framework emulator, start dependent services in containers, or deploy temporarily to a shared development account. These approaches are complementary.

2.1 Levels of local execution

Direct invocation calls a handler with a representative event. It is fast and useful for unit testing, but it does not reproduce networking, identity, queues, retries, or provider-specific runtime behaviour.

Service emulation imitates APIs such as object storage, queues, databases, or API gateways. It supports realistic workflows while keeping development local. However, an emulator may omit limits, timing effects, failure modes, or subtle differences in authentication and consistency.

Containerised dependencies package databases, message brokers, and test doubles in reproducible environments. Containers help teams share setup instructions and reduce “works on my machine” problems.

Ephemeral cloud environments deploy a branch or pull request into an isolated account or namespace. They provide higher fidelity but consume time and resources and require careful cleanup and access control.

A local environment should be described as an approximation, not as proof that the production system will behave identically. Important provider-specific behaviours must be validated in a real cloud environment.

2.2 Configuration and environment parity

Configuration should be externalised from application code. A typical service may require a database endpoint, queue name, timeout, feature flag, and identity configuration. Values should be supplied through environment-specific configuration, while sensitive values should be held in a secrets-management system rather than committed to source control.

A practical configuration hierarchy is:

1. Safe defaults for local development.
2. Explicit local overrides for a developer or test run.
3. Versioned environment configuration for development, staging, and production.
4. Secret references resolved at runtime or deployment time.

Configuration schemas should be validated when a service starts. Failing early on a missing or malformed value is safer than allowing a partially configured service to process production traffic.

2.3 Designing for local testability

A function should separate its business decision from infrastructure access. For example, order validation can be implemented as a pure function receiving an order and returning a decision. Database access, clock access, random identifiers, and message publication can be supplied through interfaces or adapters. During unit testing, these dependencies can be replaced with controlled fakes.

This separation is sometimes called hexagonal, ports-and-adapters, or clean architecture. It does not require a large framework. Its central idea is that domain logic should not be inseparably coupled to a cloud SDK.

3. A Layered Testing Strategy

No single test type provides sufficient confidence. A layered strategy places fast, focused tests near the code and reserves slower, broader tests for interactions that genuinely require them.

The objective is not a rigid numerical ratio of test types. The appropriate balance depends on architecture, risk, regulatory obligations, and change frequency.

3.1 Unit testing

A unit test isolates a small unit of behaviour and checks an expected result for a controlled input. Good unit tests are deterministic, readable, and independent of network services. They should cover normal cases, boundary conditions, invalid inputs, and important error paths.

For a function that calculates shipping eligibility, tests might include:

• an order below the minimum value;
• an order exactly at the threshold;
• an order containing a prohibited destination;
• a missing address;
• a dependency returning an error.

A test should not merely increase code coverage. Coverage indicates which statements or branches executed; it does not prove that assertions are meaningful. Mutation testing, review, and risk-based test design can reveal weak tests that execute code without checking useful outcomes.

3.2 Contract testing

A contract specifies an agreement between a service and its consumer. For an HTTP API, it may define the method, path, request fields, response fields, status codes, and error representation. For an event, it may define the event type, version, required attributes, data types, and delivery assumptions.

Consumer-driven contract testing allows a consumer to express the interaction it requires. The provider verifies that its implementation satisfies the published expectations. This can detect breaking changes before the services are deployed together.

Compatible evolution usually involves additive change: adding an optional response field is often safer than removing or renaming a required field. Producers should publish a new event version when semantics change materially. Consumers should tolerate unknown fields and, where appropriate, process duplicate messages safely.

A contract test is not a substitute for integration testing. It verifies an interface agreement, not database correctness, network configuration, or the full business workflow.

3.3 Integration testing

Integration tests exercise a service with a real dependency or a high-fidelity substitute. Examples include writing to a temporary database, publishing to a local message broker, or invoking an API gateway emulator.

Tests should control their data and clean up after execution. Isolation can be achieved through transaction rollback, unique test identifiers, disposable databases, or separate namespaces. Shared mutable test environments tend to produce order-dependent and intermittent failures.

Integration tests should verify details that unit tests cannot, such as serialization, authentication configuration, query behaviour, queue acknowledgement, and retry handling. They should also test timeouts and the service’s response to dependency errors.

3.4 End-to-end testing

End-to-end tests follow a meaningful business journey across multiple services. For example, a purchase test may create an order, authorize payment, reserve inventory, and verify a notification. These tests provide valuable confidence but are slower and more fragile than unit tests.

Use a limited number of stable end-to-end journeys for critical paths. Avoid using them to check every validation rule; those rules belong in unit or contract tests. When an end-to-end test fails, distributed tracing and correlation identifiers are essential for locating the fault.

4. Performance and Capacity Testing

Performance testing asks whether a system satisfies measurable service objectives under a defined workload. Useful measures include latency, throughput, error rate, concurrency, and resource utilisation.

A latency percentile describes the response time below which a specified proportion of observations fall. For example, a p95 latency of 400 ms means that approximately 95% of measured requests completed in 400 ms or less; it does not mean that every request completed within 400 ms.

A basic utilisation relationship for a single-server approximation is expressed by Little’s Law:

Here, L is the average number of items in the system, λ is the average arrival rate in items per unit time, and W is the average time an item spends in the system. The relationship assumes a stable system over the measurement interval and consistent definitions of the system boundary. It can help reason about queued messages or in-flight requests, but it does not by itself predict latency or capacity.

For example, if a queue receives an average of 20 messages per second and each message spends an average of 0.5 seconds in the system, the average number of messages in the system is:

This result is an average, not a maximum queue size. Bursts, retries, uneven processing times, and failures may produce larger values.

Performance tests should define:

• workload shape, such as steady, ramping, burst, or spike traffic;
• request mix and payload sizes;
• warm and cold execution conditions where relevant;
• success and error criteria;
• duration and data-reset procedure;
• acceptable latency percentiles and throughput;
• cost and resource limits.

Serverless performance testing must consider concurrency limits, throttling, cold starts, downstream bottlenecks, connection management, and queue backlogs. A function may scale rapidly while a database cannot accept the resulting connection load. Performance is therefore a property of the whole dependency path, not only of function execution time.

5. Failure and Resilience Testing

Distributed systems fail in ordinary ways: networks delay or drop messages, services return errors, credentials expire, queues deliver duplicates, and dependencies become slow. Failure testing deliberately explores these conditions before production exposes users to them.

Important scenarios include:

• dependency timeout;
• dependency returning a 5xx response;
• malformed or unexpected event;
• duplicate event delivery;
• partial completion before a crash;
• queue backlog and poison messages;
• exhausted concurrency or rate limits;
• unavailable secrets or configuration;
• regional or provider service disruption.

A resilient consumer should usually be idempotent: processing the same logical message more than once should not create an incorrect additional effect. A common approach is to assign an idempotency key and record completed keys with an appropriate retention period. The storage operation that records the key and the business effect must be designed carefully; recording a key without completing the effect, or completing the effect without recording the key, can both create inconsistencies.

Retries require particular care. Exponential backoff increases the delay between attempts, and jitter adds controlled randomness so that many clients do not retry simultaneously. Retrying a non-idempotent operation without protection may create duplicate charges or orders. Some errors are permanent and should be routed to a dead-letter queue rather than retried indefinitely.

Failure tests should have explicit safety boundaries. Use test accounts, synthetic data, restricted permissions, and an abort mechanism. Chaos experiments in production require authorisation, monitoring, a defined hypothesis, and a rollback plan.

6. Version Control and Collaborative Engineering

Git or a comparable version-control system should contain application source, test code, infrastructure definitions, dependency manifests, configuration schemas, and documentation. Secrets, generated build outputs, and personal environment files should be excluded.

A useful commit is small enough to review and describes one coherent change. Pull requests should provide context, tests performed, security implications, migration considerations, and operational impact. Automated checks should run before approval, while at least one qualified reviewer evaluates design and risk.

Branching strategy should support frequent integration. Long-lived branches increase merge conflicts and allow incompatible assumptions to persist. Trunk-based development with short-lived branches is often effective, although regulated environments may require additional approval and release controls.

Database and event-schema changes deserve special treatment. A safe migration may use an expand-and-contract sequence:

1. Add the new field or structure while retaining the old one.
2. Deploy code that can read both forms and writes the new form.
3. Migrate existing data.
4. Stop writing the old form.
5. Remove the old structure only after all consumers have migrated.

This approach avoids requiring every service to change simultaneously.

7. CI/CD and the DevSecOps Pipeline

Continuous integration (CI) automatically builds and tests changes whenever they are proposed or merged. Continuous delivery (CD) keeps a tested release ready for deployment. Continuous deployment automatically promotes approved changes into an environment according to defined controls.

A representative pipeline is:

1. Source stage: retrieve the commit and verify its provenance.
2. Dependency stage: resolve locked dependencies and inspect licences and vulnerabilities.
3. Build stage: compile or package functions and services reproducibly.
4. Static quality stage: run formatting, linting, type checks, and static analysis.
5. Unit stage: execute fast tests and publish results.
6. Contract and integration stage: verify interfaces and dependencies.
7. Security stage: scan source, dependencies, infrastructure definitions, containers, and secrets.
8. Package stage: create an immutable artefact and record its version or digest.
9. Deployment stage: apply infrastructure and release to a controlled environment.
10. Verification stage: run smoke tests and inspect health metrics.
11. Promotion stage: use approvals or automated policy gates to release progressively.

A pipeline should fail clearly and early when a mandatory control is violated. It should also preserve evidence: commit identifier, build environment, dependency versions, scan results, test reports, artefact digest, approver, and deployment target.

7.1 Security throughout the lifecycle

DevSecOps moves security from a final inspection to continuous practice. Important controls include:

• secret detection before code is merged;
• dependency and software-composition analysis;
• static application security testing;
• infrastructure-as-code scanning;
• container image scanning where containers are used;
• least-privilege identity policies;
• signed artefacts and provenance records;
• protected deployment credentials;
• logging of administrative and deployment actions.

Automated scanners generate findings, not final risk decisions. A team must assess exploitability, exposure, compensating controls, false positives, and remediation deadlines. Critical findings in an internet-facing authentication component should not be treated like a low-risk development-only warning.

Pipeline credentials should be short-lived where possible. Workloads should use separate identities for building, testing, and deploying. A build job that can both read production data and modify production infrastructure represents an unnecessarily large blast radius.

8. Deployment Strategies

Deployment strategy determines how a new version reaches users and how quickly the team can detect and reverse problems.

Serverless platforms often support versioned functions and aliases, allowing traffic to be shifted between immutable versions. Regardless of platform, a progressive release should define automatic and manual stop conditions. Examples include increased error rate, latency regression, authentication failures, queue growth, or abnormal business outcomes.

Rollback is not always safe. If a release changes a database schema or emits new event semantics, reverting code may not restore compatibility. Teams should distinguish rollback (returning to an earlier version) from roll-forward (deploying a corrective version). Backward-compatible migrations make both safer.

9. Observability and Release Decisions

Testing provides evidence before release; observability provides evidence during and after release. The three principal signals are:

• Logs: detailed event records, preferably structured and correlated.
• Metrics: numerical measurements such as request count, error rate, latency, throttling, and queue depth.
• Traces: a view of a request as it crosses services.

A correlation identifier should be propagated across synchronous calls and asynchronous messages. Logs must avoid passwords, tokens, unnecessary personal information, and sensitive payloads. Retention and access should follow organisational policy and applicable privacy law.

A release gate should combine technical and business indicators. A payment service can have a normal HTTP error rate while payment completion falls because a downstream provider is rejecting transactions. Effective monitoring therefore includes domain metrics such as completed orders, failed authorisations, and duplicate fulfilments.

10. Illustrative Case: Online Learning Enrolment Platform

Background

A fictional institution operates an online enrolment platform composed of an API service, an enrolment function, a payment service, an event bus, a notification function, and a managed database. The platform must process enrolments during a short period before each semester.

Problem

A new payment-service release passed unit tests but caused duplicate payment attempts under retry conditions. In addition, the notification service rejected events after a producer changed a field name. The team had no isolated integration environment and deployed all traffic at once.

Analysis

The unit tests verified payment validation but did not simulate a timeout after the payment provider had accepted a request. There was no idempotency key at the payment boundary. The event contract was implicit in source-code assumptions, so the producer’s rename was not detected before release. Operational dashboards showed function errors but did not connect them to enrolment completion.

Approach and solution

The team introduced the following controls:

1. A payment idempotency key derived from the enrolment transaction was stored with the provider response.
2. Timeout and retry integration tests simulated an accepted payment followed by a lost response.
3. Consumer-driven contract tests checked required event fields and version compatibility.
4. An ephemeral environment was created for pull requests using synthetic payment responses.
5. The pipeline added secret scanning, dependency analysis, infrastructure scanning, and artefact signing.
6. Deployments used a canary release beginning with 5% of traffic.
7. Dashboards tracked payment success, duplicate attempts, enrolment completion, p95 latency, and queue depth.
8. A dead-letter queue and replay procedure were established for invalid notification events.

Outcome

The revised process detected the event incompatibility before production and showed that the initial payment implementation was not safe to retry. During the next release, the canary remained within its error and latency thresholds, and the team promoted it gradually. When a notification dependency later slowed, queue growth triggered an alert without interrupting completed enrolments.

Learning points

The case demonstrates that testing must reflect failure semantics, not only successful examples. Contracts protect communication between independently deployed services. Idempotency protects business effects under retries. Progressive delivery limits exposure, while observability must include business outcomes as well as infrastructure signals.

11. Challenges, Governance, and Ethical Practice

Automated delivery can create a false sense of security. A green pipeline may still miss an untested business rule, a compromised dependency, a dangerous permission, or a privacy violation. Teams should document what each test does not prove.

Cloud delivery also raises governance questions. Who may deploy to production? How are emergency changes reviewed? How long are logs retained? Which data may be copied into test environments? Are automated decisions explainable and contestable? Are accessibility, fairness, and privacy requirements represented in acceptance criteria?

Production-like data should generally be masked or synthetically generated for testing. Personal data should not be copied into developer laptops or shared test accounts without a lawful basis and appropriate controls. Security testing must be authorised; intentionally exploiting a live service without permission is unacceptable.

Modern tools, including AI-assisted coding and test generation, can accelerate development but do not remove professional accountability. Generated code may contain insecure dependencies, licensing problems, incorrect assumptions, or fabricated tests. Human review, reproducible builds, security scanning, and ownership of the resulting system remain necessary.

12. Best-Practice Checklist

Before approving a service release, verify that:

• source, tests, infrastructure, and schemas are versioned;
• configuration is validated and secrets are not stored in source;
• unit tests cover business rules and error paths;
• contracts are tested for important consumers and producers;
• integration tests exercise real dependency behaviour where needed;
• performance assumptions and thresholds are documented;
• retries, duplicates, timeouts, and dead-letter handling are tested;
• the artefact is immutable and traceable to a commit;
• security and licence findings are reviewed and managed;
• deployment permissions follow least privilege;
• observability and business-level release criteria exist;
• rollback or roll-forward procedures have been rehearsed;
• data protection and access requirements are satisfied.

Chapter Summary

Serverless and microservice delivery requires coordinated development, testing, security, and operations. Local invocation, emulators, containers, and ephemeral cloud environments provide different levels of speed and fidelity; none should be mistaken for complete production equivalence. A layered testing strategy combines unit, contract, integration, end-to-end, performance, and failure testing according to risk.

Reliable distributed systems account for retries, duplicate delivery, timeouts, partial completion, and dependency failure. Version control, reproducible builds, immutable artefacts, and automated CI/CD make delivery repeatable. DevSecOps integrates secret detection, dependency analysis, static analysis, infrastructure scanning, least privilege, and provenance into the pipeline. Canary, blue–green, rolling, and feature-flag strategies reduce release risk when supported by meaningful technical and business observability. Finally, governance, privacy, authorised testing, and human judgement remain essential even when delivery is highly automated.

Key Terms

Local emulation: Running a local approximation of a cloud service or platform.

Unit test: A focused test of a small component in isolation.

Contract test: A test that verifies an agreement between a service and its consumer or producer.

Integration test: A test of interactions between a component and a real or faithful dependency.

End-to-end test: A test of a complete business journey across multiple system components.

Idempotency: The property that repeating an operation produces the intended single logical effect.

Dead-letter queue: A destination for messages that cannot be processed successfully after defined attempts.

Exponential backoff: A retry strategy that increases the delay between attempts.

Jitter: Controlled randomness added to timing, often to spread retries.

Continuous integration: Frequent automated building and testing of source changes.

Continuous delivery: Keeping software in a releasable state through automation and controls.

DevSecOps: Integrating development, security, and operations practices throughout delivery.

Software composition analysis: Examination of third-party components for vulnerabilities and licence issues.

Infrastructure as code: Describing infrastructure in versioned, machine-readable definitions.

Immutable artefact: A packaged build that is not modified after creation.

Canary deployment: Sending a small proportion of traffic to a new version before wider release.

Blue–green deployment: Operating old and new environments separately and switching traffic between them.

Feature flag: A runtime control that enables or disables application behaviour independently of deployment.

Observability: The ability to infer system state from logs, metrics, and traces.

Correlation identifier: A value propagated across related operations to connect their records.

Review Questions

Short Answer Questions

1. Why is local emulation useful, and why can it not prove production equivalence?
2. What is the difference between a unit test and a contract test?
3. Why must retry behaviour be considered when designing an idempotent operation?
4. What evidence should an immutable deployment artefact retain?
5. Name three security checks that can be automated in a CI/CD pipeline.

Long Answer Questions

1. Explain how unit, contract, integration, end-to-end, performance, and failure tests complement one another in a serverless system.
2. Describe a CI/CD pipeline for a microservice and identify the purpose of each major stage.
3. Compare rolling, blue–green, canary, and feature-flag deployment strategies.
4. Discuss how configuration, secrets, identity, and infrastructure-as-code affect secure delivery.
5. Explain why observability should include business metrics as well as logs, infrastructure metrics, and traces.

Analytical / Application Questions

1. A queue receives 30 messages per second and the average time in the system is 0.4 seconds. Using Little’s Law, estimate the average number of messages in the system. State the assumptions behind your calculation.
2. A payment function times out after the provider accepts a charge. Design a test sequence and implementation controls that prevent a retry from charging the customer twice.
3. A new event producer renames a required field while three consumers are deployed independently. Propose a contract, versioning, deployment, and rollback approach that maintains compatibility.

Multiple Choice Questions with Answers

Question 1

Which test primarily verifies that a service and its consumer agree on request and response structure?

A. Unit test

B. Contract test

C. Load test

D. Chaos test

Correct Answer: B. Contract test

Question 2

What is the principal purpose of an idempotency key in a payment operation?

A. To increase database throughput

B. To encrypt the payment amount

C. To prevent repeated processing from creating multiple logical charges

D. To reduce function cold-start time

Correct Answer: C. To prevent repeated processing from creating multiple logical charges

Question 3

A queue has an average arrival rate of 20 messages per second and an average time in the system of 0.5 seconds. What is the average number of messages in the system?

A. 10 messages

B. 20 messages

C. 40 messages

D. 100 messages

Correct Answer: A. 10 messages

Question 4

Which deployment strategy initially sends only a small proportion of traffic to the new version?

A. Big-bang deployment

B. Canary deployment

C. Static deployment

D. Full replacement deployment

Correct Answer: B. Canary deployment

Question 5

Which statement about p95 latency is correct?

A. Every request completes within the p95 value

B. The p95 value is always the average latency

C. Approximately 95% of measured requests complete at or below the p95 value

D. The p95 value measures only failed requests

Correct Answer: C. Approximately 95% of measured requests complete at or below the p95 value

Question 6

Which practice best protects a pipeline from excessive privilege?

A. Using one permanent administrator credential for every stage

B. Sharing production secrets with all developers

C. Using separate, least-privilege identities for build, test, and deployment activities

D. Disabling audit logs during deployment

Correct Answer: C. Using separate, least-privilege identities for build, test, and deployment activities

Question 7

What is a dead-letter queue used for?

A. Storing successfully processed messages permanently

B. Holding messages that repeatedly fail processing for inspection or later handling

C. Caching function source code

D. Routing all normal traffic to a canary

Correct Answer: B. Holding messages that repeatedly fail processing for inspection or later handling

Question 8

Why are long-lived branches often discouraged in frequently integrated systems?

A. They prevent unit tests from running

B. They increase merge conflicts and delay detection of incompatible changes

C. They automatically expose secrets

D. They eliminate the need for code review

Correct Answer: B. They increase merge conflicts and delay detection of incompatible changes

Question 9

Which activity is an example of software-composition analysis?

A. Measuring API response latency

B. Checking third-party dependencies for known vulnerabilities and licences

C. Tracing a request across services

D. Testing a user interface colour

Correct Answer: B. Checking third-party dependencies for known vulnerabilities and licences

Question 10

A service receives 100 requests per second and has an average response time of 0.2 seconds under stable conditions. According to Little’s Law, what is the average number of in-flight requests?

A. 5

B. 20

C. 100

D. 500

Correct Answer: B. 20