Backend Dev – Deployment and Observability

Code that only runs on your laptop is not finished. The last stretch — getting it onto a machine someone else uses, and being able to tell what it is doing once it is there — is a real part of the job, and it is the part self-taught developers most often skip.

One artifact, many environments

Build once. Deploy that same file everywhere. Change behaviour with configuration, never with a rebuild.

If you build a jar per environment, the thing you tested is not the thing you shipped, and the difference between them is exactly where the bug will be. So: one artifact, and environment-specific values supplied from outside.

# Activated with -Dspring-boot.run.profiles=local,docker
#
# Points the app at the MySQL from docker-compose.yml instead of a MySQL installed on
# the machine. The only difference is the port: the container publishes 3308, because
# 3306 already belongs to the local installation.
#
# The default in application.properties is left alone on purpose — someone with a
# native MySQL should keep working without knowing this profile exists.
spring.datasource.url=jdbc:mysql://127.0.0.1:3308/pizza?useSSL=false&allowPublicKeyRetrieval=true

Note what that profile does not do: it does not restate every setting. It overrides the one thing that differs and inherits the rest. A profile that duplicates the whole base file is two files that will disagree within a month.

Secrets are the same principle, one level stricter — they come from the environment or a secret manager, and never live in a file you commit. That was post 7.

Containers, and what they actually fix

A container image packages your app and the runtime it needs. What that buys you is not magic; it is one specific thing: the same bytes run in CI, in staging and in production, so "works on my machine" stops being a category of bug.

Two habits worth having from the start:

Run your backing services in containers locally. A docker-compose.yml means a new developer gets a working database in one command instead of an afternoon.
Give every service a healthcheck, and understand why. This is the comment on the demo app's MySQL container:

healthcheck:
  # `docker compose up --wait` uses this. Without a healthcheck the container is
  # "up" the instant the process starts, several seconds before MySQL accepts a
  # connection — so the app races it and dies on a connection refused.
  test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1"]
  interval: 5s
  timeout: 5s
  retries: 20
  start_period: 30s

"The process started" is not "the service is ready." That gap is the source of an enormous number of confusing startup failures, in compose files and in Kubernetes alike.

The pipeline

Deploying by hand works until the day you are tired. Automate it early, even badly. A minimum useful pipeline, triggered on a push:

Check out and compile.
Run the tests. Stop here if they fail.
Build the artifact or image, tagged with the commit.
Deploy to staging.
Deploy to production — with a human approving, at first.

Tag the artifact with the commit hash, not latest. When something breaks you need to know exactly which code is running, and latest cannot tell you. It also makes rollback possible: knowing how to roll back matters more than deploying quickly. Practise it before you need it.

Migrations during a deploy

This is where deployment and databases collide, and it catches people out once.

During a rolling deploy, old and new code run at the same time for a minute or two. So the schema must be compatible with both. Additive changes are safe. A dropped or renamed column breaks every instance still running the old code — which is most of them, at that moment.

Hence the three-step rename from post 6: add the new column, write to both while backfilling, drop the old one in a later deploy once nothing reads it. Slower, and it is the difference between a deploy and an outage.

Health checks

Your platform needs to ask two questions, and they are different:

Liveness — is this process alive, or should it be killed and restarted?
Readiness — can it serve traffic right now? A just-started app that is still opening its connection pool is alive but not ready, and sending it traffic produces errors for no reason.

In Spring Boot both come from the Actuator dependency, which exposes /actuator/health along with separate liveness and readiness probes. Two rules. The endpoint must be reachable without authentication, or the load balancer cannot use it —

.requestMatchers("/actuator/health")
.permitAll()

— and the check should be cheap. One that runs a real query on every probe, from every instance, every few seconds, becomes its own source of load.

An honest footnote, because it is a mistake worth recognising: that rule is lifted from the demo app, and the demo app does not have the Actuator dependency. So the path is permitted and nothing serves it — the rule is dead, and /actuator/health returns 404. Nothing fails at startup to tell you. If you are relying on a health check, curl it after deploying; a probe pointed at a URL that does not exist looks exactly like a healthy one that nobody asked about, right up until the load balancer starts failing every instance.

Logs you can actually use at 3am

Logging is not printing. The difference is whether you can find one request's story in ten million lines.

Log at the right level. ERROR means a human must look. WARN means something is wrong but handled. INFO is the significant events — a request arrived, an order was placed. DEBUG is for development. If everything is ERROR, nothing is.

Log the identifiers, never the secrets. Order ids, user ids, the path. Never passwords, tokens, card numbers or personal data — logs are copied to places with far weaker access control than your database, and they are retained for years.

Log the exception object, not its message. In SLF4J, passing the throwable as the last argument gives you the stack trace; string-concatenating ex.getMessage() throws it away, and the message alone is rarely enough:

log.error("Unhandled exception on {}", request.getRequestURI(), ex);

Give every request an id and put it on every line. This is the one that turns logs from noise into a tool: with a correlation id you can pull the complete story of one failed request out of a shared log. SLF4J's MDC holds it as thread-local context so you do not have to pass it through every method signature.

Which introduces a trap the demo app handles explicitly. Thread-local context does not follow work onto another thread — so the moment you use @Async from post 8, your background task logs without a correlation id and the trail goes cold. The fix is to copy the context across when the task is handed over:

class LogTaskDecorator implements TaskDecorator {

    @Override
    public Runnable decorate(Runnable runnable) {
        Map<String, String> contextMap = MDC.getCopyOfContextMap();
        return () -> {
            try {
                if (contextMap != null) {
                    MDC.setContextMap(contextMap);
                }
                runnable.run();
            } finally {
                MDC.clear();
            }
        };
    }
}

The MDC.clear() in the finally is not tidiness. Pool threads are reused, so a context left behind gets attached to the next unrelated task — and you end up with one customer's id on another customer's log lines.

The three signals

Signal	Answers	Cost
Logs	What happened, in detail, for one request	Volume
Metrics	How often, how fast, how many — aggregated over time	Cheap
Traces	Where the time went across services	Setup

Start with logs and four metrics: request rate, error rate, latency (as percentiles — the average hides the problem) and saturation (connection pool, thread pool, memory). Those four will tell you about most incidents before a customer does.

You do not need a full metrics stack to begin. The demo app's timing aspect writes a SLOW line for any service call over 250ms — one class, no infrastructure, and it will find your worst endpoint this afternoon.

Alert on symptoms, not causes

Alert on things a user would notice: the error rate is up, latency is up, the queue is growing, the site is down. Do not alert on CPU at 80% — that might be perfectly healthy, and an alert that fires when nothing is wrong teaches everyone to ignore alerts. That is worse than having none, because the one that mattered arrives looking identical to the ninety that did not.

What to remember

Build once, deploy the same artifact everywhere, configure from outside.
A profile overrides what differs and inherits the rest.
Containers make CI, staging and production run the same bytes. "Started" is not "ready" — use healthchecks.
Automate the pipeline early; tests gate the deploy; tag with the commit, not latest.
Know how to roll back before you need to.
Old and new code run together during a deploy — the schema must suit both.
Liveness and readiness are different questions. Keep the probe cheap and unauthenticated.
Log identifiers, never secrets. Pass the exception object. Use a correlation id — and copy the context onto async threads, then clear it.
Request rate, error rate, latency percentiles, saturation. Alert on symptoms.

You have reached the end

Ten posts: the job, the language, the framework, the API, the database, security, performance, tests, and getting it running. That is the shape of backend work. None of it is finished — every one of these has a deeper track on this site, and the first post lists which to read next.

The useful thing to do now is build something small and take it all the way through: a real database, a real API, real tests, deployed somewhere you do not control. One end-to-end project teaches more than ten tutorials, because only the deployment step tells you which of your assumptions were wrong.