Most engineers learn git push on day one. You commit your work, push it to a remote, and move on. What happens next is usually invisible: a few minutes later a bot comments on your change, some checks turn green, and eventually the code ends up running in production. For a lot of us, that middle part stays a black box for years.
I want to open that box. This is the path a single commit travels from your terminal to a running system in a large organization, and more importantly, why each stop on that path exists. Almost none of it is there by accident. Every stage was added because something broke without it.
I’ll keep the examples generic. The point isn’t any specific company’s pipeline, it’s the shape that most enterprise release pipelines converge on, and the engineering reasoning behind that shape.
Table of contents
Open Table of contents
-
- 1. The push lands on a server, not “the cloud”
- 2. Code review: the part that isn’t really about finding bugs
- 3. Continuous Integration: proving the change is safe to merge
- 4. Build systems: reproducibility is the whole point
- 5. Artifact management: build once, promote many times
- 6. Dependency management: the code you didn’t write
- 7. Release validation: staging, release candidates, and canaries
- 8. Deployment and the rollback you hope you never use
Why Release Engineering exists at all
Start with a question worth sitting with: if every developer can build and run the code on their laptop, why does a company need an entire discipline devoted to shipping it?
The short answer is that “works on my machine” does not scale to “works for a million users, on infrastructure I don’t control, alongside forty other teams’ changes, and can be undone in ninety seconds if it’s wrong.” Release Engineering is the practice of making software delivery repeatable, auditable, and reversible. When you have three engineers, you can ship by hand and remember what you did. When you have three hundred, tribal memory becomes an outage waiting to happen.
A useful way to think about it: developers optimize for change, Release Engineering optimizes for safe change. Those two goals pull in different directions, and the pipeline is where the tension gets resolved.
The journey, end to end
Here is the flow at a high level. The details vary between organizations, but the stages and their order are remarkably consistent.
Developer Code Review Continuous Integration
┌───────────┐ push ┌──────────────┐ ┌──────────────────────────┐
│ git push │ ──────► │ Gerrit / PR │ ─────► │ Build │
└───────────┘ │ + reviewers │ │ Static analysis / lint │
└──────────────┘ │ Unit + integration test │
▲ └────────────┬─────────────┘
│ feedback │ signed, versioned
└───────────────────────┐ │ artifact
│ ▼
Production Canary Staging ┌──────────────────────────┐
┌──────────┐ ┌──────────┐ ┌─────────┐ │ Artifact Repository │
│ rollout │ ◄─│ 1% of │◄─│ prod- │◄│ (e.g. Artifactory) │
│ + monitor│ │ traffic │ │ like env│ │ immutable, provenance │
└────┬─────┘ └──────────┘ └─────────┘ └──────────────────────────┘
│
└──► rollback path (always available)Let’s walk each stage and what it’s actually protecting against.
1. The push lands on a server, not “the cloud”
When you push, your commits go to a central system: Gerrit, GitHub, GitLab, or similar. The first thing that can happen is a server-side hook rejecting your push before anyone sees it. Branch protection rules, commit message format checks, file size limits, secret scanning that blocks an accidentally committed private key.
This is the cheapest place to catch a mistake, so a lot of policy lives here. A rejected push costs you thirty seconds. The same secret discovered in production costs a credential rotation and an incident review.
2. Code review: the part that isn’t really about finding bugs
Review is where a lot of newcomers misread the intent. Yes, reviewers catch defects. But in a mature organization the more valuable outcomes are shared ownership and knowledge transfer. After a review, at least two people understand the change. The bus factor goes up. Conventions get enforced by humans who can explain the reasoning, not just a linter.
Enterprises tend to use one of two models. Gerrit centers on the individual commit as the unit of review, with a scoring system (the classic +2 to merge) and a strong preference for small, self-contained changes. GitHub and GitLab center on the pull request as a branch of related commits. Both work. The Gerrit style tends to produce a cleaner linear history and forces smaller changes, which is a real advantage when you’re bisecting a regression later.
A practical observation: the single biggest lever on review quality is change size. A 40-line change gets a genuine review. A 2,000-line change gets a “looks good to me” because no human can hold that much context. If your reviews feel like rubber-stamping, the problem usually isn’t the reviewers.
3. Continuous Integration: proving the change is safe to merge
Once a change is up for review, CI kicks in automatically. This is the stage people mean when they say “the pipeline.” Its job is to answer one question: if we merged this right now, would anything break?
CI typically runs, in rough order of cost:
- Build — does the code even compile and link?
- Static analysis and linting — style, obvious bug patterns, security anti-patterns.
- Unit tests — fast, isolated checks of individual functions and modules.
- Integration tests — do the pieces work together, talk to a database, respect an API contract?
The ordering matters and it’s deliberate. Cheap, fast checks run first so a broken build fails in two minutes instead of forty. This is the same idea as the test pyramid: many fast unit tests at the base, fewer integration tests in the middle, a small number of slow end-to-end tests at the top. Invert that pyramid and your pipeline becomes so slow that people start looking for ways to skip it, which defeats the purpose.
There’s a deeper reason CI exists, and it’s in the name. “Continuous integration” was a response to integration hell: teams working in isolation for weeks, then spending days painfully merging divergent branches. Integrating small changes constantly means the merges are trivial and the conflicts are tiny. The automated tests are how you make constant integration safe.
4. Build systems: reproducibility is the whole point
Compiling on your laptop and compiling in the pipeline should produce the same result. That sounds obvious and is surprisingly hard. This is what build systems like Make, CMake, Bazel, and Gradle are really for. Not just “turn source into binary,” but turn this exact source into this exact binary, every time, regardless of who runs it or on what machine.
Two properties are worth naming:
- Incremental builds — rebuild only what changed. On a large C++ codebase, a clean build might take an hour; an incremental build takes seconds. Tools like CMake (often driving Make or Ninja) track dependencies between files so they can skip untouched work.
- Hermeticity — the build depends only on declared inputs, not on whatever happens to be installed on the machine. This is where newer tools like Bazel push hardest, and it’s what makes a build genuinely reproducible.
The failure mode this prevents is the one every engineer has hit: the build passes for you and fails for your colleague because your machine had a library theirs didn’t. Multiply that across a release and you have a shipment that can’t be reproduced or debugged.
5. Artifact management: build once, promote many times
Here’s a principle that took me a while to fully appreciate: you build a release artifact exactly once, and then you promote that same artifact through every environment. You do not rebuild for staging and rebuild again for production.
Why? Because if you rebuild, you can no longer prove that the thing you tested is the thing you shipped. A dependency could have shifted, a base image could have updated, a compiler flag could differ. The artifact that passed all your tests in staging must be the identical artifact, byte for byte, that reaches production.
This is what artifact repositories like JFrog Artifactory (or Nexus, or a container registry) provide: a place to store immutable, versioned, addressable build outputs, jar files, container images, packages, binaries, with metadata about where they came from. Immutable means version 1.4.2 is always the same bytes; you can’t quietly overwrite it. That immutability is what makes rollback trustworthy. When you roll back to 1.4.1, you know precisely what you’re getting because it was never allowed to change.
6. Dependency management: the code you didn’t write
Modern software is mostly other people’s code. A service with 200 lines of your logic can pull in thousands of transitive dependencies. Managing that is its own stage, and it’s increasingly a security concern, not just a build concern.
The problems you’re solving here:
- Transitive dependency resolution — your dependency has dependencies, and so on down the tree. Two of them may demand conflicting versions of a third. Something has to resolve that.
- Pinning and reproducibility — lock files exist so that “install the dependencies” means the same thing today and in six months. Without pinning, an upstream release can change your build with no change on your side.
- Supply chain security — scanning dependencies for known vulnerabilities (CVEs) and, increasingly, verifying their provenance. A compromised upstream package is now a mainstream attack vector, not a hypothetical.
A good artifact repository doubles as a controlled proxy for external dependencies, so you’re not pulling directly from the public internet during a build, and so a package that disappears upstream doesn’t break you.
7. Release validation: staging, release candidates, and canaries
Passing CI means a change is safe to merge. It does not mean it’s safe to release. Those are different bars. A release usually bundles many merged changes, and the interactions between them are what validation is checking.
Typical layers:
- Staging / pre-production — an environment built to resemble production as closely as budget allows: similar data shapes, similar dependencies, similar scale where possible. The gap between staging and production is where a lot of incidents are born, so reducing that gap is high-leverage work.
- Release candidates — a specific artifact version nominated as “this might be the release.” It gets extra scrutiny: longer-running tests, manual validation, sign-off from stakeholders.
- Canary releases — instead of flipping everyone to the new version at once, you route a small slice of real traffic (say 1%) to it and watch. If error rates or latency degrade, you stop before most users ever notice.
The theme across all of these is limiting blast radius. You are trying to discover problems while they’re still cheap to fix.
8. Deployment and the rollback you hope you never use
Finally the artifact reaches production. Even here, how you get it there matters. Common strategies:
- Rolling deployment — replace instances gradually, a few at a time, so the service stays up throughout.
- Blue-green — stand up a complete second environment (green) with the new version, then switch traffic over from the old one (blue) all at once. Rollback is just switching back.
- Canary — as above, but as a deployment mechanism: increase the new version’s traffic share in steps.
The single most important property of a good deployment system is not how fast it ships. It’s how fast it can un-ship. A release you can roll back in ninety seconds lets you take risks. A release that takes four hours to reverse makes every deploy terrifying, and fear makes teams deploy less often, which makes each deploy bigger and riskier. It’s a vicious cycle, and fast rollback is how you break it.
Common misconceptions
“CI/CD is just a tool we install.” CI/CD is a set of practices. The tool (Jenkins, GitLab CI, GitHub Actions) is the easy part. The hard part is a test suite you trust, small changes, and a team culture that keeps the pipeline green. A pipeline nobody trusts gets bypassed.
“If the tests pass, the code is correct.” Passing tests mean the code does what the tests check. That’s a floor, not a ceiling. Tests are a safety net whose holes you can’t see.
“We rebuild for each environment to be safe.” The opposite is true. Rebuilding per environment breaks the guarantee that what you tested is what you shipped. Build once, promote the same artifact.
“Release Engineering slows developers down.” Done badly, sure. Done well, it’s what lets developers move fast without being afraid, because the guardrails catch mistakes early and cheaply. Speed and safety aren’t opposites here; a good pipeline gives you both.
“Rollback means undoing the code.” Rollback usually means redeploying a previous known-good artifact, not reverting commits under pressure. This is exactly why immutable, versioned artifacts matter.
Best practices worth internalizing
- Keep changes small. Small changes are easier to review, faster to test, and trivial to roll back. This single habit improves almost every stage downstream.
- Fail fast and cheap. Order your pipeline so the quickest checks run first. Developers should learn about a broken build in minutes, not after a forty-minute test run.
- Make artifacts immutable and versioned. If you can overwrite a release, you can’t trust a rollback.
- Build once, promote everywhere. The artifact that reaches production must be the one that passed validation.
- Invest in staging parity. Most surprises come from the gap between staging and production. Closing that gap pays for itself.
- Automate the boring, gate the dangerous. Automate builds, tests, and promotion. Keep human sign-off for the genuinely irreversible steps.
- Optimize for rollback, not just deploy. The question isn’t only “how do we ship this?” but “how do we take it back in under two minutes?”
Key takeaways
- The path from
git pushto production is a series of stages, each of which exists to catch a specific class of failure as early and cheaply as possible. - Code review is as much about shared ownership and knowledge as about finding bugs.
- CI answers “is this safe to merge?”; release validation answers “is this safe to ship?” They are different questions.
- Build reproducibility and immutable artifacts are what make testing meaningful and rollback trustworthy.
- Release Engineering exists to make change repeatable, auditable, and reversible. Its real product is confidence.
FAQ
Is Release Engineering the same as DevOps? They overlap heavily but aren’t identical. DevOps is a broad cultural and operational movement about collaboration between development and operations. Release Engineering is a more specific discipline focused on the build, packaging, and delivery pipeline. In many organizations Release Engineers are the people who own that pipeline day to day.
Do I need to know a specific tool like Jenkins or Artifactory to work in this space? Knowing one tool in each category helps, but the concepts transfer. If you understand why an artifact repository exists, you can pick up Artifactory or Nexus or a container registry quickly. Interviews tend to probe the reasoning, not the button locations.
Why do some teams use Gerrit instead of pull requests? Gerrit’s commit-centric model encourages very small, individually reviewed changes and a clean linear history, which is valuable in large codebases where bisecting regressions is common. It has a steeper learning curve than pull requests, so it shows up more in large engineering organizations than in small teams.
What’s the difference between continuous delivery and continuous deployment? Continuous delivery means every change that passes the pipeline is ready to ship, with the actual release being a deliberate decision (often a button someone presses). Continuous deployment goes one step further and ships automatically once checks pass, with no human gate. Many enterprises practice delivery rather than full deployment because of compliance and change-control requirements.
How do canary releases and blue-green deployments differ? Blue-green switches all traffic from an old environment to a new one at once, with rollback being a switch back. Canary shifts traffic gradually and watches metrics at each step. Canary catches problems with a smaller initial blast radius; blue-green gives a cleaner, more instant rollback. Some teams combine them.
Closing thought
The next time you type git push, picture the road ahead of that commit. It gets reviewed by people who now share ownership of it. It gets compiled reproducibly, tested in layers, packaged into an immutable artifact, promoted through environments that get progressively closer to real, and finally rolled out in a way designed to be undone. None of that machinery is bureaucracy for its own sake. Each piece is a scar from something that went wrong before it existed.
Understanding that road is what separates someone who can run a pipeline from someone who can reason about one. If you’re moving into DevOps or Release Engineering, that reasoning is the thing worth building. The tools will keep changing. The questions they answer won’t.