The industry is determined to deliver value ever more rapidly, safely, and securely across software development lifecycle environments. One way to achieve this is through GitOps. In broad terms, GitOps is about applying the Git workflow to operations. I took this idea and narrowed it down to exactly one use case, using Git to drive deployments on OpenShift. This is a walk-through of the concepts and components we used to enable GitOps driven deployments all the way from development to production for a real-world enterprise client.
Another way to define GitOps is applying a developer experience (DX) to operations. DX for APIs, SDKs, CLIs, and documentation tend to take up all of the oxygen in the room. But I propose DX for operations is an idea whose time has come. What speaks more to the experience of being a developer than interacting with Git all day. It’s natural to extend that interaction and apply it to operations as well.
GitOps follows this flow across four major actors to drive deployments on OpenShift:
Underlying the pipelines and environments is OpenShift. Conceivably you could also run Git on OpenShift but that was not the case at this client.
The users of this system are the people either proposing or merging a change to an environment (env). For example, a developer has completed a feature and produced version 1.2.3 of an API service in a development environment. Once that feature is ready for testing, the developer proposes version 1.2.3 of that API service to a test env via a pull request. The test env owner reviews the pull request, approves it, and merges it. The merge fires a webhook that kicks off the pipeline run that deploys version 1.2.3 of that API service to the test env.
Naturally, Git provides source control but it also provide excellent access control and audit capabilities. In Git we have two types of repositories (repos).
Environment repos contain the configuration and versions for the services in a particular env (e.g. prod). The env repos also contain all of the settings for the repo itself to enable GitOps. This is achieved by setting repository and branch permissions, pull request rules and reviewers, and webhooks.
Service repos container the source code of the services that compose the application(s) running in the env. Depending on how your platform creates conatiner images, the source code may be laid out in a certain structure (e.g. Source-to-Image). It can also contain deployment artifacts particular to the service such as a DeploymentConfig, ConfigMap(s), and Jenkinsfile.
Pipelines are a sequence of steps executed in Jenkins, running on OpenShift. As input, pipelines consume environment configuration and source code from git. As output, pipelines produce container images and deployments of the EdPay application for an environment. For day to day development, the pipelines can be considered as your interface to the OpenShift Container Platform.
There are two types of pipeline and one type of build:
- Env Pipeline: Orchestrates the deployment of the entire EdPay stack and follows a
deploy-<env>naming convention (e.g.
- Service Pipeline: Builds and deploys a specific service from a container image and follows a
deploy-<service>naming convention (e.g.
- Image Build: A build of a service produces a container image stored in the registry and follows a
build-<service>naming convention (e.g.
An environment (an OpenShift project) is a combination of running services and configuration, primarily specific versions of the running services and the DB they are configured connect to. Each environment contains its own instances of all of the services that make up the EdPay application.
There are two types of environments:
- Dev Envs: Build images and deploy services.
- Higher Envs: Deploy services from existing images (prod is just another higher env with the exception that the images must be copied to the prod cluster first).
The primary differentiator is that dev envs do the builds that create the container images for all of the services in the stack. Dev master and dev stable are where all application builds take place. All envs do deployments of these container images.
It’s worth noting that the environment itself is versioned. Ultimately the environment is just another application dependency.
There are two OpenShift clusters, one for production and one for non-production. The production cluster only has a staging and a prod environment. The non-production cluster has all other environments, include dev, test, demo, and pre-prod. Again, the only difference between the staging/prod pipelines from the other higher env pipelines is that the images must be copied to the prod cluster first.
The deck below is a walk-through of the concepts and components used to enable the GitOps driven deployments described above.
Because there are a fair number of concepts and components at play, I also did a screencast to explain the system.
Note: Always set limits on compute resources. Always! We learned this one the hard way when we deployed Jaeger Tracing on OpenShift and it began to cause node instability. My solution was to add max traces and compute resource parameters to the Jaeger Tracing OpenShift all-in-one template.
The ideas and concepts for a system like this have been rattling around in my head for a while. It’s been great getting the opportunity to build it out. I had help from a few people around me at the client that truly helped bring this to life. Without their help and support this wouldn’t have been possible and my thanks go out to them.