Skip to content

The principles of DevOps

A common goal

The separation of DEV and OPS leads to distinct objectives:

  • DEVs want to deliver evolutions quickly.
  • OPS must above all ensure availability.

This results in the wall of confusion.

Wall of confusion

To remedy this situation, it will be imperative to put the product and value creation back at the center by setting a common goal for DEV and OPS:

Deliver evolutions quickly while ensuring availability

A process unifying DEV and OPS

Sharing this same objective will lead to unifying development and deployment processes:

In this process, we will recognize the Deming wheel, well known in the field of quality and agility:

(Source: wikimedia.org - Michel Weinachter)

With DevOps, it is mainly about not having two distinct processes for development and deployment.

A process subject to continuous improvement

We will agree that implementing such a process will take time and that it will always be perfectible.

As such, it will need to be continuously improved.

The CALMS model

Culture

Infrastructure management is a sensitive subject subject to a regulatory framework (information system security, GDPR,...) and recommendations (e.g.: cyber.gouv.fr - Recommendations relating to the secure administration of IS).

To evolve practices and processes, a shared understanding will first be needed:

  • Of what agility in development is (and what it implies in terms of operations, predictability of costs and feature delivery schedules...)
  • Of the limits of traditional operating methods (hence the many slides)
  • Of the bottlenecks and possible improvements in current deployment processes
  • ...

Before targeting an agile infrastructure, many will need to observe a problem when:

  • Delivering an application with a simple dependency update takes days (recent case: Log4Shell vulnerability)
  • The deployment procedure is not up to date
  • Adding a VM to cope with a peak load is impossible (5 days to get the machine, 5 days to configure the load balancer,...)
  • ...

In practice, moving towards the DevOps method will be difficult without a global policy allowing agility at the company level. Without going into details:

  • It will be difficult to involve DEVs in operations with a project logic where work stops at production deployment.
  • It will be difficult to identify problems and improvements with projects managed with heterogeneous methods (Excel, JIRA, Teams, Redmine, GitHub/GitLab issues, or even by email...)
  • It will also become difficult to effectively contact DEVs and OPS to deal with them after feature additions have stopped.

Concretely, it will be necessary to go beyond Microsoft Office Excel for project management (support.microsoft.com) and review application lifecycle management (ALM) to:

  • Move from a project logic to a product logic (with MCO and MCS, DEVs and OPS are needed until a service is stopped!)
  • Maintain a list of these products with regular review to turn off those that have become useless (example: gcemetery.co - THE GOOGLE CEMETERY)
  • Be able to highlight problems and improvements by systematizing the use of a ticket manager (which will also allow effective contact with DEVs and OPS).

MCO = operational condition maintenance / MCS = security condition maintenance.

As such, we will find scaled agile frameworks such as Scaled Agile Framework (SAFe) which will include DevOps in a more global approach.

The same applies to the ITIL 4 framework which focuses on value streams and continuous improvement (which was not necessarily the case in previous versions).

See www.alenia.io - ITIL 4 versus ITIL v3: what's changing? and www.atlassian.com - DevOps and ITIL: which is essential for your team?

Automation

DevOps will place a strong emphasis on automation. It will take several forms:

  • Deployment automation to avoid human errors, deliver quickly,...
  • Test automation to limit risks related to automation, reduce manual testing times...
  • Monitoring automation to quickly detect and address problems.
  • Documentation generation automation to ensure by design that it matches the system's state.
  • ...

Lean

The Lean concept originates from Toyota with two fundamentals:

  • Respect for people
  • Continuous improvement

Reading LEAN PRIMER by Craig Larman and Bas Vodde will give a more precise idea, but we will emphasize that the focus is on:

  • A multitude of daily improvements through automation and permanent questioning of processes ("my job is to do my job, and to improve my job")
  • The horizontal dissemination of knowledge
  • On-site observation ("Don't look with your eyes, look with your legs... people who only look at numbers are the worst.")
  • The distinction between value production and waste classified into three categories: Muda (non-value-added action), Mura (variability), and Muri (overload)

In software development, Lean will be found through the following principles:

  • Eliminate waste (useless/undelivered/non-functional features, waiting times, double reporting,...)
  • Foster learning (expert groups, pair-programming, training, self-training, sharing standards...)
  • Decide as late as possible (don't try to plan everything, postpone decisions,...)
  • Deliver fast
  • Empower the team (clear objectives, self-organization, customer feedback, "context, not control" at Netflix,...)
  • Build quality in (integrate quality into construction rather than checking afterwards)
  • Optimize the system as a whole

c.f. www.journaldunet.com - Lean Software Development and project management: decryption

We will note that the principles of Lean resonate with:

  • Agile methods and tools (Scrum, Kanban,...).
  • The desire to bring DEVs and OPS closer (knowledge sharing, reduced waiting times,...).
  • Automating as many tasks as possible.

Lean should not be perverted by simply relying on figures ensuring that resources are 100% occupied (1). To truly gain efficiency, it is necessary to observe the creation process as a whole and allow for optimizations. For example:

  • Directly providing an exploitable deployment script vs. a text-format procedure.
  • Making a pull-request vs. a validation request in an email.
  • Using a project management tool vs. entering task progress in a spreadsheet.

(1) www.usinenouvelle.com - Lean deviations: why the method strayed from original principles

Measurement

The adage says that "what is not measured does not exist". At least, what is displayed in red on a graph will be more visible to management:

Illustration of the problem of waiting times with the traditional approach.

As such, we will strive to define objectives and associated metrics.

Sharing

Sharing and transparency will be important at several levels. They will promote:

  • Trust between different stakeholders
  • A shared understanding of objectives and challenges
  • Skill transfers between teams
  • ...

Metrics

Introduction

Defining relevant metrics and being able to calculate them will be far from trivial. Simply having metrics and having irrelevant metrics will be counterproductive.

Here we will limit ourselves to a few telling metrics and discuss in session the methods for calculating them. You will encounter many others.

See effectivesoftwaredesign.com - Lean Startup Principles: Vanity Metrics and Actionable Metrics which describes the distinction between Vanity Metrics and Actionable Metrics proposed by Eric RIES in "The Lean Startup".

DevOps Metrics

The following metrics will be characteristic of the level of deployment automation:

  • Deployment frequency (per year, per month,... variable depending on maturity)
  • Deployment duration where it will be interesting to distinguish:
  • The deployment duration of an evolution or a minor fix (automation of application deployment)
  • The deployment duration of a major evolution (automation of infrastructure management)

The following metrics will provide an indication of the quality of deployment automation:

  • Change Failure Rate
  • Mean Time To Recovery (MTTR)

The measurement of the Mean Time To Detect (MTTD) will, in turn, provide an indication of the quality of infrastructure instrumentation.

Code Quality Metrics

To measure code quality, it will be possible to rely on:

  • Code coverage by tests ("coverage")
  • Indicators produced by static code analysis tools such as SonarQube
  • Indicators produced by dependency analysis tools (number of obsolete, vulnerable dependencies,...) such as Trivy.

SonarQube has the merit of offering a unified interface for different languages (which appeals to decision-makers) but code analysis tools dedicated to languages (jshint, jslint, phpmd, phpstan, cppcheck...) will often offer more relevant alerts as well as an "as code" approach to exception management.

Development Quality Metrics

The following indicators will provide a more global view of an application's quality:

  • Number of incident tickets (per month, per year,...)
  • Number of open incident tickets
  • Average lifespan of incident tickets

Example of visualizing metrics on tickets.

Calculating such metrics will be difficult without:

  • The use of a ticketing system (Jira, Redmine, source code manager issues,...)
  • Rigorous ticket management (requests vs. incidents)
  • Adding the metadata necessary for calculating metrics:
  • Ticket creation date
  • Ticket handling date
  • Resolution date (to be distinguished from the resolution confirmation date)

Furthermore, to reason about comparable elements and identify areas for improvement, it will be interesting to:

  • Categorize requests (parameter change, minor/major release,...)
  • Categorize incidents (application, infrastructure, operations)

Observability

System Monitoring

System monitoring tools (Grafana/Prometheus, Centreon, Munin, Netdata,...) will allow monitoring resource consumption (RAM, CPU, storage, bandwidth,...):

Monitoring tools will also allow implementing alerts to:

  • Detect problems (reaching bandwidth limits, number of simultaneous connections, cryptomining,...)
  • Avoid problems (if >=90% of storage is used)

They will also offer an interesting discussion ground between DEVs and OPS.

Web Probes

A web probe periodically queries a URL, checking the response (return code 200) and measuring the response time. It thus allows to:

  • Measure an availability rate
  • Measure an average response time (performance indicator)
  • Implement an alert to deal with unavailability as quickly as possible

For services exposed online, implementation will be trivial with tools such as UptimeRobot, Uptrends...

Source uptimerobot.com - Geoservices probes

We can also extend monitoring tools to add this type of probe, which will be particularly interesting for unexposed services:

Monitoring a URL using prometheus/blackbox_exporter

Unit Web Probes

Implementing web probes will easily improve the Mean Time To Detect (MTTD). However, to improve the Mean Time To Recovery (MTTR), we will find that:

  • Interpreting a failure on a probe involving several services behind the scenes is not trivial.
  • Triggering automatisms in case of a problem (e.g., restarting a service) with procedures or tools such as StackStorm is even less so.

To quickly identify the origin of problems, it will be interesting to provide dedicated URLs for monitoring at the application level:

  • /health/db testing database access
  • /health/wfs-geoplateforme testing the availability of Géoplateforme WFS
  • ...

See learn.microsoft.com - Health Endpoint Monitoring pattern.

Application Logs

In case of a problem, it will be necessary to have visibility into the processing performed in an application. In this sense, it is appropriate to:

  • Produce exploitable application logs
  • Centralize these logs in a log sink offering an efficient search interface.

Log Sink

The historical reference for log sinks is the ELK stack (ElasticSearch, Logstash, and Kibana):

Illustration of the ELK stack architecture.

It will also be possible to choose variants and alternatives:

(1) Fork of ElasticSearch and Kibana for licensing reasons.

We will also find SaaS offerings specific to hosting providers such as:

Call Traces

To diagnose performance problems in complex systems, it will be interesting to instrument the system with a tool such as Jaeger which will provide details of processing times behind a request.

Infrastructure as Code

Principle

The Infrastructure as Code (IaC) approach will be fundamental for deployment automation. It consists of managing IT infrastructure using programs:

  • Deployment procedures become deployment scripts.
  • Information trapped in documents becomes parameters or secrets for these deployment scripts.

Several layers to configure

Deployment automation will concern several layers of the system:

  • Creation of system resources (machines, networks, storage...)
  • System configuration (installing and configuring frameworks such as PHP, NodeJS, Java,...)
  • Application deployment
  • Configuration of infrastructure services (reverse proxy/load balancer, DNS,...)

What good practices?

We will ensure that deployment scripts can:

  • Be executed multiple times (idempotence)
  • Be interrupted and restarted (atomicity)
  • Coexist with others (~orthogonality)
  • Be tested, for example, with qualification and pre-production environments.
  • Allow both system updates and reconstruction

We will favor a declarative approach over an imperative approach to facilitate the implementation of these principles.

What prerequisites for the infrastructure?

The IaC approach will leave great freedom of choice in the technical framework tools as long as they are compatible with automation. It will mainly be necessary to pay attention to the available configuration methods:

  • Is the configuration based on environment variables?
  • Is the configuration based on configuration files?
  • Is the configuration done via command-line calls (CLI)?
  • Is a WEB API available to configure the tool?

In essence, only tools that can be configured only via a GUI are to be banned!

Partial automation possible but limiting

Depending on the possibilities offered by the infrastructure and the company's policy, automation may be partial, but it will be necessary to be aware of the consequences.

In particular, if the reverse proxy/load balancer configuration is not automated:

  • How will you be able to have a system that adapts to the load?
  • How will you be able to avoid unavailability during new version deployments?

What tools for deployment automation?

We will find different tools for implementing IaC (e.g., Terraform, Ansible, Docker, Kubernetes, Helm,...).

We will note at this stage that:

  • The choice of tools will depend on the target (IaaS, CaaS, public cloud,...)
  • These tools will compete in certain areas (which will not make the choice easier)

We will see them later with concrete examples.

GitOps

The GitOps approach will go a step further than Infrastructure as Code:

  • The main branch of the repository containing the deployment code will reflect the system state.
  • The validation of a pull request on the main branch will trigger the deployment.

The GitOps approach thus allows:

  • Implementing continuous deployment
  • Relying on the source code manager's framework for rights management and evolutions:
  • Protected branches
  • Roles authorized to validate pull requests
  • Solving problems of deployment traceability:
  • Who launched which version of the deployment script?
  • Who proposed/validated the modification?

GitOps is thus an illustration of the use of the pull request concept instead of a more traditional validation process.

Docs as Code

Principle

The Docs as Code approach invites to manage documentation with the same tools used to build applications:

  • The source code manager (GIT) allows versioning the "source code" of the documentation and previewing the content.
  • The document source is in a text format allowing differential calculation (Markdown, reStructuredText, Asciidoc).
  • The ticketing system allows managing evolutions and anomalies.
  • The merge request mechanism is used to validate modifications.
  • The CI/CD chain allows generating and publishing documentation in HTML format.

Interest

This approach leads to the production of documentation in HTML format, which has many advantages, particularly in terms of ability to structure documentation:

  • Offer several levels of reading (c4model).
  • Effectively reference appendices (e.g., sizing configuration).

In the case of DevOps, it is important to:

  • Ensure consistency between the system description and the system state (the DAT and DEX becoming, with IaC, the documentation of deployment scripts and configuration files).
  • Ensure that document production does not slow down the pace of deployments (pull-requests allowing to streamline validation processes, time savings on formatting,...).
  • Not waste energy by manually processing document updates (e.g., entering the number of CPUs, RAM, IP address for each machine vs. generating the corresponding table).

Some examples

Everything as Code!

Ultimately, it will very quickly be tempting to manage as many elements as possible using the source code manager because it offers a framework for managing evolutions.

We will thus find:

The only exception will concern secrets, which should never be stored in a repository! (not even in a private repository, because it might be opened one day!)