The origins of DevOps¶

The democratization of the internet¶

The end of the 90s marked the beginning of the democratization of the internet:

The number of potential customers for an online application will follow this curve.

The web giants¶

The web giants were born during this period and quickly faced requests from several million users:

Amazon (1994)
Netflix (1997)
Google (1998)
Facebook (2004)
...

They responded with new approaches at the processing level (e.g., MapReduce (2004)) and storage (e.g., NoSQL storage) allowing for horizontal scalability:

This approach will lead to:

The use of a reverse proxy acting as a load balancer at the web services level.
The multiplication of equipment and services to deploy, administer, and monitor.

Thus, these players will also be pioneers in DevOps by responding with a new approach to infrastructure management.

The role of Site Reliability Engineer (SRE) established by Google in 2003 to ensure a high level of service availability in close collaboration with developers will be a precursor in DevOps.

Agility in development¶

The democratization of the internet and networked applications will also result in the ability to deliver evolutions and fixes at a lower cost.

The publication of the Agile Manifesto in 2001 marked a turning point in development methods that would fully exploit the possibility of delivering at a lower cost.

Indeed, agility will include, among other things:

Delivering a new version of the application quickly and regularly.
Making business or affairs people and implementation people work together daily throughout the project.

Source management¶

Development agility will require improving tools and methods for managing application source code to facilitate collaboration and better manage the history of evolutions and corrections.

We will remember the dark hours when the use of a source code manager was not a widespread practice:

Traditional history management

Managing a student project's history
(before the use of a source code manager was taught at ENSG)

We will highlight the evolution of source code managers (SCM) with:

Client/server history management: CVS (1990), SVN (2000)
Decentralized history management: GIT (2005)
The introduction of the concept of pull-request with GitHub (2008)

We will endeavor to measure the contribution of the pull-request concept in terms of traceability compared to traditional validation processes. It will motivate the use of SCMs for other use cases (infrastructure management, documentation management, etc.)

Test automation on code¶

Delivering quickly and regularly will be incompatible with performing long manual acceptance tests to avoid introducing bugs.

Agility will therefore be declined into several development methods that will encourage reducing the risk of regression with the help of unit and functional tests.

We will cite, for example, the Test-driven development (TDD) method formalized in 2002 by Kent Beck, where the idea is to start by writing unit tests.

Continuous integration¶

Continuous integration tools such as Hudson released in 2005 (forked and renamed to Jenkins) will naturally gain popularity. They will be used, among other things, to:

Centralize test execution (and ensure they are properly executed)
Present test execution reports for everyone
Produce deliverables to deploy (zip/tar archive, .deb package for debian, .rpm for centos, windows installer,...)

www.jenkins.io - Creating a Jenkinsfile

It will very quickly be tempting to run tests before accepting code modification requests (pull requests).

Continuous integration solutions such as GitHub actions and GitLab-CI will therefore be integrated into source code managers a few years later.

mborne/node-extract - actions

Virtualization¶

The popularization of virtualization¶

Virtualization gained popularity among developers thanks to the release of free software such as VirtualBox supporting x86 architectures:

Illustration of using VirtualBox to build a development environment

The different types of hypervisors¶

For cultural knowledge, we will highlight that there are different types of hypervisors:

The different types of hypervisors.

Application virtualization¶

In practice, we will remember that the use of VMs allows abstraction from the infrastructure:

Application virtualization.

It will be possible to host several applications on the same VM but we will note that this implies sharing system libraries posing compatibility problems (python 2/3, PHP 5.6/7.4/8.1,...)

The democratization of the cloud¶

A low entry cost for scalability¶

In 2006, Amazon launched two services that would help popularize the concept of cloud computing:

Amazon S3 for data storage.
Amazon Elastic Compute Cloud (EC2) for application execution.

With these services:

Having a storage and computing infrastructure capable of adapting to the load is no longer reserved for a few large groups.
Pay-as-you-go billing gives small businesses the opportunity to launch a new service that will not be a victim of its own success.

A design allowing automation¶

At AWS, Jeff BEZOS established an important architectural rule: All communications between projects must go through the exposure and use of network APIs (cf. The API Mandate: How a mythical memo from Jeff Bezos changed software forever). This will play a big role in:

AWS's ability to effectively expand its service offering (loose coupling, clearly defined interfaces,...)
Customers' ability to automate the use of resources made available by AWS.

NB: This rule applies to communications between services (e.g., striving to build all applications solely on the basis of REST/JSON APIs developed by others is quite limiting...)

The generalization of IaaS¶

More broadly, the principle of making available an API to control a virtualized infrastructure will subsequently become widespread under the name IaaS (Infrastructure as a Service):

Illustration of IaaS principle

VMs will be initialized by the host based on a catalog offering several OS.
Updates and application installations will then be the responsibility of the client.

The diversification of cloud offerings¶

Cloud offerings will diversify quite naturally with variable support for different system layers by service providers:

Source: www.redhat.com - IaaS, PaaS, SaaS: what are the differences?

The limits of traditional operations¶

A strict separation of roles¶

At this stage, the possibilities for deployment automation are there, but methods inherited from traditional operations generally result in a process that sanctifies a strict compartmentalization of roles between DEV and OPS.

This leads to situations where the infrastructure is virtualized and equipped with an API allowing the automation of VM management, without the DEV team being aware of this possibility.

Communication centered on documents¶

For example, we will find the following process for deploying an application:

Traditional operations process

A seemingly perfect approach!¶

Developers are relieved of operational issues
Only system administrators configure the system

This is forgetting Murphy's Law!

This approach prevents fast delivery!¶

The initial production deployment of the application will easily take 1 month for various reasons:

Problem of understanding documents
Problem of completeness of documents
Problem of timeliness of documents
Problem of team availability

The same will apply to each evolution leading to the slightest architectural change (adding a parameter, adding a support service,...) which will leave two options:

Voluntarily maintain a non-optimal architecture.
Group many transformations into a major version where delivery will be long and risky.

An approach that generates problems in production!¶

An operational request or procedure may be misunderstood or poorly translated into operations without everyone's expertise being utilized:

OPS not participating in the application design cannot have a critical eye and understanding of requests (an update becomes a version upgrade, an action requested in QUALIFICATION is applied in PRODUCTION,...)
DEV not participating in the infrastructure design do not operate the system optimally (use of the wrong storage system, lack of compression, bandwidth saturation,...)

An approach leading to finding a culprit rather than a solution!¶

The boundary between roles humanly translates into a tendency for DEV and OPS teams to seek to limit their responsibility in case of a problem before seeking to prevent problems from appearing or even to quickly correct a problem.

Typically, before looking for a solution:

It is necessary to prove on the DEV side that the problem is not in the code for OPS to start analyzing supervision and logs.
It is necessary to prove on the OPS side that the problem is related to the code or design for DEV to review their work.

See the comic www.commitstrip.com - How to know if your company is DevOps? which summarizes the situation well.

The birth of the term DevOps¶

The term DevOps was born from an awareness of these issues:

In 2008, a conference allowed a meeting between the organizer of a meeting on the theme "Agile Infrastructure" and a project manager Patrick Debois facing the lack of cohesion between application development teams and operations teams.
In 2009, Patrick Debois contracted developments (DEV) and operations (OPS) in a hashtag to announce the first DevOpsDays: DevOps was born.

devopssec.fr - The history of DevOps covers this genesis in more detail.

DevOps thus goes beyond the simple problem of deployment automation. DevOps is above all an observation:

To be able to deliver applications regularly and frequently and adapt to the load, agility in infrastructure management is needed.
To introduce agility into infrastructures, a rapprochement between development (DEV) and operations (OPS) activities is needed.

In the section the principles of DevOps, we will try to see more precisely how to proceed to achieve this goal.