# The Path to a Creative Deployment Automation

> **Note**:
>
> The diagrams shown are in code (mermaid), like the document itself (Markdown). The code is available [here](https://raw.githubusercontent.com/imjoseangel/sandbox/devel/doc/automation.md).

## Introduction - Fail Fast and Teams

Many Teams try to implement automated deployment for their applications or Cloud components. Most of them fall in the same hole: **Fail Fast** and **Confidence**.

Different reasons prevent a **Growth Mindset**:

* Non-sense strict deadlines
* Lack of confidence
* Complain of Failure
* Judging performance
* Lack of innovation

We can improve the manual creation of repetitive content by using Human creativity and Input.

### Ever tried. Ever failed. No matter. Try Again. Fail again. Fail better. - Samuel Beckett

Automation requires time to start, creativity, investigation, and innovation. Failing fast is part of the learning process.

Check if the implemented process is *optimal and scalable*. Investigate alternative solutions and improvements. With the right mindset to *refactor* the process and the code, every Team will reach its full potential.

## From Human Intervention to Full Automation

Let's define a possible timeline of how could be the process from a manual run to a fully automated procedure.

### The first stage - Fear of losing a job

When a new technology knocks on the door, multiple reasons could stop a Team from progressing on innovation and adopting new techniques.

* The *damned* comfort zone
* Lost one's job
* False sense of control

Having the right coaching and time, Teams will gain confidence and improve the above procedure.

With these feelings over the table and other external factors, the best approach could be:

```mermaid
sequenceDiagram
    autonumber
    Local->>Script: Manual Run
    Script->>Server: It works!
    Server-->>Local: Automated!
```

It works and is automated. But there is still some field for improvement.

### The second stage - Starting the Dunning-Kruger Effect

At this stage, the Team starts to gain confidence and is pretty damn sure they know a lot. Automation starts to grow, and the Team initiates to learn new techniques. They use *GIT* and *pipelines*.

```mermaid
sequenceDiagram
    autonumber
    GIT->>Script: Manual Pipeline
    GIT->>GIT: History and Rollback
    Script->>Server: It works!
    Server-->>GIT: Automated!
```

It seems similar to the *first stage*, but not in the least. The process is:

* Centralized in a GIT repository.
* Runnable by anyone on the Team.
* It has a history log.
* It can be rolled back and traced.

#### Using a CI/CD Tool with inline code

> **Warning!!!**
>
> Having all the logic in the pipeline or using graphical options is not recommended.
>
>It is not scalable, has a lot of dependency on the CI/CD tool, and doesn't offer the advantages like rollbacks, traces, and logging given the *"as code"* options.
>
>It could be something like:

```mermaid
sequenceDiagram
    autonumber
    Note over CI/CD Tool: Inline Code
    CI/CD Tool->>Server: Manual Pipeline
    Server->>CI/CD Tool: It works!
```

### The third stage - Questioning our existing implementation

Hosting the code in GIT is a further important step towards automation.

Curiosity opens new ways of working with GIT, like *GIT Hooks*. The Team discovers how to apply Continuous Integration in their code. The original scripts could be custom *Bash*, *Python*, or *Powershell* and use *ShellCheck*, *PyLint*, or *PSScriptAnalyzer*. The **Pre-Commit**, **Static Analysis**, and **Pull Requests** have arrived to stay.

```mermaid
sequenceDiagram
    autonumber
    GIT->>Script: Manual Pipeline
    GIT->>GIT: History and Rollback
    GIT->>GIT: Pre-Commit
    Script->>Script: Static Code Analysis
    Script->>Continuous Integration: Pull Request
    Script->>Server: It works Faster!
    Server-->>GIT: Automated!
```

Check if the automation in place fits with the following rules:

* Can Create new components
* Can Modify existing components
* Can Remove existing components

In this stage, thoughts about recovery speed and rolling back in case of disaster emerge.

### The fourth stage - Realizing there is still a long way to go

With Continuous Integration, the process changes dramatically. The Teams start to look for new automation techniques and think about changing their scripts for standard tools like *Terraform*, *Pulumi*, *Ansible*, or *Chef*, among others. Using a tool like *Terraform* doesn't mean the Team will improve its process. Compared with Bash, the *Terraform* code is just another script launched from the local machine.

**Speed** is becoming an important asset, and the referred external tools help to achieve it. Most of these tools are prepared to run different jobs in parallel.

**Security** is key to automation, and with standard tools, it is possible to do **Static Security Analysis** before running the deployments.

**Refactoring** is a new word and becomes part of the process. The Team realizes it is time to review the existing implementation and learn from previous mistakes.

*GIT* is now the standard, and the Pull Request Process is improving. The Team starts to implement other testing and quality gates before the approvals.

```mermaid
sequenceDiagram
    autonumber
    GIT->>as Code Tool: Manual Pipeline
    GIT->>GIT: History and Rollback
    GIT->>GIT: Pre-Commit
    as Code Tool->>as Code Tool: Static Code Analysis
    as Code Tool->>as Code Tool: Static Security Analysis
    as Code Tool->>as Code Tool: Refactor
    as Code Tool->>Continuous Integration: Pull Request
    Continuous Integration ->> Continuous Integration: Quality
    as Code Tool->>Server: It works!
    Server-->>GIT: Automated!
```

### The fifth stage - The Grown-Up Mountain

**Continuous Delivery** and other techniques are now knocking on the door. The Team is out of its comfort zone and starts to investigate and learn new ways of improving. Probably, it is too early for Continuous Deployment.

**Tags** and **releases** are part of the process, just as creating **artifacts** and pushing to the respective servers.

**Documentation as code**, **Containers**, **Dynamic Security**, and **Integration Testing** is part of the improvement plan.

**Makefiles** facilitate self-documentation and serve as a startup point for all the implementations.

The diagram is now larger with lots of components. The process is becoming better and better:

```mermaid
sequenceDiagram
    autonumber
    GIT->>GIT: History and Rollback
    GIT->>GIT: Pre-Commit
    as Code Tool->>as Code Tool: Static Code Analysis
    as Code Tool->>as Code Tool: Static Security Analysis
    as Code Tool->>as Code Tool: Refactor
    as Code Tool->>Continuous Integration: Pull Request
    Continuous Integration ->> Continuous Integration: Quality
    Continuous Integration ->> Continuous Delivery: DAST and Testing
    GIT->>Continuous Delivery: Automated Pipeline
    Continuous Delivery ->> Continuous Delivery: Tags and Releases
    Continuous Delivery ->> Server: It works!
    Server-->>GIT: Automated!
```

Everything should be [modular](https://en.wikipedia.org/wiki/Orthogonality_(programming)) and repeatable now. As examples:

* Jinja2 Templates
* Code generation Templates
* Golang Templates
* CookieCutter
* Modules, Roles, Providers, or Packages in Terraform, Ansible, and Pulumi

### The sixth stage - Plateau of Productivity

In this stage, **GITOps**, **APIs**, **WebHooks**, **Trunk Based**, or **Containers** are part of the game. But also **Observability**, **Chaos Testing**, **Continuous Deployment**, or **Canary and Blue-Green** deployments, among others.

The whole process can be summarized like this:

```mermaid
sequenceDiagram
    autonumber
    GIT->>Code: Automation
    Code ->>CI: Pull Request
    CI ->> CD: Testing
    CD ->> Container: It works!
```

Don't stop here. Think about:

* Deliver Automation as a Service
* Deliver APIs for Self-Service
* Add central logging
* Add orchestration monitoring
* Packetize solutions with Containers

## Summary

Hitting excellence is the result given by the confidence gained by the Team during the whole process, but also from the business.

**Automation doesn’t mean taking away the creative process.** Investing time in new ways of working and techniques is essential to continue improving. Getting out of our comfort zone, discussing and applying our learning to our job are symbols of a well-balanced Team.