Addressing the Problem of Drift Detection and Drift Cause Analysis

You run [.code]terraform apply[.code], it fails partway through, and the next command you run refuses to move: [.code]Error acquiring the state lock[.code]. Nothing is broken yet, but nothing will proceed either until the lock is cleared.

This guide covers what [.code]force-unlock[.code] actually does, when it is safe to use, how to find the lock ID on every major backend, and what to do if the command itself does not fix things.

What does terraform force-unlock do?

Terraform locks state before any operation that could write to it. The lock stops two processes from writing to the same state file at once, which is the most common way a state file gets corrupted. Locking happens automatically and silently on every plan and apply where the backend supports it. You will not see a message unless acquiring the lock takes longer than expected.

[.code]force-unlock[.code] is the manual override for when that automatic process gets stuck. It removes the lock record so a new operation can proceed. According to HashiCorp, the command does not modify your infrastructure, and on most backends it does not touch your state data either. It just clears the record that says the state is currently held.

Usage:

terraform force-unlock [options] LOCK_ID

The only option is [.code]-force[.code], which skips the yes or no confirmation prompt. That is useful inside a script or CI job where nothing is available to type “yes” into. Otherwise leave it off, since the confirmation step is the last chance to catch a mistake before you unlock something someone else is actively using.

The lock ID is not optional and is not guessable. Terraform prints it in the error message when a lock is already held, and [.code]force-unlock[.code] requires an exact match. Per HashiCorp's own documentation, the ID works as a nonce, a one-time verification token that ensures a lock and an unlock target the same lock. That is deliberate: you can only release a lock you can identify, not just any lock on the state file.

One thing worth flagging up front: on the local backend, a stuck lock can only be cleared by the same machine and user that created it. There is no separate process to force it from elsewhere, which is one more reason most teams move to a remote backend, such as the ones covered in this backend configuration guide, before this becomes a live problem.

When should you use the force-unlock command?

Treat [.code]force-unlock[.code] as a last resort, not a first response. If two operations are genuinely running against the same state at the same time, forcing a lock open defeats the entire purpose of state locking and can leave you with a corrupted state file. Only run it when you are certain the process that created the lock is no longer active.

A stuck lock usually traces back to one of a few causes:

• A [.code]terraform apply[.code] or [.code]plan[.code] was cancelled or errored mid-run, for example because a CI job timed out or someone hit Ctrl+C, so Terraform never reached the step where it releases the lock.
• The machine or build agent running Terraform lost its network connection to the backend before the lock could be released.
• The backend storage itself changed mid-operation, for example a Terraform run modifying firewall rules, private endpoints, or access policies on the very storage account that holds the state file.

If none of those match your situation and you are not sure why the lock exists, treat that as a reason to investigate before clearing it, not a reason to assume it is safe to force.

Where to find the lock ID for every backend

In most cases you will not need to go looking. The lock ID appears directly in the [.code]Error acquiring the state lock[.code] message, under the [.code]ID:[.code] field, alongside who holds it and when it was created. The backend-specific detail below matters mainly when you are troubleshooting secondhand, for example clearing a lock a teammate reported without a fresh error message in front of you.

Local backend

Terraform writes a [.code].terraform.tfstate.lock.info[.code] file next to the state file while an operation is in progress. It is a small JSON object containing the lock [.code]ID[.code], the operation type, and who created it. On clean exit, Terraform deletes this file automatically. As noted above, a lock created by one machine cannot be released by [.code]force-unlock[.code] from a different one.

Amazon S3

As of Terraform 1.11, the S3 backend supports native state locking through the [.code]use_lockfile[.code] argument, and no longer requires a separate DynamoDB table. Setting it to [.code]true[.code] tells Terraform to create a lock object in the same S3 bucket as your state, using conditional writes so only one process can create it at a time.

terraform {
  backend "s3" {
    bucket       = "mybucket"
    key          = "path/to/my/key"
    region       = "us-east-1"
    use_lockfile = true
  }
}

With native locking, the lock ID is whatever the error message reports; there is no separate table to query. If your configuration still uses the older [.code]dynamodb_table[.code] argument, note that HashiCorp has deprecated it in favor of [.code]use_lockfile[.code]. On that legacy path, the lock lives as an item in the DynamoDB table, keyed by a partition key named [.code]LockID[.code], and you can inspect it directly:

aws dynamodb get-item \
  --table-name your-lock-table \
  --key '{"LockID": {"S": "your-bucket/path/to/terraform.tfstate"}}'

Azure Blob Storage

Azure Blob Storage implements locking through native blob leases, with no extra backend configuration required. If a run is interrupted mid-apply, the lease can be left in place. The lock ID appears in the error message, but you can also inspect the lease state directly:

az storage blob show \
  --account-name YOUR_STORAGE_ACCOUNT \
  --container-name YOUR_CONTAINER \
  --name path/to/terraform.tfstate \
  --query 'properties.lease'

If [.code]force-unlock[.code] is not an option, for example the lock ID is unavailable, you can break the lease directly through the Azure CLI, which achieves the same result at the storage layer:

az storage blob lease break \
  --account-name YOUR_STORAGE_ACCOUNT \
  --container-name YOUR_CONTAINER \
  --blob-name path/to/terraform.tfstate

Google Cloud Storage

The GCS backend also locks natively with zero extra configuration. Terraform writes a lock object at [.code]/.tflock[.code] in the same bucket as your state, and the lock ID is the object's generation number, which is included in the error message. Deleting that object directly is the manual equivalent of [.code]force-unlock[.code] if the CLI command fails for some reason.

HCP Terraform and Terraform Enterprise

This is a common point of confusion: [.code]terraform force-unlock[.code] is a CLI command that works against backends where Terraform itself manages the lock file. HCP Terraform and Terraform Enterprise instead lock and unlock workspaces through their own UI and API, not the CLI command. In the workspace's Actions menu, you can select Lock workspace or Unlock workspace directly, or call the workspaces API endpoint to do the same thing from automation.

Consul

With the Consul backend, lock information lives in the Consul key-value store rather than in a file. You can list it with the [.code]consul kv get [.code] command, or query the same data through Consul's HTTP API.

Using terraform force-unlock: a worked example

1. Identify the lock ID from the error message. For example: [.code]Lock Info: ID: b8814894-4a5f-217b-e97b-c4f5c02a1f88[.code].
2. Confirm nobody else is running an operation against this state. Check your CI/CD dashboard, ask your team, or check the environment's deployment history if you are running on a platform that centralizes this, before assuming the lock is actually stale.
3. Run the command with the ID from step one: [.code]terraform force-unlock b8814894-4a5f-217b-e97b-c4f5c02a1f88[.code]. Confirm the prompt with [.code]yes[.code], or add [.code]-force[.code] if you are running this non-interactively.
4. Verify the fix by re-running the command that originally failed, such as [.code]terraform plan[.code]. If it proceeds past the locking step without error, the lock is cleared.

Unlocking remote state: alternatives to force-unlock

Wait instead of forcing: -lock-timeout

If two operations occasionally overlap for a few seconds, for example two CI jobs kicking off close together, [.code]force-unlock[.code] is the wrong tool. The [.code]-lock-timeout[.code] flag tells Terraform to wait for the lock to clear on its own instead of failing immediately:

terraform plan -lock-timeout=5m

This is worth setting as a default in CI pipelines that run plan, apply, or destroy operations back to back, so a brief overlap resolves itself instead of surfacing as a lock error at all.

Manual removal as a last resort

Occasionally [.code]force-unlock[.code] itself fails, usually because the backend is unreachable or credentials cannot reach the lock record. HashiCorp's guide to recovering state from backup covers this scenario directly. In that situation, the remaining options are backend-specific: delete the lock object from S3 or GCS, edit or remove the DynamoDB item, or break the Azure blob lease as shown above. All of these bypass Terraform entirely, so treat them with the same caution as [.code]force-unlock[.code] itself.

Coordinate before you unlock

Whichever method you use, confirm no other process is mid-write before you touch the lock, and consider pulling a backup first with [.code]terraform state pull[.code]. Never use [.code]-lock=false[.code] as a standing workaround for frequent lock errors. It disables the protection entirely rather than resolving whatever is causing the contention.

Troubleshooting force-unlock errors

The lock ID does not match

[.code]force-unlock[.code] will refuse an ID that does not match the current lock. This almost always means you are using a stale ID from an old error message. Re-run the failing command to get the current lock's ID and try again.

Permission errors during force-unlock

Clearing a lock requires write or delete access to wherever the lock record lives, for example [.code]s3:DeleteObject[.code] on the lock object, or the equivalent DynamoDB, GCS, or Azure permission. A permissions error here usually points to the credentials Terraform is running with, not the lock itself.

The same lock error comes back immediately

If you clear a lock and it reappears right away, something is still actively writing to that state. Stop and investigate before unlocking again. This pattern usually means step two of the worked example above was skipped.

Managing state locking at scale with env zero

Clearing a stuck lock by hand does not scale once a platform team is managing hundreds of environments across multiple backends. Ad Hoc Tasks in env zero let you run a command, including [.code]terraform force-unlock -force LOCK_ID[.code], directly on the environment's deployment container from the UI. That means resolving a stuck lock does not require local CLI access, a checked-out copy of the Terraform configuration, or direct credentials to the backend that holds the state.

By default, ad hoc tasks are restricted to organization administrators, since they allow arbitrary commands against a live deployment container. Teams that want to delegate lock-clearing to platform engineers without granting full admin access can do that with a custom role scoped to just that permission.

It is worth distinguishing this from Environment Locking in env zero, which is a separate, deliberate governance control rather than Terraform's automatic state lock. Locking an environment in env zero blocks deploys, destroys, plans, and drift detection outright, with a reason attached for anyone else who looks at it, and it stays in effect until someone with permission unlocks it. A Terraform state lock, by contrast, is transient by design and normally clears itself within seconds. If you are troubleshooting a “locked” environment in env zero and [.code]force-unlock[.code] does not seem relevant, this distinction is usually why.

Key takeaways

• [.code]terraform force-unlock LOCK_ID[.code] manually clears a stuck state lock. It does not touch your infrastructure, and on most backends it does not touch your state data either.
• Only use it when you are certain the process that created the lock is no longer running. Unlocking an active operation risks a corrupted state file.
• The lock ID is usually sitting right in the error message. You only need to hunt through backend-specific tooling when troubleshooting without that message in hand.
• S3 no longer needs DynamoDB for locking. [.code]use_lockfile = true[.code] has been the supported path since Terraform 1.11.
• [.code]-lock-timeout[.code] prevents most stuck-lock situations in CI before they happen, by waiting instead of failing immediately.

Frequently asked questions

Q. How do I fix a Terraform state lock?

Run [.code]terraform force-unlock LOCK_ID[.code], using the ID from the [.code]Error acquiring the state lock[.code] message. Only do this once you are certain no other operation is currently running against the same state.

Q. What is Terraform state locking for?

State locking prevents two operations from writing to the same state file at the same time, which is one of the most common causes of state corruption. Terraform acquires the lock automatically before any operation that could write state and releases it when the operation finishes.

Q. Can force-unlock corrupt my Terraform state?

[.code]force-unlock[.code] itself does not modify your infrastructure or your state data; it only removes the lock record. The risk is indirect: if you unlock a state that another process is actively writing to, that process and yours can both write at once, which can corrupt the state file.

Q. Does the S3 backend still need DynamoDB for state locking?

No. Since Terraform 1.11, the S3 backend supports native locking through [.code]use_lockfile = true[.code], using S3 conditional writes instead of a separate DynamoDB table. The older [.code]dynamodb_table[.code] argument still works but is deprecated.

Q. How do I avoid stuck state locks in the first place?

Avoid cancelling Terraform runs mid-operation, set [.code]-lock-timeout[.code] in CI so brief overlaps wait instead of failing, and use a platform that centralizes deployment history so you can quickly confirm whether a lock is stale before clearing it.

What is Infrastructure-as-Code

Infrastructure-as-Code (IaC) is a method of automating the management and provisioning of infrastructure resources. Instead of manually clicking buttons on a web console, IaC enables organizations to describe their system architecture using code, allowing them to store, version, and track changes to their systems and application infrastructure. 

The goal is to automate the process of setting up, configuring, deploying, and managing applications. IaC is a powerful technology that allows you to provision and manage any cloud resource in an automated, declarative way. Infrastructure-as-Code is now the de facto standard for new projects and the focus of many organizations is now migrating from legacy architecture to IaC.

Before Infrastructure-as-Code: Pre-IaC Architecture

IaC’s major transformation was that developers could now create a consistent, repeatable workflow, bringing about wider-scale deployments across a range of resources, environments, and locations.

Delving a bit deeper, how did it achieve this? IaC provisions infrastructure and application resources through machine-readable definition files instead of through physical hardware configuration or interactive configuration tools. 

Before, infrastructure management was a costly, manual process that hindered scale and availability. There was extreme variability in infrastructure largely due to manual configuration. Manual processes were more error-prone and could not be scaled, much less standardized. Remote access tools slowly entered the market, but system administrators (sysadmins) still had to provision new hardware and resources manually by connecting to remote cloud providers via APIs.

Environment drift: When infrastructure for an application's software development process – development, staging, and production environments falls out of sync. Environment drift, or configuration drift, causes inefficiencies and can be expensive in direct cost and potential user experience impacts. If your app’s development environment varies from the production environment, this can lead to failure in production or bugs, and even prevent recovery in the event of disaster.

Automation changed that, reducing the problem of forgotten tasks, automating configuration drift detection, and allowing other features to automatically manage infrastructure problems or remedy issues. Among those revolutionary features were version control systems (VCS), configuration management tools, and orchestration capabilities.

Infrastructure-as-Code Benefits

Now, IaC has made IT more efficient than ever before, solving numerous IT challenges and enabling new capabilities such as:

Recreating environments

It used to be challenging to recreate an identical environment after deployment because the systems it interacted with also had to be updated.

With Infrastructure-as-Code, users can recreate infrastructure from scratch, and on-demand, simply by replaying code. The pipeline uses a prescribed set of parameters for deployment and creates a new environment that is identical in terms of the number of hosts, networks, data centers, clusters, data stores, etc., every time that it runs. The infrastructure code can even be versioned with the product, making it easy for engineers to recreate the infrastructure as it was when a previous version of the product was released.

Minimizing errors

IaC minimizes the need for manual infrastructure management, reducing the risk of human error. Rather than depending on engineers to remember past configurations or respond to failures, everything is in the code, under your source control system.

When changes go to production, the infrastructure code is checked in a code review or in a review by a gatekeeper.

Supporting teamwork and collaboration

Using IaC, engineers don’t have to deal with problems caused by conflicting changes in a shared environment. Infrastructure-as-Code makes it easier to work as a team and to share code with colleagues and other teams, so they can utilize it to set up their own environments. Using a VCS, different teams can each work on a separate piece of the infrastructure, rolling out their changes in a controlled manner.

Reducing cloud expenditure

The shift from bare metal infrastructure investments to the cloud reduced CapEx, and IaC has reduced them even further by enabling auto-scaling capabilities. With IaC, a software developer writes code and configuration management instructions that trigger actions according to actual needs and accurately reflects the structure of the real operating environment. Infrastructure-as-Code lets you manage your environments easily and automatically deactivates environments you no longer need.

DevOps and Infrastructure-as-Code

DevOps emphasizes automating manual tasks that typically take up a lot of software developers’ and IT operators’ time. IaC is one of the key technical practices that enable DevOps within an organization, by automating the provisioning and management of IT infrastructure. With IaC, developers can self-serve the provisioning of environments, saving time for them and the operations team.

How Infrastructure-as-Code Works

Key Concepts

1. GitOps – This involves integrations between your IaC tech stack and the infrastructure itself via your Git repository (on GitHub, GitLab, Bitbucket, etc.). This includes streamlining changes as much as possible, such as embedded PR commands.
2. Version Control – This is related to GitOps, where you will want to have a firm grasp on what versions of a framework, module, provider, or code you are using for your current work or for a specific kind of deployment.
3. State Management – This refers to the storage and maintenance of your desired state. Some IaC tools do not encrypt state files by default. For example, Terraform does not encrypt (it’s a premium feature in Terraform Cloud) while OpenTofu does. 
4. Registry – A registry is a marketplace for finding add-ons, integrations, packages, and policies. It often refers to the Terraform Registry. 
5. Templates – Templates refer to reusable packages of code or files that provision resources in certain configurations. They should be git-based.
6. Modules – This is the term for a configuration package, or collection of config files, in Terraform.
7. Providers – This is the term for an integration mechanism, akin to an API, between Terraform and a third-party app.
8. FinOps – This refers to the automation of cost monitoring, spending projections (cost estimation), and budget notifications/alerts so users can track the expense of their cloud deployments (in IaC and other sectors of DevOps).
9. IaC Pipelines – This is an ordered sequence of common or repetitive tasks that is configured to run automatically so as to save teams time with projects.
10. IaC Workflows – This refers to the sequence of status changes of infrastructure within a pipeline.

Declarative vs. Imperative Approach for Infrastructure Configuration

As with other subjects in DevOps, infrastructure has declarative and imperative approaches. Think of it like a means to an end; or rather, the imperative approach defines the means and the declarative approach defines the end. 

The imperative approach focuses on the sequence of commands needed to reach the desired state of your application, specifically in this case your infrastructure. In contrast, the declarative approach is becoming more popular thanks to better automation tools, as devs can define the endgame state and a given tool will configure an environment to reach that stated goal. 

Chef is the most prominent tool relying on imperative programming for IaC. Some have a mix of imperative and declarative implementations, namely Pulumi, Salt, and Ansible. However, declarative is gaining traction and effectiveness thanks to advances in automation. Declarative IaC tools include OpenTofu, Terraform, AWS CloudFormation, and Puppet.

Challenges and Best Practices 

Many best practices for IaC overlap with DevOps best practices in general. However, there are caveats specific to maintaining code-based infrastructure. 

Idempotency

Yeah, read that word carefully. This refers to being able to reapply code multiple times while getting a consistent result every time. This is as much a principle as it is a requirement to automate infrastructure, and templating will reduce or outright eliminate errors in many use cases. The goal of consistency also relates to testing, making sure that a deployment works in multiple environments and avoids the ‘it works on my machine’ problem.

CI/CD & Testing

Many teams have not instilled continuous integration and continuous deployment into their infrastructure deployments. CI/CD should be standardized in all layers of development and operations, including IaC. Constant changes to infra require testing and full VCS integration.

Observability – Logs & Debugging

Depending on the kind of deployment, you should have logging configured across your entire tech stack. Additionally, consider metrics and tracing to monitor every level of your infrastructure. Finally, debugging should be standard protocol with any code changes, especially if you’re changing code within a resource instead of switching out resources.

Immutability (when applicable)

Immutability refers to making code unchangeable. In such cases, changes mean replacing a resource entirely rather than editing its internal code. This is not always practical, but when it is, it eliminates an area prone to frustrating errors.

Version Control (including environmental parity)

As mentioned with CI/CD, VCS can protect you from influencing the wrong environment or pushing changes that aren’t applicable in some versions of your Infrastructure-as-Code framework. This is even more essential when dealing with multi-framework deployments, which get confusing.

Cost Management/FinOps

Cost management and cost projection/prediction are getting better with newer tooling available to all classes of developers, and the same with IaC FinOps for system architects. Tracking cloud spending gets tricky,  especially with the long list of internal features that cloud providers like AWS or Azure offer.

State Management

Storing the state of your IaC framework is fundamental. With many tools moving toward declarative programming, keeping that well-defined state protected is crucial.

Modularization

Relating to templates and paralleling containers, IaC frameworks like Terraform and OpenTofu rely on modules to organize resources defined by configuration files in the same directory. In the case of Terraform, they will be .tf or .tfjson files. There are three primary reasons behind using a Terraform module: 1) packaging resources together that will be used together in a reusable configuration, 2) sharing standardized configurations across organizations, and 3) don’t-repeat-yourself programming (DRY).

Access (Roles and Users)

This is part of the security concerns of an IaC setup. You want to manage and allow access to as many people in your organization as possible, but make sure that levels of access are well-defined in specific roles. This makes RBAC, role-based access control, as essential in IaC as any other sector of DevOps. 

Watch out for these IaC Pitfalls...

While IaC has clear advantages, it also presents unique challenges that usually emerge as you scale.

1. Integration with management tools

To harness the full benefits of IaC, it must be integrated into all processes, including CI/CD workflows, notification tools like Slack, security tools, system administration, IT operations teams, and DevOps teams, with well-documented policies and procedures. Without full integration, errors can quickly spread across the system.

2. Longer turnaround

When using IaC, every change has to be coded, tested, and reviewed before it is applied. Changes are more complex and must be planned carefully to avoid significant downtime. Learn more: Video: Top IaC Challenges

3. Lack of cloud expense oversight

Since IaC deploys infrastructure components automatically, it can be hard to keep track of expenses. Development teams are often unaware of the financial ramifications of their code, and expenses can build up quickly without monitoring tools that are designed for IaC.

That’s why some would explicitly include FinOps in the rubric of IaC. Regardless, it’s an essential part of managing complex infrastructure.  For instance, env zero includes cloud cost monitoring and optimization in its feature set.

IaC Toolchain Sprawl

One of the primary benefits of adopting Infrastructure-as-Code is consistency, which is only possible if teams across your organization are using different IaC tools and approaches. In many cases, implementing IaC requires a cultural shift in addition to the technical one to ensure success. The advantages far outweigh any overhead associated with implementing and managing IaC.

We’ll try to make some sense of that tool sprawl with the following section, covering the major frameworks and associated platforms in the world of infra.

Infrastructure-as-Code Frameworks

IaC’s major tools are frameworks that incorporate multiple functions into a single platform. The list below starts with those assets and then continues with IaC tools that are popular for one or multiple functions within IaC tech stacks. The following Venn diagram shows what kind of features go into a complete IaC framework, but note its complex structure that shows some tools can cover much of what you need for a deployment, but not everything.

Terraform & OpenTofu

Terraform is an IaC tool created and maintained by HashiCorp; it is currently the most widely used Infrastructure-as-Code tool in the industry. It is widely credited with creating common best practices including arguably the use of declarative programming.

In Summer 2023, Terraform moved away from Open Source licensing. As a response, several companies (including env0) collaborated to create an open-source, alternative known as OpenTofu. OpenTofu is currently managed by the Linux Foundation. Its initial release, v1.6.alpha, seeks to be a drop-in replacement for the Terraform version of the same number.

Terragrunt

Terragrunt is a thin wrapper for Terraform that provides additional tools for deploying hooks, managing dependencies, remote states and multiple environments, as well as keeping your Terraform configuration files DRY (Don't Repeat Yourself). Terragrunt is open-source and a popular choice for Terraform users looking for ways to keep their codebase efficient, clean and well-organized.

AWS CloudFormation

CloudFormation is the AWS service for IaC. It uses JSON or YAML to define resources. Its added advantage is that it works seamlessly with other AWS tools. On the flip side, its main disadvantage is that it only handles AWS infrastructure resources. Additionally, it limits templates to only 500 resources apiece, arbitrarily still keeps some processes manual, and has confusing documentation.

Pulumi

Pulumi is an open-source IaC framework that uses common programming languages to configure and provision resources rather than a domain-specific language like HCL. That also allows it to take advantage of inherent features of languages like Python, JavaScript, C#, and Go among others, as well as various implementations of those languages like TypeScript, Node.js, .NET, etc. 

Like Terraform and OpenTofu, Pulumi supports major cloud providers - AWS, Azure, and GCP cloud providers. It also features its own state management and language hosting, plus a command-line interface (CLI).

Crossplane

Crossplane is an open-source IaC framework managed by the Cloud Native Computing Foundation (CNCF) with a specific focus on managing Kubernetes infrastructure. It keeps application and infrastructure configuration in the same control plane (Kubernetes application layer), and uses other common k8s tools like Helm or Kustomize to launch IaC templates. 

Atlantis

Atlantis is a GitOps-focused tool that often acts as an add-on to basic IaC frameworks. It applies infrastructure automation with Terraform actions by use of commands embedded in pull requests (PRs) and to work from within their VCS. It still uses the webhooks native to Terraform to manage this, trying to get more done in Terraform by working through comments and PRs from GitHub, GitLab, and other version control systems.

CI/CD & Configuration Tools Used for IaC

Ansible

Ansible is an open-source CI/CD application that applies automation to pipelines but also functions as a configuration manager and orchestration tool. It is often compared with Jenkins, though the two tools can also function together in certain environments. In addition, Ansible integrates with Terraform. It is written in Python and works from the command line/terminal.

Argo CD

Argo CD is an open-source continuous delivery tool focused on Kubernetes that uses declarative programming. It monitors activity in Kubernetes clusters and compares infrastructure there to the version stored in a specified git repository. It will resolve any differences between the two versions to maintain the desired state. ArgoCD is commonly used in conjunction with IaC tools for managing and orchestrating applications alongside infrastructure.

Jenkins

Jenkins is mainly an open-source continuous integration tool. It automates testing, packaging, building, and deployment. It is more broadly considered a CI/CD tool, as it also handles continuous delivery. It supports several VCSs from the most popular to more niche options: GitHub, GitLab, Bitbucket, Git, Mercurial, Subversion, etc. Many developers use Jenkins to deploy infrastructure components, but it has limitations relative to fully IaC-dedicated frameworks. It can run multiple jobs through multi-threading.

CircleCI

CircleCI is, despite the limiting name, a full CI/CD tool for automating builds, testing, and deployments. Through its integration with a VCS, any change in a repository will trigger a CircleCI run job and run jobs simultaneously through parallelism/parallel processing (in contrast to Jenkins’ multi-threaded approach).

SaltStack

SaltStack, also known as the Salt Project or simply Salt, mainly serves as an orchestration and configuration tool. It has an emphasis on automating repeated DRY tasks. It uses the push method to make changes to code.

Chef

Chef is usually defined as a configuration management tool, which automates – writes, tests, and deploys – code. It can also be defined broadly as an infrastructure-as-code framework and automation platform. Its DSL is based on Ruby. To draw an analogy with Terraform’s modules, Chef’s “cookbooks” package together multiple “recipes,” e.g. config files that cover which resources to manage and in what order to execute them. As mentioned above, Chef relies mainly on imperative programming. Its client-side server architecture is known to support popular operating systems like Ubuntu and Windows.

Puppet

Puppet is a configuration management tool for automating code; it is often directly compared with Chef. It can also be defined broadly as an IaC framework with uses for orchestration, CI/CD, and monitoring. It mainly supports declarative programming. It supports different implementations of Linux in addition to other operating systems (MacOS, Windows, Ubuntu, Debian, etc.). It relies more on the pull method to make changes.

Infrastructure Management at Scale with env0

env zero is a self-service automation platform and management layer that sits above an IaC framework. It provides a simplified user interface for administering environment templates, controlling access roles, managing variables, defining policies, overseeing FinOps mech anisms, setting parameters for different developer environments (including ephemeral), and more.  

All in all, env0’s product reflects what the company sees as best practices for Infrastructure-as-Code, and therefore offers a suite of services:

Infrastructure Automation

env zero extends the creation of pipelines and workflows to Infrastructure-as-Code, using what are now established best practices in other segments of DevOps. env zero integrates with tools from different parts of the IaC tech stack – version control systems, configuration managers, orchestration tools, and CI/CD platforms – to create a consistent workflow with persistent changes pushed/pulled to your infrastructure.

Self-Service & Visibility

The emphasis on self-service leads to an emphasis on ‘granular RBAC’, where admins can add numerous specifications to custom roles in order to extend secure access across an entire organization as widely as possible. Utilizing Policy-as-Code and integrations with tools like OPA or Checkov, you can be confident that the right people have the right amount of access and let teams function independently to push/pull their changes to code.

With that, teams do not have to wait for someone else’s okay to be productive. Organization members can achieve that by using ephemeral environments (with time-to-live settings) to test new features, automated scheduling, and configurable templates.

Additional features like dashboarding and audit logs, plus available integrations with several major observability platforms, give admins even more data to adjust those policies in the long-term.

Covering All Frameworks

env zero is framework-agnostic. In other words, env zero covers Terraform, Pulumi, CloudFormation, Terragrunt, and others. While some companies (HashiCorp, AWS) provide a premium service on top of their IaC frameworks, they often encourage vendor lock-in and cover their own frameworks at the expense of others. 

Fair Pricing, FinOps Built-in

env zero encourages scale by using deployment-based pricing. However, other services such as Terraform Cloud price by RUM – or resources under management. RUM guarantees a higher bill for companies month to month, as teams are always adding more complex code and configuration changes. 

Deployment pricing provides flexibility to team managers to customize their environments in such a way to be smart with their cloud spending. env zero encourages this further with its slew of FinOps features like cost management, budget notifications, and project-based calculations. Those analyses inform future policies to limit or increase budgets for users, teams, specific resources, or particular deployments.

Imagine managing your cloud infrastructure using the programming languages you already love—Python, Go, JavaScript, you name it. No more wrestling with YAML, JSON, or HCL (HashiCorp Configuration Language) files!

Pulumi gives you that power, offering a robust CLI and service backend to manage both state and secrets. It's like the Swiss Army knife for cloud infrastructure, supporting all the major providers like AWS, Azure, and Google Cloud.

Today we're diving into the world of Pulumi and its integration with env0. We'll explore what Pulumi is, its features, how to set it up, and even throw in a real-world example (provisioning an EKS cluster). Also, we’ll weigh the pros against the cons and look at how it stacks up against other options. So buckle up; this is going to be a fun ride!

Video Walk-through

‍

Requirements:

• A GitHub account
• An AWS account
• An env zero account
• A Pulumi account

TL;DR: You can find the main repo here.

What is Pulumi?

Pulumi is an open source Infrastructure-as-Code (IaC) framework that provisions resources utilizing common programming languages. Pulumi also supports the major cloud providers: AWS, Azure, and Google Cloud. Its leaning on common languages eliminates the time it would otherwise take to get used to a new domain-specific language like HCL. 

If you're wondering how it stacks up against Terraform, check out my previous blog comparing Pulumi vs. Terraform. But the main benefits come in three main cores: the Pulumi SDK(s), the service backend, and finally the automation API.

Pulumi SDKs

First up, the SDKs. Pulumi's SDKs are what make it super versatile. These SDKs allow you to use languages like Python, JavaScript, TypeScript, Go, or .NET for defining and deploying your infrastructure. This is super cool because it means you can use the same language you're already comfortable with for your application development. 

That all means that you end up with the following advantages: strong familiarity with the core languages, a long list of library resources (according to language), and reusable custom abstractions.

Pulumi Service Backend

Pulumi's SaaS offering comes replete with CI/CD integrations, Policy-as-Code, role-based access, and state management. 

State Management  – Safely stores and manages the state of your infrastructure. This means less headache worrying about where your infrastructure's "truth" lives. There is also an option for self-managed state through your own cloud account on AWS, Azure, or GCP.

Collaboration Features  – You can collaborate with your team on infrastructure updates, with features like RBAC, stacks history, and more.

Policy-as-Code – Enforce security, compliance, and best practices across your infrastructure using Pulumi’s Policy as Code offering called CrossGuard.

CI/CD Integration  – Pulumi CI/CD integrations work with popular systems like GitHub Actions, GitLab CI, Jenkins, TravicCI, AWS Code Services, Azure DevOps, and more.

Automation API

This Automation API can embed Pulumi directly into your application code, offering a hassle-free way to manage infrastructure. 

In essence, this concept encapsulates the core functionalities offered by the Pulumi Command Line Interface (CLI), such as executing commands like [.code]pulumi up[.code], [.code]pulumi preview[.code], [.code]pulumi destroy[.code], and [.code]pulumi stack init[.code]. 

However, it extends beyond this by offering enhanced flexibility and control. This approach is designed to be strongly typed and secure, facilitating the use of Pulumi within embedded environments, for instance, within web servers.

Importantly, this method eliminates the need for running the CLI through a shell process, streamlining operations, and integrating infrastructure management more seamlessly into application environments.

Pulumi Features

Alright, let’s dig into some of the Pulumi concepts and features that it offers:

1. Component Resources

Pulumi lets you define reusable building blocks known as "component resources." These are like your typical cloud resources but bundled with additional logic. If you are familiar with Terraform, these would be your modules.

2. Stack References

Manage dependencies between multiple Pulumi stacks effortlessly. This feature is a real game-changer for managing infrastructure at scale.

3. Templates and Packages

Think of these as the ultimate cheat codes for your IaC. Instead of starting from scratch, you can kick things off with a pre-baked setup. Here’s why they're great:

• Speedy Setup: No more blank-slate syndrome. You’ve got a starting point that’s not just a blank file – it’s a springboard that gets you coding your infra in record time.
• Best Practices: These templates aren't just thrown together – they're crafted with best practices in mind. So you're not just starting faster, you're starting smarter.
• Learning Resources: New to Pulumi or a particular cloud service? Templates can be great learning tools, showing you the ropes of how things are structured and pieced together.

How to Install Pulumi

Alright, time to get our hands dirty. Installing Pulumi is a breeze. You can reference this from Pulumi's documentation.

Since I'm running this in my Windows for Subsystem Linux environment, I can run the install script as shown:

curl -fsSL https://get.pulumi.com | sh -s -- --version 3.91.1

Pulumi Stack Example

Let's get into the meat and potatoes: stacks. A Pulumi stack is essentially an isolated, independently configurable instance of a Pulumi program. Let's first work with the Pulumi CLI then later we'll see how to use env0.

Create a New Pulumi Project

First, create a Pulumi project by creating a new directory and running the [.code]pulumi new[.code] command with the [.code]kubernetes-aws-python[.code] Pulumi template.

mkdir Pulumi-EKS
cd Pulumi-EKS
pulumi new kubernetes-aws-python

Continue by providing a project name, description, and stack name along with the AWS region and some other parameters.

Pulumi installs the necessary dependencies and your new project is ready.

Run Pulumi

Next, make sure you export your AWS cloud credentials as environment variables and run [.code]pulumi up[.code].

export AWS_ACCESS_KEY_ID=your-access-key-id
export AWS_SECRET_ACCESS_KEY=your-secret-access-key
pulumi up

Read what Pulumi is about to do, then answer [.code]yes[.code] when asked if you want to perform this update.

Now, Pulumi will start to provision resources and you will see the resources get created in the terminal as shown below.

Observe the Output Results

If all goes well, you should have your new EKS cluster up and running. You can also check the Pulumi UI for your new stack where you can view all the resources created along with the output.

You can view the output in the UI or the CLI for the vpcId and the kubeconfig.

Access the EKS Cluster

To get the kubeconfig for the EKS cluster, run the following command:

echo $(pulumi stack output kubeconfig) > mykubeconfig
export KUBECONFIG=./mykubeconfig

Now run [.code]kubectl[.code] commands to interact with the EKS cluster:

kubectl get nodes

Congratulations! You've successfully provisioned an EKS cluster in AWS.

Examine the Infrastructure Code

Take a look at the actual code that provisions our EKS cluster. Notice how it's written in simple Python. I could have built the cluster from scratch by calling on each resource, but why reinvent the wheel? There is an excellent Pulumi package called Amazon EKS in the Pulumi Registry. I decided to go with this.

As you see, in under 40 lines of code, we have our EKS cluster defined:.

import pulumi
import pulumi_awsx as awsx
import pulumi_eks as eks

# Get some values from the Pulumi configuration (or use defaults)
config = pulumi.Config()
min_cluster_size = config.get_float("minClusterSize", 3)
max_cluster_size = config.get_float("maxClusterSize", 6)
desired_cluster_size = config.get_float("desiredClusterSize", 3)
eks_node_instance_type = config.get("eksNodeInstanceType", "t3.medium")
vpc_network_cidr = config.get("vpcNetworkCidr", "10.0.0.0/16")

# Create a VPC for the EKS cluster
eks_vpc = awsx.ec2.Vpc("eks-vpc",
    enable_dns_hostnames=True,
    cidr_block=vpc_network_cidr)

# Create the EKS cluster
eks_cluster = eks.Cluster("eks-cluster",
    # Put the cluster in the new VPC created earlier
    vpc_id=eks_vpc.vpc_id,
    # Public subnets will be used for load balancers
    public_subnet_ids=eks_vpc.public_subnet_ids,
    # Private subnets will be used for cluster nodes
    private_subnet_ids=eks_vpc.private_subnet_ids,
    # Change configuration values to change any of the following settings
    instance_type=eks_node_instance_type,
    desired_capacity=desired_cluster_size,
    min_size=min_cluster_size,
    max_size=max_cluster_size,
    # Do not give worker nodes a public IP address
    node_associate_public_ip_address=False,
    # Change these values for a private cluster (VPN access required)
    endpoint_private_access=False,
    endpoint_public_access=True
    )

# Export values to use elsewhere
pulumi.export("kubeconfig", eks_cluster.kubeconfig)
pulumi.export("vpcId", eks_vpc.vpc_id)

Pulumi makes it very easy to choose between many languages right in the documentation.

If you need to tweak the cluster configuration, it's easy to do so with the very well-documented eks.Cluster package.

Pulumi Configuration Files

When we ran the [.code]pulumi new kubernetes-aws-python[.code] command, Pulumi 1) created a new folder for us, 2) downloaded dependencies in a virtual environment for Python, and 3) also created two config files. 

Let's take a look at them now.

1. pulumi.yaml

This file acts as the manifest for your Pulumi project. It's a key part of the project configuration and provides metadata about the project itself.

name: my-pulumi-eks-env0
runtime:
  name: python
  options:
    virtualenv: venv
description: A Python program to deploy a Kubernetes cluster on AWS

‍

Here's what each part of the content you've provided does:

• name  – This is the name of your Pulumi project. When you run [.code]pulumi new[.code], it sets this name, and it's used as a default prefix for the resources Pulumi creates.
• runtime  – This specifies the runtime environment that your Pulumi program is expected to run in. In your case, it's set to python, meaning the Pulumi CLI expects your Infrastructure-as-Code to be written in Python.
• options  – These are additional settings related to the runtime environment.
• virtualenv –  This option tells Pulumi to use a Python virtual environment located in the venv directory within your project directory. This is important for Python-based projects to ensure dependencies are isolated from other Python projects on the same system.
• description  – This provides a human-readable description of what the Pulumi project does. It's a string that helps you and others understand the project's purpose at a glance.

So, when you initialize a new Pulumi stack or when Pulumi interacts with your project, it uses this file to understand the project structure, runtime requirements, and other metadata that influence how it deploys and manages your infrastructure resources.

2. pulumi.dev.yaml

When you run the pulumi new command and answer the setup wizard's questions, Pulumi automatically saves these answers as configurations in the pulumi.dev.yaml file. This file acts as a record of the initial setup parameters you specified for your project.

Now, if you enter commands or make changes at a different time (i.e., not during the initial Pulumi new setup) these changes won't automatically update the pulumi.dev.yaml file. Instead, you have two main alternatives for updating configurations after the initial setup:

1. Manual Editing – You can directly edit the pulumi.dev.yaml file to change or add configurations. This is like tweaking the settings of your project by hand.

2. Using Pulumi CLI Commands – You can use specific Pulumi CLI commands to update your configuration. For example, if you want to change the AWS region, you could use a command like [.code]pulumi config set aws:region us-west-2[.code]. This command updates the configuration in your pulumi.dev.yaml file without you having to manually edit the file.

Here is the content of the file:

config:
  aws:region: us-east-1
  my-pulumi-eks-env0:desiredClusterSize: "2"
  my-pulumi-eks-env0:eksNodeInstanceType: t2.small
  my-pulumi-eks-env0:maxClusterSize: "3"
  my-pulumi-eks-env0:minClusterSize: "1"
  my-pulumi-eks-env0:vpcNetworkCidr: 10.0.0.0/16

To clean up simply run [.code]pulumi destroy[.code].‍

Pros and Cons of Using Pulumi

Pros

1. Language Choice – Use your favorite programming language.
2. Rich Ecosystem – Supports a ton of cloud providers.
3. Dynamic Providers – Extend its capabilities as you see fit.

Cons

1. Language Overload – Sometimes, choosing a language can be a burden.
2. Learning Curve – If you're coming from dedicated DSL tools like Terraform's HCL, there might be an initial hump.

Pulumi Alternatives

The most obvious alternative to Pulumi is Terraform. But hey, keep an eye out for OpenTofu, an upcoming open-source alternative following a BSL license change. Crossplane is another alternative for those who enjoy building infrastructure using Kubernetes CRDs. Check out more details below.

1. Terraform

Overview: Terraform is a big player in the IaC field. It uses its own domain-specific language, HCL (HashiCorp Configuration Language), which is designed to describe infrastructure in a declarative way.

Why It's Popular: Terraform's been around for a while and has a huge community and support base. Plus, it works across many cloud providers, making it super versatile.

Key Differences from Pulumi: Unlike Pulumi, Terraform isn’t based on conventional programming languages. So, if you're not into learning HCL, it might be a bit of a curve.

2. Crossplane

Crossplane is perfect for those who are all-in with Kubernetes. It allows you to manage your infrastructure using Kubernetes CRDs (Custom Resource Definitions).

If you’re comfortable with Kubernetes and want to manage cloud resources as Kubernetes objects, Crossplane is your go-to. Being Kubernetes-focused, it fits well in ecosystems already heavy with Kubernetes usage and has a growing community.

Thoughts

Each of these alternatives has its own flavor. Terraform is the established giant with a dedicated language, OpenTofu promises to always be open-source along with new approaches to IaC, and Crossplane merges the worlds of Kubernetes and IaC.

Depending on your needs, comfort with certain technologies, and the specifics of your infrastructure, one of these might be a better fit for you than Pulumi.

Tutorial: Using Pulumi with env0

Now let's see how to use env zero to create the same Pulumi stack. We will create the same EKS cluster but this time by using env zero to trigger Pulumi.

Let's start by creating a new project in env0.

Next, you'll need to create a Pulumi template as shown:

Then connect to your VCS. Make sure to select the Pulumi folder, in our case Pulumi-EKS.

Under variables, add your PULUMI_ACCESS_TOKEN environment variable.

Then finally, make sure this template is deployable in our 'eks-demo' project.

AWS Cloud Provider Credentials

Make sure you have your AWS credentials set up in the Project settings

Create an Environment

Now we're ready to create a new environment. Head over to 'Project Environments' then create a new e.

When you see the eks-template, click the 'Run Now' button. There are some options to use such as enabling drift detection and the ability to automatically destroy the environment. When you're ready click the Run button.

Notice in the deployment logs how we have a 'Before: Pulumi Preview' step. This is defined in the env0.yaml file at the root of our repo to provide our configuration variables.

Below you can see how our env0.yaml looks like. Notice that we are specifying the same configuration variables that were in our pulumi.dev.yaml file.

version: 1
deploy:
  steps:
    pulumiPreview:
      before:
        - cd Pulumi-EKS && pulumi config set-all \
          --plaintext aws:region=us-east-1 \
          --plaintext my-pulumi-eks-env0:desiredClusterSize="2" \
          --plaintext my-pulumi-eks-env0:eksNodeInstanceType=t2.small \
          --plaintext my-pulumi-eks-env0:maxClusterSize="3" \
          --plaintext my-pulumi-eks-env0:minClusterSize="1" \
          --plaintext my-pulumi-eks-env0:vpcNetworkCidr=10.0.0.0/16

If you left the option to approve the plan automatically unchecked, you will need to confirm the execution of the [,code]pulumi up[.code] command.

View the Output

Finally, once the deployment completes, you can view the outputs under the 'Resources' tab.

Once again, to access the Kubernetes cluster, you can simply save the kubeconfig in a file and export as an environment variable as shown below:

export KUBECONFIG=./mykubeconfig
kubectl get nodes
NAME                           STATUS   ROLES    AGE   VERSION
ip-10-0-144-143.ec2.internal   Ready       73m   v1.28.2-eks-a5df82a
ip-10-0-29-122.ec2.internal    Ready       73m   v1.28.2-eks-a5df82a

Congratulations! You've just used env zero to deploy the Pulumi stack and provision an EKS cluster, and it probably took less than 5 minutes.

To clean up, just click the 'Destroy' button. One click and it gone.

In Summary

We've covered a lot of ground in this post—from the nuts and bolts of what Pulumi is to its nifty features, and even how it plays nice with env0.

If you're in the DevOps or Platform Engineering space, Pulumi offers a refreshing take on infrastructure-as-code. By marrying traditional programming languages with cloud resources, you get a level of flexibility and power that’s hard to beat.

So, what's the takeaway? If you’re looking to step up your infrastructure game, Pulumi is worth a shot.

Not ready for your entire team to move from Terraform to Pulumi? That's the benefit of a framework agnostic IaC platform such as env0.

Here are some of the key features that I like about env0:

• Drift detection – env zero provides drift detection that can help you detect drifts and alert you about them automatically.
• Governance – our platform allows you to define custom policies and guardrails to both secure and keep your infrastructure compliant.
• Multiple frameworks – env zero supports multiple frameworks such as Pulumi, Terraform, OpenTofu, and more.
• Ephemeral environments – Developers can set up an environment with a timer to self-destruct reducing wasted resources.
• Flexibility – With pre- and post-hooks that reduce the need for a full external  CI/CD pipeline.

For more information on env0's support of Pulumi, please reference this guide.

‍

Join us as we sit down with Sergey Korolev of Rakuten Viber to learn how his team automates and manages Infrastructure as Code, and how mature IaC practices improve developer experience, reduce technical debt, and streamline operations at scale.

Discover practical strategies and best practices for:

• Automatically detecting drift from manual IaC changes
• Resolving deployment tracking issues
• Setting up managed developer self-service for IaC
• Integrating IaC with GitOps workflows
• Optimizing IaC-related costs
• Unifying IaC-related processes

and more!

-----

Transcript

Andrew

Welcome, everyone, and thank you for attending today's webinar, “Using env zero to Pave the Path to Infrastructure as Code Maturity.” Today I have Sergey Korolev with me from Viber. Sergey, would you like to introduce yourself? 

‍Sergey

A few words about myself: I've been in the area of DevOps and managing teams that are working in this field of DevOps for the last ten years. Besides that I like traveling, I like snowboarding and doing martial arts in my free time. Thank you. 

‍Andrew

Nice. My name is Andrew. I'm the Director of Sales Engineering here at env0. And today I'll be helping lead this conversation. We do have the chat open, so please feel free to ask your questions in there, and then we'll address that during the Q&A portion of this conversation. 

Before we begin, a little quick overview of what we'll be talking about today. We'll be reviewing some challenges with Viber’s previous CI/CD process, how they managed Infrastructure as Code, the cultural shifts related to that, and what that really meant in terms of managing resources and deployments. And at what point did the previous process come to a breaking point? So what was that tipping point for change? And just before we begin this conversation, let's quickly review env0.

env zero gives you the ability to fast track your Infrastructure as Code’s maturity. What we're seeing in the market is a lot of times people are starting with no Infrastructures as Code, click ops and manual deployments, which is slow, and they start scripting and adding some Infrastructure as Code. But that creates cost overruns and security challenges. At env0, we're trying to optimize both productivity and governance through four main pillars: automation, cost controls, managed self-service, and governance.

We'll be addressing some of these features that Viber has been able to take advantage of and talk more about how we are helping Viber move along this maturity curve. So on a high level, what was their biggest game changer?

‍Sergey

When we started with env0, I guess the first thing that gave us the most value was drift detection. It's probably also the first topic we are going to be discussing today. Basically just understanding where we stand in terms of what our Terraform states look like. And I will be describing exactly where we started from and where we are today. Drift detection is the thing that actually helped us a lot, especially in the first few phases of onboarding.

‍Andrew

Awesome. So let's just jump right into it then. In terms of drift detection. Let's go into before and after. What did you see? 

‍Sergey

So before, onboarding to env zero would be… I guess a lot of the participants in this chat know this feeling when you try to apply some Terraform. And then you are like, oh, it's been changed two years ago and for some reason I don't know who did that and well, I can’t find him now or why he made the change he did. Detecting drifts, this is the term that is being used, a long time after the actual apply or the actual change happened. So you basically just see it whenever you run a plan or apply and you can check that list. 

How we use that, we would check it only at this phase, like you need to make a change. You make the change. Oh no, the state is not what I was expecting it to be. And now I need to fix the previous change that somebody else made.

And of course, there was no process to regularly check whether the states have been changed. If the resources, the actual resources, have been changed or not. And once we have onboarded env0, we detected that basically 60% of all our states, of all our Terraform so-called environments in env0, are like that. 

And just to let you know, we've been using Terraform at least for the scanned environments for about two years. We have between 7 to 10 committers I think. So plenty of hands touch the code, and not just the code. It could be either the code or the actual resources in AWS in our case. 

Now, after integrating the environments and code into env0, you can see in the screenshot that today we have more than 900 environments. Most of them are not drifted. Of course the drift happens due to some manual changes or other changes people or some automation that might be running and changing resources like AWS or ECS for example, there are things that are being changed by the service itself and you need to correlate the Infrastructure as Code to that.

We do these scans every other day. We are running a plan on all of our environments every other day and detecting whether those environments, whether the Terraform that represents them, are the actual resources that we have in our cloud environment, and they can see it easily in this dashboard. It's very convenient, something I do every day, and me, as a manager, I’m bugging my engineers like “you need to fix that, why does it drift?” 

Eventually we reduced it to zero. And as time goes on, you can see that the drift is going back and forth, but we try to eliminate it as much as possible. Some drifts are eliminated easier than others. But yeah, and as we go on with the talk, I will also describe how this, in my opinion, is leading to the cultural shift in our team.

‍Andrew

Awesome. Let's move on to the next topic then. Environment Discovery. So let me just quickly inform the audience on what Environment Discovery is at env0. So it's basically the ability to be able to scan for code changes within your git repo and automatically on board that into env0. And another way to describe it is kind of self-service through code.

So as your developers are creating here, you can see in the image a new Terraform stack and a folder. env zero essentially detects that through the PR process and will generate the plan and then deploy that resource once it's been approved. 

‍Sergey

I think it was about a year and a half ago, Environment Discovery was not as advanced as it is now. So that's a good fit for whoever is joining now will have it much easier for themselves. The cool thing about that is that any Terraform I’m adding, let's say I have this kind of a folder structure. When I have an integration environment, I have some infrastructure. Let's say I have EKS and I have some integration cluster now.

So basically the integration clusters and environment. And now I want to, for example, add another cluster. All I want to add is the new service that I was not managing before, for example ECS, and all I need to do after I configure the environment discovery in env zero – I just add the folder, right? And the same GitOps that again will basically trigger env zero to add an environment according to the new structure of the folder.

And this is something that we benefit a lot from at this point, in my view, and we benefit the most from it because it's just straightforward. You add your Terraform and it’s being run somewhere else, not on your computer, for example. Something I will also probably be talking about in the next few slides. 

I guess that's as far as it goes in terms of the Environment Discovery itself. Another thing I would add is that once you discover the environment, all the newly discovered environments go through the same process. If you need linting, governance checks, or tests to apply to your Terraform, it will be consistently applied to all Terraform environments.

Previously, it wasn't like that. People would run Terraform on their machines, and each team member might or might not follow best practices. For example, everyone should run lint, but I didn't have the tool to enforce that. I couldn't ensure that everyone was running lint regardless of where they were running Terraform from.

I think we can move on to the next topic.

‍Andrew

Okay. Let's talk about GitOps then.

‍Sergey

GitOps is something that is very close to my heart. We've been using ArgoCD for the past three years or so and we find it very convenient just making changes in Git and having this change go on as you proceed with whatever strategy that you choose. But for example, if you open a merge request, let's say in Terraform, you would have a plan. If you merge this merge request or pull request, you will actually run an apply. This is basically the general idea here and we try to implement it ourselves.

So we implemented the kind of testing mechanism where we would run a plan and then apply to our environments prior to onboarding env0. And it was a hassle to implement the solution itself, it was a hassle to maintain it and actually make sure that people are not just skipping the pipeline itself and just submitting the changes.

And because also, as I mentioned, we did not have the actual apply and plan of the actual infrastructure as part of the pipeline. And we've reviewed Atlantis. I tried to see why it fell short exactly but didn't find the exact things that made it not work for us. I think it was something involving us using GitLab Enterprise and Atlantis not supporting it at the time.

And also it's basically a CI tool which you need to orchestrate more things around to make it actually work. You will need to implement the whole thing in your CI/CD pipelines to make it work, as you would expect. At the time, about a year and a half ago, two years ago when we tested it, maybe it's better now.

Another thing is that, as I mentioned, everybody could just run Terraform as they wished. And me as a manager, I would not see these changes. And that from personal experience, one of the things that could happen is that you will run the Terraform locally and you go on to another task but not submit the commit. 

And so then you get the drift in your Terraform because it did not submit the code to Git. And you now need to understand who made the change and why he made it. And eventually you discover that somebody just forgot to submit the commit. 

‍Andrew

There's a question from the audience about ArgoCD and whether or not you… Is it replacing Argo CD?

Are you still running Argos CD on the side? Does env zero interact with Argos CD In which way?

‍Sergey

For us it was pretty straightforward that in our use case we will probably not use env zero for deploying Kubernetes just because we already had that and we had this expertise and were totally fine with ArgoCD, but it's complementary.

One of the cool things that you could do is, for example, and this is what we are doing, deploying the whole ArgoCD infrastructure itself in Kubernetes via env0. So you need to deploy that. So to say, operating ArgoCD, you do it for env zero for Terraform. Similarly, you can deploy basically any Helm chart either via env0's capabilities to deploy Helm charts or for Terraform providers to deploy Helm charts, both ways work.

But we chose to stay with ArgoCD for deploying Kubernetes. We really like this ecosystem and they remain as a complementary system to one another. 

‍Andrew

Right. Thank you. Let's talk about the next topic: Cost Management. How does env zero help you with managing costs? 

‍Sergey

So if you look at the right screen here, you will probably see that one of our environments is pretty costly, right? I reviewed it because I saw it in env zero discovery and found this is basically a cloud fund distribution that is getting a lot of requests, a lot of traffic. And this is why it costs as much as it does. 

And basically in cost management, initially when starting, I didn't think of it as a thing that I might be so interested in because we had so many other FinOps tools that we are grabbing information from, we have anomaly detection, you have the cost Explorer, you might have some third party that does that for you and you have all these kinds of things that your management costs. 

And it depends on your company, you might have dedicated FinOps engineers, you might have developers doing the FinOps and you might have your software engineers doing FinOps. So it depends on the company’s strategy regarding FinOps in general. And I think from the perspective of a DevOps engineer, you're not going to cost-explore every commit or every change. You're doing infrastructure and you will probably review your environments and your Terraform code more often than reviewing your costs. Especially if a FinOps is a different group or a different team or a different person in your organization. So putting it in your face basically is something that helps us extend our FinOps practices, extend our FinOps options and visibility. It does not necessarily replace those.

But there is another thing regarding that, we can see exactly which resources are costing what amount of money. I said that an environment represents a Terraform folder, for example, so you know exactly which resources are in this Terraform, following this Terraform state. And you can see exactly how much they cost, which is really cool.

And also it is relatively simple to configure. Basically I think Terratag is env0’s open source, so it's just incorporated in env zero itself. And basically what it does, on top of your state file it adds the needed tags for your environment ID and project ID. And then you configure your cost explorer and your billing so that those tags are tagged and you can see by these tags of your env zero environment, for example, how much this specific resource costs you according to the tag. And all that was just an addition that you basically just enable. It's relatively easy to configure. I think it took me an hour or so to do the whole configuration.

So it's a feature that I like a lot. I would not say again that it changed the whole FinOps practices in our company, but it for sure led me to some interesting environments where I immediately saw that something was wrong. This is a tool that I use each and every day. 

‍Andrew

So with all these different topics I’d love to better understand, how did it affect your company culture?

‍Sergey

I wouldn't say company because we – and in the next few slides you'll see that we are going to talk about – our future plans and how we see env zero part of it. Today we're mostly using env zero as a tool for our DevOps engineers and culturally I think that things such as auditing and governing our code became more of an everyday practice, something that we can introduce to the Infrastructure as Code more easily.

And for Viber, for example, it's very important in terms of auditing and governance. Let's move to a different topic: Git and commit, commits and changes in Git. So we have the governance of it, we need audits and we have for example to 4 eye principle practice. Well, we have somebody mandatorily check our code and approve it.

The same goes for Infrastructure as Code. With this in mind we also implement the governance and auditing of everything via env zero as well. And another thing is that in the culture of our team, something that changed is the state of mind of not breaking states and keeping the broken states as low as possible and reducing the tech debt.

And I think when you're looking at the long run, you want to keep your tech debt as low as possible for the team and the future of people to come. You don't want to come to a place where you'll need to find those things breaking all the time or broken things that are there for years. And I guess many of the participants today know exactly what I'm talking about. And of course, it did not eliminate everything. But I think the right life mindset will eventually get us to this point. 

‍Andrew

So yeah, cultural change definitely is hard and takes the right people in place and the processes and in order to help move that along and that tools can only do so much. We'd love to talk next about your expectations, your experience from onboarding with env0.

So there are naturally expectations. And I think this will be really interesting for people who are just going to start looking at env zero or considering. So let's look at these five different parts here. When you're talking about transferring pipelines, what was your expectation and what was the reality?

‍Sergey

I would expect onboarding all of our environments, especially talking about a Terraform-like liability that consists of all those kinds of Terraform and different structures and different ways that were applied.

I expected it to be less effort, it took me about, I think, two weeks to do it end to end. Today we are using the auto Discovery feature to just continuously onboard new environments automatically. So now it doesn't take any more effort anymore.

 Andrew

What about visibility? You mentioned here that you were concerned about how you actually see the resources in env zero and policies and things like that?

‍Sergey

Yeah, so I didn't have the chance to introduce it here, but in general, what you can see besides the dashboard is all the environments and those statuses and whether they are drifted or not. So if you click on any of the environments, for example, you will see an audit of all the deployments that have been done.

And you can see who did it, why they did it, when all the phases of the lifecycle of a specific deployment, let's say it's starting by initializing the Terraform and then planning and applying it and some steps in between, which can be the tagging that I told you about with Terratag. It can be linting as a step. You can add custom scripts to it, OPA, a plugin that you can add. So again, from a manager’s perspective it's so convenient for me to understand what my team is working on. As an engineer, just going back and just reviewing: Okay, which changes were done to some resources that I own? That’s also something that we value a lot.

‍Andrew

Definitely. A lot of our customers and prospective clients are concerned about auditability and being able to track exactly what happened, especially when you're coming from a manual deployment process or even just simple pipelines, because in general, CI pipelines are harder to go through as opposed to what you have here in env0, you can click on the infrastructure and see exactly what those changes were.

I love this. There's this third expectation we would love if everyone loved env0. What was your actual experience onboarding with env zero and people's reaction to using it? 

‍Sergey

So as I said, you can run all those kinds of things, such as limiting and OPA and other scripts. You have more steps that a plan would take locally. You can initialize it once, you have all the files that you need locally downloaded from your backend and basically rerun it as much as you want so you don't need to initialize it again. The plan will probably run faster on the apply and everything will be running relatively faster on a local machine due to the fact that you have all the files locally all the time.

People were hanging onto that. They would tell me, “it was fast for me, I don't see the reason why I need to run all this additional stuff. What does it give me?” And the answer is consistency. If you have policies in the company that you regulatory policies that you must follow, you must follow those policies.

It's not a matter of question. It's a matter of you have to do it. And people who are running things locally without applying those policies on their runs, eventually, hopefully not, but it can lead to issues. And I don't say that planning it in a managed environment and going through all the policies and all the tests and whatever will result in zero errors and no issues and no mistakes from people.

But at least you will have an audit, at least you will understand what happened and you hopefully will find out about the issue faster. And that's what I tend to believe. But yeah, people kept holding to this philosophy of running things locally. Now, again, it's not false at this point as well, but I see that people are very careful about when to do it and why.

‍Andrew

Let's get to this next topic about drifts.

This is an interesting one because it kind of turns into a double edged sword. Can you tell us more about your experience?

‍Sergey

We started off with about 60% drift to the environments. There is a specific engineer in my team which is very dedicated to eliminating drifted environments, he's been doing it for quite a while now, like for the past year or so. He's especially into those specific cases which are really hard to tend to, some changes that were made years ago. First thing first, handling drifted environments is a pain in ass. And as long as you keep this drift, you will have a harder time in the future. And this is exactly why I tend to encourage everybody to treat it as fast as possible.

But more than that, the things that we started also doing is thinking about how to modularized our environments and our deployments better. It depends on the use case and the scenario, but there are use cases where you say, I want to just try the Terraform as plainly as possible and you do so, but after some time you realize that this request is coming back to you again and again and again. So now we need to understand what the request is and how you can modularized it better. This is actually not a simple task. You need to understand exactly what your customers, or in our case, the engineers, need. And what is the recurring request that requires a specific solution.

We have some use cases. We use the env zero templates, which is basically like collateral for modules, and we use that to deploy some resources in the cloud which are reusable. A simple example – we have a team that uses Lambda functions a lot. So we basically have this module which is representing their needs for the lambda function and its environment, and we redeploy it as a module using the env zero templates and we find it to be very time saving eventually. But initially we would have been like, how are they able to do it? 

‍Andrew

Because it wasn't modularized yet. There's a question from the audience about where the drift is coming from. So earlier you mentioned there are 60% of the drift. Was it because people were doing local applies or what else was causing that drift?

‍Sergey

Yeah, that's a simple scenario from the perspective of a DevOps engineer. Our DevOps engineers have full admin access to our environments, managing the entire cloud environment end-to-end. So, let's say we have an issue with a team in production and need to make changes.

The simplest scenario would be changing the number of instances. If there's a production issue, you might change these things manually in the AWS console, and you'll likely need to document it. The drift will indicate that you had ten instances, changed it to twenty, and now you need to revert it.

This is the drift. While this is a specific small change, it can be related to any service and any change. Urgent changes often cause drifts, as they require immediate action.

‍Andrew

This is also the last expectation here we get asked a lot, especially since CICD needs so many tooling and what scope does env zero have. So what can tell us more about this expectation? Does env zero replace everything? 

‍Sergey

No, not really. I describe it in the part we talked about GitOps. So you can also see the pipeline itself. So basically env zero is integrated into all Git pipelines, you can see the pipeline itself showing you the whole like plan and apply, and all the steps that are happening in env zero itself. 

So technically you can avoid going into the env zero system and just using your pipeline as-is for Git, for example. Also you can use the API and all of the Terraform modules of env zero API to integrate it with different other tools that you use in your company, for example IVP, and this is something that we plan for the future.

‍Andrew

So that's a good transition to talk about future plans. So you've been using env zero for a while now. Tell us about your journey and your future journey? 

‍Sergey

A few things are important to me, and one is the governance aspect. I've talked about policy enforcement quite a lot today. Tools like OPA are something we are looking to incorporate. We're also expanding our developer portal for self-service via env zero for cloud resources. This answers a question from the audience about integrating env zero for developers. In our use case, we do not expose env zero directly to the developers; it operates in the background.

We are waiting for a few features that we hope will come soon. One is Infrastructure as Code coverage control and drift blame. So I can blame the engineers for being… No, I'm kidding. Drift blame is a term used to describe identifying who made changes in the console. For example, if someone changed the number of minimum instances from 10 to 20, I would want to know who did that.

So this is something that we all are eager to get as a feature in the future. And the last point here is talking about a maturity journey. We are not there yet. We’re using the GitOps approach and we have env zero managing and governing our environment. But we are still not at the self-service level. We still have work to do in componentization. You need to understand what you want to component inside and which features you want to introduce as a self-service. So we are working on those. 

‍Andrew

Thank you for sharing your experience. Let's open up to open Q&A. There's a question about your separation of Argos CD. How do you separate resources that need to be automatically deployed by env zero and others that need to be deployed by ArgoCD? 

‍Sergey

At this point we don't have a solution just yet. The way we are looking at it is that we will probably use the IDP and together with ArgoCD detection of new applications, right? So basically env zero will detect Terraform and ArgoCD will detect the application. Basically the same approach on both ends. So you will need to add the Terraform code and the Helm chart or manifest it.

And each of the systems will separately deploy the resources that are needed. So for us, the approach is to use IDP and a developer portal. A developer can trigger the process and there are two different stages and each stage whether the Helm chart or the cloud resources.

‍Andrew

I'll put in a chat a blog post where it shows env zero deploying. So you mentioned this, you deploy your Kubernetes cluster, but you need to bootstrap ArgosCD onto your company's cluster so you are using a Helm chart or Terraform deploying Helm chart, you can deploy ArgoCD’s operator into your Kubernetes cluster. Now, in order to configure ArgosCD to essentially listen to new applications, you can deploy the application YAML, which is the Argos CD’S CRD for managing applications.

You can use env zero to manage this application YAML as well. If you want to deploy Kubernetes manifests directly into your ArgoCD cluster, env zero will continue to listen for any Infrastructure as Code changes, such as updates to Terraform. All of this is managed in GitOps with ArgoCD. Because you've configured ArgoCD to listen, it will automatically respond to any image changes or manifest changes, managing its own processes accordingly.

So essentially GitOps on both sides, listening to their own hooks within Git. 

‍Sergey

Yeah there's just another one about this topic is that you have application sets you know in ArgoCD you also have an approach which is called app of apps, one application that holds many applications. So for us it would be at least at this point, we are at the point where we have the app of apps deployed via Terraform and it is managing the whole other applications that also might be useful there.

‍Andrew

There is a question here. Have you used env0’s Environment Workflow? You answer the question, I’ll describe Workflow. So Workflow for our audience in env zero is essentially an orchestrator of multiple templates, it allows you to deploy a multilayer infrastructure. So say you have a three tier infrastructure network, compute and services.

You can keep each of these different services as separate Terraform templates or Terraform resources and env zero will orchestrate, make sure it deploys the right in the right order. So that's the high level of what env zero workflow is. The question is, are you using it? And if so, what has your experience been?

‍Sergey

Yeah, so I would say no. We've evaluated some use cases as we have before, but eventually we end up not using it at this point. 

‍Andrew

Okay, let's look at another question here. I'm interested in order of magnitude, how many workspaces, how many of these are templated and auto discovered and how long do they actually live? So how many env zero environments or Terraform Workspaces, are you managing with env0? Roughly, yeah.

‍Sergey

So as I presented it before in the dashboard and then one of the slides, it’s about 900 of those. Most of those are environments that are pointing and discovered in some GitHub repository that we have that holds most of all the Terraform code and basically each folder eventually is an environment. Most of those are not templated. 

But we have some use cases which are the templates that as I describe I gave one example, the Lambda functions. We have a lot of static websites that we deployed that use templates. And I think the other one that we consider using is with the self-service that we want to introduce. Eventually those will also be templated environments.

‍Andrew

Here's a question. We are planning to force all the development to to be deployed by a central code catalog that's maintained by a single infra team. Is this something env zero you're seeing more of, or are teams still able to develop Infrastructure as Code themselves? I guess I will answer that. So as you heard from Sergey's perspective right now, they are developing Infrastructure as Code themselves and slowly introducing templates to other team members.

‍

At env0, working with dozens and dozens of customers, we see a broad spectrum. Some are completely siloed and command and control, I would say. So basically the central team is managing all of the resources and deploying all the resources, and then some are having a catalog approach. They create essentially a set of resources that their teams can onboard onto and then let the dev teams choose.

And then some are mixed in the sense that they have subject matter experts within each of the dev teams. So those people would like to write Infrastructure as Code, but then you also have the people on the other end of the spectrum who don't know any Infrastructure as Code and need a catalog in order to get access to resources.

So in env zero with templates, you can essentially create these quote unquote modules that will be able to give access to resources to these dev developers who aren't familiar with the infrastructure. At the same time, those modules you can publish in MS model registry for the developers who are familiar with infrastructure and want to compose their own resources, you can use what we described earlier with environment discovery and give them the ability to get access and deploy their own Infrastructure as Code through a PR process as well.

So we're definitely seeing the spectrum and it really kind of depends on the kind of people you have within your organization and the type of processes that you want to put in place. And env zero can really help cover both sides of that. And I hope that helps give you a sense of capabilities within env0. Maybe we'll take one more question from the audience.

For the questions we didn't get to cover, which are a bit more about env zero in general, maybe we'll segue into closing statements. So thank you again, Sergey, for taking this time to meet with us. Any last words about your experience with env0?

‍Sergey

I guess that a motto in life for me is taking things slow. And this is a process. I think we are in it for the last one and a half years and it will take more time. 

‍Andrew

Absolutely, let's keep on making steps towards climbing that mountain. So thank you for taking your time. So I want to end on this one note, as since you're now here and you're probably interested in seeing how you can get started at env0, we have this concept of a one-day proof of value, so essentially we can help unlock and help you get an understanding of how env zero can build value for you.

It's an easy four-step process. We create an organization, add credentials, connect your Git repo, and start deploying resources in env0. And what we hope to achieve and can achieve within this one day POC is simply be able to show you the setup process, show you the cost estimation and start tracking cost and setup drift detection.

We've done this with a few of our customers already and immediately added value and we’ll show you what we can do. We'll start with a one-hour pre-onboarding session to get you started. Then, on that day, we'll go through the entire checklist and get you fully onboarded. We'll set up Slack or Teams support for easy communication and give you a 30-day trial of env zero so you can truly start experiencing the ROI.

After the 30 day trial, we expect you to reduce up to 45% of your cloud cost, increase your deployment speeds and decrease your time to merge. And again, all we need to do is create an organization, set up some cloud credentials, connect to whatever version control system you're using and deploy some Terraform or other Infrastructure as Code that you're using. If you're not ready for a one day POC just yet and you want to learn more, you can book a demo through our website, envzero.com/demo-request or sign up for free trial on envzero.com and check out envzero.com for more docs and blogs and resources.

I want to thank everyone attending again today. If you have any more questions, please feel free to reach out and we'll be happy to address those on a future call or through chat. All right, So that concludes our webinar for today. Thank you again, Sergey, for joining us and hope to hear from everyone soon.

Cheers!

AWS GovCloud

AWS GovCloud is limited-access regions intended for US Government Agencies and organizations, and related contractors and vendors. Because of its setup and implementation, even information about the deployed instances (for example, IP address space) is considered sensitive, and subject to ITAR regulation. As a result, access is limited to US citizens and approved organizations only.

Because GovCloud is a completely separate AWS instance, it cannot be accessed using your accounts’ normal IAM users, API keys, and secrets. This means that many tools need to be specially configured or implemented to hit the correct API target in order to deploy and manage resources in GovCloud. If your organization is using GovCloud in order to ease the path to securing FedRAMP authorization for your application, be prepared for a lot of duplicative efforts and parallel environments since you cannot commingle non-FedRAMP resources with those inside a FedRAMP system boundary. 

Terraform certainly is in wide use across AWS, including to deploy GovCloud resources. Furthermore, the same reasons for needing an IaC or Terraform manager in standard regions apply to GovCloud use. Because they capture a known state and configuration, the Terraform state and plan files themselves are likely internal to your defined FedRAMP system boundary. However, it is possible to keep the platform that manages the Terraform files themselves outside of your defined system boundary.

There are limitations, though. Because many applications deployed into GovCloud are subject to a variety of security frameworks such as FedRAMP or NIST 800-53, deploying organizations need to be aware of what software must be included in their System Security Plan (SSP). Additionally, best practices of GovCloud deployment mean that GovCloud resources tend to be highly isolated in multiple VPCs with strictly defined security groups.

These deployments have a few impacts:

• GovCloud deployments can be a bit more complex than many standard region deployments in order to meet the stringent Government Security and Compliance requirements. 
• Local execution from a SaaS-connected agent needs to be well understood and implemented to have a minimal impact on the organization's SSP boundary.

Terraform and AWS GovCloud

If you’ve already tried to run Terraform code from AWS commercial, then you likely already know that there are many nuanced differences between AWS GovCloud and commercial cloud implementations. If you have NOT yet attempted to apply your Terraform plans in GovCloud, just know that they’re unlikely to work as written. You will almost certainly need to make updates to your IaC code in order for it to properly create GovCloud infrastructure.

During that porting effort, and once the IaC is complete, env zero with the self-hosted agent is a fantastic way to manage your IaC in GovCloud.

Implementation

The good news is that implementing the env zero Self-Hosted Agent in GovCloud is very similar to other Kubernetes environments. We provide a helm chart that works with AWS EKS in GovCloud. Note that in order to communicate with the env zero SaaS platform, the env zero self-hosted agent for kubernetes needs to have some amount of access to the outside world, so plan your EKS cluster appropriately. Once installed, the agent’s communication abilities can be restricted to a limited number of outbound domains required for the agent to operate correctly. Additionally, limiting the agent’s communication to only the required services, it’s possible to strictly limit what the agent can access.

The Self-Hosted Agent uses EFS to store metadata, terraform plan files, etc. that are required for execution in your environment. While non-sensitive variables are stored within the env zero SaaS system, any sensitive variables and secrets are stored in your selected secrets manager (i.e. AWS Secret Manager, Vault, etc.), which keeps sensitive information inside your SSP boundary and will greatly ease your compliance process.

In order to securely access and run your Terraform code, the Self-Hosted Agent also provides multiple different authentication methods in order to assign an appropriate role to the agent container.

Chances are your organization also already has a defined container image that is the foundation for a number of your container and cloud-native workloads. In GovCloud, as in other environments, it’s not recommended to run container images from unknown or untrusted 3rd parties. The env zero Self-Hosted Agent can use your existing container image.

Implementing the Self-Hosted Agent in this configuration will be compliant with a broad range of frameworks, and withstand your 3rd party assessments and audits as required by FedRAMP and CMMC.

Next steps

Despite a number of our team having lived in the world of compliance, it would be a good idea to consult your compliance readiness firm (if you have one) to more fully understand the impact installing the env zero Self-Hosted Agent in your environment, and to determine the impact from installing an agent to move plan execution internal to your system boundary.

If you’re ready to get started, reach out, and our US-based personnel will be happy to work on an env zero in GovCloud POC with your team.e

‍What is ArgoCD?

ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes applications that uses Git repositories as the source of truth for defining the desired application state. It automates application deployment and lifecycle management, making it easy to understand and audit.

Key Features and Benefits

Argo CD offers several key features and benefits, including:

• Single source of truth: Git repositories serve as the source of truth for application definitions, configurations, and environments.
• User interface: Argo CD provides a user-friendly interface for managing applications and visualizing deployment status.
• Faster updates directly in the cluster: Argo CD enables faster application updates by directly synchronizing the desired state with the live state in the cluster.
• Easy rollback: Argo CD allows for easy rollback of application deployments in case of issues.
• Cluster disaster recovery: Argo CD helps in recovering from cluster disasters by maintaining a version-controlled history of application deployments.
• Security: Argo CD provides role-based access control (RBAC) for managing the authorization of teams to specific services and clusters.
• Scalability: Argo CD can scale to manage multiple applications and clusters.
• Multi-tenancy: Argo CD supports multi-tenancy, allowing multiple teams to manage their applications independently.
• Automated configuration drift detection and visualization.
• Automated or manual syncing of applications to their desired state.

Argo CD Architecture

Argo CD's architecture follows a client-server architecture, where the server component runs within the Kubernetes cluster, and the client component is a command-line interface (CLI) or a web UI used to interact with the server.

Below are some high-level points regarding its architecture.

• Argo CD is implemented as a Kubernetes controller which continuously monitors running applications
• It compares the current, live state against the desired target state (as specified in the Git repo)
• A deployed application whose live state deviates from the target state is considered OutOfSync
• Argo CD reports & visualizes the differences while providing facilities to automatically or manually sync the live state back to the desired target state
• Any modifications made to the desired target state in the Git repo can be automatically applied and reflected in the specified target environments

ArgoCD Components

ArgoCD components include:

• Argo CD Application Controller: A Kubernetes controller that continuously monitors application definitions and configurations defined in a Git repository, comparing the specified state with the live state of the applications.
• Argo CD ApplicationSet Controller: A controller that manages multiple Argo CD Applications as a single ApplicationSet unit, supporting deployments to large numbers of clusters and improving multi-cluster and multi-tenant support within Argo CD.
• Argo CD Dex Server: An identity service that provides authentication and authorization for Argo CD, delegating authentication to external identity providers like GitHub, SAML, and others.
• Argo CD Notifications Controller: A controller that continuously monitors Argo CD applications and provides a flexible way to notify users about important changes in the application state
• Argo CD Redis: A throw-away cache used by Argo CD for storing application state. Redis can be lost and rebuilt without loss of service, as all data is persisted as Kubernetes objects in etcd.
• Argo CD Repository Server: A server that manages access to Git repositories and Helm chart repositories, providing a centralized location for storing and retrieving application configurations.
• Argo CD Server: A gRPC/REST server that exposes the API consumed by the Web UI, CLI, and CI/CD systems, responsible for application management, status reporting, invoking application operations, repository and cluster credential management, authentication, and RBAC enforcement.

What are Helm Charts?

Helm Charts are YAML configurations that define a package of pre-configured resources that can be deployed in a Kubernetes cluster. They allow users to define and deploy applications in a declarative manner, making it easier to maintain configurations.

Overview of Using ArgoCD with Helm Charts

When used together, ArgoCD and Helm Charts provide a powerful combination for managing and deploying cloud-native applications. ArgoCD automates the deployment process and the lifecycle management of applications, while Helm Charts provide a declarative way to define and package applications. In our demo, we will see how to manage applications using helm charts without Argo CD and then how deploying helm charts with Argo CD is a better option.

‍

Video Walk-through

‍Requirements: A GitHub account

TL;DR: You can find the main repo here.

PS, We will also make use of another repo for our School App but it's not essential for this blog post.

Argo CD Design Principles

ArgoCD follows the GitOps pattern, which emphasizes declarative infrastructure and version control. This approach allows for better collaboration among team members and ensures that application configurations are version-controlled. ArgoCD automates the deployment process of the desired states in the specified target environments, tracking updates to branches, tags, or pinned to a specific version of manifests at a Git commit.

Understanding Argo CD Workflow

GitOps

GitOps is a pattern that uses Git repositories as the source of truth for application definitions, configurations, and environments. This approach emphasizes declarative infrastructure and version control, allowing for better collaboration among team members and ensuring that application configurations are version-controlled.

Application Lifecycle Management

Argo CD continuously monitors application definitions and configurations defined in a Git repository, comparing the specified state with the live state of the applications. This enables developers to manage application deployments and updates through continuous delivery of simple Git commit events without the need for complicated Continuous Integration and/or Deployment Pipelines.

Configuration Synchronization

ArgoCD synchronizes the state with the declared state of configurations and ensures that new configurations are correctly deployed to a Kubernetes cluster. As all the records of all changes, including all details of the environment, are stored in Git, ArgoCD provides an auditable history of application deployments, making it easier to track changes and maintain security.

Getting started with Argo CD

To get started with ArgoCD, you'll need to install the ArgoCD CLI and have access to a Kubernetes cluster. You can run the following commands to install argo cd.

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

For more details, you can follow the official getting started guide to install Argo CD and set up your environment.

Argo CD Services

Let's take a look at all the services that Argo CD creates.

Argo CD Custom Resource Definitions (CRDs)

Below are the custom resource definitions that Argo CD creates:

kubectl get crd
NAME CREATED AT
applications.argoproj.io 2023-07-12T19:51:43Z
applicationsets.argoproj.io   2023-07-12T19:51:44Z
appprojects.argoproj.io 2023-07-12T19:51:44Z

Access ArgoCD

Let's now access the Argo CD UI.

Get the admin password

Run the command below to retrieve the admin password for Argo CD.

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

Expose the ArgoCD API Server

In a new terminal run this command:

kubectl port-forward svc/argocd-server -n argocd 8002:443

Login using the UI

You can log in to the UI by opening a browser window and going to https://127.0.0.1:8002

username: admin
password: THE_PASSWORD_YOU_GOT_ABOVE

While experimenting, you may encounter a warning that the page is insecure due to the use of a self-signed certificate. Although you can proceed with this setup, it is recommended that you follow the detailed instructions in the Argo documentation to configure TLS with an Ingress route before transitioning to a production environment.

Update the Admin Password

Now go ahead and update the initial admin password that was automatically generated for you.

Delete the Initial Admin Password

Run the command below:

kubectl -n argocd delete secret argocd-initial-admin-secret

Application Deployment with Helm Charts

Let's now deploy helm charts for our application with a helm chart per service. Run the following commands:

# Setup MongoDB DB in K8s
kubectl create ns schoolapp
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install schoolapp-mongodb --namespace schoolapp \
--set auth.enabled=true \
--set auth.rootUser=schoolapp \
--set auth.rootPassword=mongoRootPass \
  bitnami/mongodb

# Add project repo to helm
helm repo add schoolapp https://gitlab.com/api/v4/projects/34240616/packages/helm/stable

# Install the Frontend
helm install frontend -n schoolapp schoolapp/schoolapp-frontend
# Install the API
helm install api -n schoolapp schoolapp/schoolapp-api

Here is the explanation for the above:

• The code sets up a MongoDB database in a Kubernetes cluster and installs a frontend and API for a school application using Helm charts.
• A new namespace called schoolapp is created using the kubectl create ns command.
• The Bitnami Helm chart repository is added using helm repo add.
• The MongoDB Helm chart is installed using helm install, with the --namespace flag specifying the namespace where the MongoDB chart will be installed, and the --set flag used to enable authentication and set the root user and password.
• The schoolapp Helm chart repository is added using helm repo add.
• The frontend and API Helm charts are installed using helm install, with the frontend and API being the names of the releases that will be created for the frontend and API Helm charts, respectively.
• The -n flag specifies the namespace where the Helm charts will be installed, which is schoolapp in this case.

Now open two new terminals to port forward the frontend and the API for our school application deployment.

# Port forward the frontend
kubectl -n schoolapp port-forward service/frontend 8001:8080
# Port forward the API
kubectl -n schoolapp port-forward service/api 5000:5000

School App Frontend

You can test the school app by going to the URL: http://127.0.0.1:8001 in your browser. You should get a welcome screen. Interact with the app and go ahead and add a course and enroll a student.

School App Architecture

Below is a diagram showing the different components of the school app.

It uses the following technology stack:

• Vue.js for the Frontend
• Python's FastAPI framework for the API
• MongoDB for its Database

Application Deployment with ArgoCD and Helm Charts

Now let's get the school app deployed using charts and our favorite GitOps continuous delivery tool, Argo CD.

Delete the Current School App

kubectl delete ns schoolapp

Create the School Application in ArgoCD

Run the following command to create the application.

kubectl apply -f argocdSchoolApp.yaml

‍

As you see in the image above, the App is deployed but the status shows that it is OutOfSync.

Syncing the Application

Now back to the Argo CD UI, you'll need to click on the sync button to synchronize the application. This will reconcile the live state with the target state. Since this is the first time we're deploying, the live state doesn't contain anything.

Now our application is synced and healthy! You can see how Argo CD gives us a nice diagram of the different components of our application if you click on the application box above.

The Application Manifest

Here is the content of the Application manifest file argocdSchoolApp.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: schoolapp
  namespace: argocd
spec:
  destination:
    namespace: schoolapp
    server: 'https://kubernetes.default.svc'
  source:
    path: schoolapp-subchart
    repoURL: 'https://github.com/samgabrail/env0-argocd.git'
    targetRevision: HEAD
  project: default
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

This K8s manifest defines an ArgoCD application called schoolapp that should be deployed to the schoolapp namespace on the Kubernetes server. The application is sourced from a Git repository and uses the "HEAD" revision. The synchronization policy specifies that the namespace should be created if it does not already exist. More details are below.

• apiVersion: argoproj.io/v1alpha1: Specifies the API version of the resource.
• kind: Application: Specifies the type of Kubernetes resource, which is an ArgoCD application.
• metadata: Contains metadata about the application, including its name and namespace.

• name: schoolapp: Specifies the name of the application.
• namespace: argocd: Specifies the namespace where the application is deployed.

• spec: Specifies the desired state of the application.

• destination: Specifies the destination where the application should be deployed.

• namespace: schoolapp: Specifies the namespace where the application should be deployed.
• server: 'https://kubernetes.default.svc': Specifies the Kubernetes server where the application should be deployed.

• source: Specifies the source of the application.

• path: schoolapp-subchart: Specifies the path to the application within the Git repository.
• repoURL: 'https://github.com/samgabrail/env0-argocd.git': Specifies the URL of the Git repository where the application is stored.
• targetRevision: HEAD: Specifies the Git revision to use for the application.

• project: default: Specifies the project where the application belongs.
• syncPolicy: Specifies the synchronization policy for the application.

• syncOptions: Specifies the synchronization options for the application.

• - CreateNamespace=true: Specifies that the namespace should be created if it does not already exist.

Test the School App with ArgoCD

Once again open two terminal windows to port-forward the frontend and API.

# Port forward the frontend
kubectl port-forward service/frontend 8001:8080 -n schoolapp
# Port forward the api
kubectl port-forward service/api 5000:5000 -n schoolapp

Try creating and deleting a course as before. Everything should be working the same.

Making a Change in Git

Now let's see how Argo CD can detect drift between the live and target states. Let's make a simple change in our Git repo for the file called schoolapp-subchart/values.yaml. Change the mongodb.auth.enabled value from true to false.

mongodb:
  auth:
    enabled: false
    rootUser: 'schoolapp'
    rootPassword: 'mongoRootPass'

Then commit and push your changes.

In the Argo CD UI, Click the Refresh button and notice how the application is now OutOfSync. Also, notice that we have auto sync disabled. We'll enable it later.

You can see the components in our Argo CD tree that are out of sync.

If you click on the APP DIFF button, you will see the manifest files. Notice how the MongoDB secret will get deleted.

Notice also that the School App MongoDB Deployment will remove some environment variables and will allow authentication with an empty password.

Go ahead and click on the sync button and notice again the components that will change (the MongoDB secret and the School App MongoDB deployment).

Once you sync, Argo CD will go ahead and make the changes. However, it will not delete the secret. We can now enable auto sync along with

Enabling Auto Sync

Click on APP DETAILS and scroll down to the SYNC POLICY section. You can now enable automated sync along with pruning resources and self heal. When you do that, it will automatically sync and delete the MongoDB secret. Go ahead and have some fun in your git repo toggling the mongodb.auth.enabled from true to false and back again a few times to see auto sync in action.

History and Rollback

By clicking on the History and Rollback button, you can access previous deployments and view all the syncs that Argo has performed. This screen provides an option to restore an older version, which can be useful if a deployment introduces a bug. By rolling back to a previous version, you can avoid pushing a fix to your repository until the issue has been resolved.

ArgoCD Integrations

ArgoCD can be integrated with various tools and platforms to enhance its capabilities. Some of these integrations include:

• GitHub Actions: Automate CI/CD workflows using GitHub Actions and ArgoCD to deploy applications to Kubernetes.
• Helm Charts: Use Helm Charts to define, install, and update pre-packaged applications or consume prebuilt applications from trusted repositories.
• Datadog: Monitor your Argo CD clusters with Datadog, providing insights into key performance metrics and enabling alerts for changes in those metrics.
• New Relic: Gain observability into Argo CD performance with New Relic's instant observability quickstart, which includes alerts for detecting changes in key performance metrics.
• Tekton: Integrate Argo CD with Tekton to create a CI/CD system that includes feature branch testing and automation for creating new features with feature branches and environments.
• Red Hat OpenShift GitOps: Use Argo CD's ApplicationSets and pull request generator with Tekton and Red Hat OpenShift tools to bring GitOps workflows into your CI/CD processes.

Security in ArgoCD

ArgoCD offers several security features to ensure the safety and integrity of your Kubernetes applications. It has undergone stringent internal security evaluations and penetration tests for PCI compliance. Below are some of the highlights mentioned in the official documentation around security.

• Authentication: It uses JSON Web Tokens (JWTs) for API server authentication, with tokens obtained in one of three ways:

• For local admin users, JWTs are issued after exchanging username/password via a specific API endpoint.
• For Single Sign-On users, JWTs are obtained via an OAuth2 login flow.
• For project automation, tokens are generated using another API endpoint and are limited in their scope and privileges.

• Authorization is done by comparing group membership in a user's JWT groups claims with the roles/rules in the RBAC policy.
• All network communication uses TLS, and the Argo CD API server can enforce TLS 1.2 usage.
• Git and Helm repositories are managed by a standalone service, the repo-server, which does not store any service credentials.
• Careful: Unauthorized access to git repositories trusted by Argo CD can lead to serious security breaches.
• Sensitive Data:

• Argo CD does not return sensitive data from its API and redacts all sensitive data in payloads and logs.
• For managing external clusters, Argo CD stores their credentials as a Kubernetes Secret in the Argo CD namespace.

• Cluster RBAC:

• Argo CD's access can be revoked by deleting RBAC artifacts and removing the cluster entry from Argo CD.
• Argo CD's default ClusterRole can be fine-tuned to limit write privileges.

• Auditing: Argo CD logs application activity as Kubernetes Events, complementing the Git revision history.
• Webhook Payloads: Payloads from webhook events are considered untrusted and only used to refresh the related application.
• Security-related logs are tagged with a security field for easier analysis.
• API Logs: Argo CD does not log sensitive API requests or IP addresses of clients.
• The ApplicationSets feature of Argo CD has its own security considerations.
• Denial of Service on Directory Type Apps: Directory-type applications can consume significant repo-server memory; a config option can limit this memory usage.

Best Practices

To ensure optimal results when using ArgoCD, consider the following best practices:

1. Use a Git repository as the single source of truth for application definitions, configurations, and environments.
2. Keep application configurations declarative and version-controlled.
3. Follow the GitOps pattern for managing and deploying applications.
4. Use Helm Charts to define applications in a declarative manner and store them in a Helm repository.
5. Regularly update and maintain Helm Charts to ensure they are up-to-date and secure.

Conclusion

In this blog post, we explored ArgoCD, a powerful GitOps continuous delivery tool for Kubernetes applications. We discussed its key features, benefits, architecture, and components, as well as the role of Helm Charts in defining and managing Kubernetes applications. Additionally, we delved into application lifecycle management, configuration synchronization, and the application deployment process

By understanding and leveraging the capabilities of ArgoCD, you can streamline your application management and deployment processes, ensuring a more efficient and secure Kubernetes environment.

Try env0's CI/CD platform. env zero is a powerful platform that offers numerous benefits for managing Infrastructure as Code (IaC). By using env0, organizations can streamline their infrastructure management processes, promote collaboration, and maintain control over their infrastructure code while enjoying a more cost-effective solution. env zero also supports using Helm, check it out.