.svg)
Nine seconds. That's roughly how long it took an AI coding agent to delete a production database and its attached backups in one widely documented 2026 incident, and the agent hadn't been tricked, jailbroken, or compromised by an outside attacker. It had a real API credential, a set of approved operations, and a goal it decided, on its own reasoning, that deleting the obstacle in front of it was the way to accomplish.
That incident is one of the latest on a growing list. In July 2025, an AI agent from Replit deleted a live production database during an active code freeze, then told the founder testing it that recovery was impossible. The claim was false, and luckily, a rollback attempt was successful. Around the same time, Google's Gemini CLI cascaded a routine file-reorganization request into total data loss when a failed step silently continued as if it had succeeded. None of this required compromising the model. The agent's intent was usually mundane, clear a cache, reorganize a folder, remove a blocker. The damage happened one layer below, in the gap between what the agent was allowed to do and what anyone had actually reviewed.
The numbers back up what the incidents suggest. Enterprises reported an average of 54 AI agent incidents in 2025 alone, with close to one in five rated high severity, and a large majority of technology leaders admitting adoption is outpacing governance. Separately, fewer than a quarter of organizations with a formal generative AI strategy believe their agentic governance is actually adequate, a figure that falls to just 4% among organizations with no strategy at all. Agents are already inside the infrastructure workflow. Governance is not keeping up.
This Is a Different Governance Problem
Traditional infrastructure risk models assume one of two actors: a person who makes a mistake, or an attacker who shouldn't have access in the first place. AI agents introduce a third case that doesn't fit either schema. An agent with legitimate credentials, operating inside its approved permissions, can reason its way to destructive actions that no human ever reviewed, and do it with internally consistent logic all the way through. It isn't confused. It isn't compromised. It just decided.
That makes soft guardrails — the kind written into an agent's own instructions or system prompt — unreliable as a control. An instruction not to touch production is just another token in the same context window the agent is reasoning over. When that instruction conflicts with the agent's read of the task, the instruction competes on equal footing with everything else it's weighing, rather than sitting above the reasoning as an enforced rule. More than one of the incidents above happened with exactly this kind of instruction already given, and overridden.
Terraform itself illustrates the gap well. The -auto-approve flag exists specifically so automated pipelines don't sit waiting for someone to type "yes," and it applies to terraform destroy exactly the same way it applies to terraform apply, deletions included. An agent scripting infrastructure changes has every reason to reach for it, the same reason a CI pipeline does. If the agent runs a saved plan file instead, Terraform treats passing that file as the approval already given, whether or not anyone reviewed the plan first. Terraform's real guardrail against a bad apply is a human typing a confirmation. An agent doesn't need one, and increasingly doesn't have one to type.
None of this means agents shouldn't touch infrastructure. It means the enforcement has to live outside the agent, in the platform it operates through, where a policy or an approval gate holds regardless of what the requester decided.
The Four Pillars of Agent Governance
Much has been written about the guardrails that keep every deploy under control: approvals, policy, and cost. AI agents don't need a new framework. They need the same one applied without the assumption that the requester is a person who gets tired, feels a budget, or stops when they read a warning.
- Identity, not tokens – Every agent action should trace to a real, scoped identity, never a shared API key passed between tools and teammates. If everything an agent does shows up in the logs as "the deploy bot," there's no way to tell which agent, which run, or which engineer's intent produced a given change.
- Approval gates that hold regardless of who's asking – A deployment that requires human sign-off needs to pause for a human whether a person or an agent triggered it. The gate can't be something the requester can reason its way around, because the requester might be the thing you're trying to gate.
- Policy-as-code as a deterministic backstop – A policy blocking a public S3 bucket, an oversized instance type, or an unrestricted IAM role has to stop the change regardless of the reasoning that produced it. That's the property you need once the requester behind a change might be a model instead of a person: something that enforces the rule rather than negotiates it.
- Cost guardrails sized for how agents actually spend – Agents provision fast and run around the clock. Budgets, pre-deploy cost estimates, and TTLs on anything an agent creates matter more, not less, when the thing spinning up resources never gets tired of doing it or feels the bill.
What This Looks Like in env zero
This is exactly what the env zero Agentic Experience was built to solve. Point Claude Code, Cursor, Codex, or Copilot at your env zero infrastructure, and the agent authenticates as a real identity, per user through the browser or per service through an API key, never a shared token. Every read and every action is attributable to someone, not fired anonymously from a bot account nobody can trace back.
Because your existing role-based access control applies to that identity, you decide what an agent can do by choosing the identity you give it. Scope an agent to a read-only role and it can diagnose a failed deployment, summarize drift, or list stale environments and their cost, but deploy, destroy, and approve simply aren't available to it. Give it a broader identity and the approval gates still hold: a deployment that requires sign-off pauses for a human no matter what asked for it, and destroying infrastructure still takes an explicit confirmation.
In practice, that looks like:
which pulls an environment's state, recent deployments, and drift or failure summary in a single call, so an agent can diagnose why something broke without guessing. Asking for a change looks like:
which starts an approval-gated deployment. The plan runs, any resource that would be replaced gets flagged, and nothing applies until a person signs off, regardless of how confident the agent's reasoning was.
Setup itself is one command:
which writes a version-matched usage guide into the agent's config, so it works from the commands and flags that version actually supports instead of inventing ones it half-remembers. Because that file lives in your repository, it's the natural place to write down your team's actual agent operating policy: a rule that no agent touches prod without approval, the golden-path templates a new environment should start from, a list of projects that are off limits entirely. Commit it, and every agent every engineer runs inherits the same rules, reviewed in a pull request like any other change.
None of this replaces governance you may already have in place, it inherits it. The same OPA policy-as-code evaluation that blocks a risky change from a human blocks it from an agent, including the ready-to-use policies that cover things like public buckets and over-permissioned IAM roles out of the box. The same custom RBAC roles that scope a contractor or a new hire scope an agent's identity too. Pre-deploy cost estimation and budget thresholds apply the same way, and every approval, deploy, or cancel an agent triggers lands in the same audit log as everything else. For anything that changes infrastructure outside IaC entirely, agent-driven or not, drift detection and Cloud Compass still catch it.
The original env zero MCP server needed a self-hosted container and a single shared credential, and it only ever saw an environment's latest deployment. The new Agent CLI exists because we found that for reliable provenance, agents need per-identity logins and a real read surface, not a workaround bolted onto an API that was never built for a machine to reason over.
Building Your Own Agent Operating Policy
Whatever platform you run, the controls that actually stop an agent from becoming an incident are the same:
- Default to read-only. An agent identity should be able to diagnose everything and change nothing until someone deliberately elevates it.
- Never share a token across agents, environments, or teammates. Shared credentials are exactly why nobody can tell which agent, run, or person caused a given change.
- Keep approval logic in the platform, not the prompt. An instruction inside an agent's context is a suggestion the same reasoning process can talk itself past. An approval gate outside that context isn't.
- Enforce policy as code. A hard-coded rule that blocks a risky configuration doesn't care how well-reasoned the request behind it sounded.
- Put cost checks before apply, not after the invoice. An agent that provisions all day needs a budget and a TTL more than a person who provisions once a sprint.
- Write the policy down where the agent reads it. A rule in a chat channel doesn't govern anything. A rule committed to the same repo an agent's skill file lives in does.
These are the same principles behind our own cloud governance framework and cost governance model, and they stay the same regardless of where a request comes from.
The Governance Gap Is the Story Right Now
We've been tracking this pattern beyond our own platform too, including in our ongoing coverage for The New Stack on how a single compromised dependency turns your own pipeline against you and on what AI agents actually do with the instructions you give them. The through-line across all of it is the same: controls that used to be optional for humans, identity, approval, policy, and cost visibility, are no longer optional for anything with write access to your infrastructure.
Agents are a genuinely useful way to expedite infrastructure work. They're also, without the right controls, a new way to lose a production database in nine seconds. The difference between the two is governance that holds regardless of who, or what, is asking.
Want to see it in action? Schedule a demo of the env zero Agentic Experience and the guardrails behind it.
FAQ's
What is AI agent governance in an infrastructure context?
AI agent governance is the set of controls, identity scoping, approval gates, policy enforcement, and cost checks, that apply to AI coding agents with access to deploy, modify, or destroy cloud infrastructure. It extends the same governance model already used for human engineers to agents acting through APIs and CLIs, so enforcement doesn't depend on trusting the agent's own judgment.
Can AI coding agents actually cause serious infrastructure damage or cost overruns?
Yes. Documented incidents in 2025 and 2026 include AI coding agents deleting production databases and their backups, cascading a routine file operation into total data loss, and bypassing explicit instructions not to make changes. These incidents involved legitimate credentials and approved operations rather than exploits, which is what makes them a governance problem rather than a security breach in the traditional sense.
How is governing an AI agent different from governing a human engineer?
A human engineer who reads a warning generally stops, feels the weight of a production outage, and notices a rising cloud bill. An agent doesn't reliably do any of that on its own. Instructions inside an agent's prompt are inputs its reasoning can weigh against everything else it's considering, not enforced rules, so the controls that matter for agents, identity, approval gates, and policy-as-code, need to sit outside the agent, in the platform it operates through, rather than inside its instructions.
Does env zero support AI coding agents like Claude Code, Cursor, Codex, or Copilot?
Yes, through the env zero Agentic Experience, powered by the env zero Agent CLI. Point any of these coding agents at your env zero infrastructure and it authenticates as a real, scoped identity, reads real environment state and deployment history, and can take action through the same roles and approval gates a person would.
What's the single most important control for AI agents with infrastructure access?
Defaulting the agent's identity to read-only and making write access an explicit, audited grant. An agent scoped this way can diagnose, summarize, and explain anything in your environment, but deploying, destroying, or approving changes simply isn't available to it until a person deliberately extends that access.
Related Content
Nine seconds. That's roughly how long it took an AI coding agent to delete a production database and its attached backups in one widely documented 2026 incident, and the agent hadn't been tricked, jailbroken, or compromised by an outside attacker. It had a real API credential, a set of approved operations, and a goal it decided, on its own reasoning, that deleting the obstacle in front of it was the way to accomplish.
That incident is one of the latest on a growing list. In July 2025, an AI agent from Replit deleted a live production database during an active code freeze, then told the founder testing it that recovery was impossible. The claim was false, and luckily, a rollback attempt was successful. Around the same time, Google's Gemini CLI cascaded a routine file-reorganization request into total data loss when a failed step silently continued as if it had succeeded. None of this required compromising the model. The agent's intent was usually mundane, clear a cache, reorganize a folder, remove a blocker. The damage happened one layer below, in the gap between what the agent was allowed to do and what anyone had actually reviewed.
The numbers back up what the incidents suggest. Enterprises reported an average of 54 AI agent incidents in 2025 alone, with close to one in five rated high severity, and a large majority of technology leaders admitting adoption is outpacing governance. Separately, fewer than a quarter of organizations with a formal generative AI strategy believe their agentic governance is actually adequate, a figure that falls to just 4% among organizations with no strategy at all. Agents are already inside the infrastructure workflow. Governance is not keeping up.
This Is a Different Governance Problem
Traditional infrastructure risk models assume one of two actors: a person who makes a mistake, or an attacker who shouldn't have access in the first place. AI agents introduce a third case that doesn't fit either schema. An agent with legitimate credentials, operating inside its approved permissions, can reason its way to destructive actions that no human ever reviewed, and do it with internally consistent logic all the way through. It isn't confused. It isn't compromised. It just decided.
That makes soft guardrails — the kind written into an agent's own instructions or system prompt — unreliable as a control. An instruction not to touch production is just another token in the same context window the agent is reasoning over. When that instruction conflicts with the agent's read of the task, the instruction competes on equal footing with everything else it's weighing, rather than sitting above the reasoning as an enforced rule. More than one of the incidents above happened with exactly this kind of instruction already given, and overridden.
Terraform itself illustrates the gap well. The -auto-approve flag exists specifically so automated pipelines don't sit waiting for someone to type "yes," and it applies to terraform destroy exactly the same way it applies to terraform apply, deletions included. An agent scripting infrastructure changes has every reason to reach for it, the same reason a CI pipeline does. If the agent runs a saved plan file instead, Terraform treats passing that file as the approval already given, whether or not anyone reviewed the plan first. Terraform's real guardrail against a bad apply is a human typing a confirmation. An agent doesn't need one, and increasingly doesn't have one to type.
None of this means agents shouldn't touch infrastructure. It means the enforcement has to live outside the agent, in the platform it operates through, where a policy or an approval gate holds regardless of what the requester decided.
The Four Pillars of Agent Governance
Much has been written about the guardrails that keep every deploy under control: approvals, policy, and cost. AI agents don't need a new framework. They need the same one applied without the assumption that the requester is a person who gets tired, feels a budget, or stops when they read a warning.
- Identity, not tokens – Every agent action should trace to a real, scoped identity, never a shared API key passed between tools and teammates. If everything an agent does shows up in the logs as "the deploy bot," there's no way to tell which agent, which run, or which engineer's intent produced a given change.
- Approval gates that hold regardless of who's asking – A deployment that requires human sign-off needs to pause for a human whether a person or an agent triggered it. The gate can't be something the requester can reason its way around, because the requester might be the thing you're trying to gate.
- Policy-as-code as a deterministic backstop – A policy blocking a public S3 bucket, an oversized instance type, or an unrestricted IAM role has to stop the change regardless of the reasoning that produced it. That's the property you need once the requester behind a change might be a model instead of a person: something that enforces the rule rather than negotiates it.
- Cost guardrails sized for how agents actually spend – Agents provision fast and run around the clock. Budgets, pre-deploy cost estimates, and TTLs on anything an agent creates matter more, not less, when the thing spinning up resources never gets tired of doing it or feels the bill.
What This Looks Like in env zero
This is exactly what the env zero Agentic Experience was built to solve. Point Claude Code, Cursor, Codex, or Copilot at your env zero infrastructure, and the agent authenticates as a real identity, per user through the browser or per service through an API key, never a shared token. Every read and every action is attributable to someone, not fired anonymously from a bot account nobody can trace back.
Because your existing role-based access control applies to that identity, you decide what an agent can do by choosing the identity you give it. Scope an agent to a read-only role and it can diagnose a failed deployment, summarize drift, or list stale environments and their cost, but deploy, destroy, and approve simply aren't available to it. Give it a broader identity and the approval gates still hold: a deployment that requires sign-off pauses for a human no matter what asked for it, and destroying infrastructure still takes an explicit confirmation.
In practice, that looks like:
which pulls an environment's state, recent deployments, and drift or failure summary in a single call, so an agent can diagnose why something broke without guessing. Asking for a change looks like:
which starts an approval-gated deployment. The plan runs, any resource that would be replaced gets flagged, and nothing applies until a person signs off, regardless of how confident the agent's reasoning was.
Setup itself is one command:
which writes a version-matched usage guide into the agent's config, so it works from the commands and flags that version actually supports instead of inventing ones it half-remembers. Because that file lives in your repository, it's the natural place to write down your team's actual agent operating policy: a rule that no agent touches prod without approval, the golden-path templates a new environment should start from, a list of projects that are off limits entirely. Commit it, and every agent every engineer runs inherits the same rules, reviewed in a pull request like any other change.
None of this replaces governance you may already have in place, it inherits it. The same OPA policy-as-code evaluation that blocks a risky change from a human blocks it from an agent, including the ready-to-use policies that cover things like public buckets and over-permissioned IAM roles out of the box. The same custom RBAC roles that scope a contractor or a new hire scope an agent's identity too. Pre-deploy cost estimation and budget thresholds apply the same way, and every approval, deploy, or cancel an agent triggers lands in the same audit log as everything else. For anything that changes infrastructure outside IaC entirely, agent-driven or not, drift detection and Cloud Compass still catch it.
The original env zero MCP server needed a self-hosted container and a single shared credential, and it only ever saw an environment's latest deployment. The new Agent CLI exists because we found that for reliable provenance, agents need per-identity logins and a real read surface, not a workaround bolted onto an API that was never built for a machine to reason over.
Building Your Own Agent Operating Policy
Whatever platform you run, the controls that actually stop an agent from becoming an incident are the same:
- Default to read-only. An agent identity should be able to diagnose everything and change nothing until someone deliberately elevates it.
- Never share a token across agents, environments, or teammates. Shared credentials are exactly why nobody can tell which agent, run, or person caused a given change.
- Keep approval logic in the platform, not the prompt. An instruction inside an agent's context is a suggestion the same reasoning process can talk itself past. An approval gate outside that context isn't.
- Enforce policy as code. A hard-coded rule that blocks a risky configuration doesn't care how well-reasoned the request behind it sounded.
- Put cost checks before apply, not after the invoice. An agent that provisions all day needs a budget and a TTL more than a person who provisions once a sprint.
- Write the policy down where the agent reads it. A rule in a chat channel doesn't govern anything. A rule committed to the same repo an agent's skill file lives in does.
These are the same principles behind our own cloud governance framework and cost governance model, and they stay the same regardless of where a request comes from.
The Governance Gap Is the Story Right Now
We've been tracking this pattern beyond our own platform too, including in our ongoing coverage for The New Stack on how a single compromised dependency turns your own pipeline against you and on what AI agents actually do with the instructions you give them. The through-line across all of it is the same: controls that used to be optional for humans, identity, approval, policy, and cost visibility, are no longer optional for anything with write access to your infrastructure.
Agents are a genuinely useful way to expedite infrastructure work. They're also, without the right controls, a new way to lose a production database in nine seconds. The difference between the two is governance that holds regardless of who, or what, is asking.
Want to see it in action? Schedule a demo of the env zero Agentic Experience and the guardrails behind it.
FAQ's
What is AI agent governance in an infrastructure context?
AI agent governance is the set of controls, identity scoping, approval gates, policy enforcement, and cost checks, that apply to AI coding agents with access to deploy, modify, or destroy cloud infrastructure. It extends the same governance model already used for human engineers to agents acting through APIs and CLIs, so enforcement doesn't depend on trusting the agent's own judgment.
Can AI coding agents actually cause serious infrastructure damage or cost overruns?
Yes. Documented incidents in 2025 and 2026 include AI coding agents deleting production databases and their backups, cascading a routine file operation into total data loss, and bypassing explicit instructions not to make changes. These incidents involved legitimate credentials and approved operations rather than exploits, which is what makes them a governance problem rather than a security breach in the traditional sense.
How is governing an AI agent different from governing a human engineer?
A human engineer who reads a warning generally stops, feels the weight of a production outage, and notices a rising cloud bill. An agent doesn't reliably do any of that on its own. Instructions inside an agent's prompt are inputs its reasoning can weigh against everything else it's considering, not enforced rules, so the controls that matter for agents, identity, approval gates, and policy-as-code, need to sit outside the agent, in the platform it operates through, rather than inside its instructions.
Does env zero support AI coding agents like Claude Code, Cursor, Codex, or Copilot?
Yes, through the env zero Agentic Experience, powered by the env zero Agent CLI. Point any of these coding agents at your env zero infrastructure and it authenticates as a real, scoped identity, reads real environment state and deployment history, and can take action through the same roles and approval gates a person would.
What's the single most important control for AI agents with infrastructure access?
Defaulting the agent's identity to read-only and making write access an explicit, audited grant. An agent scoped this way can diagnose, summarize, and explain anything in your environment, but deploying, destroying, or approving changes simply isn't available to it until a person deliberately extends that access.
AI Agent Governance: How to Stop Autonomous Agents From Wrecking Your Infrastructure and Cloud Bill
Imagine managing your cloud infrastructure using the programming languages you already love—Python, Go, JavaScript, you name it. No more wrestling with YAML, JSON, or HCL (HashiCorp Configuration Language) files!
Pulumi gives you that power, offering a robust CLI and service backend to manage both state and secrets. It's like the Swiss Army knife for cloud infrastructure, supporting all the major providers like AWS, Azure, and Google Cloud.
Today we're diving into the world of Pulumi and its integration with env0. We'll explore what Pulumi is, its features, how to set it up, and even throw in a real-world example (provisioning an EKS cluster). Also, we’ll weigh the pros against the cons and look at how it stacks up against other options. So buckle up; this is going to be a fun ride!
Video Walk-through
Requirements:
- A GitHub account
- An AWS account
- An env zero account
- A Pulumi account
TL;DR: You can find the main repo here.
What is Pulumi?
Pulumi is an open source Infrastructure-as-Code (IaC) framework that provisions resources utilizing common programming languages. Pulumi also supports the major cloud providers: AWS, Azure, and Google Cloud. Its leaning on common languages eliminates the time it would otherwise take to get used to a new domain-specific language like HCL.
If you're wondering how it stacks up against Terraform, check out my previous blog comparing Pulumi vs. Terraform. But the main benefits come in three main cores: the Pulumi SDK(s), the service backend, and finally the automation API.
Pulumi SDKs
First up, the SDKs. Pulumi's SDKs are what make it super versatile. These SDKs allow you to use languages like Python, JavaScript, TypeScript, Go, or .NET for defining and deploying your infrastructure. This is super cool because it means you can use the same language you're already comfortable with for your application development.
That all means that you end up with the following advantages: strong familiarity with the core languages, a long list of library resources (according to language), and reusable custom abstractions.
Pulumi Service Backend
Pulumi's SaaS offering comes replete with CI/CD integrations, Policy-as-Code, role-based access, and state management.
State Management – Safely stores and manages the state of your infrastructure. This means less headache worrying about where your infrastructure's "truth" lives. There is also an option for self-managed state through your own cloud account on AWS, Azure, or GCP.
Collaboration Features – You can collaborate with your team on infrastructure updates, with features like RBAC, stacks history, and more.
Policy-as-Code – Enforce security, compliance, and best practices across your infrastructure using Pulumi’s Policy as Code offering called CrossGuard.
CI/CD Integration – Pulumi CI/CD integrations work with popular systems like GitHub Actions, GitLab CI, Jenkins, TravicCI, AWS Code Services, Azure DevOps, and more.
Automation API
This Automation API can embed Pulumi directly into your application code, offering a hassle-free way to manage infrastructure.
In essence, this concept encapsulates the core functionalities offered by the Pulumi Command Line Interface (CLI), such as executing commands like [.code]pulumi up[.code], [.code]pulumi preview[.code], [.code]pulumi destroy[.code], and [.code]pulumi stack init[.code].
However, it extends beyond this by offering enhanced flexibility and control. This approach is designed to be strongly typed and secure, facilitating the use of Pulumi within embedded environments, for instance, within web servers.
Importantly, this method eliminates the need for running the CLI through a shell process, streamlining operations, and integrating infrastructure management more seamlessly into application environments.
Pulumi Features
Alright, let’s dig into some of the Pulumi concepts and features that it offers:
1. Component Resources
Pulumi lets you define reusable building blocks known as "component resources." These are like your typical cloud resources but bundled with additional logic. If you are familiar with Terraform, these would be your modules.
2. Stack References
Manage dependencies between multiple Pulumi stacks effortlessly. This feature is a real game-changer for managing infrastructure at scale.
3. Templates and Packages
Think of these as the ultimate cheat codes for your IaC. Instead of starting from scratch, you can kick things off with a pre-baked setup. Here’s why they're great:
- Speedy Setup: No more blank-slate syndrome. You’ve got a starting point that’s not just a blank file – it’s a springboard that gets you coding your infra in record time.
- Best Practices: These templates aren't just thrown together – they're crafted with best practices in mind. So you're not just starting faster, you're starting smarter.
- Learning Resources: New to Pulumi or a particular cloud service? Templates can be great learning tools, showing you the ropes of how things are structured and pieced together.
How to Install Pulumi
Alright, time to get our hands dirty. Installing Pulumi is a breeze. You can reference this from Pulumi's documentation.
Since I'm running this in my Windows for Subsystem Linux environment, I can run the install script as shown:
Pulumi Stack Example
Let's get into the meat and potatoes: stacks. A Pulumi stack is essentially an isolated, independently configurable instance of a Pulumi program. Let's first work with the Pulumi CLI then later we'll see how to use env0.
Create a New Pulumi Project
First, create a Pulumi project by creating a new directory and running the [.code]pulumi new[.code] command with the [.code]kubernetes-aws-python[.code] Pulumi template.
Continue by providing a project name, description, and stack name along with the AWS region and some other parameters.
Pulumi installs the necessary dependencies and your new project is ready.
Run Pulumi
Next, make sure you export your AWS cloud credentials as environment variables and run [.code]pulumi up[.code].
Read what Pulumi is about to do, then answer [.code]yes[.code] when asked if you want to perform this update.
Now, Pulumi will start to provision resources and you will see the resources get created in the terminal as shown below.
Observe the Output Results
If all goes well, you should have your new EKS cluster up and running. You can also check the Pulumi UI for your new stack where you can view all the resources created along with the output.
You can view the output in the UI or the CLI for the vpcId and the kubeconfig.
Access the EKS Cluster
To get the kubeconfig for the EKS cluster, run the following command:
Now run [.code]kubectl[.code] commands to interact with the EKS cluster:
Congratulations! You've successfully provisioned an EKS cluster in AWS.
Examine the Infrastructure Code
Take a look at the actual code that provisions our EKS cluster. Notice how it's written in simple Python. I could have built the cluster from scratch by calling on each resource, but why reinvent the wheel? There is an excellent Pulumi package called Amazon EKS in the Pulumi Registry. I decided to go with this.
As you see, in under 40 lines of code, we have our EKS cluster defined:.
Pulumi makes it very easy to choose between many languages right in the documentation.
If you need to tweak the cluster configuration, it's easy to do so with the very well-documented eks.Cluster package.
Pulumi Configuration Files
When we ran the [.code]pulumi new kubernetes-aws-python[.code] command, Pulumi 1) created a new folder for us, 2) downloaded dependencies in a virtual environment for Python, and 3) also created two config files.
Let's take a look at them now.
1. pulumi.yaml
This file acts as the manifest for your Pulumi project. It's a key part of the project configuration and provides metadata about the project itself.
Here's what each part of the content you've provided does:
- name – This is the name of your Pulumi project. When you run [.code]pulumi new[.code], it sets this name, and it's used as a default prefix for the resources Pulumi creates.
- runtime – This specifies the runtime environment that your Pulumi program is expected to run in. In your case, it's set to python, meaning the Pulumi CLI expects your Infrastructure-as-Code to be written in Python.
- options – These are additional settings related to the runtime environment.
- virtualenv – This option tells Pulumi to use a Python virtual environment located in the venv directory within your project directory. This is important for Python-based projects to ensure dependencies are isolated from other Python projects on the same system.
- description – This provides a human-readable description of what the Pulumi project does. It's a string that helps you and others understand the project's purpose at a glance.
So, when you initialize a new Pulumi stack or when Pulumi interacts with your project, it uses this file to understand the project structure, runtime requirements, and other metadata that influence how it deploys and manages your infrastructure resources.
2. pulumi.dev.yaml
When you run the pulumi new command and answer the setup wizard's questions, Pulumi automatically saves these answers as configurations in the pulumi.dev.yaml file. This file acts as a record of the initial setup parameters you specified for your project.
Now, if you enter commands or make changes at a different time (i.e., not during the initial Pulumi new setup) these changes won't automatically update the pulumi.dev.yaml file. Instead, you have two main alternatives for updating configurations after the initial setup:
1. Manual Editing – You can directly edit the pulumi.dev.yaml file to change or add configurations. This is like tweaking the settings of your project by hand.
2. Using Pulumi CLI Commands – You can use specific Pulumi CLI commands to update your configuration. For example, if you want to change the AWS region, you could use a command like [.code]pulumi config set aws:region us-west-2[.code]. This command updates the configuration in your pulumi.dev.yaml file without you having to manually edit the file.
Here is the content of the file:
To clean up simply run [.code]pulumi destroy[.code].
Pros and Cons of Using Pulumi
Pros
- Language Choice – Use your favorite programming language.
- Rich Ecosystem – Supports a ton of cloud providers.
- Dynamic Providers – Extend its capabilities as you see fit.
Cons
- Language Overload – Sometimes, choosing a language can be a burden.
- Learning Curve – If you're coming from dedicated DSL tools like Terraform's HCL, there might be an initial hump.
Pulumi Alternatives
The most obvious alternative to Pulumi is Terraform. But hey, keep an eye out for OpenTofu, an upcoming open-source alternative following a BSL license change. Crossplane is another alternative for those who enjoy building infrastructure using Kubernetes CRDs. Check out more details below.
1. Terraform
Overview: Terraform is a big player in the IaC field. It uses its own domain-specific language, HCL (HashiCorp Configuration Language), which is designed to describe infrastructure in a declarative way.
Why It's Popular: Terraform's been around for a while and has a huge community and support base. Plus, it works across many cloud providers, making it super versatile.
Key Differences from Pulumi: Unlike Pulumi, Terraform isn’t based on conventional programming languages. So, if you're not into learning HCL, it might be a bit of a curve.
2. Crossplane
Crossplane is perfect for those who are all-in with Kubernetes. It allows you to manage your infrastructure using Kubernetes CRDs (Custom Resource Definitions).
If you’re comfortable with Kubernetes and want to manage cloud resources as Kubernetes objects, Crossplane is your go-to. Being Kubernetes-focused, it fits well in ecosystems already heavy with Kubernetes usage and has a growing community.
Thoughts
Each of these alternatives has its own flavor. Terraform is the established giant with a dedicated language, OpenTofu promises to always be open-source along with new approaches to IaC, and Crossplane merges the worlds of Kubernetes and IaC.
Depending on your needs, comfort with certain technologies, and the specifics of your infrastructure, one of these might be a better fit for you than Pulumi.
Tutorial: Using Pulumi with env0
Now let's see how to use env zero to create the same Pulumi stack. We will create the same EKS cluster but this time by using env zero to trigger Pulumi.
Let's start by creating a new project in env0.
Next, you'll need to create a Pulumi template as shown:
Then connect to your VCS. Make sure to select the Pulumi folder, in our case Pulumi-EKS.
Under variables, add your PULUMI_ACCESS_TOKEN environment variable.
Then finally, make sure this template is deployable in our 'eks-demo' project.
AWS Cloud Provider Credentials
Make sure you have your AWS credentials set up in the Project settings
Create an Environment
Now we're ready to create a new environment. Head over to 'Project Environments' then create a new e.
When you see the eks-template, click the 'Run Now' button. There are some options to use such as enabling drift detection and the ability to automatically destroy the environment. When you're ready click the Run button.
Notice in the deployment logs how we have a 'Before: Pulumi Preview' step. This is defined in the env0.yaml file at the root of our repo to provide our configuration variables.
Below you can see how our env0.yaml looks like. Notice that we are specifying the same configuration variables that were in our pulumi.dev.yaml file.
If you left the option to approve the plan automatically unchecked, you will need to confirm the execution of the [,code]pulumi up[.code] command.
View the Output
Finally, once the deployment completes, you can view the outputs under the 'Resources' tab.
Once again, to access the Kubernetes cluster, you can simply save the kubeconfig in a file and export as an environment variable as shown below:
Congratulations! You've just used env zero to deploy the Pulumi stack and provision an EKS cluster, and it probably took less than 5 minutes.
To clean up, just click the 'Destroy' button. One click and it gone.
In Summary
We've covered a lot of ground in this post—from the nuts and bolts of what Pulumi is to its nifty features, and even how it plays nice with env0.
If you're in the DevOps or Platform Engineering space, Pulumi offers a refreshing take on infrastructure-as-code. By marrying traditional programming languages with cloud resources, you get a level of flexibility and power that’s hard to beat.
So, what's the takeaway? If you’re looking to step up your infrastructure game, Pulumi is worth a shot.
Not ready for your entire team to move from Terraform to Pulumi? That's the benefit of a framework agnostic IaC platform such as env0.
Here are some of the key features that I like about env0:
- Drift detection – env zero provides drift detection that can help you detect drifts and alert you about them automatically.
- Governance – our platform allows you to define custom policies and guardrails to both secure and keep your infrastructure compliant.
- Multiple frameworks – env zero supports multiple frameworks such as Pulumi, Terraform, OpenTofu, and more.
- Ephemeral environments – Developers can set up an environment with a timer to self-destruct reducing wasted resources.
- Flexibility – With pre- and post-hooks that reduce the need for a full external CI/CD pipeline.
For more information on env0's support of Pulumi, please reference this guide.
What Is Pulumi And How To Use It With env zero
Join us as we sit down with Sergey Korolev of Rakuten Viber to learn how his team automates and manages Infrastructure as Code, and how mature IaC practices improve developer experience, reduce technical debt, and streamline operations at scale.
Discover practical strategies and best practices for:
- Automatically detecting drift from manual IaC changes
- Resolving deployment tracking issues
- Setting up managed developer self-service for IaC
- Integrating IaC with GitOps workflows
- Optimizing IaC-related costs
- Unifying IaC-related processes
and more!
-----
Transcript
Andrew
Welcome, everyone, and thank you for attending today's webinar, “Using env zero to Pave the Path to Infrastructure as Code Maturity.” Today I have Sergey Korolev with me from Viber. Sergey, would you like to introduce yourself?
Sergey
A few words about myself: I've been in the area of DevOps and managing teams that are working in this field of DevOps for the last ten years. Besides that I like traveling, I like snowboarding and doing martial arts in my free time. Thank you.
Andrew
Nice. My name is Andrew. I'm the Director of Sales Engineering here at env0. And today I'll be helping lead this conversation. We do have the chat open, so please feel free to ask your questions in there, and then we'll address that during the Q&A portion of this conversation.
Before we begin, a little quick overview of what we'll be talking about today. We'll be reviewing some challenges with Viber’s previous CI/CD process, how they managed Infrastructure as Code, the cultural shifts related to that, and what that really meant in terms of managing resources and deployments. And at what point did the previous process come to a breaking point? So what was that tipping point for change? And just before we begin this conversation, let's quickly review env0.
env zero gives you the ability to fast track your Infrastructure as Code’s maturity. What we're seeing in the market is a lot of times people are starting with no Infrastructures as Code, click ops and manual deployments, which is slow, and they start scripting and adding some Infrastructure as Code. But that creates cost overruns and security challenges. At env0, we're trying to optimize both productivity and governance through four main pillars: automation, cost controls, managed self-service, and governance.
We'll be addressing some of these features that Viber has been able to take advantage of and talk more about how we are helping Viber move along this maturity curve. So on a high level, what was their biggest game changer?
Sergey
When we started with env0, I guess the first thing that gave us the most value was drift detection. It's probably also the first topic we are going to be discussing today. Basically just understanding where we stand in terms of what our Terraform states look like. And I will be describing exactly where we started from and where we are today. Drift detection is the thing that actually helped us a lot, especially in the first few phases of onboarding.
Andrew
Awesome. So let's just jump right into it then. In terms of drift detection. Let's go into before and after. What did you see?
Sergey
So before, onboarding to env zero would be… I guess a lot of the participants in this chat know this feeling when you try to apply some Terraform. And then you are like, oh, it's been changed two years ago and for some reason I don't know who did that and well, I can’t find him now or why he made the change he did. Detecting drifts, this is the term that is being used, a long time after the actual apply or the actual change happened. So you basically just see it whenever you run a plan or apply and you can check that list.
How we use that, we would check it only at this phase, like you need to make a change. You make the change. Oh no, the state is not what I was expecting it to be. And now I need to fix the previous change that somebody else made.
And of course, there was no process to regularly check whether the states have been changed. If the resources, the actual resources, have been changed or not. And once we have onboarded env0, we detected that basically 60% of all our states, of all our Terraform so-called environments in env0, are like that.
And just to let you know, we've been using Terraform at least for the scanned environments for about two years. We have between 7 to 10 committers I think. So plenty of hands touch the code, and not just the code. It could be either the code or the actual resources in AWS in our case.
Now, after integrating the environments and code into env0, you can see in the screenshot that today we have more than 900 environments. Most of them are not drifted. Of course the drift happens due to some manual changes or other changes people or some automation that might be running and changing resources like AWS or ECS for example, there are things that are being changed by the service itself and you need to correlate the Infrastructure as Code to that.
We do these scans every other day. We are running a plan on all of our environments every other day and detecting whether those environments, whether the Terraform that represents them, are the actual resources that we have in our cloud environment, and they can see it easily in this dashboard. It's very convenient, something I do every day, and me, as a manager, I’m bugging my engineers like “you need to fix that, why does it drift?”
Eventually we reduced it to zero. And as time goes on, you can see that the drift is going back and forth, but we try to eliminate it as much as possible. Some drifts are eliminated easier than others. But yeah, and as we go on with the talk, I will also describe how this, in my opinion, is leading to the cultural shift in our team.
Andrew
Awesome. Let's move on to the next topic then. Environment Discovery. So let me just quickly inform the audience on what Environment Discovery is at env0. So it's basically the ability to be able to scan for code changes within your git repo and automatically on board that into env0. And another way to describe it is kind of self-service through code.
So as your developers are creating here, you can see in the image a new Terraform stack and a folder. env zero essentially detects that through the PR process and will generate the plan and then deploy that resource once it's been approved.
Sergey
I think it was about a year and a half ago, Environment Discovery was not as advanced as it is now. So that's a good fit for whoever is joining now will have it much easier for themselves. The cool thing about that is that any Terraform I’m adding, let's say I have this kind of a folder structure. When I have an integration environment, I have some infrastructure. Let's say I have EKS and I have some integration cluster now.
So basically the integration clusters and environment. And now I want to, for example, add another cluster. All I want to add is the new service that I was not managing before, for example ECS, and all I need to do after I configure the environment discovery in env zero – I just add the folder, right? And the same GitOps that again will basically trigger env zero to add an environment according to the new structure of the folder.
And this is something that we benefit a lot from at this point, in my view, and we benefit the most from it because it's just straightforward. You add your Terraform and it’s being run somewhere else, not on your computer, for example. Something I will also probably be talking about in the next few slides.
I guess that's as far as it goes in terms of the Environment Discovery itself. Another thing I would add is that once you discover the environment, all the newly discovered environments go through the same process. If you need linting, governance checks, or tests to apply to your Terraform, it will be consistently applied to all Terraform environments.
Previously, it wasn't like that. People would run Terraform on their machines, and each team member might or might not follow best practices. For example, everyone should run lint, but I didn't have the tool to enforce that. I couldn't ensure that everyone was running lint regardless of where they were running Terraform from.
I think we can move on to the next topic.
Andrew
Okay. Let's talk about GitOps then.
Sergey
GitOps is something that is very close to my heart. We've been using ArgoCD for the past three years or so and we find it very convenient just making changes in Git and having this change go on as you proceed with whatever strategy that you choose. But for example, if you open a merge request, let's say in Terraform, you would have a plan. If you merge this merge request or pull request, you will actually run an apply. This is basically the general idea here and we try to implement it ourselves.
So we implemented the kind of testing mechanism where we would run a plan and then apply to our environments prior to onboarding env0. And it was a hassle to implement the solution itself, it was a hassle to maintain it and actually make sure that people are not just skipping the pipeline itself and just submitting the changes.
And because also, as I mentioned, we did not have the actual apply and plan of the actual infrastructure as part of the pipeline. And we've reviewed Atlantis. I tried to see why it fell short exactly but didn't find the exact things that made it not work for us. I think it was something involving us using GitLab Enterprise and Atlantis not supporting it at the time.
And also it's basically a CI tool which you need to orchestrate more things around to make it actually work. You will need to implement the whole thing in your CI/CD pipelines to make it work, as you would expect. At the time, about a year and a half ago, two years ago when we tested it, maybe it's better now.
Another thing is that, as I mentioned, everybody could just run Terraform as they wished. And me as a manager, I would not see these changes. And that from personal experience, one of the things that could happen is that you will run the Terraform locally and you go on to another task but not submit the commit.
And so then you get the drift in your Terraform because it did not submit the code to Git. And you now need to understand who made the change and why he made it. And eventually you discover that somebody just forgot to submit the commit.
Andrew
There's a question from the audience about ArgoCD and whether or not you… Is it replacing Argo CD?
Are you still running Argos CD on the side? Does env zero interact with Argos CD In which way?
Sergey
For us it was pretty straightforward that in our use case we will probably not use env zero for deploying Kubernetes just because we already had that and we had this expertise and were totally fine with ArgoCD, but it's complementary.
One of the cool things that you could do is, for example, and this is what we are doing, deploying the whole ArgoCD infrastructure itself in Kubernetes via env0. So you need to deploy that. So to say, operating ArgoCD, you do it for env zero for Terraform. Similarly, you can deploy basically any Helm chart either via env0's capabilities to deploy Helm charts or for Terraform providers to deploy Helm charts, both ways work.
But we chose to stay with ArgoCD for deploying Kubernetes. We really like this ecosystem and they remain as a complementary system to one another.
Andrew
Right. Thank you. Let's talk about the next topic: Cost Management. How does env zero help you with managing costs?
Sergey
So if you look at the right screen here, you will probably see that one of our environments is pretty costly, right? I reviewed it because I saw it in env zero discovery and found this is basically a cloud fund distribution that is getting a lot of requests, a lot of traffic. And this is why it costs as much as it does.
And basically in cost management, initially when starting, I didn't think of it as a thing that I might be so interested in because we had so many other FinOps tools that we are grabbing information from, we have anomaly detection, you have the cost Explorer, you might have some third party that does that for you and you have all these kinds of things that your management costs.
And it depends on your company, you might have dedicated FinOps engineers, you might have developers doing the FinOps and you might have your software engineers doing FinOps. So it depends on the company’s strategy regarding FinOps in general. And I think from the perspective of a DevOps engineer, you're not going to cost-explore every commit or every change. You're doing infrastructure and you will probably review your environments and your Terraform code more often than reviewing your costs. Especially if a FinOps is a different group or a different team or a different person in your organization. So putting it in your face basically is something that helps us extend our FinOps practices, extend our FinOps options and visibility. It does not necessarily replace those.
But there is another thing regarding that, we can see exactly which resources are costing what amount of money. I said that an environment represents a Terraform folder, for example, so you know exactly which resources are in this Terraform, following this Terraform state. And you can see exactly how much they cost, which is really cool.
And also it is relatively simple to configure. Basically I think Terratag is env0’s open source, so it's just incorporated in env zero itself. And basically what it does, on top of your state file it adds the needed tags for your environment ID and project ID. And then you configure your cost explorer and your billing so that those tags are tagged and you can see by these tags of your env zero environment, for example, how much this specific resource costs you according to the tag. And all that was just an addition that you basically just enable. It's relatively easy to configure. I think it took me an hour or so to do the whole configuration.
So it's a feature that I like a lot. I would not say again that it changed the whole FinOps practices in our company, but it for sure led me to some interesting environments where I immediately saw that something was wrong. This is a tool that I use each and every day.
Andrew
So with all these different topics I’d love to better understand, how did it affect your company culture?
Sergey
I wouldn't say company because we – and in the next few slides you'll see that we are going to talk about – our future plans and how we see env zero part of it. Today we're mostly using env zero as a tool for our DevOps engineers and culturally I think that things such as auditing and governing our code became more of an everyday practice, something that we can introduce to the Infrastructure as Code more easily.
And for Viber, for example, it's very important in terms of auditing and governance. Let's move to a different topic: Git and commit, commits and changes in Git. So we have the governance of it, we need audits and we have for example to 4 eye principle practice. Well, we have somebody mandatorily check our code and approve it.
The same goes for Infrastructure as Code. With this in mind we also implement the governance and auditing of everything via env zero as well. And another thing is that in the culture of our team, something that changed is the state of mind of not breaking states and keeping the broken states as low as possible and reducing the tech debt.
And I think when you're looking at the long run, you want to keep your tech debt as low as possible for the team and the future of people to come. You don't want to come to a place where you'll need to find those things breaking all the time or broken things that are there for years. And I guess many of the participants today know exactly what I'm talking about. And of course, it did not eliminate everything. But I think the right life mindset will eventually get us to this point.
Andrew
So yeah, cultural change definitely is hard and takes the right people in place and the processes and in order to help move that along and that tools can only do so much. We'd love to talk next about your expectations, your experience from onboarding with env0.
So there are naturally expectations. And I think this will be really interesting for people who are just going to start looking at env zero or considering. So let's look at these five different parts here. When you're talking about transferring pipelines, what was your expectation and what was the reality?
Sergey
I would expect onboarding all of our environments, especially talking about a Terraform-like liability that consists of all those kinds of Terraform and different structures and different ways that were applied.
I expected it to be less effort, it took me about, I think, two weeks to do it end to end. Today we are using the auto Discovery feature to just continuously onboard new environments automatically. So now it doesn't take any more effort anymore.
Andrew
What about visibility? You mentioned here that you were concerned about how you actually see the resources in env zero and policies and things like that?
Sergey
Yeah, so I didn't have the chance to introduce it here, but in general, what you can see besides the dashboard is all the environments and those statuses and whether they are drifted or not. So if you click on any of the environments, for example, you will see an audit of all the deployments that have been done.
And you can see who did it, why they did it, when all the phases of the lifecycle of a specific deployment, let's say it's starting by initializing the Terraform and then planning and applying it and some steps in between, which can be the tagging that I told you about with Terratag. It can be linting as a step. You can add custom scripts to it, OPA, a plugin that you can add. So again, from a manager’s perspective it's so convenient for me to understand what my team is working on. As an engineer, just going back and just reviewing: Okay, which changes were done to some resources that I own? That’s also something that we value a lot.
Andrew
Definitely. A lot of our customers and prospective clients are concerned about auditability and being able to track exactly what happened, especially when you're coming from a manual deployment process or even just simple pipelines, because in general, CI pipelines are harder to go through as opposed to what you have here in env0, you can click on the infrastructure and see exactly what those changes were.
I love this. There's this third expectation we would love if everyone loved env0. What was your actual experience onboarding with env zero and people's reaction to using it?
Sergey
So as I said, you can run all those kinds of things, such as limiting and OPA and other scripts. You have more steps that a plan would take locally. You can initialize it once, you have all the files that you need locally downloaded from your backend and basically rerun it as much as you want so you don't need to initialize it again. The plan will probably run faster on the apply and everything will be running relatively faster on a local machine due to the fact that you have all the files locally all the time.
People were hanging onto that. They would tell me, “it was fast for me, I don't see the reason why I need to run all this additional stuff. What does it give me?” And the answer is consistency. If you have policies in the company that you regulatory policies that you must follow, you must follow those policies.
It's not a matter of question. It's a matter of you have to do it. And people who are running things locally without applying those policies on their runs, eventually, hopefully not, but it can lead to issues. And I don't say that planning it in a managed environment and going through all the policies and all the tests and whatever will result in zero errors and no issues and no mistakes from people.
But at least you will have an audit, at least you will understand what happened and you hopefully will find out about the issue faster. And that's what I tend to believe. But yeah, people kept holding to this philosophy of running things locally. Now, again, it's not false at this point as well, but I see that people are very careful about when to do it and why.
Andrew
Let's get to this next topic about drifts.
This is an interesting one because it kind of turns into a double edged sword. Can you tell us more about your experience?
Sergey
We started off with about 60% drift to the environments. There is a specific engineer in my team which is very dedicated to eliminating drifted environments, he's been doing it for quite a while now, like for the past year or so. He's especially into those specific cases which are really hard to tend to, some changes that were made years ago. First thing first, handling drifted environments is a pain in ass. And as long as you keep this drift, you will have a harder time in the future. And this is exactly why I tend to encourage everybody to treat it as fast as possible.
But more than that, the things that we started also doing is thinking about how to modularized our environments and our deployments better. It depends on the use case and the scenario, but there are use cases where you say, I want to just try the Terraform as plainly as possible and you do so, but after some time you realize that this request is coming back to you again and again and again. So now we need to understand what the request is and how you can modularized it better. This is actually not a simple task. You need to understand exactly what your customers, or in our case, the engineers, need. And what is the recurring request that requires a specific solution.
We have some use cases. We use the env zero templates, which is basically like collateral for modules, and we use that to deploy some resources in the cloud which are reusable. A simple example – we have a team that uses Lambda functions a lot. So we basically have this module which is representing their needs for the lambda function and its environment, and we redeploy it as a module using the env zero templates and we find it to be very time saving eventually. But initially we would have been like, how are they able to do it?
Andrew
Because it wasn't modularized yet. There's a question from the audience about where the drift is coming from. So earlier you mentioned there are 60% of the drift. Was it because people were doing local applies or what else was causing that drift?
Sergey
Yeah, that's a simple scenario from the perspective of a DevOps engineer. Our DevOps engineers have full admin access to our environments, managing the entire cloud environment end-to-end. So, let's say we have an issue with a team in production and need to make changes.
The simplest scenario would be changing the number of instances. If there's a production issue, you might change these things manually in the AWS console, and you'll likely need to document it. The drift will indicate that you had ten instances, changed it to twenty, and now you need to revert it.
This is the drift. While this is a specific small change, it can be related to any service and any change. Urgent changes often cause drifts, as they require immediate action.
Andrew
This is also the last expectation here we get asked a lot, especially since CICD needs so many tooling and what scope does env zero have. So what can tell us more about this expectation? Does env zero replace everything?
Sergey
No, not really. I describe it in the part we talked about GitOps. So you can also see the pipeline itself. So basically env zero is integrated into all Git pipelines, you can see the pipeline itself showing you the whole like plan and apply, and all the steps that are happening in env zero itself.
So technically you can avoid going into the env zero system and just using your pipeline as-is for Git, for example. Also you can use the API and all of the Terraform modules of env zero API to integrate it with different other tools that you use in your company, for example IVP, and this is something that we plan for the future.
Andrew
So that's a good transition to talk about future plans. So you've been using env zero for a while now. Tell us about your journey and your future journey?
Sergey
A few things are important to me, and one is the governance aspect. I've talked about policy enforcement quite a lot today. Tools like OPA are something we are looking to incorporate. We're also expanding our developer portal for self-service via env zero for cloud resources. This answers a question from the audience about integrating env zero for developers. In our use case, we do not expose env zero directly to the developers; it operates in the background.
We are waiting for a few features that we hope will come soon. One is Infrastructure as Code coverage control and drift blame. So I can blame the engineers for being… No, I'm kidding. Drift blame is a term used to describe identifying who made changes in the console. For example, if someone changed the number of minimum instances from 10 to 20, I would want to know who did that.
So this is something that we all are eager to get as a feature in the future. And the last point here is talking about a maturity journey. We are not there yet. We’re using the GitOps approach and we have env zero managing and governing our environment. But we are still not at the self-service level. We still have work to do in componentization. You need to understand what you want to component inside and which features you want to introduce as a self-service. So we are working on those.
Andrew
Thank you for sharing your experience. Let's open up to open Q&A. There's a question about your separation of Argos CD. How do you separate resources that need to be automatically deployed by env zero and others that need to be deployed by ArgoCD?
Sergey
At this point we don't have a solution just yet. The way we are looking at it is that we will probably use the IDP and together with ArgoCD detection of new applications, right? So basically env zero will detect Terraform and ArgoCD will detect the application. Basically the same approach on both ends. So you will need to add the Terraform code and the Helm chart or manifest it.
And each of the systems will separately deploy the resources that are needed. So for us, the approach is to use IDP and a developer portal. A developer can trigger the process and there are two different stages and each stage whether the Helm chart or the cloud resources.
Andrew
I'll put in a chat a blog post where it shows env zero deploying. So you mentioned this, you deploy your Kubernetes cluster, but you need to bootstrap ArgosCD onto your company's cluster so you are using a Helm chart or Terraform deploying Helm chart, you can deploy ArgoCD’s operator into your Kubernetes cluster. Now, in order to configure ArgosCD to essentially listen to new applications, you can deploy the application YAML, which is the Argos CD’S CRD for managing applications.
You can use env zero to manage this application YAML as well. If you want to deploy Kubernetes manifests directly into your ArgoCD cluster, env zero will continue to listen for any Infrastructure as Code changes, such as updates to Terraform. All of this is managed in GitOps with ArgoCD. Because you've configured ArgoCD to listen, it will automatically respond to any image changes or manifest changes, managing its own processes accordingly.
So essentially GitOps on both sides, listening to their own hooks within Git.
Sergey
Yeah there's just another one about this topic is that you have application sets you know in ArgoCD you also have an approach which is called app of apps, one application that holds many applications. So for us it would be at least at this point, we are at the point where we have the app of apps deployed via Terraform and it is managing the whole other applications that also might be useful there.
Andrew
There is a question here. Have you used env0’s Environment Workflow? You answer the question, I’ll describe Workflow. So Workflow for our audience in env zero is essentially an orchestrator of multiple templates, it allows you to deploy a multilayer infrastructure. So say you have a three tier infrastructure network, compute and services.
You can keep each of these different services as separate Terraform templates or Terraform resources and env zero will orchestrate, make sure it deploys the right in the right order. So that's the high level of what env zero workflow is. The question is, are you using it? And if so, what has your experience been?
Sergey
Yeah, so I would say no. We've evaluated some use cases as we have before, but eventually we end up not using it at this point.
Andrew
Okay, let's look at another question here. I'm interested in order of magnitude, how many workspaces, how many of these are templated and auto discovered and how long do they actually live? So how many env zero environments or Terraform Workspaces, are you managing with env0? Roughly, yeah.
Sergey
So as I presented it before in the dashboard and then one of the slides, it’s about 900 of those. Most of those are environments that are pointing and discovered in some GitHub repository that we have that holds most of all the Terraform code and basically each folder eventually is an environment. Most of those are not templated.
But we have some use cases which are the templates that as I describe I gave one example, the Lambda functions. We have a lot of static websites that we deployed that use templates. And I think the other one that we consider using is with the self-service that we want to introduce. Eventually those will also be templated environments.
Andrew
Here's a question. We are planning to force all the development to to be deployed by a central code catalog that's maintained by a single infra team. Is this something env zero you're seeing more of, or are teams still able to develop Infrastructure as Code themselves? I guess I will answer that. So as you heard from Sergey's perspective right now, they are developing Infrastructure as Code themselves and slowly introducing templates to other team members.
At env0, working with dozens and dozens of customers, we see a broad spectrum. Some are completely siloed and command and control, I would say. So basically the central team is managing all of the resources and deploying all the resources, and then some are having a catalog approach. They create essentially a set of resources that their teams can onboard onto and then let the dev teams choose.
And then some are mixed in the sense that they have subject matter experts within each of the dev teams. So those people would like to write Infrastructure as Code, but then you also have the people on the other end of the spectrum who don't know any Infrastructure as Code and need a catalog in order to get access to resources.
So in env zero with templates, you can essentially create these quote unquote modules that will be able to give access to resources to these dev developers who aren't familiar with the infrastructure. At the same time, those modules you can publish in MS model registry for the developers who are familiar with infrastructure and want to compose their own resources, you can use what we described earlier with environment discovery and give them the ability to get access and deploy their own Infrastructure as Code through a PR process as well.
So we're definitely seeing the spectrum and it really kind of depends on the kind of people you have within your organization and the type of processes that you want to put in place. And env zero can really help cover both sides of that. And I hope that helps give you a sense of capabilities within env0. Maybe we'll take one more question from the audience.
For the questions we didn't get to cover, which are a bit more about env zero in general, maybe we'll segue into closing statements. So thank you again, Sergey, for taking this time to meet with us. Any last words about your experience with env0?
Sergey
I guess that a motto in life for me is taking things slow. And this is a process. I think we are in it for the last one and a half years and it will take more time.
Andrew
Absolutely, let's keep on making steps towards climbing that mountain. So thank you for taking your time. So I want to end on this one note, as since you're now here and you're probably interested in seeing how you can get started at env0, we have this concept of a one-day proof of value, so essentially we can help unlock and help you get an understanding of how env zero can build value for you.
It's an easy four-step process. We create an organization, add credentials, connect your Git repo, and start deploying resources in env0. And what we hope to achieve and can achieve within this one day POC is simply be able to show you the setup process, show you the cost estimation and start tracking cost and setup drift detection.
We've done this with a few of our customers already and immediately added value and we’ll show you what we can do. We'll start with a one-hour pre-onboarding session to get you started. Then, on that day, we'll go through the entire checklist and get you fully onboarded. We'll set up Slack or Teams support for easy communication and give you a 30-day trial of env zero so you can truly start experiencing the ROI.
After the 30 day trial, we expect you to reduce up to 45% of your cloud cost, increase your deployment speeds and decrease your time to merge. And again, all we need to do is create an organization, set up some cloud credentials, connect to whatever version control system you're using and deploy some Terraform or other Infrastructure as Code that you're using. If you're not ready for a one day POC just yet and you want to learn more, you can book a demo through our website, envzero.com/demo-request or sign up for free trial on envzero.com and check out envzero.com for more docs and blogs and resources.
I want to thank everyone attending again today. If you have any more questions, please feel free to reach out and we'll be happy to address those on a future call or through chat. All right, So that concludes our webinar for today. Thank you again, Sergey, for joining us and hope to hear from everyone soon.
Cheers!
Webinar: Lessons Learned from Managing IaC Across 800+ Environments
Infrastructure as Code (IaC) offers many benefits but also introduces new security and compliance concerns, along with challenges in controlling cloud budgets.
Join us for a 30-minute lightning talk, where we’ll share practical tips on how env zero addresses these and other risks with a suite of analytics, governance, and drift management tools.
Learn how to make the most of our platform from the best in-house experts!
Watch the video below to learn about:
- The impact of cloud cost overruns and how to control budgets effectively
- Best practices for securing access and enforcing compliance in IaC environments
- How to detect and remediate infrastructure drift before it disrupts operations
and more!
Roni: Welcom, everyone, to our webinar in env0. Today we will be discussing risk, mostly in the context of Infrastructure as Code.
We have Kosta with us, a Senior Sales Engineer on our team, and Yuval, our Director of Product. My name is Roni. I'm the Director of Engineering here at env0.
And we're here to talk about risk in the context of Infrastructure as Code.
Infrastructure as Code, in its own ways, is a way to mitigate risks of traditional ClickOps and other methods of managing infrastructure. Not knowing who changed what and when is a huge problem.
IaC helps solve this by providing approval flows for pull requests, version control, and rollbacks—all of which come out of the box just by using any kind of Infrastructure as Code.
But despite being a major step forward, IaC does have loopholes that need to be addressed with best practices and tools. Without them, you're introducing new risks simply by using IaC.
Yuval, can you share some examples of such risks?
Yuval: Yeah, sure. Thanks, Roni.
As you said, IaC presents a huge opportunity for IT and DevOps teams. But it also introduces some risks:
- Cost management: How do we ensure that costs don’t spiral out of control? Developers and testers may leave resources running over the weekend or provision oversized instances that aren’t needed.
- Access control: How do we make sure unauthorized users aren’t able to modify or delete critical resources, such as those in production environments?
Many organizations implement different access models—allowing more access in staging but restricting production to the DevOps team. But when we open infrastructure up to more users via IaC, these issues need to be managed carefully.
Roni: Great points. We also hear the term Policy as Code thrown around a lot to help mitigate these risks.
Kosta, can you share some scenarios where Policy as Code and enforcement platforms help minimize risk?
Kosta: Yes, certainly. As the saying goes, "With great power comes great responsibility." That's where Policy as Code comes into play.
One common use case is cost controls. Organizations often want to set limits—for example, preventing deployments estimated to cost over $100 unless they receive admin approval. This ensures cost efficiency while maintaining flexibility.
Another example is environment-based restrictions. You might enforce stricter policies in production than in testing environments. This ensures that the right guardrails are in place while still allowing teams to work efficiently.
Roni: The Infrastructure as Code on its own does not guarantee an organization benefits from it simply by using it. I think we've heard some examples right here.
One of the bigger risks out there—and we see this all the time, especially with larger organizations—is that someone, in certain cases, steps in manually. Sometimes they need to make a change by going into the web console of a cloud provider or using an API.
That is what we call drift.
Yuval, let’s talk about this term. Why is it so dreaded?
Yuval: So drift is definitely a dreaded term in the world of IaC, and for a good reason. Drift happens when the actual state of infrastructure no longer matches the desired state defined in IaC. This can occur if someone makes a manual change in the cloud console or if external updates happen outside the IaC pipeline.
Why does this matter?
- Unpredictability: If the cloud infrastructure no longer matches the code, unexpected behaviors can occur when making changes.
- Compliance risks: Policies enforced via IaC can be bypassed with manual changes.
- Operational complexity: Engineers must spend extra time troubleshooting and investigating issues caused by drift.
For many organizations, finding the right tools to detect and fix drift is a top priority.
Roni: Gotcha. Thanks, Yuval. Managing drift is a constant challenge, and one of the biggest questions is trying to understand how drift happens—who made the change and why?
Instead of just talking about it, I thought we’d actually see this in action. This demo will wrap up everything we've talked about and show how env zero helps mitigate exactly these risks.
Kosta, over to you.
Kosta: Thanks, Roni. Let’s dive into the demo.
So that's Cloud Compass.
And I will also point out that with Cloud Compass, being able to read the details of each of the resources, we can pretty much understand what caused the drift.
To demonstrate that, I have this S3 bucket that I deployed some time ago, and I know for a fact that it got drifted.
Under drift detection, I can see that a specific tag—the env zero project ID—was originally assigned, but drift detection was triggered and resulted in a drifted state.
Looking at the plan output, I can see that someone added a new tag that wasn't part of my code, which is why it is now flagged for remediation.
At the same time, another parameter that used to be false is now true.
In env0, we have Drift Cause Analysis, where I can click the Analyze Drift Cause button and drill down into the historical events that led to this drift.
For instance, I see that this bucket was originally created via IaC. The event log confirms it was created in December, and I can also see who created it.
Later on, I notice a ClickOps operation occurred on that bucket. Looking at the logs, I see a put bucket tagging operation was performed, and I can even see exactly who made that change.
This allows me to not only detect when the drift happened but also who made the change and why it happened.
So that's Drift Cause Analysis in env0, helping teams pinpoint and remediate drift efficiently.
Roni: Thanks, Kosta. That was a great walkthrough! To wrap things up, we've covered how:
- Infrastructure as Code helps mitigate risk but also introduces new challenges
- Policy as Code helps enforce security, compliance, and cost controls
- Drift detection and remediation are critical to maintaining infrastructure integrity
We’ve seen how env zero offers tools to help organizations stay compliant, efficient, and in control of their infrastructure. Now, let’s open it up for some Q&A.
Roni: We have a few questions from the audience.
Question: If an organization already experienced cost overruns, how does env zero assist in identifying and fixing those issues?
Yuval: Great question. env zero allows organizations to monitor cloud costs, tag resources, and enforce policies that prevent excessive spending. We provide insights into where overruns occur and allow teams to take corrective actions before costs spiral out of control.
Question: How does env zero prevent unauthorized manual changes to infrastructure?
Kosta: env0’s Policy as Code ensures that only approved deployments are allowed. If someone makes a manual change, drift detection alerts the team, and Drift Cause Analysis helps pinpoint exactly what happened and by whom, enabling fast remediation.
Roni: Those were great questions! If anyone has more, feel free to reach out to us.
Before we wrap up, I’d like to thank everyone for joining us today. A big thank you to Yuval and Kosta for sharing their insights.
If you’d like to learn more, visit envzero.com.
Have a great rest of your day!
For more detail, download our solution brief: Reducing Risks in Infrastructure Provisioning.
Three Ways to Manage Cloud Provisioning Risks with env zero
As infrastructure scales, managing its increasing complexity and the dependencies between different components becomes more challenging. This is where the env zero Workflows feature becomes essential, offering a solution to address today's complexities in Infrastructure as Code (IaC) workloads.
In this write-up, we’ll discuss the capabilities, use cases, and features of Workflows in detail.
What are env zero Workflows?
env zero Workflows provide a structured approach to managing groups of related environments by defining their dependencies and orchestrating the entire process. This ensures that all dependencies are satisfied and the workflow runs smoothly from start to finish.
Workflows are particularly useful for managing systems divided into multiple sub-systems, allowing you to oversee the entire system as a cohesive unit while retaining control over individual stacks. They simplify the process of breaking down IaC stacks into smaller, more manageable parts.
With Workflows, you can describe complex deployment relationships and initiate partial or full runs to create and update sub-environments.
This ensures proper sequencing of resource deployment and allows changes to any sub-environment to auto-update dependent components.
Workflows also help manage environment variables and sensitive credentials across different stacks, ensuring consistency and security.
Whether your IaC is monolithic or split into microservice-based stacks, Workflows are a massive step forward in describing and managing complex deployments.
Disclaimer
Workflows is an Enterprise plan feature. Visit here to learn more about env zero pricing and plans.
How Workflows Work
A workflow is based on a workflow template along with the IaC stack templates that you intend to deploy in your workflow environment.
The Workflow YAML file, env0.workflow.yml, is used to define the hierarchy of the dependencies and configuration of IaC sub-environments within a workflow declaratively.
To better understand how Workflows work, let us take an example.
Below is the definition of the env0.workflow.yml consisting of two environments that deploy VPC, and an EC2 instance that depends on that VPC:
This method works when we have the respective IaC templates created in place for Virtual Network and VM stacks.
It is also important to note that each Workflow sub-environment can also reference the directories for individual IaC components, not only from different repositories but even from different VCS providers.
Let’s take the same example above to make it clearer.
Here we are referencing the VPC directory from our GitHub VCS repo and the EC2 directory from our Gitlab VCS repo to define our sub-environments in the Workflow YAML:
Notice the needs block makes one environment dependent on another (EC2 depends on VPC, in this case). You can add more parameters as you see fit to define your custom Workflow YAML file.
Following on the template creation approach, with our VPC and EC2 templates now in place, we are set to develop our workflow template, which references the env0.workflow.yml file located in the root directory of our repository.
The next step is to deploy a workflow environment using our workflow template. This workflow environment shows us a vivid dependency graph that illustrates the relationship between our Virtual Network and VM sub-environments.
In this example, no variables or information are shared between the sub-environments. This presents an opportunity to leverage the Workflows' core capability of Environment Outputs, which allows one IaC sub-environment's output to serve as another's input.
For instance, you can output the VPC subnet ID from the Virtual Network environment and use it as input for the VM environment, ensuring the EC2 instance is deployed into the correct VPC.
This method facilitates the secure transfer of sensitive credentials or data between sub-environments dynamically via the env zero platform abd eliminates the need for local storage of sensitive information or hard-coded variables in your IaC -enhancing the security of your sub-environments.
We’ll go into the detailed mechanics of Workflows with Environment Outputs in a moment.
Advantages of env zero Workflows
The capabilities of Workflows prove to be invaluable, particularly in scenarios like those discussed below:
- Dependency Management: As emphasized earlier, Workflows excels at managing dependencies between different environments. The Workflow YAML file allows you to declaratively define all the IaC stacks you plan to deploy and specifies any dependencies between them. Additionally, Workflows visually represent these dependencies, making it easier to understand and manage complex infrastructure setups. This visual aid simplifies troubleshooting and planning, enhancing overall deployment efficiency.
- Targeted Deploy/Destroy: With env zero Workflows, you can deploy or destroy specific parts of your infrastructure without affecting the entire workflow. This partial or selective deployment feature is handy in scenarios where you might only need to update or roll back certain IaC components. You could also destroy a specific IaC environment without affecting others, reducing the risk of unintended changes to other parts of your infrastructure.
- Multiple IaC Tools Support: Workflows enable teams to use different IaC tools within the same workflow. For instance, one team can use Terraform to provision infrastructure, while another team uses CloudFormation. Both teams can still share environment variables or sensitive credentials through Environment Outputs for their infrastructure, ensuring seamless collaboration and integration.
- Custom integration: Workflows can seamlessly integrate with other env zero features, such as custom flows, drift detection, and approval policies. For example, you can configure custom flows to run hooks (like installing a web server) after deploying an EC2 IaC stack in a workflow.
You can also configure the workflow to run policies using an OPA plugin after every environment is deployed to ensure meeting security and compliance requirements.
Practical Example
Now, let us delve into a practical working example of how Workflows operate.
Here are three environments where we need to deploy a Virtual Network, an AKS cluster that depends on the virtual network, and a Helm chart that installs FluxCD, which depends on the AKS cluster.
The respective env0.workflow.yml looks like this:
First, make sure that you have created your IaC stack templates:
Create your workflow template and select Run Now to create a workflow environment.
Make sure that your environment name exactly matches that of the name defined in the Workflow YAML file.
We have defined the subnet ID as an output value to be used by the AKS cluster in the VNet config, and the subnet ID as an input value in the AKS config.
Before deploying the VNet environment, we’ll switch to the AKS environment and set the input value for the AKS environment (vnet_subnet_id) as the environment output value of the VNet environment (aks_subnet_id):
Similarly, the Flux environment expects input values from the AKS environment, and the outputs are defined in the AKS config.
We’ll switch to the Flux environment and set our input variables.
Here, we are able to pass sensitive Kubernetes connection credentials as environment output to Helm for configuring and installing flux in our AKS cluster.
Note: Environment Outputs cannot be marked as sensitive in UI on the fly. Therefore, it is necessary to make the sensitive variables marked as “sensitive=true” in your IaC config itself.
Lastly, after configuring all the values, we switch to the VNet environment (as it should deploy first) and run the workflow:
After a while, we can see our entire workflow functioning properly:
Less is More: Partial Workflow Deployment
Often, we might forget to set up values correctly between different environments. Redeploying all environments to fix these issues is not ideal and can be time-consuming.
With Partial Workflow Deployment, you can redeploy or destroy a single sub-environment without affecting others.
Let's consider the previous example and take a scenario where we forgot to configure the environment output for the AKS environment.
In this case, the workflow would fail, as it relies on these outputs to function correctly.
To solve this issue, navigate to the AKS environment, configure the environment output, then click on the three dots in the AKS cluster block and select the Run from here option.
We will select the AKS cluster and re-run the environment. After this, we can confirm that our workflow is operating correctly again.
Additional Use Cases
There are more possibilities for utilizing Workflows than what we have explored so far. Here are key use cases to consider:
1. Multi-tool Setup
As infrastructure grows more complex with the business expansion, different teams may handle multiple environments.
For example, one team can manage the DB stack and its dependencies (Billing and Configuration Services), while another team handles the EKS stack and its dependencies (Configuration and Notification Services).
Over time, as organizations scale, IaC deployments can grow even more intricate than the example we saw above. However, by using Workflows to define your IaC resources in a declarative approach, you can efficiently manage and track your entire IaC stack. Due to a clear presentation of their IaC stack statuses, the teams can collaborate and ensure smoother operations.
2. Bulk Operations
Bulk operations refer to performing the same operation across multiple environments or resources simultaneously.
In this example, the EC2 and S3 environments depend on an IAM role, both configured with a Deny effect for any operation on SNS topics. Later, the team decides to change the IAM role policy to Allow operations on the SNS topics, enabling the necessary permissions for EC2 and S3 environments to interact with those topics.
Utilizing Workflows simplifies making changes to one environment, automatically applying them to all dependent environments.
This streamlines tasks such as updating software versions, applying security patches, and modifying configurations across multiple IaC environments, thereby reducing the time these tasks take.
Wrap Up
In conclusion, env zero Workflows stand out as a powerful and efficient way to manage complex infrastructure setups and dependencies in an Infrastructure-as-Code (IaC) environment.
Interested to learn more? Schedule a technical demo to see env zero in action!
For the complete framework, download our solution brief: Breaking the IaC Monolith.
env zero Workflows: Simplifying Advanced IaC Setups and Managing Dependencies
