The questions each chapter ends with — levelled, searchable, and answerable before you look.
215 questions, extracted from the chapters. Not ranked by popularity — there is no usage data yet.
Running the same playbook twice produces no further change. You prove it by running it again and seeing `changed=0`. Idempotency comes from using modules that check state rather than `command`/`shell` — and where a shell step is unavoidable, guarding it with `creates`, `removes` or a `when` condition.
Almost always `defaults/`. It sits near the bottom of the precedence order so callers can override it; `vars/` sits near the top and is effectively unoverridable. A role whose tunables live in `vars/` cannot be reused, which defeats the purpose of writing one.
Terraform provisions the hardware (creates the AWS EC2 server). Ansible manages the configuration (installs Docker and Kubernetes onto that server).
No agent is required. Ansible connects natively via SSH (Port 22) using standard SSH key pairs.
A Handler is a special task that only runs if notified by another task. For example, if Task A modifies the `nginx.conf` file, it notifies the `Restart Nginx` handler. If you run the playbook again, and `nginx.conf` does NOT change, the handler is not notified, and Nginx is not unnecessarily restarted. This is critical for preventing production downtime during configuration runs.
You integrate it at the image level or the boot level. Ideally, you use Packer to bake the Ansible configuration into a golden AMI, so instances boot instantly. If dynamic runtime configuration is strictly required, you use `cloud-init` (User Data) in the EC2 Launch Template to trigger an Ansible `ansible-pull` command. The instance reaches out to a Git repository on boot, downloads its own configuration, and configures itself before attaching to the Load Balancer, removing the need for a central Ansible push server.
Jenkins is an automation engine primarily used for Continuous Integration (compiling code and running tests). ArgoCD is a deployment engine for Continuous Delivery, specifically built for Kubernetes, that ensures the live cluster matches the Git repository.
It means the live state of the Kubernetes cluster does not match the YAML files stored in the Git repository. Either someone manually changed the live cluster (Configuration Drift), or a new commit was pushed to Git and ArgoCD hasn't applied it yet.
If you have 100 microservices, managing 100 ArgoCD `Application` YAMLs manually is tedious. The App-of-Apps pattern defines a single "Root" Application that points to a folder containing the 100 child Application YAMLs. When you want to add a new microservice, you just commit a new child YAML to that folder. The Root Application synchronizes it, which creates the child Application, which synchronizes the actual microservice. It is complete automation.
Both are CNCF graduated GitOps tools. ArgoCD is famous for its exceptional GUI, its strong multi-cluster management (Hub-and-Spoke model), and its Application CRD architecture, making it highly visible and easy for developers to use. FluxCD (often Flux v2) is fundamentally designed around the Kubernetes API (using Kustomization and HelmRelease CRDs) without a native GUI, focusing strictly on native controller patterns and high performance. An enterprise heavily invested in developer self-service and visual dashboards will choose ArgoCD; an enterprise focused purely on infrastructure-as-code automation via CLI/API integrations might prefer Flux.
A user is a permanent identity with long-lived credentials. A role is assumed temporarily and issues credentials that expire. Anything that is not a human — an EC2 instance, a CI job, a Pod — should use a role, so there is no long-lived key to leak.
First `aws sts get-caller-identity` — you are often not the principal you assumed. Then check whether the denial is on `sts:AssumeRole` (a trust policy problem) or the action itself (a permissions problem); whether an explicit Deny or an SCP overrides the Allow; and whether the resource ARN matches, remembering that S3 bucket and object operations need both `bucket` and `bucket/*`.
It lets instances in private subnets reach the internet without being reachable from it. It is billed hourly plus per GB processed, is zonal (one per AZ for real availability), and pulling container images through it is a common surprise on the bill — VPC endpoints for S3 and ECR remove most of that traffic.
No. Multi-AZ keeps a synchronous standby that serves no traffic and exists only for failover — it buys availability, not capacity. Read replicas are asynchronous and do serve reads. They solve different problems and are often needed together.
When the workload is sustained rather than bursty. `t` instances earn CPU credits while idle and spend them under load; once exhausted they throttle to a fraction of a core. The instance stays 'healthy' while the application becomes unusably slow, and nothing says throttled unless you look at `CPUCreditBalance`.
In order: visibility, then elimination, then commitment. Enforce tagging and group spend by team and service; delete unattached EBS volumes, idle Elastic IPs and forgotten snapshots; right-size what is oversized; then buy Savings Plans for what remains. Committing to three years for an instance you should have deleted locks the waste in.
An Auto Scaling Group creates and destroys servers based on traffic. A Load Balancer takes the traffic and distributes it evenly among whatever servers the Auto Scaling Group has created.
The ASG Health Checks will detect that the instance was terminated. Because the current capacity (1) is now below the `min_size` (2), the ASG will instantly launch a brand new instance to replace the terminated one.
`user_data` is a script passed to the EC2 instance metadata service. The `cloud-init` daemon executes this script exactly once during the first boot of the OS. It allows the server to dynamically configure itself (e.g., downloading Ansible, running `kubeadm join`, fetching secrets) without any human intervention. This makes the infrastructure immutable: if a server breaks, you do not SSH in to fix it; you terminate it, and the ASG boots a fresh one that perfectly configures itself via `user_data`.
EBS volumes are locked to a specific Availability Zone (AZ). If a Node in `us-east-1a` dies, the ASG might spin up the replacement Node in `us-east-1b` to maintain balance. The Pod will schedule in `1b`, but the disk is in `1a`.
"The Cloud" is just a marketing term. In reality, it simply means "someone else's computer." You are paying Amazon to rent a server in a massive warehouse.
Both store data, but completely differently. **EBS (Elastic Block Store)** is like a hard drive plugged directly into an EC2 server; it's fast, used for operating systems, and dies when the server dies. **S3 (Simple Storage Service)** is object storage accessed via the internet (HTTPS); it is infinitely scalable, extremely cheap, and survives regardless of what happens to your EC2 servers.
AWS bills EC2 instances by the second. A `t3.xlarge` costs roughly $0.1664 per hour. If left running 24/7 for a 730-hour month, it will cost ~$121. This is why automated tear-down scripts and Auto Scaling down to 0 during non-business hours are critical cost-optimization strategies.
In a multi-account organization, an identity (User or CI/CD runner) in Account A needs to build resources in Account B. Instead of giving the runner static credentials for Account B, Account B creates an IAM Role with a Trust Policy granting Account A permission to assume it. The runner in Account A calls the AWS Security Token Service (STS) `AssumeRole` API. STS returns temporary, short-lived (e.g., 1 hour) credentials. The runner uses these credentials to execute Terraform against Account B. This adheres to the Principle of Least Privilege and eliminates static key rotation overhead.
Both are container registries used to store Docker images. Docker Hub is the default, generic public registry. AWS ECR is Amazon's fully managed, highly secure private registry designed for enterprise AWS integration.
This usually means Kubernetes cannot download the image from the registry. The two most common causes are: a typo in the image name/tag, or Kubernetes lacks the authentication permissions (`imagePullSecrets` or IAM Roles) to access the private ECR repository.
You implement an ECR Lifecycle Policy. The policy evaluates rules daily, such as deleting untagged images older than 7 days, or keeping only the 50 most recently pushed images for tagged releases. This ensures the registry only retains artifacts actively used for production or immediate rollbacks.
You do not have the EU cluster pull images across the Atlantic from the US registry. That incurs high cross-region data transfer out (DTO) costs and slows down pod startup. Instead, you configure **ECR Cross-Region Replication**. You push the image once to `us-east-1`. ECR asynchronously replicates the image to an identical registry in `eu-west-1`. The EU Kubernetes cluster then pulls the image from its local EU ECR, keeping traffic on the AWS backbone and eliminating internet DTO costs while minimizing latency.
A User is a permanent identity with a static password or access key. A Role is a temporary identity without a password; it is assumed dynamically by users, AWS services (like EC2), or federated external identities.
In AWS IAM, an explicit `Deny` always wins. Always. The user will be blocked from accessing S3.
This requires Cross-Account IAM.
You use AWS Organizations and Service Control Policies (SCPs). You attach an SCP to the root of the Organization (or specific Organizational Units). The SCP uses a `Deny` effect for `ec2:RunInstances` with a condition key `aws:RequestedRegion` specifying `StringNotEquals` for `us-east-1` and `eu-west-1`. Because SCPs act as a master filter over all IAM policies in the child accounts, this explicitly blocks the action regardless of the local user's Administrator privileges.
The Load Balancer's health check will fail for the crashed server. The Load Balancer will immediately stop sending traffic to the dead server and distribute 100% of the traffic evenly between the two remaining healthy servers.
Security. If the servers were in the public subnet, hackers could bypass the Load Balancer and attack the servers directly via SSH or open ports. By putting the servers in a private subnet, the Load Balancer becomes the *only* possible way into the system.
An ALB is a physical/managed AWS resource that balances traffic across EC2 instances. An Ingress Controller is a software router running *inside* the cluster that balances traffic across Pods. In a production EKS/kubeadm setup, you typically use both: The ALB receives public traffic and routes it to the EC2 nodes on a NodePort. The Nginx Ingress Controller running on those nodes receives the traffic and uses internal Kubernetes DNS to route it to the specific Pods. Alternatively, using the AWS Load Balancer Controller with "IP Mode", the ALB can bypass Nginx entirely and route traffic directly to the Pod IPs via the CNI.
You must decouple the traffic from the compute layer using Edge caching and Web Application Firewalls (WAF).
Giving a database a public IP is the number one cause of enterprise data breaches. Hackers scan the internet for open Port 3306 (MySQL). The database must be in a Private Subnet. Developers should use an AWS Client VPN or an SSM Session Manager jump-host to access it.
**Multi-AZ** is for Disaster Recovery; it creates a hidden, synchronous standby node that you cannot read from. If the primary dies, AWS fails over to the standby automatically. **Read Replicas** are for Performance; they create asynchronous copies of the database that you *can* read from. If your application has heavy read traffic (like generating reports), you point the read queries to the Read Replica to take the load off the Primary.
In our code, `password = var.db_password`. This is a variable. We never hardcode it. In production, we execute Terraform via a CI/CD pipeline. The pipeline pulls the password securely from HashiCorp Vault or AWS Secrets Manager, and injects it into Terraform at runtime as an environment variable (`TF_VAR_db_password`). Furthermore, the state file must be stored in an encrypted S3 bucket, because the password *will* be stored in plaintext inside `terraform.tfstate`.
Standard RDS Multi-AZ uses block-level synchronous replication, which is sensitive to heavy I/O spikes. If write latency is unacceptable, the architecture must transition from standard RDS MySQL to **Amazon Aurora MySQL**. Aurora decouples compute from storage. It does not use block-level replication to a standby node. Instead, it writes redo logs directly to a distributed storage fleet spanning 3 AZs (6 copies of data). This eliminates the traditional replication lag bottleneck entirely, resulting in vastly higher write throughput and sub-10 millisecond failover times.
Base64 encoding is just a way to translate text into a format computers can easily transmit; anyone can instantly translate it back without a key. Encryption (like AES-256) scrambles the data mathematically; you cannot read the data unless you possess the specific cryptographic key used to lock it. Kubernetes Secrets use Base64. AWS Secrets Manager uses Encryption.
GitOps requires all YAML to be in Git. ESO allows us to store an `ExternalSecret` YAML in Git that contains *pointers* to AWS (e.g., `key: prod/rds`), rather than the actual passwords. ArgoCD syncs the pointers, and ESO dynamically fetches the real passwords from AWS at runtime.
While ESO successfully pulled the new password and updated the Kubernetes `Secret`, the Java Spring Boot application only reads environment variables or mounted files *at startup*. It does not hot-reload secrets. You must architect a solution to restart the Pod when the Secret changes. Tools like `Reloader` (Stakater) watch Kubernetes Secrets; when a Secret updates, `Reloader` automatically performs a rolling restart of the associated Deployment, ensuring the Java app reads the new password without downtime.
The Kubernetes API Server is configured as an OpenID Connect (OIDC) Identity Provider. AWS IAM is configured to trust this specific OIDC provider endpoint. When a Pod (with a designated ServiceAccount) is scheduled, the kubelet injects a cryptographically signed JWT into the Pod's filesystem. The AWS SDK inside the Pod reads this JWT and sends an `AssumeRoleWithWebIdentity` API call to AWS STS. STS verifies the JWT's signature against the Kubernetes API Server's public OIDC discovery keys. If valid, STS returns temporary AWS session credentials to the Pod. This entire chain relies on cryptographic trust, eliminating the need to ever create or rotate long-lived static AWS access keys.
No. There are still physical servers in an AWS data center. "Serverless" simply means the cloud provider completely manages the servers, the operating system, and the capacity provisioning. The customer only manages the application code.
A Cold Start occurs when a Lambda function is triggered after being idle. AWS must allocate compute resources, download the deployment package, start the runtime environment, and run initialization code. This initialization process introduces latency (delay) to the request. Subsequent requests to the same warm instance do not experience this delay.
No. AWS Lambda has a strict hard timeout of 15 minutes. If the process takes 25 minutes, AWS will violently terminate the execution at the 15-minute mark. For long-running serverless tasks, you should use AWS Fargate (Serverless Containers) or AWS Step Functions to orchestrate a distributed workflow.
It is the component attached to a VPC that provides a target in your VPC route tables for internet-routable traffic, and performs network address translation (NAT) for instances that have been assigned public IPv4 addresses.
A NAT Gateway is used for IPv4 traffic. It translates the private IPv4 address of an instance to its own elastic public IPv4 address. An Egress-Only Internet Gateway is exclusively for IPv6 traffic. Because IPv6 addresses are globally routable (no private IPs), NAT is mathematically unnecessary. The Egress-Only IGW simply acts as a stateful router that allows outbound IPv6 traffic to the internet but blocks inbound IPv6 connections.
Because the CIDR blocks overlap identically, you cannot use VPC Peering or a Transit Gateway directly; the routing tables would have no idea which `10.0.x.x` IP to send traffic to.
`clean` deletes the `target/` directory so nothing stale is packaged. `package`
Docker layer caching. Dependency resolution is expensive and changes rarely;
Coverage data is not being produced or not being found. Most likely the
Four things. Pin every dependency to a release version — no `SNAPSHOT` and no
In the cloud, you pay by the second. A weekend is roughly 60 hours. If a Development cluster costs $5 an hour, leaving it running while all the developers are at home sleeping costs the company $300 every single weekend for absolutely no reason.
A `request` is a guaranteed reservation; the scheduler uses it to find a node with enough available capacity to host the pod. A `limit` is a hard boundary enforced by the OS; if the pod exceeds its memory limit, it is OOMKilled. If it exceeds its CPU limit, it is throttled.
Orphaned resources are infrastructure components that are no longer actively used by any application but continue to generate hourly billing charges.
The high NAT Gateway cost is almost certainly caused by Pods in the private subnets pulling massive Docker images from ECR, or downloading large files from S3. Because these AWS services live outside the VPC, traffic routes through the NAT Gateway.
An image is an immutable stack of read-only layers — the template. A container is a running instance of that image with a thin writable layer on top. The writable layer is discarded when the container is removed, which is why persistent data needs a volume.
Docker caches a layer per instruction and invalidates everything below the first change. Copying source before installing dependencies means every code edit re-downloads dependencies. Copy the manifest, install, then copy source — the difference between a 30-second and a 6-minute build.
Use a multi-stage build so compilers and build dependencies never reach the final image; start from a minimal base (alpine or distroless); combine `RUN` steps; add a `.dockerignore`; and run as a non-root `USER`. Smaller images pull faster and have far less to patch.
Kubernetes removed dockershim in 1.24, so it no longer uses the Docker *daemon* as a runtime; it talks to containerd through the CRI. Images are unaffected — they are an OCI standard, so an image built with `docker build` runs unchanged. The practical impact is on the node: `docker ps` no longer shows Kubernetes containers, `crictl ps` does.
No. Layers are additive — the deletion is a marker in a higher layer, and the original file is still in the lower one and extractable from the image. The secret must be rotated, and builds must take secrets from build secrets or the runtime environment instead.
An Image is the static, read-only file (the blueprint). A Container is the running, active process created from that Image.
The Docker philosophy is "One Process Per Container." If you put both in one container, they compete for resources, you cannot scale the web server without also scaling the database, and if the web server crashes, the container dies, taking the database down with it.
Every line in a Dockerfile (e.g., `RUN`, `COPY`) creates a new read-only filesystem layer. Docker caches these layers. If you change a line of code at the very end of the Dockerfile, Docker instantly reuses the cached layers above it, making the build take 1 second. If you change a line at the *top* of the Dockerfile (like updating the base image), it invalidates the cache for *every layer below it*, forcing a completely new, slow build. Therefore, files that change frequently (source code) should be `COPY`'d at the very bottom of the Dockerfile.
Standard Docker containers share the exact same Linux Kernel as the host. Despite cgroups and namespaces, a kernel-level vulnerability (e.g., Dirty COW) allows a tenant in Container A to crash the kernel, taking down Container B (another tenant). AWS Firecracker uses KVM to launch hardware-virtualized MicroVMs in milliseconds. Each MicroVM runs its own isolated kernel. This provides the security boundary of a full Virtual Machine with the speed and density of a Docker container, making it the required architectural choice for true multi-tenant untrusted workloads (e.g., AWS Lambda).
`merge` creates a commit joining two histories and preserves what actually happened. `rebase` replays your commits on top of another branch, producing a linear history but rewriting commit hashes. Rebase your own unpushed work; never rebase a shared branch other people have pulled.
Treat the secret as compromised and rotate it first — that is the only step that actually protects you. Then remove it from history (`git filter-repo` or BFG) and force-push. Deleting the file in a new commit does nothing: the value is still in history and in every clone.
`git fetch` downloads commits from the remote but does not change your working
`git revert <sha>` and push. That creates a new commit undoing the change,
Because the pipeline itself creates that commit. Without the marker, the
You control write access to the branch ArgoCD watches, because in this model
They must be placed inside the `.github/workflows/` directory at the root of the repository.
A Job is a collection of Steps that run on a single virtual machine (runner). Multiple Jobs run in parallel by default on separate virtual machines. A Step is a sequential, individual task (like running a shell script or calling a pre-built Action) within that Job.
With static IAM User keys, you must store long-lived secrets in GitHub. If they leak, your AWS account is compromised. OIDC establishes a trust relationship. GitHub acts as the Identity Provider (IdP). The workflow requests a short-lived JWT from GitHub. AWS IAM verifies the cryptographically signed JWT. If it matches the Trust Policy (e.g., verifying the repo name), AWS STS issues temporary session credentials valid for only 1 hour. There are zero long-lived secrets to rotate or steal.
You implement **Required Workflows** (a feature of GitHub Enterprise). You define the security scanning workflow in a centralized `.github` repository for the organization. You then configure the Organization's Branch Protection Rules to require that specific workflow to pass before any Pull Request can be merged into `main`. Developers cannot bypass or remove this workflow from their local repositories.
It replaces push with pull and makes drift visible. A controller in the cluster continuously compares running state to Git and reports or corrects the difference, so Git is genuinely the source of truth. It also means CI credentials never need cluster access — the cluster pulls, rather than the pipeline pushing.
Argo CD marks the application OutOfSync, and with `selfHeal: true` reverts the change within minutes — correct behaviour that feels hostile the first time. The right response is to commit the fix so Git and the cluster agree. Genuinely controller-owned fields, such as replicas managed by an HPA, belong in `ignoreDifferences`.
ArgoCD immediately notices that the Deployment is missing from the cluster but still exists in the Git repository. Because Git is the Single Source of Truth, ArgoCD instantly recreates the Deployment, effectively fighting off the hacker automatically.
Configuration Drift happens when the actual running state of an environment differs from the documented or desired state (e.g., an engineer manually patches a server in the middle of the night but forgets to update the codebase). GitOps eliminates drift through continuous reconciliation.
You use branching or folder structures. A common pattern is folder-based: `namespaces/dev`, `namespaces/staging`, `namespaces/prod`. When Jenkins tests `v2.0` in Dev, it updates the YAML in the `dev/` folder. A human engineer then opens a Pull Request to copy that YAML change into the `prod/` folder. When the PR is approved and merged, ArgoCD detects the change in the `prod/` folder and deploys it.
In a Push-based model, the CI/CD server sits outside the cluster (e.g., Jenkins in a management VPC). To deploy, the Kubernetes API server must be reachable from Jenkins, requiring inbound firewall rules. Furthermore, Jenkins must hold highly privileged cluster-admin credentials. If Jenkins is compromised, the attacker owns the cluster. In a Pull-based GitOps model, ArgoCD lives inside the cluster. It polls Git via outbound HTTPS only. The cluster API requires zero inbound firewall ports from the CI system, and the CI system holds zero cluster credentials, drastically reducing the IAM blast radius and shrinking the network attack surface.
No. Grafana is just the UI layer. It relies entirely on backend data sources (like Prometheus or AWS CloudWatch) to store the actual data.
You practice "Dashboards as Code". You export the dashboard as a JSON file, commit it to your Git repository, and deploy it to the cluster via a `ConfigMap`. Grafana reads the ConfigMap on boot.
You are currently graphing by the `pod` label, which is ephemeral (e.g., `pod="api-1234"`). When the pod dies, that specific time series permanently ends. To create a continuous line, you must aggregate away the ephemeral labels using `sum by ()` or `avg by ()`. You modify the query to aggregate by the static `app` label instead: `sum by (app) (rate(container_cpu_usage_seconds_total{app="ivolve-api"}[5m]))`. This combines all the pods into a single logical metric that survives pod churn.
The developers are likely running highly complex, long-range aggregation queries (e.g., calculating the 99th percentile of 100,000 time series over a 30-day window) directly inside the Grafana panels. Every time they refresh the dashboard, Prometheus has to recalculate the massive equation on the fly.
It is the configuration file for a Helm chart. It allows you to customize the application (like changing the password, the number of replicas, or the memory limits) without having to edit the actual underlying YAML templates.
Helm is a templating engine (it uses `{{ }}` brackets to inject variables into templates). Kustomize is an overlay engine (it takes standard, valid YAML files and mathematically merges new lines into them). Helm is better for sharing generic packages publicly. Kustomize is often better for simple, internal environment variations (dev/prod).
`helm install` talks directly to the Kubernetes API and deploys the application. `helm template` renders the Go templates locally on your machine and simply prints the raw YAML to the screen, without connecting to a cluster. In CI/CD, we run `helm template` and pipe the output to a tool like `kubeval` or `polaris` to validate the syntax and security of the generated YAML *before* it is ever deployed.
Helm's `lookup` function attempts to query the live Kubernetes API during template rendering to fetch data (like an existing Secret). GitOps tools like ArgoCD render Helm charts purely deterministically, often in a vacuum without live cluster access, causing `lookup` to fail or produce non-deterministic results (Configuration Drift). Similarly, Helm lifecycle hooks rely on the Helm binary orchestrating the deployment order. ArgoCD bypasses the Helm binary, using it only for rendering, and applies the YAML using its own synchronization waves. Therefore, Helm hooks are often ignored or mishandled by GitOps tools, requiring a migration to ArgoCD SyncHooks instead.
Integration merges and verifies every change automatically. Delivery keeps every verified change *releasable*, with a human choosing when. Deployment removes that human — every change that passes goes to production. Most teams practise delivery and describe it as deployment.
It fails the build. A stage that reports findings and exits zero is a notification, not a gate — so scanners run with `--exit-code 1` and analysis waits for its verdict rather than firing and forgetting. Gates should also be scoped to new code, or a legacy codebase makes them unadoptable and they get disabled.
Inject them at run time from a credential store and never write them to disk or the log. Prefer identity over secrets entirely — an IAM role assumed by the runner, or OIDC federation from the CI provider — so there is no long-lived key to rotate or leak.
A text file that contains the definition of a Jenkins Pipeline. It is committed to source control (Git) alongside the application code.
Declarative pipelines (starting with `pipeline {}`) have a strict, rigid syntax defined by CloudBees. They are easier to read and have built-in error checking. Scripted pipelines (starting with `node {}`) are raw Groovy code. They offer infinite flexibility but are much harder to maintain and debug.
Jenkins uses the JNLP (Java Network Launch Protocol) / Remoting architecture. The Jenkins Kubernetes plugin makes an API call to the Kubernetes cluster to create the Pod. The Pod contains a special hidden container (the `jnlp` container). Once booted, this `jnlp` container establishes a reverse TCP connection back to the Jenkins Master's JNLP port (usually 50000). The Master then sends the shell commands over this established TCP tunnel.
Increasing the heap is a band-aid. Master instability is almost always caused by executing heavy logic on the Master instead of the Agents.
A Pod is one or more containers sharing a network namespace and storage, always scheduled together and reachable on `localhost`. It is the smallest unit Kubernetes schedules. Usually there is one container; the exception is a sidecar that needs the same lifecycle and localhost access.
A Deployment declares desired state and manages rollouts; it creates a ReplicaSet per version, and the ReplicaSet keeps the requested number of Pods running. Rolling back works because the previous ReplicaSet is retained at zero replicas.
Liveness failing restarts the container — for a process that has deadlocked. Readiness failing removes the Pod from Service endpoints without restarting it — for a temporarily busy or dependency-blocked process. Startup suspends the other two while a slow application boots. Pointing liveness at a database check turns a slow database into a cluster-wide restart storm.
`kubectl get endpoints <service>` first. An empty list means the selector matches no ready Pod — a label mismatch or a failing readiness probe — and the network is not involved at all. If endpoints exist, check `targetPort` against the container's actual port, then test DNS from inside a Pod.
`ClusterIP` for internal traffic, which is nearly everything. `NodePort` mainly for debugging or behind an external balancer. `LoadBalancer` provisions a real cloud load balancer per Service, which is why production uses one Ingress in front of many ClusterIP Services rather than a LoadBalancer each.
`kubectl logs <pod> --previous` — the current container has just started and knows nothing; the evidence is in the one that died. Then `kubectl describe pod` for events and the exit code: 137 is OOMKilled (raise the memory limit or fix the leak), a config or missing-secret error appears in the logs, and a liveness probe that is too aggressive shows as repeated restarts of an otherwise healthy process.
The container exceeded its memory limit and the Linux kernel's OOM killer terminated it inside its cgroup — Kubernetes only reports what the kernel did. Fix it by raising the limit if the workload genuinely needs it, or by fixing the leak. Setting requests and limits equal gives the Pod the Guaranteed QoS class and makes it the last to be evicted.
A Role (or ClusterRole) is a list of permissions; a RoleBinding attaches it to a user, group or ServiceAccount. RBAC is purely additive — a subject can do the union of what its bindings grant — so you restrict by granting less, never by subtracting. Verify with `kubectl auth can-i --list --as system:serviceaccount:ns:name`.
By default they are base64-encoded, not encrypted — anyone who can read the Secret can decode it instantly. Real protection needs encryption at rest in etcd, RBAC limiting who can read Secrets, `automountServiceAccountToken: false` where the API is not used, and ideally an external store such as AWS Secrets Manager.
etcd needs a strict majority to accept a write. Three nodes tolerate one failure; four also tolerate only one, because losing two of four leaves no majority. A two-node 'HA' cluster is less available than a single node — it can lose quorum and refuse writes while both machines are still running.
`kubeadm` builds the cluster (you run it once). `kubectl` talks to the cluster after it is built (you run it every day to deploy apps).
This is the `kubeconfig` file. It contains the cluster's IP address and the administrative cryptographic certificates required to authenticate as the `kubernetes-admin` user. Without this file, `kubectl` will return a "Connection Refused" or "Unauthorized" error.
Kubernetes is a highly precise orchestrator. It needs to know exactly how much RAM every Pod is using to make scheduling decisions and enforce memory Limits. If the Linux kernel starts moving RAM into a Swap file on the hard drive, Kubernetes loses track of the memory consumption, leading to severe performance degradation and unpredictable `OOMKilled` behavior. (Note: As of Kubernetes v1.28+, swap support is available in beta, but disabling it remains the standard best practice).
A Pod is the smallest deployable unit in Kubernetes. It is a wrapper around one or more containers. We use Pods because Kubernetes needs to attach metadata (labels, IP addresses, storage volumes) to the workload, and it attaches these to the Pod, not the raw Docker container.
A Deployment is for stateless applications (like web servers); the Pods are identical, interchangeable, and have random names (e.g., `api-5f8g9`). A StatefulSet is for stateful applications (like databases); the Pods have sticky, predictable identities (e.g., `db-0`, `db-1`) and are guaranteed to start and stop in a strict order.
A Service gets a virtual IP address (ClusterIP) managed by `kube-proxy`. The Service has a label selector (e.g., `app: my-api`). The Kubernetes Endpoints controller continuously watches for Pods matching that label. When it finds them, it adds their real IPs to an `Endpoints` object. When traffic hits the Service IP, `kube-proxy` (using Linux `iptables` or `IPVS`) load-balances the TCP packets to the IPs in the Endpoints list.
The `Pending` state means the Scheduler cannot find a Worker Node with sufficient aggregate CPU/RAM capacity to satisfy the Pod's `requests`. The cluster is out of physical resources. To automate resolution, we must implement the **Cluster Autoscaler** (or Karpenter in AWS). The Cluster Autoscaler watches for `Pending` pods. When it sees them, it makes an API call to the AWS Auto Scaling Group to provision a new EC2 instance. Once the new EC2 instance boots and joins the cluster, the Kubernetes Scheduler recognizes the new capacity and schedules the pending Pods onto it.
A sidecar is a secondary container that runs inside the exact same Pod as the primary application container. It shares the same network namespace and IP address. It is used to add functionality (like logging or proxying) without changing the main application code.
Because the Service Mesh Proxy (Envoy) intercepts 100% of the HTTP traffic going in and out of the Pod, the proxy can natively measure how many requests are failing and how long they take. The proxy exposes these metrics to Prometheus. The developer gets golden signal metrics (Latency, Traffic, Errors) without having to import an OpenTelemetry SDK into their Java/Python code.
An API Gateway handles "North-South" traffic (traffic entering the cluster from the public internet). It focuses on external authentication, rate limiting, and monetization. A Service Mesh handles "East-West" traffic (internal traffic between microservices). It focuses on mTLS, internal retries, and circuit breaking. In a mature enterprise, you absolutely need both, as they solve fundamentally different problems.
The `kubelet` (running on the Worker Node) executes the `livenessProbe` by sending a plaintext HTTP request to the Pod's health endpoint. Because Istio is enforcing strict mTLS, the Envoy proxy intercepts the kubelet's plaintext request and rejects it, causing the probe to fail.
The base is a complete, valid set of manifests that could be applied as-is.
For manifests I own, deployed to environments I control. The base stays
Because `sed` matches text and `kustomize edit` understands structure. A `sed`
Structurally, by making divergence expensive to express. The base holds
Every running program is a process with a numeric PID. A daemon is simply a process that runs in the background with no controlling terminal — a service. `dockerd`, `kubelet` and a database server are all daemons.
`df -h` shows which filesystem is full, then `du -sh /var/*` narrows it down directory by directory. The usual culprits are `/var/log` and unpruned container images. A full disk breaks things that look unrelated — Docker cannot pull, Kubernetes evicts Pods, databases refuse writes.
Each digit sums read (4), write (2) and execute (1) for owner, group and others. `755` gives the owner everything and everyone else read and execute. `777` grants write access to every process on the machine, including a compromised one — the fix is nearly always correct ownership (`chown`) plus a shared group, not wider permissions.
It was started but never enabled. `systemctl start` runs it now; `systemctl enable` registers it to start at boot. They are independent, and confusing them produces a service that runs perfectly for months and then never comes back after a 3am reboot.
`SIGTERM` (`kill`) asks a process to shut down cleanly; `SIGKILL` (`kill -9`) terminates it immediately with no chance to clean up. Kubernetes sends SIGTERM when removing a Pod and waits `terminationGracePeriodSeconds` before SIGKILL. An application that ignores SIGTERM drops in-flight user requests on every deploy.
`sudo` stands for "SuperUser Do". It allows a normal user to execute a single command with root (Administrator) privileges.
You use the `tail` command with the follow flag: `tail -n 20 -f /var/log/syslog`.
`OOMKilled` stands for Out Of Memory Killed. When a container exceeds its defined memory limit, the Linux Kernel's Out-Of-Memory Killer process intervenes to protect the host node from crashing. It terminates the offending process inside the `cgroup`. Kubernetes detects the exit code and updates the Pod status to `OOMKilled`.
A Virtual Machine utilizes a Hypervisor (like ESXi or KVM) to emulate physical hardware. Every VM runs a complete, heavy, independent Operating System kernel. A Container (like Docker) does not emulate hardware. It uses the Host's existing Linux Kernel, utilizing `Namespaces` for isolation and `cgroups` for resource limitation. Because containers share a single kernel, they boot in milliseconds and have vastly less overhead than VMs.
`kubectl logs` only reads the logs that are currently on the Worker Node's hard drive. If the Pod crashes, scales down, or the Node itself dies, the logs are permanently deleted. You need a centralized system to store the logs off the cluster.
A DaemonSet ensures that exactly one copy of a Pod runs on every single node in the cluster. Because logs are generated by applications on every node and saved to that specific node's local hard drive, you need a Log Agent running on every single node to read those local files.
Elasticsearch is an inverted index search engine; it indexes every word of every log line, which requires massive compute and RAM, often making the logging cluster more expensive than the actual application cluster. Loki only indexes the metadata labels (like the Kubernetes namespace and pod name), leaving the actual log text unindexed and compressed in cheap object storage (S3). It trades query speed for massive ingestion efficiency and cost reduction, which aligns perfectly with Kubernetes label architectures.
You implement a **Log Routing/Observability Pipeline** (using tools like Vector, FluentBit, or Cribl LogStream) *before* the logs reach the expensive SIEM/Search database.
DNS resolves the name to an IP (resolver → root → TLD → authoritative, cached at every step by TTL); TCP connects; TLS negotiates encryption and verifies the certificate chain; HTTP carries the request; the server responds. Most 'network' incidents are really DNS caching or certificate expiry.
Its route table. A public subnet has a `0.0.0.0/0` route to an Internet Gateway; a private one either routes `0.0.0.0/0` to a NAT Gateway (outbound only) or has no default route at all. The name is a convention — the route is the mechanism.
A security group attaches to an instance, is stateful (return traffic is automatic) and only has allow rules. A NACL attaches to a subnet, is stateless (you must allow the reply explicitly) and supports deny rules. Forgetting the return rule on a NACL produces a hang, not a refusal, which sends people debugging the application.
Resolvers are still serving the old answer until its TTL expires. Check the remaining TTL with `dig`. The fix is procedural: lower the TTL a day *before* a migration, cut over, then raise it again — lowering it at cutover is too late, because the old long TTL is already cached.
The first 24 bits are the network portion, leaving 8 bits for hosts — 256
Through a NAT gateway in a public subnet. The private instance's route table
They are separate routing domains sharing one kernel routing table. On
Allocate from a single `/8` and delegate downward: a `/12` per
Three reasons. Availability — a Central outage would stop every build in the
A release repository holds immutable versions — write-once, never
An attacker publishes to a public registry using the name of an internal
You have to have prepared for the question before it was asked. Concretely:
Monitoring answers questions you knew to ask — dashboards and thresholds for known failure modes. Observability is whether you can answer *new* questions from the telemetry you already emit, without shipping code. High-cardinality, well-structured data is what makes the difference.
Symptoms users are feeling — error rate, latency, an SLO burning fast — not causes like CPU at 90%. The test is whether the recipient can act on it now and whether it matters if they do not. Alerting on causes is the fastest route to a team that ignores its alerts.
A count threshold fires on a busy night and stays silent during a quiet outage. A ratio means the same thing at any traffic level, and a burn rate expresses how fast you are consuming the error budget — so a page fires in proportion to how much trouble you are actually in, and a slow leak becomes a ticket rather than a 3am call.
Metrics, Logs, and Traces.
Separation of concerns. Prometheus is a highly specialized time-series database optimized for scraping and storing millions of numbers per second. It is terrible at making pretty graphs. Grafana is a dedicated UI tool that is amazing at graphs, but it doesn't store data. Grafana can also query data from other sources (like AWS CloudWatch or MySQL) simultaneously.
In a Push model (like DataDog or StatsD), the application actively connects to the monitoring server and sends its data. If you have 10,000 Pods, they can easily DDoS your monitoring server. In a Pull model (Prometheus), the monitoring server reaches out to the applications on a schedule (e.g., every 15 seconds) and scrapes the data. This allows Prometheus to rate-limit the ingestion and prevents the monitoring system from being overwhelmed by the applications.
You must decouple local ingestion from global querying.
DevOps is a culture where Dev and Ops work closely together (often combining the roles). Platform Engineering is a specialized discipline where a dedicated team builds a self-service product (the platform) for the developers to consume, reducing the cognitive load on the developers.
An IDP is a self-service portal (often built with tools like Backstage) that standardizes and automates the creation of infrastructure, CI/CD pipelines, and application scaffolding. It allows developers to spin up new, fully-compliant microservices in minutes without needing to understand Terraform or Kubernetes.
Strict IT mandates force developers to use specific tools, causing friction and slowing down innovation. A Golden Path is a fully supported, highly automated, secure route provided by the Platform team. Developers are technically allowed to choose different tools (going off-road), but they lose the automation, support, and on-call protection of the Golden Path. Ultimately, 95% of developers will voluntarily choose the Golden Path simply because it is the easiest way to get their job done.
Adoption requires treating the Platform as a Product.
IaaS (Infrastructure as a Service) is renting the raw hardware (like AWS EC2). PaaS (Platform as a Service) provides the hardware *and* the operating system/tools so developers can just focus on code (like Heroku, or the Kubernetes platform we are building in this project).
They serve different purposes. Terraform is an Infrastructure Provisioner (it declares the existence of a server, network, or database in AWS). Ansible is a Configuration Manager (it connects to the server Terraform created and installs software on it). While they have overlapping features, using Terraform for hardware and Ansible for software is the industry standard.
Our Control Plane utilizes `etcd` as its distributed key-value store. `etcd` requires a strict quorum (majority) to elect a leader and commit changes (using the Raft consensus algorithm). By distributing exactly 3 Control Plane nodes across 3 Availability Zones, the system can sustain the total loss of 1 AZ. The remaining 2 nodes maintain a quorum (2/3) and the cluster remains fully operational and writeable.
A GitOps-driven Self-Managed cluster (our architecture) shifts the operational burden to the Platform team (managing etcd backups, API certificate rotations, and kubelet upgrades). However, it guarantees absolute configuration synchronization; the cluster continuously pulls state from Git, neutralizing configuration drift and ensuring disaster recovery is deterministic. EKS reduces the control plane management burden significantly, but if paired with a push-based CI (e.g., Jenkins running `kubectl apply`), it introduces security vulnerabilities (CI requires cluster admin credentials) and risks drift if engineers manually edit cluster state via the AWS console. The architectural tradeoff is Operational Effort (Self-Managed) vs. Security/Determinism (GitOps).
Organization and predictability. If a team member needs to fix a Jenkins pipeline, they know exactly where to look (`jenkins/pipelines/`) without having to search through hundreds of Terraform and Kubernetes files.
`modules` contains generic, reusable templates (like a blueprint for a house). `environments` contains the specific instances of that blueprint (like building the house at a specific address in `dev` or `prod`). You write the code once in `modules`, and call it multiple times from `environments`.
If you use the Terraform `kubernetes` provider to apply YAML manifests in the exact same `main.tf` file that builds the EKS/kubeadm cluster, Terraform will evaluate the plan before creating anything. It will try to connect to the Kubernetes API to plan the YAML changes, but the API doesn't exist yet because the cluster hasn't been built. This causes Terraform to crash. This is why we physically separate `infrastructure/` from `kubernetes/` in the repository structure.
You never store plaintext secrets in the repo. You structure the repo to integrate with a dynamic secrets manager (like HashiCorp Vault or AWS Secrets Manager). For Terraform, you use data sources to fetch secrets at runtime. For Kubernetes, you use the External Secrets Operator (ESO) configured in the `kubernetes/` directory. ESO authenticates with AWS Secrets Manager via IRSA (IAM Roles for Service Accounts) and dynamically injects the secrets into the cluster memory, keeping the Git repository completely devoid of sensitive data while maintaining a unified directory structure.
The AWS Command Line Interface allows you to type commands into your terminal to control AWS, instead of clicking around the website. Tools like Terraform use these credentials in the background to automatically build servers.
You missed the SSH Key requirement. Ansible is trying to log into the AWS EC2 instance, but it doesn't have the correct private SSH key (`.pem` file) that corresponds to the public key injected into the server by AWS.
This causes State File corruption. If an engineer uses Terraform 1.6, the remote `terraform.tfstate` file is upgraded to the 1.6 format. When the engineer with 1.4 tries to run a command, Terraform will throw an error and refuse to run, because older versions cannot parse newer state file formats. Version locking is critical.
You use OIDC (OpenID Connect). The CI/CD provider (e.g., GitHub Actions) acts as an Identity Provider. AWS IAM is configured to trust the GitHub OIDC provider for a specific repository. When a workflow runs, it requests a short-lived JSON Web Token (JWT) from GitHub, presents it to AWS STS (Security Token Service), and receives temporary, short-lived session credentials. This eliminates the risk of static keys leaking in source code.
A Public Subnet has a routing table entry pointing to an Internet Gateway, allowing servers inside it to be reached from the public internet. A Private Subnet does not; its servers are completely hidden from the outside world.
Private servers often need to download updates or API data from the internet. They send their request to the NAT (Network Address Translation) Gateway. The NAT Gateway, sitting in the Public Subnet, acts as a middleman. It forwards the request to the internet on behalf of the private server, and sends the response back, ensuring the private server's IP address is never exposed.
You can use VPC Peering (for simple 1-to-1 connections) or AWS Transit Gateway (for complex hub-and-spoke topologies). Both route traffic entirely across the internal AWS backbone. Alternatively, for exposing specific services, AWS PrivateLink allows one VPC to consume an endpoint in another VPC securely.
Routing is handled via Route53 latency-based or geolocation routing. For stateful data, we must utilize global databases (like Amazon Aurora Global Database or DynamoDB Global Tables) which replicate storage synchronously or asynchronously at the block level. The architectural tradeoff is latency vs. consistency (CAP Theorem): synchronous replication guarantees consistency but adds high latency across regions, whereas asynchronous replication is fast but risks data loss during an abrupt region failure.
A database specifically optimized for handling data indexed by time. Instead of storing complex relationships between tables (like a relational DB), it just stores a timestamp, a metric name, and a value (e.g., `[12:00:00], CPU, 80%`).
The Prometheus Operator automates Prometheus configuration. Instead of manually editing `prometheus.yml`, developers deploy a `ServiceMonitor` Kubernetes object. The Operator watches for these objects, dynamically discovers the associated Kubernetes Services via labels, and instructs Prometheus to begin scraping them without any downtime or manual restarts.
A "Counter" metric in Prometheus only ever goes up (e.g., total HTTP requests served since the server booted). If you just graph the raw counter, you get a useless diagonal line that goes up forever. `rate()` calculates the per-second average rate of increase of the time series in the specified range. It answers "How many requests per second are we getting right now?", which is the actual actionable data you need for a dashboard.
Cardinality Explosion occurs when a metric has too many unique label combinations. For example, if a developer adds a `user_id` label to an HTTP request metric, and you have 1 million users, Prometheus must create and hold 1 million distinct time-series in RAM, instantly causing an OOM.
At three points: on pull requests, in the build pipeline before the push, and continuously in the registry. The third is the one teams forget — an image that passed on Monday can be critically vulnerable on Friday without a single line changing, because the vulnerability was published, not introduced.
`runAsNonRoot: true`, a numeric `runAsUser`, `allowPrivilegeEscalation: false`, `readOnlyRootFilesystem: true`, and `capabilities: drop: ["ALL"]`, adding back only what genuinely breaks. Enforce it at the namespace boundary with Pod Security Admission rather than relying on every author remembering.
A Pod selected by no policy accepts all traffic; security starts only once something selects it, which is why the first policy you write is a default-deny. Policies are additive with no deny rule — traffic is allowed if any policy allows it — and an egress policy that forgets DNS to CoreDNS breaks every hostname lookup in the namespace.
If a hacker finds a vulnerability in your application and breaks in, they inherit the permissions of the user running the application. If the user is `root`, the hacker can install malware, alter files, or attempt a container-escape attack to take over the underlying host node.
It refers to moving security checks (like vulnerability scanning and static code analysis) to the earliest possible stages of the software development lifecycle (e.g., local IDEs, Git pre-commit hooks, and CI pipelines), rather than waiting for a QA or Security audit right before production deployment.
You do not remove `readOnlyRootFilesystem: true`. Instead, you provide a temporary, isolated writable space specifically for that folder by mounting an `emptyDir` volume backed by memory (tmpfs) to the `/tmp` path in the Pod specification. The rest of the OS remains strictly read-only.
Traditional sidecar security requires injecting a proxy container into every single Pod. This consumes massive overhead (CPU/RAM per pod) and only has visibility into network traffic or specific application layers; it cannot easily see kernel-level file modifications or process executions.
A Default Deny policy is a baseline security rule that explicitly blocks all incoming (Ingress) and outgoing (Egress) traffic for all Pods in a namespace. Once applied, engineers must write explicit "Allow" rules to permit necessary traffic.
Kubernetes is an orchestrator, not a router. It relies on the Container Network Interface (CNI) to handle the actual packet routing. If you install a basic CNI like Flannel (which only does routing), policies are ignored. You must install an advanced CNI like Calico or Cilium, which integrates with the Linux Kernel (iptables/eBPF) to actively drop packets.
When you block all Egress traffic, you also block outbound UDP Port 53 traffic to the Kubernetes `CoreDNS` service. The Pod cannot resolve IP addresses. You must write an explicit Egress rule allowing outbound traffic on Port 53 (UDP/TCP) specifically to the `kube-system` namespace where CoreDNS resides.
In a traditional iptables CNI, every Network Policy translates into sequential iptables rules on the Linux node. In a massive cluster, evaluating a packet against 50,000 iptables rules takes a long time, causing severe CPU spikes and network latency (the `iptables` bottleneck).
RTO is how long you may take to restore service; RPO is how much data you may lose, measured in time. RTO drives standby capacity and automation, RPO drives backup and replication frequency. Both are business decisions, and a backup nobody has ever restored satisfies neither.
No. HA handles a component failing inside a region, automatically and in seconds. DR handles losing a region — or a bad decision such as a dropped table or ransomware — deliberately, in minutes to hours. HA replicates faults as faithfully as data, which is exactly why backups still exist.
Declare it out loud early, assign an incident commander who decides rather than debugs, and stop the bleeding before finding the cause — roll back or fail over first. Check what changed recently, because most incidents are a change. Communicate on a timer even when there is nothing new, then write a blameless postmortem with specific, owned, dated actions.
To discover hidden vulnerabilities, test automated failover mechanisms, and ensure the engineering team knows how to respond to an outage during normal business hours, rather than panicking at 3:00 AM during a real emergency.
The blast radius is the extent of the damage caused by a failure or a chaos experiment. In Chaos Engineering, you always start with the smallest possible blast radius (e.g., affecting one user or one pod) and slowly expand it (e.g., affecting an entire availability zone) as your confidence in the system's resilience grows.
The experiment exposed a lack of timeouts and Circuit Breaking in the Frontend code. When the network partitioned, the Backend stopped responding. The Frontend kept accepting new user requests, creating thousands of open HTTP connections waiting for a response that would never come. These open connections exhausted the RAM, causing the OOM. The fix is to configure strict network timeouts in the code or implement an Istio Circuit Breaker to instantly reject traffic when the backend is unreachable.
You start small and focus on the Staging environment. You do not touch Production.
High Availability (like having 3 servers behind a Load Balancer) protects against the failure of a single component. Disaster Recovery protects against the catastrophic failure of the entire primary location (like a data center burning down) by failing over to a secondary location.
**RPO (Recovery Point Objective):** The maximum acceptable amount of data loss measured in time (e.g., if you back up every hour, your RPO is 1 hour). **RTO (Recovery Time Objective):** The maximum acceptable amount of time the system can be offline before it causes unacceptable damage to the business.
ArgoCD only restores the stateless declarative configuration (the YAML files stored in Git). It has absolutely no knowledge of stateful data. If you have an application using a Persistent Volume (like a StatefulSet running a local database or file cache), ArgoCD will recreate the Pod, but the disk will be completely empty. Velero is required to snapshot and restore the actual block-storage data residing on the physical disks.
True Active/Active across regions is impossible for standard relational databases (like PostgreSQL) due to the CAP Theorem and the speed of light. If a user writes to US-East, and a user simultaneously reads from US-West before the data has time to cross the continent, they will see stale data. Attempting synchronous replication across regions will introduce massive write latency (destroying performance).
It maps your configuration to real resource IDs — Terraform's memory of what it created. Without it Terraform cannot tell an existing resource from one to create. It belongs in an encrypted remote backend with locking, never in Git: it frequently contains secrets in plain text.
State locking. With an S3 backend, a DynamoDB table holds the lock: whoever starts first acquires it and the second run fails with a clear message rather than writing concurrently. Locking is the reason a shared backend is safe to use.
Stop. `-/+` means destroy and recreate, because you changed an attribute that cannot be modified in place — on a database that is your data. Identify the forcing attribute in the plan output, and either revert it, use `ignore_changes`, or plan a proper migration with a snapshot first.
`terraform plan` queries the provider, finds the resource missing — this is configuration drift — and proposes recreating it to match the declared state. `terraform apply` repairs it. This self-healing property is the point of declarative infrastructure.
`terraform state mv` moves resource addresses between states, and `terraform import` adopts existing resources. Both change the mapping only — no cloud API calls that alter real infrastructure — so production keeps running while the layout changes.
Separate directories, usually. Workspaces share one configuration, so the only thing between a staging apply and a production apply is which workspace you are in — something you can forget. Directories make the environment explicit in the path and let production legitimately differ.
`plan` is a safe dry-run that tells you what will happen. `apply` actually executes those changes against the cloud provider.
It is Terraform's memory file that maps your code to real AWS resource IDs. You must never commit it to Git because it often contains plaintext secrets (like initial database passwords) that were returned by the AWS API during creation. It should always be stored in an encrypted S3 bucket.
You run `terraform plan`. Terraform will contact the AWS API, realize the instance is missing (detecting Configuration Drift), and the plan will show that it intends to recreate the missing instance to match the declarative code. Running `terraform apply` fixes it automatically.
You must decouple the state file using the `terraform state mv` command. You define the new backend configurations, then surgically move the resource addresses from the monolithic state file to the new micro-state files. This updates the memory mapping without touching the actual AWS APIs, ensuring zero downtime for the production resources.
`kubectl describe pod` to read the Events and see the last termination
`kubectl get endpoints <service>`. If it is empty, no pod is passing its
`describe pod` first for the Events — usually a missing Secret, an unbound
Three things, in order of value. First, every alert carries a `runbook_url`