Pods can crash unexpectedly. Stateful applications may enter CrashLoopBackOff. Deployments can silently disappear after failed rollouts. A container may remain stuck in Pending, Terminating, or an Unhealthy state without immediate awareness from the engineering team.

While Kubernetes provides native observability primitives, many teams still need a lightweight, automated alerting mechanism tailored to specific business-critical workloads — without introducing the operational overhead of managing a full monitoring stack with minimal cost.

To solve this, I built a serverless EKS pod monitoring solution using:

• AWS Lambda for health evaluation logic
• Amazon EventBridge for scheduled execution
• Amazon EKS API + Kubernetes Python Client for cluster inspection
• Slack Incoming Webhooks for alert delivery

The result is a production-friendly monitoring workflow that checks pod health every minute, compares the current state with the previous execution state, and sends actionable alerts only when something changes.

This solution detects scenarios such as the following:

✅ Pod crashes (CrashLoopBackOff)
✅ Pods stuck in unhealthy states (Pending, Terminating, etc.)
✅ Complete workload disappearance
✅ Recovery events when failed workloads become healthy again

Unlike heavyweight observability platforms that require additional infrastructure, this approach stays intentionally lightweight while still providing meaningful operational monitoring for critical Kubernetes workloads.

In this article, I’ll walk through the complete implementation — from architecture and RBAC configuration to Lambda authentication with EKS, Slack integration, Terraform setup, and production hardening considerations.

Overview

Objective

Implement an automated monitoring solution for critical Kubernetes pods running in an AWS EKS cluster.

The solution continuously monitors specific workloads inside a namespace and sends Slack alerts when:

• A pod enters unhealthy state
• A pod crashes
• A pod becomes unavailable
• A monitored workload disappears
• A failed workload recovers

This implementation uses:

• AWS Lambda
• AWS EventBridge
• AWS IAM
• AWS SSM Parameter Store
• AWS EKS API
• Kubernetes Python Client
• RBAC
• Slack Incoming Webhooks

Architecture

EventBridge (every 1 minute)
        ↓
AWS Lambda (Python 3.12)
        ↓
Generate EKS auth token via STS
        ↓
Call EKS DescribeCluster API
        ↓
Connect to Kubernetes API
        ↓
Read monitored pods
        ↓
Compare current state vs previous state (SSM)
        ↓
If unhealthy: Send Slack alert
        ↓
Save latest state in SSM

Monitored Workloads

Current monitored workloads: watched = [“mongodb”, “neo4jnode”]

Monitoring logic matches pod name prefixes.

Prerequisites

Before implementation, ensure:

• Existing AWS account
• Existing EKS cluster
• kubectl access to cluster
• Terraform configured for Kubernetes provider
• Slack workspace access
• Permission to create: IAM roles, Lambda, Lambda layers, EventBridge rules, SSM parameters

Terraform Kubernetes RBAC Configuration (only if you’re using Terraform)

Lambda authenticates to EKS using its IAM role.

EKS authentication succeeds via AWS IAM, but Kubernetes RBAC authorization must also allow pod access.

Add below Terraform:

resource "kubernetes_role" "wbd_lambda_pod_reader" {
  metadata {
    name      = "wbd-lambda-pod-reader"
    namespace = "wbd-kavia-clone"
  }

  rule {
    api_groups = [""]
    resources  = ["pods"]
    verbs      = ["get", "list", "watch"]
  }
}

resource "kubernetes_role_binding" "wbd_lambda_pod_reader_binding" {
  metadata {
    name      = "wbd-lambda-pod-reader-binding"
    namespace = "wbd-kavia-clone"
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "Role"
    name      = kubernetes_role.wbd_lambda_pod_reader.metadata[0].name
  }

  subject {
    kind      = "User"
    name      = "arn:aws:iam::031749757330:role/wbd-eks-monitor-role"
    api_group = "rbac.authorization.k8s.io"
  }
}

This grants Lambda permission to:

• get pods
• list pods
• watch pods

inside your namespace.

Lambda Layer Creation

Kubernetes Python dependencies are too large for inline Lambda packaging. That’s why we need to create Lambda layer.

Required dependencies:

Create local directory

mkdir k8s-layer
cd k8s-layer

Create python folder:

mkdir python

Install dependencies:

pip install kubernetes boto3 urllib3 -t python/

Zip:

zip -r k8s-python-layer.zip python

Upload Layer → AWS Console → Lambda → Layers → Create layer

Configuration:

Name: k8s-python-layer

Upload: k8s-python-layer.zip

Compatible runtime: Python 3.12

Architecture: x86_64

Create layer.

IAM Role Creation

Create Lambda execution role.

AWS Console → IAM → Roles → Create Role

Trusted entity: AWS Service

Service: Lambda

Role name: wbd-eks-monitor-role

Attach Managed Policy

Attach: AWSLambdaBasicExecutionRole

This grants:

• CloudWatch log creation
• CloudWatch log stream creation
• log publishing

Custom IAM Policy for EKS + SSM

Add inline policy:

{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Sid": "EKSAccess",
     "Effect": "Allow",
     "Action": [
       "eks:DescribeCluster"
     ],
     "Resource": "*"
   },
   {
     "Sid": "SSMStateAccess",
     "Effect": "Allow",
     "Action": [
       "ssm:GetParameter",
       "ssm:PutParameter"
     ],
     "Resource": "arn:aws:ssm:eu-north-1:031749757330:parameter/eks-monitor/*"
   }
 ]
}

Attach to: wbd-eks-monitor-role

Lambda Function Creation

AWS Console → Lambda → Create Function

Configuration:

Function name: wbd-eks-pod-monitor

Runtime: Python 3.12

Architecture: x86_64

Execution role: Use existing role

Select: wbd-eks-monitor-role

Create function.

Lambda General Configuration

Update Memory: 256 MB

Timeout: 10 seconds

Ephemeral storage: 512 MB

Attach Lambda Layer

Open Lambda → Configuration → Layers → Add layer

Select: Custom layers

Choose: k8s-python-layer

Version: 1

Attach.

Lambda Environment Variables

CLUSTER_NAME

MONGO_DEPLOYMENT

NAMESPACE

NEO_STATEFULSET

SLACK_WEBHOOK_URL

Lambda Code Deployment

import os
import json
import base64
import tempfile
import urllib3
import boto3

from kubernetes import client
from kubernetes.client import Configuration
from botocore.signers import RequestSigner

http = urllib3.PoolManager()

CLUSTER_NAME = os.environ["CLUSTER_NAME"]
NAMESPACE = os.environ["NAMESPACE"]
SLACK_WEBHOOK_URL = os.environ["SLACK_WEBHOOK_URL"]
AWS_REGION = os.environ["AWS_REGION"]

SSM_PARAM_NAME = f"/eks-monitor/{CLUSTER_NAME}/pod-states"


def send_slack(message):
    try:
        response = http.request(
            "POST",
            SLACK_WEBHOOK_URL,
            body=json.dumps({"text": message}),
            headers={"Content-Type": "application/json"},
        )
        if response.status != 200:
            print(f"Slack webhook failed with status: {response.status}")
    except Exception as e:
        print(f"Failed to send Slack alert: {e}")


def get_previous_states():
    ssm = boto3.client("ssm", region_name=AWS_REGION)
    try:
        response = ssm.get_parameter(Name=SSM_PARAM_NAME)
        return json.loads(response["Parameter"]["Value"])
    except ssm.exceptions.ParameterNotFound:
        return {}
    except Exception as e:
        print(f"Failed to get SSM state: {e}")
        return {}


def save_current_states(states):
    ssm = boto3.client("ssm", region_name=AWS_REGION)
    try:
        ssm.put_parameter(
            Name=SSM_PARAM_NAME,
            Value=json.dumps(states),
            Type="String",
            Overwrite=True,
        )
    except Exception as e:
        print(f"Failed to save SSM state: {e}")


def get_pod_status(pod):
    if pod.metadata.deletion_timestamp is not None:
        return "Terminating"
    if pod.status.phase != "Running":
        return pod.status.phase
    for cs in (pod.status.container_statuses or []):
        if not cs.ready:
            waiting = cs.state.waiting
            terminated = cs.state.terminated
            if waiting and waiting.reason:
                return waiting.reason
            if terminated and terminated.reason:
                return terminated.reason
    return "Running"


def get_eks_token(cluster_name, region):
    session = boto3.session.Session()
    client_sts = session.client("sts", region_name=region)

    signer = RequestSigner(
        client_sts.meta.service_model.service_id,
        region,
        "sts",
        "v4",
        session.get_credentials(),
        session.events,
    )

    params = {
        "method": "GET",
        "url": f"https://sts.{region}.amazonaws.com/?Action=GetCallerIdentity&Version=2011-06-15",
        "body": {},
        "headers": {"x-k8s-aws-id": cluster_name},
        "context": {},
    }

    signed_url = signer.generate_presigned_url(
        params,
        region_name=region,
        expires_in=60,
        operation_name="",
    )

    return "k8s-aws-v1." + base64.urlsafe_b64encode(
        signed_url.encode("utf-8")
    ).decode("utf-8").rstrip("=")


def get_k8s_client():
    eks = boto3.client("eks", region_name=AWS_REGION)
    cluster = eks.describe_cluster(name=CLUSTER_NAME)["cluster"]

    token = get_eks_token(CLUSTER_NAME, AWS_REGION)

    config = Configuration()
    config.host = cluster["endpoint"]
    config.verify_ssl = True

    ca_data = base64.b64decode(cluster["certificateAuthority"]["data"])

    with tempfile.NamedTemporaryFile(delete=False) as cert:
        cert.write(ca_data)
        config.ssl_ca_cert = cert.name

    config.api_key = {"authorization": "Bearer " + token}

    return client.CoreV1Api(client.ApiClient(config))


def lambda_handler(event, context):
    api = get_k8s_client()
    pods = api.list_namespaced_pod(namespace=NAMESPACE).items

    watched = ["mongodb", "neo4jnode"] #Add the pod name here
    previous_states = get_previous_states()
    current_states = {}
    issues = []

    found = {name: False for name in watched}

    for pod in pods:
        pod_name = pod.metadata.name

        for name in watched:
            if not pod_name.startswith(name):
                continue

            found[name] = True
            current_status = get_pod_status(pod)
            previous_status = previous_states.get(name, "Running")

            if name not in current_states or current_status != "Running":
                current_states[name] = current_status

            if current_status != "Running" and previous_status == "Running":
                issues.append(
                    f" Pod issue\n"
                    f"Namespace: {NAMESPACE}\n"
                    f"Pod: {pod_name}\n"
                    f"Workload: {name}\n"
                    f"Status: {current_status}"
                )

            elif current_status == "Running" and previous_status != "Running":
                issues.append(
                    f"✅ Pod recovered\n"
                    f"Namespace: {NAMESPACE}\n"
                    f"Pod: {pod_name}\n"
                    f"Workload: {name}\n"
                    f"Previous status: {previous_status}"
                )

    for name in watched:
        if not found[name]:
            previous_status = previous_states.get(name, "Running")
            current_states[name] = "Missing"

            if previous_status != "Missing":
                issues.append(
                    f" Pod missing\n"
                    f"Namespace: {NAMESPACE}\n"
                    f"Workload: {name}"
                )

        elif previous_states.get(name) == "Missing":
            issues.append(
                f"✅ Pod found again\n"
                f"Namespace: {NAMESPACE}\n"
                f"Workload: {name}"
            )

    save_current_states(current_states)

    if not issues:
        return {"statusCode": 200, "body": "all healthy"}

    for issue in issues:
        send_slack(issue)

    return {"statusCode": 200, "body": "alerts sent"}

Deploy.

EventBridge Trigger Creation

Open Lambda.

Configuration → Triggers → Add trigger

Select: EventBridge (CloudWatch Events)

Create new rule.

Rule name: wbd-eks-pod-monitor-lambda-eventbridge

Schedule: rate(1 minute)

State: Enabled

Add trigger.

Lambda Resource-based Permission

EventBridge needs permission to invoke Lambda.

After trigger creation, Lambda resource policy should contain:

Principal: events.amazonaws.com

Action: lambda:InvokeFunction

Source: EventBridge rule ARN.

Verified under: Lambda → Configuration → Permissions

Example Slack Alerts

Failure:

 Pod issue
Namespace: wbd-kavia-clone
Pod: mongodb-0
Workload: mongodb
Status: CrashLoopBackOff

Recovery:

✅ Pod recovered
Namespace: wbd-kavia-clone
Pod: mongodb-0
Workload: mongodb
Previous status: CrashLoopBackOff

Missing:

 Pod missing
Namespace: wbd-kavia-clone
Workload: neo4jnode

Recovered:

✅ Pod found again
Namespace: wbd-kavia-clone
Workload: neo4jnode

CloudWatch Validation

Check logs: CloudWatch → Log groups

Lambda log group: /aws/lambda/wbd-eks-pod-monitor

Verify: successful execution