Fix: AWS CloudWatch Logs Not Appearing

The Error

You expect logs to appear in CloudWatch Logs but the log group is empty or missing:

Log group: /aws/lambda/my-function
Log streams: (none)

Or the log group exists but new log streams aren’t being created:

The log stream '/aws/lambda/my-function/2026/03/18/[$LATEST]abc123' does not exist.

Or your application logs using the CloudWatch agent but nothing appears:

# In CloudWatch console — log group exists but no events
aws logs describe-log-streams --log-group-name /myapp/application
# {
#     "logStreams": []
# }

Or Lambda logs appear for some invocations but not others. Or EC2 instance logs stopped streaming after a restart.

Why This Happens

CloudWatch log delivery fails for several reasons depending on the source:

• Missing IAM permissions — the Lambda execution role or EC2 instance profile lacks logs:CreateLogGroup, logs:CreateLogStream, or logs:PutLogEvents permissions.
• Log group not created — some services create log groups automatically on first use; others require the log group to exist before logs are sent.
• CloudWatch agent not running or misconfigured — on EC2, the CloudWatch agent is responsible for shipping logs. If it’s stopped or has a bad config, logs are silently dropped.
• Log stream errors — each Lambda invocation creates its own log stream. If the IAM role can’t create streams, logs are lost silently.
• VPC endpoint missing — Lambda or EC2 in a private VPC without internet access needs a VPC endpoint for CloudWatch Logs. Without it, log delivery fails silently.
• Log retention deleting old logs — if you set a short retention period, logs older than the retention window are deleted automatically.
• Region mismatch — you’re looking at the wrong region in the CloudWatch console.

The failure is rarely a single permission. It is almost always a chain: the execution role has PutLogEvents but not CreateLogStream, the function lives in a VPC subnet without a NAT route, or the log group exists in us-east-1 while you scroll through us-west-2. Each link silently drops the event with no error returned to your code, which is why “no logs” is the canonical AWS debugging black hole.

A second category of failure is delivery delay. CloudWatch Logs is eventually consistent. Lambda buffers up to 4 KB or 250 ms of output before flushing, and the console UI itself polls on a one-minute interval. New invocations can take 30 to 60 seconds to surface in the console even when delivery is working perfectly. If you watch the console for ten seconds and conclude that logging is broken, you may be looking at a buffering artifact, not a misconfiguration.

The third category is structured data. Lambda’s print() and console.log() write to stdout, which the Lambda runtime captures and ships. But sidecar libraries (Powertools for Lambda, Lambda Insights, Embedded Metric Format clients) write to dedicated streams that require the right runtime version, the right log format, and in some cases the right Lambda extension. A log line that looks fine in your code may be discarded if the runtime can’t parse it as the expected format.

Version History That Changes the Failure Mode

CloudWatch Logs has shifted significantly since 2018, and the version of the AWS managed policy, runtime, and feature surface you target changes how this error presents.

Log retention defaults (2014 onward). For most of CloudWatch Logs’ history, the default retention was “never expire.” Logs accumulated forever and the bill grew silently. In 2023 AWS added a configurable account-level default retention via PutAccountPolicy (DATA_PROTECTION_POLICY family), but Lambda-created log groups still default to “never expire” unless you set retention explicitly. If your “missing logs” complaint is actually “old logs deleted,” check whether someone applied an account-level policy after the group was created.

CloudWatch Logs Insights GA (November 2018). Before Insights, you searched logs with filter-log-events and grep. After Insights, you write a query language that runs server-side. The Fix 6 examples below use Insights — they will not work on accounts running pre-2018 tooling, and the start-query API is region-aware in a way the older filter API is not.

Embedded Metric Format / EMF (November 2019). Lambda runtimes started supporting JSON log lines that CloudWatch parses as metrics. If your “log line is missing” is actually an EMF-formatted line that failed validation, CloudWatch silently drops the metric but still keeps the raw line — so check the raw stream, not just the metric dashboard.

Lambda Extensions GA (October 2020). Extensions added a /logs API that lets agents (Datadog, New Relic, Dynatrace) intercept Lambda logs before they reach CloudWatch. If an extension is misconfigured, logs go to the third-party destination only, never to CloudWatch. Check the function’s layer list for any logging extensions before assuming the IAM role is at fault.

CloudWatchLogsFullAccess deprecation track (2021 onward). AWS has progressively split the legacy CloudWatchLogsFullAccess policy into narrower managed policies (CloudWatchLogsReadOnlyAccess, CloudWatchAgentServerPolicy, AWSLambdaBasicExecutionRole). Roles created with the old wildcard policy still work, but new accounts under SCP guardrails may block the wildcard. If you copy an old role template into a new org account, the Logs:* action may be denied by an SCP you don’t control.

Lambda runtime deprecations (ongoing). When a Node.js or Python runtime is deprecated, AWS first blocks updates, then blocks new function creation, then disables invocations entirely. A deprecated runtime stops logging silently before it stops running. If a function suddenly stopped logging “for no reason,” check whether its runtime hit the deprecation date in the last 30 days.

Fix 1: Check IAM Permissions

The most common cause is missing IAM permissions on the execution role or instance profile.

Required permissions for Lambda:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    }
  ]
}

Check Lambda function’s execution role:

# Get the execution role
aws lambda get-function-configuration \
  --function-name my-function \
  --query 'Role' \
  --output text
# arn:aws:iam::123456789012:role/my-lambda-role

# List policies attached to the role
aws iam list-attached-role-policies --role-name my-lambda-role

# Check if AWSLambdaBasicExecutionRole is attached
# This managed policy includes the required CloudWatch Logs permissions
aws iam get-policy \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

Attach the basic execution role if missing:

aws iam attach-role-policy \
  --role-name my-lambda-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

For EC2 instances running the CloudWatch agent:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:PutMetricData",
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams",
        "logs:DescribeLogGroups"
      ],
      "Resource": "*"
    }
  ]
}

Attach the managed policy for EC2:

# CloudWatchAgentServerPolicy includes all required permissions
aws iam attach-role-policy \
  --role-name my-ec2-instance-role \
  --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy

Fix 2: Check the Correct Region

CloudWatch Logs is region-specific. If your Lambda or EC2 is in us-east-1 but you’re looking at eu-west-1 in the console, the log group won’t appear.

# Check which region your Lambda is in
aws lambda get-function-configuration \
  --function-name my-function \
  --query 'FunctionArn' \
  --output text
# arn:aws:lambda:us-east-1:123456789012:function:my-function
#                 ^^^^^^^^^^
# The region is us-east-1

# Check CloudWatch log groups in the correct region
aws logs describe-log-groups \
  --region us-east-1 \
  --log-group-name-prefix /aws/lambda/my-function

In the AWS Console: Always check the region selector (top right) matches where your service runs.

Fix 3: Create the Log Group Manually

Some services require the log group to exist before they can send logs. Create it manually:

# Create the log group
aws logs create-log-group \
  --log-group-name /myapp/application \
  --region us-east-1

# Set a retention policy (optional — default is never expire)
aws logs put-retention-policy \
  --log-group-name /myapp/application \
  --retention-in-days 30

For Lambda — the log group is /aws/lambda/<function-name>:

aws logs create-log-group \
  --log-group-name /aws/lambda/my-function

aws logs put-retention-policy \
  --log-group-name /aws/lambda/my-function \
  --retention-in-days 14

Using Terraform to pre-create log groups:

resource "aws_cloudwatch_log_group" "lambda_logs" {
  name              = "/aws/lambda/${var.function_name}"
  retention_in_days = 14

  tags = {
    Environment = var.environment
  }
}

resource "aws_lambda_function" "my_function" {
  # ... lambda config
  depends_on = [aws_cloudwatch_log_group.lambda_logs]
}

Fix 4: Fix the CloudWatch Agent on EC2

The CloudWatch agent must be installed, configured, and running to ship logs from EC2 instances.

Check agent status:

# Amazon Linux / RHEL / CentOS
sudo systemctl status amazon-cloudwatch-agent

# If stopped, start it
sudo systemctl start amazon-cloudwatch-agent
sudo systemctl enable amazon-cloudwatch-agent

# Check agent logs for errors
sudo tail -f /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

Verify the agent configuration:

# Default config location
cat /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

# Or use the wizard to regenerate config
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

Example CloudWatch agent config for application logs:

{
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/myapp/application.log",
            "log_group_name": "/myapp/application",
            "log_stream_name": "{instance_id}",
            "retention_in_days": 30,
            "timezone": "UTC"
          },
          {
            "file_path": "/var/log/nginx/access.log",
            "log_group_name": "/myapp/nginx-access",
            "log_stream_name": "{instance_id}-access",
            "retention_in_days": 7
          }
        ]
      }
    }
  }
}

Apply the new config and restart:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config \
  -m ec2 \
  -s \
  -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

Fix 5: Fix Lambda in a VPC Without Internet Access

Lambda functions in a private VPC cannot reach CloudWatch Logs endpoints over the internet. Add a VPC endpoint for CloudWatch Logs:

# Create a VPC endpoint for CloudWatch Logs
aws ec2 create-vpc-endpoint \
  --vpc-id vpc-abc12345 \
  --service-name com.amazonaws.us-east-1.logs \
  --vpc-endpoint-type Interface \
  --subnet-ids subnet-abc12345 subnet-def67890 \
  --security-group-ids sg-abc12345

Using Terraform:

resource "aws_vpc_endpoint" "cloudwatch_logs" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.logs"
  vpc_endpoint_type   = "Interface"

  subnet_ids         = aws_subnet.private[*].id
  security_group_ids = [aws_security_group.vpc_endpoint.id]

  private_dns_enabled = true
}

Alternatively — attach the Lambda to a subnet with a NAT Gateway that routes traffic to the internet.

Fix 6: Verify Log Delivery with AWS CLI

Use the AWS CLI to check if log events are actually arriving:

# List log streams in a log group
aws logs describe-log-streams \
  --log-group-name /aws/lambda/my-function \
  --order-by LastEventTime \
  --descending \
  --max-items 5

# Get events from the most recent stream
aws logs get-log-events \
  --log-group-name /aws/lambda/my-function \
  --log-stream-name '2026/03/18/[$LATEST]abc123' \
  --limit 50

# Filter log events across all streams (CloudWatch Logs Insights)
aws logs filter-log-events \
  --log-group-name /aws/lambda/my-function \
  --filter-pattern "ERROR" \
  --start-time $(date -d '1 hour ago' +%s000) \
  --end-time $(date +%s000)

CloudWatch Logs Insights query (in the console or CLI):

# Query the last hour of logs
aws logs start-query \
  --log-group-name /aws/lambda/my-function \
  --start-time $(date -d '1 hour ago' +%s) \
  --end-time $(date +%s) \
  --query-string 'fields @timestamp, @message | sort @timestamp desc | limit 20'

Fix 7: Check Log Retention Settings

If logs appeared previously but now seem deleted, check if a short retention period is set:

# Check retention settings for a log group
aws logs describe-log-groups \
  --log-group-name-prefix /aws/lambda/my-function \
  --query 'logGroups[*].{name:logGroupName, retentionDays:retentionInDays}'

# Update retention — 0 means never expire
aws logs put-retention-policy \
  --log-group-name /aws/lambda/my-function \
  --retention-in-days 0   # Never delete

# Or a reasonable retention period
aws logs put-retention-policy \
  --log-group-name /aws/lambda/my-function \
  --retention-in-days 90

Still Not Working?

Test permissions directly using the AWS CLI from the failing instance or role:

# Test if you can write to CloudWatch Logs
aws logs put-log-events \
  --log-group-name /test/permissions \
  --log-stream-name test-stream \
  --log-events timestamp=$(date +%s000),message="Test log entry"
# If this fails, the IAM permissions are the problem

Check CloudTrail for CloudWatch Logs API errors:

# Search CloudTrail for failed PutLogEvents calls
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=PutLogEvents \
  --start-time $(date -d '1 hour ago' --iso-8601=seconds) | \
  python3 -c "import json,sys; events=json.load(sys.stdin)['Events']; [print(e['CloudTrailEvent']) for e in events if json.loads(e['CloudTrailEvent']).get('errorCode')]"

Check for resource-based policies on the log group that might deny access:

aws logs describe-resource-policies --region us-east-1
# Look for policies that might deny PutLogEvents

For ECS tasks — check the task definition’s logConfiguration. The awslogs driver requires awslogs-group, awslogs-region, and optionally awslogs-create-group:

{
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-group": "/ecs/my-task",
      "awslogs-region": "us-east-1",
      "awslogs-stream-prefix": "ecs",
      "awslogs-create-group": "true"
    }
  }
}

Check for a Lambda Extension intercepting logs. Run aws lambda get-function-configuration --function-name my-function --query 'Layers' and look for any layer whose ARN contains datadog, newrelic, dynatrace, lambda-insights, or cloudwatch-lambda-extension. Extensions that subscribe to the Logs API can route output away from CloudWatch entirely. If you find one, check that extension’s destination configuration before re-blaming IAM. Removing a misconfigured logging extension is often the only way to get logs back in CloudWatch without a function redeploy. Related: Fix: AWS Lambda Layer Not Working.

Confirm your runtime is not deprecated. Run aws lambda get-function-configuration --query 'Runtime'. Cross-reference against the AWS Lambda runtime support policy. A deprecated runtime can hit “blocked from invocation” silently, which surfaces as “logs stopped without error.” Update the runtime and redeploy — see Fix: AWS Lambda Cold Start Timeout for related runtime issues.

Check for SCP denial in AWS Organizations. If your account belongs to an Organization, a service control policy on the parent OU can deny logs:PutLogEvents regardless of the role’s policy. Run aws organizations describe-effective-policy --policy-type SERVICE_CONTROL_POLICY --target-id <account-id> (you need org admin) or ask whoever owns the org to verify. SCP denials do not show up in the role’s simulate-principal-policy output unless you pass --caller-arn and the policy explicitly references aws:PrincipalOrgID.

Throttling. PutLogEvents has a per-region account quota (default 5,000 transactions/second per region, up to 1 MB/s per stream). If you scale a Lambda from 10 to 5,000 concurrent executions overnight, CloudWatch will throttle a chunk of PutLogEvents calls and lose those events. The Lambda invocation itself succeeds with no error. Check IncomingLogEvents and ThrottledRecords metrics in the AWS/Logs namespace.

For related AWS issues, see Fix: AWS IAM AccessDeniedException and Fix: AWS Lambda Import Module Error.