AWS Fargate Troubleshooting: ENI Limits, Timeouts, Debugging

Fargate hides resource constraints (ENI limits, memory leaks, and subnet exhaustion) behind green dashboards until a traffic spike makes them impossible to ignore. These failures share a pattern: the root cause is invisible to standard CloudWatch metrics, so tasks stall or crash with misleading errors. Five failure modes account for most of them, and each one has a check that identifies it in minutes.

Previous parts of this Fargate series (101, 102) covered the basics and advanced patterns. The next part, 104, covers Infrastructure-as-Code deployment patterns.

ENI Exhaustion

Symptom: Tasks never leave PENDING, and the service event log shows:

ResourcesNotReady: The ENI allocation could not be completed

Deployments never finish and auto-scaling cannot add capacity. Service and task dashboards look normal, because the exhausted resource is the account’s network interface quota, which no ECS metric reports.

Where the ENIs Go

Each Fargate task in awsvpc mode gets its own ENI. The quota is not scoped to your service, and it is not scoped to your VPC either: Network interfaces per Region is an account-level quota of 5,000, enforced per Availability Zone. A few hundred tasks are harmless on their own. They exhaust the quota once every other consumer in the same account and zone is counted:

• Default quota: 5,000 network interfaces per Region, enforced per Availability Zone
• Each Fargate task: 1 ENI
• Each RDS instance: 1 ENI
• Each Lambda in VPC: Shares ENI pool
• Each ELB: Multiple ENIs

Checking usage against the quota:

# Interfaces currently allocated in the account
aws ec2 describe-network-interfaces \
  --query 'length(NetworkInterfaces)'

# The quota itself
aws service-quotas get-service-quota \
  --service-code vpc \
  --quota-code L-DF5E4CA3  # Network interfaces per Region

Load tests that only exercise the application will never surface this. The counter that matters is cumulative interface usage across every service in the account, and it keeps climbing while the application looks healthy.

Resolution Approach

Immediate steps:

# Scale down non-critical services in development
aws ecs update-service \
  --cluster development \
  --service api \
  --desired-count 0

# Request quota increase through AWS Support
aws support create-case \
  --subject "ENI quota increase needed - production capacity planning" \
  --service-code "service-quota-increase"

Longer-term improvements:

1. Separate accounts: The quota is per account and per Region, so moving dev and staging into their own accounts is what actually buys headroom
2. ENI monitoring: CloudWatch custom metric tracking ENI usage
3. Right-sizing: Reduce over-provisioned tasks
4. Lambda optimization: Move Lambdas out of the VPC where possible

// ENI monitoring Lambda
import { EC2Client, paginateDescribeNetworkInterfaces } from '@aws-sdk/client-ec2';
import { CloudWatchClient, PutMetricDataCommand } from '@aws-sdk/client-cloudwatch';

const ec2 = new EC2Client({});
const cloudWatch = new CloudWatchClient({});

export const monitorENIs = async () => {
  // DescribeNetworkInterfaces is paginated; a single page undercounts
  let inUse = 0;
  for await (const page of paginateDescribeNetworkInterfaces({ client: ec2 }, {})) {
    inUse += page.NetworkInterfaces?.length ?? 0;
  }

  await cloudWatch.send(new PutMetricDataCommand({
    Namespace: 'Custom/VPC',
    MetricData: [{
      MetricName: 'ENIsInUse',
      Value: inUse,
      Unit: 'Count'
    }]
  }));
};

Lessons learned:

• Load testing has to cover infrastructure quotas as well as application throughput
• The interface quota is account-wide and enforced per Availability Zone, so a quiet VPC next door still consumes it
• Quota increases go through Service Quotas or Support and are not instant, so request headroom before a traffic event

Subnet Routing Failure

Setup: Multi-AZ Fargate deployment across three private subnets.

Symptom: Intermittent connectivity. Some HTTP requests succeed, others time out after 30 seconds, and only tasks in one subnet are affected.

The Standard Checks

Start with the checks that rule out the task itself:

# Check task health
aws ecs list-tasks --cluster production --service-name api
aws ecs describe-tasks --cluster production --tasks task-abc123

# Check network interfaces
aws ec2 describe-network-interfaces \
  --filters "Name=subnet-id,Values=subnet-12345" \
  --query 'NetworkInterfaces[*].[NetworkInterfaceId,Status,PrivateIpAddress]'

Tasks report healthy. Network interfaces are attached and active. Neither view says anything about the path out of the subnet.

Flow logs answer that question:

# Enable VPC Flow Logs for the problem subnet
aws ec2 create-flow-logs \
  --resource-type Subnet \
  --resource-ids subnet-12345 \
  --traffic-type ALL \
  --log-destination-type cloud-watch-logs \
  --log-group-name /aws/vpc/flowlogs

Flow logs show packets leaving the subnet with no matching return traffic. Something on the egress path is dropping them.

The Root Cause

The usual cause is a route table edit made for a different workload. Changing the default route from 0.0.0.0/0 → nat-gateway-123 to 0.0.0.0/0 → nat-gateway-456 is a one-line change that nobody associates with a running Fargate service, because the service keeps reporting healthy.

If the replacement NAT gateway sits in another Availability Zone, every packet takes a cross-AZ hop and the network ACL on its subnet decides whether return traffic survives. NAT gateways have no security group of their own, so the subnet NACL is the control to check.

The fix:

# Check which route table is associated with the subnet
aws ec2 describe-route-tables \
  --filters "Name=association.subnet-id,Values=subnet-12345"

# Verify the routes
aws ec2 describe-route-tables --route-table-ids rtb-abc123 \
  --query 'RouteTables[*].Routes[*].[DestinationCidrBlock,GatewayId,State]'

# Fix the route (revert to original NAT gateway)
aws ec2 replace-route \
  --route-table-id rtb-abc123 \
  --destination-cidr-block 0.0.0.0/0 \
  --nat-gateway-id nat-gateway-123

Lessons learned:

• Always test routing changes in non-production first
• VPC Flow Logs answer the question no ECS metric can: did the packet leave, and did anything come back
• Document which route tables serve which services
• Alarm on route table changes through CloudTrail and EventBridge

Tracking a Memory Leak Without SSH Access

Setup: Node.js API on Fargate with a 2 GB memory limit per task.

Symptom: Memory climbs steadily over a few hours, then the task is OOM killed and replaced. The graph is a sawtooth, and there is no shell to SSH into.

Debugging Without a Shell

Three tools cover almost every case:

1. ECS Exec (the primary tool):

# First, enable it on the service
aws ecs update-service \
  --cluster production \
  --service api \
  --enable-execute-command

# Then connect to a running task
aws ecs execute-command \
  --cluster production \
  --task task-abc123 \
  --container api \
  --interactive \
  --command "/bin/bash"

# Inside the container, check memory usage
> ps aux --sort=-%mem | head -20
> cat /proc/meminfo
> pmap -x 1  # Memory map of PID 1

2. Application-level monitoring:

// Add to your Node.js app
const express = require('express');
const app = express();

// Memory monitoring endpoint
app.get('/debug/memory', (req, res) => {
  const used = process.memoryUsage();
  const stats = {
    rss: Math.round(used.rss / 1024 / 1024 * 100) / 100,  // MB
    heapTotal: Math.round(used.heapTotal / 1024 / 1024 * 100) / 100,
    heapUsed: Math.round(used.heapUsed / 1024 / 1024 * 100) / 100,
    external: Math.round(used.external / 1024 / 1024 * 100) / 100,
    arrayBuffers: Math.round(used.arrayBuffers / 1024 / 1024 * 100) / 100
  };
  
  res.json(stats);
});

// Heap snapshot endpoint, using the built-in v8 module
const v8 = require('v8');

app.get('/debug/heapdump', (req, res) => {
  const filename = v8.writeHeapSnapshot(`/tmp/heapdump-${Date.now()}.heapsnapshot`);
  res.download(filename);
});

Writing a snapshot pauses the event loop and allocates roughly the size of the heap, so keep the endpoint behind an internal path and never call it on every task at once.

3. Finding the leak:

ECS Exec lets you install tooling inside a running task. For a slow climb with no obvious allocation site, start with sockets rather than the heap:

# Inside the container
> npm install -g clinic
> clinic doctor --on-port 8080 -- node index.js &
> curl http://localhost:8080/debug/memory

# Check open file descriptors
> ls -la /proc/1/fd | wc -l
> lsof -p 1 | grep TCP

A large count of TCP connections stuck in CLOSE_WAIT is the HTTP client’s signature. The application heap is fine.

The Root Cause

The code that causes it looks harmless:

// The problematic code
const axios = require('axios');

async function callExternalAPI() {
  const response = await axios.get('https://api.example.com/data');
  return response.data;
}

With no agent configured, each call can open a fresh socket, and nothing closes the sockets the peer has already half-closed. Node keeps the descriptor, and resident memory tracks the descriptor count.

The fix:

// Fixed version with proper configuration
const axios = require('axios');
const https = require('https');
const http = require('http');

// Configure connection pooling
const httpAgent = new http.Agent({
  keepAlive: true,
  maxSockets: 50,
  timeout: 5000,
});

const httpsAgent = new https.Agent({
  keepAlive: true,
  maxSockets: 50,
  timeout: 5000,
});

const axiosInstance = axios.create({
  httpAgent,
  httpsAgent,
  timeout: 10000, // 10 seconds
});

// Graceful shutdown
process.on('SIGTERM', () => {
  httpAgent.destroy();
  httpsAgent.destroy();
});

async function callExternalAPI() {
  const response = await axiosInstance.get('https://api.example.com/data');
  return response.data;
}

Lessons learned:

• ECS Exec is invaluable for containerized debugging
• Always configure HTTP clients properly in production
• File descriptor count is the leading indicator; memory is the lagging one
• Connection pools matter, even for “simple” HTTP clients

The 30-Second Connection Timeout

Setup: Internal service-to-service calls between two Fargate services, routed through a load balancer.

Symptom: A small fraction of requests hang for exactly 30 seconds and then fail with a connection timeout. There is no correlation with load, time of day, or deployment history.

Layer by Layer

Network layer investigation:

# VPC Flow Logs analysis
aws logs filter-log-events \
  --log-group-name /aws/vpc/flowlogs \
  --start-time 1645564800000 \
  --filter-pattern "REJECT"

# Security group rules audit
aws ec2 describe-security-groups \
  --group-ids sg-12345 \
  --query 'SecurityGroups[*].{GroupId:GroupId,IpPermissions:IpPermissions}'

Security groups and flow logs both look normal, which rules out a blocked path and points at connection setup instead.

Application layer investigation:

// Added detailed connection tracking
const net = require('net');
const original_connect = net.Socket.prototype.connect;

net.Socket.prototype.connect = function(...args) {
  const startTime = Date.now();
  console.log(`[${new Date().toISOString()}] Starting connection to ${args[0]?.host || args[0]?.path}`);
  
  const result = original_connect.apply(this, args);
  
  this.on('connect', () => {
    const duration = Date.now() - startTime;
    console.log(`[${new Date().toISOString()}] Connected after ${duration}ms`);
  });
  
  this.on('error', (err) => {
    const duration = Date.now() - startTime;
    console.log(`[${new Date().toISOString()}] Connection error after ${duration}ms:`, err.message);
  });
  
  return result;
};

What the Logs Showed

Successful connections complete in single-digit milliseconds. The hanging ones sit at exactly the client timeout value, which is the signature of a connection that was never established rather than one that was refused.

The condition that reproduces it: the caller is itself a registered target of the load balancer it is calling.

The cause is NAT loopback, also called hairpinning. AWS documents it for Network Load Balancers: with client IP preservation enabled on the target group, a target that calls its own load balancer succeeds only when the request is routed to a different target. Routed back to the caller, source and destination addresses are identical and the connection times out. The same applies to containers that share a host.

The fix: AWS recommends disabling client IP preservation and reading the client address from Proxy Protocol v2 instead. Where the load balancer adds nothing to an internal call, two options remove the hop entirely:

1. Direct service-to-service calls:

// Service discovery with AZ awareness
const { ECSClient, ListTasksCommand, DescribeTasksCommand } = require('@aws-sdk/client-ecs');
const ecs = new ECSClient({});

// Fargate does not inject the Availability Zone as an environment variable.
// The task metadata endpoint is the supported source.
async function currentAvailabilityZone() {
  const res = await fetch(`${process.env.ECS_CONTAINER_METADATA_URI_V4}/task`);
  const meta = await res.json();
  return meta.AvailabilityZone;
}

async function getServiceEndpoints() {
  const { taskArns = [] } = await ecs.send(new ListTasksCommand({
    cluster: 'production',
    serviceName: 'target-service'
  }));
  if (taskArns.length === 0) return [];

  const { tasks = [] } = await ecs.send(new DescribeTasksCommand({
    cluster: 'production',
    tasks: taskArns
  }));

  return tasks.map(task => ({
    ip: task.attachments[0].details.find(d => d.name === 'privateIPv4Address').value,
    az: task.availabilityZone,
    port: 8080
  }));
}

// Smart routing
async function callService(endpoint, data) {
  const currentAZ = await currentAvailabilityZone();
  const endpoints = await getServiceEndpoints();
  
  // Try same-AZ direct connection first
  const sameAZEndpoint = endpoints.find(e => e.az === currentAZ);
  if (sameAZEndpoint) {
    try {
      return await axios.post(`http://${sameAZEndpoint.ip}:${sameAZEndpoint.port}${endpoint}`, data);
    } catch (error) {
      // Fall back to load balancer
      return await axios.post(`https://internal-service.example.com${endpoint}`, data);
    }
  }
  
  // Use load balancer for cross-AZ
  return await axios.post(`https://internal-service.example.com${endpoint}`, data);
}

Cache that endpoint list. ListTasks and DescribeTasks are throttled APIs, and calling them once per request will hit the limit long before the load balancer would have. ECS Service Connect or Cloud Map does the same job without the API traffic.

2. Connection timeout tuning:

const axiosInstance = axios.create({
  timeout: 5000,  // Fail fast instead of waiting 30s
  httpsAgent: new https.Agent({
    timeout: 2000,  // Connection timeout
    keepAlive: true,
  })
});

Lessons learned:

• A target that calls its own load balancer can hang until the client timeout fires; check client IP preservation first
• Service discovery enables direct communication patterns
• Always implement connection timeouts shorter than your SLA
• Internal traffic does not have to leave through a load balancer at all

Stalled Deployments

Setup: Standard blue-green deployment using CodeDeploy.

Symptom: The deployment stalls partway through. Some tasks run the new revision, the rest run the old one, and the CodeDeploy console shows InProgress with no error.

Auto-rollback never fires, because nothing has reported a failure.

The Investigation

CodeDeploy tells you nothing:

aws deploy get-deployment --deployment-id d-XXXXXXXXX
# Status: InProgress, no error information

aws logs filter-log-events \
  --log-group-name /aws/codedeploy-agent \
  --start-time $(date -d '1 hour ago' +%s)000

ECS service events do:

aws ecs describe-services \
  --cluster production \
  --services api \
  --query 'services[0].events[0:10]'

The events showed:

"(service api) failed to launch a task with (error ECS was unable to assume role...)"

The Root Cause

The task execution role’s trust policy no longer names ECS. An edit made for an unrelated service is enough to do it: the trust relationship is a single principal, and replacing it breaks nothing until the next task launch, which can be days later.

A trust policy in this state fails every launch:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"  // WRONG!
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

It should have been:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ecs-tasks.amazonaws.com"  // CORRECT
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

The fix:

# Check the role trust policy
aws iam get-role --role-name fargate-task-execution-role \
  --query 'Role.AssumeRolePolicyDocument'

# Update it
aws iam update-assume-role-policy \
  --role-name fargate-task-execution-role \
  --policy-document file://trust-policy.json

Prevention strategy:

// Automated role validation
import { IAMClient, GetRoleCommand } from '@aws-sdk/client-iam';

const iam = new IAMClient({});

export const validateTaskRoles = async () => {
  const { Role } = await iam.send(new GetRoleCommand({
    RoleName: 'fargate-task-execution-role'
  }));

  // AssumeRolePolicyDocument comes back URL-encoded
  const trustPolicy = JSON.parse(decodeURIComponent(Role.AssumeRolePolicyDocument));

  // Principal.Service is either a string or an array of strings
  const trustsECS = trustPolicy.Statement.some(statement =>
    [statement.Principal?.Service ?? []].flat().includes('ecs-tasks.amazonaws.com')
  );

  if (!trustsECS) {
    await sendAlert('Task execution role missing ECS trust relationship');
    return false;
  }

  return true;
};

Lessons learned:

• ECS service events are more detailed than CodeDeploy logs
• Role trust policies are fragile and need monitoring
• Blue-green deployments can get stuck in limbo
• Always check IAM when things mysteriously stop working

Debug Toolbox

Three pieces cover most of what the sections above need:

1. A Debug-Capable Container Image

FROM node:18-alpine
RUN apk add --no-cache \
    curl \
    wget \
    netcat-openbsd \
    bind-tools \
    tcpdump \
    strace \
    htop \
    iotop \
    lsof \
    procps \
    net-tools

# Add your app
COPY . /app
WORKDIR /app

# Debug endpoints
RUN npm install express clinic

strace also needs SYS_PTRACE in the task definition under linuxParameters.capabilities.add. It is the only capability Fargate lets you add, and it requires platform version 1.4.0 or later.

2. Monitoring Stack

// Health check endpoint with detailed diagnostics
app.get('/health/detailed', async (req, res) => {
  const health = {
    timestamp: new Date().toISOString(),
    uptime: process.uptime(),
    memory: process.memoryUsage(),
    cpu: process.cpuUsage(),
    connections: {
      active: await getActiveConnections(),
      waiting: await getWaitingConnections()
    },
    environment: {
      nodeVersion: process.version,
      availabilityZone: await currentAvailabilityZone(),  // task metadata endpoint
      region: process.env.AWS_REGION || 'unknown'
    }
  };
  
  res.json(health);
});

async function getActiveConnections() {
  return new Promise((resolve) => {
    require('child_process').exec('netstat -an | grep ESTABLISHED | wc -l', 
      (error, stdout) => {
        resolve(parseInt(stdout.trim()) || 0);
      }
    );
  });
}

3. Automated Incident Response

# Alarms for the two limits that fail silently
ENIUtilizationAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmName: High-ENI-Utilization
    MetricName: ENIsInUse
    Namespace: Custom/VPC
    Statistic: Maximum
    Period: 300
    EvaluationPeriods: 2
    Threshold: 4500  # 90% of the default 5,000 quota
    ComparisonOperator: GreaterThanThreshold
    AlarmActions:
      - !Ref SNSTopic

MemoryUtilizationAlarm:
  Type: AWS::CloudWatch::Alarm  
  Properties:
    AlarmName: Fargate-Memory-High
    MetricName: MemoryUtilized
    Namespace: ECS/ContainerInsights
    Statistic: Average
    Period: 300
    EvaluationPeriods: 3
    Threshold: 80  # 80% memory usage
    ComparisonOperator: GreaterThanThreshold

Triage Order by Symptom

When something breaks, this is the order that reaches the cause fastest:

1. When tasks won’t start: Check quotas, security groups, and IAM trust policies (in that order)

2. When tasks are slow: Look at the network first (route tables, NAT gateways, DNS)

3. When memory keeps climbing: Check connection pooling and event listeners before the application heap

4. When deployments hang: Check service events, not deployment logs

5. When a small fraction of requests fail: Look for load balancer loopback or cross-AZ paths

6. When nothing makes sense: Enable VPC Flow Logs and ECS Exec

Fargate removes most of the infrastructure you would otherwise operate, and the abstraction holds everywhere except quotas, routing, and connection lifecycle. Those three stay yours, so keep the diagnostics for them in place before you need them: ECS Exec enabled on the service, VPC Flow Logs available on the subnets, and an alarm on interface usage. Retrofitting any of them mid-incident costs more time than the fix itself.

References

• Architect for AWS Fargate for Amazon ECS - Core Fargate architecture reference covering task lifecycle, networking, and the fully managed infrastructure model.
• Amazon ECS task networking options for Fargate - How Fargate manages ENIs per task and the constraints that come with awsvpc network mode.
• Amazon VPC quotas - The network interface quota (5,000 per Region, enforced per Availability Zone), plus route table, NACL, and security group limits.
• Troubleshoot your Network Load Balancer - The loopback timeout that hits a target calling its own load balancer with client IP preservation enabled, and the two supported workarounds.
• Using Amazon ECS Exec to access your containers - Prerequisites, IAM permissions, and platform version requirements for opening a shell in a running Fargate task.
• Task metadata endpoint version 4 for Fargate - The supported way to read the Availability Zone, task ARN, and container stats from inside a task.
• Fargate security best practices in Amazon ECS - Security hardening for Fargate workloads including read-only root filesystems, resource limits, and network policies.
• Amazon ECS service quotas and API throttling limits - ENI and task quotas that cause production surprises at scale, and when to request increases.
• Updating an Amazon ECS service - Service update strategies including rolling updates and deployment circuit breaker configuration.
• Cost Optimization Checklist for Amazon ECS and AWS Fargate - AWS Containers blog checklist covering Fargate Spot, Savings Plans, and right-sizing to reduce production costs.