Fix: gRPC Error Not Working — Status Codes, Connection Failed, or Deadline Exceeded

The Problem

A gRPC client throws UNAVAILABLE when connecting to a server:

Error: 14 UNAVAILABLE: Connection refused
Error: 14 UNAVAILABLE: failed to connect to all addresses

Or a call fails with DEADLINE_EXCEEDED even though the server is running:

Error: 4 DEADLINE_EXCEEDED: Deadline exceeded

Or the server returns UNIMPLEMENTED:

Error: 12 UNIMPLEMENTED: Method not found

Or TLS setup fails in production:

Error: 14 UNAVAILABLE: upstream connect error or disconnect/reset before headers. reset reason: connection failure

Why This Happens

gRPC uses HTTP/2 and has its own status code system separate from HTTP. Common failure causes:

• Wrong port or address — gRPC servers default to port 50051 (convention, not a standard). If the client targets the wrong address, it gets UNAVAILABLE.
• TLS mismatch — using a TLS client against a plaintext server (or vice versa) causes connection failures. InsecureChannel vs SslChannel must match the server.
• No deadline set — without a deadline, gRPC calls can hang indefinitely if the server is slow or stuck. This also triggers the default DEADLINE_EXCEEDED if the framework sets one.
• Proto mismatch — if client and server use different versions of the .proto file, the server doesn’t recognize the method name and returns UNIMPLEMENTED.
• Load balancer not gRPC-aware — standard HTTP/1.1 load balancers don’t understand gRPC’s HTTP/2 multiplexing. All requests go to one backend, and the connection drops after the LB’s HTTP timeout.

Fix 1: Verify Connection and Credentials

Check the most basic issues first — address and TLS:

// Node.js — @grpc/grpc-js
const grpc = require('@grpc/grpc-js');
const protoLoader = require('@grpc/proto-loader');

const packageDefinition = protoLoader.loadSync('service.proto');
const proto = grpc.loadPackageDefinition(packageDefinition);

// WRONG — using secure credentials against a plaintext server
const client = new proto.MyService(
  'localhost:50051',
  grpc.credentials.createSsl()  // Server not using TLS → UNAVAILABLE
);

// CORRECT — plaintext for development
const client = new proto.MyService(
  'localhost:50051',
  grpc.credentials.createInsecure()
);

// CORRECT — TLS for production
const client = new proto.MyService(
  'api.example.com:443',
  grpc.credentials.createSsl()  // Uses system root CAs
);

// CORRECT — TLS with custom certificate
const rootCert = fs.readFileSync('ca.crt');
const client = new proto.MyService(
  'api.example.com:443',
  grpc.credentials.createSsl(rootCert)
);

Go client:

import (
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials"
    "google.golang.org/grpc/credentials/insecure"
)

// Plaintext (development)
conn, err := grpc.NewClient("localhost:50051",
    grpc.WithTransportCredentials(insecure.NewCredentials()),
)

// TLS (production)
creds, err := credentials.NewClientTLSFromFile("ca.crt", "")
conn, err := grpc.NewClient("api.example.com:443",
    grpc.WithTransportCredentials(creds),
)

Fix 2: Set Deadlines on Every Call

Always set a deadline (timeout) on gRPC calls:

// Node.js — set deadline as a Date object
const deadline = new Date();
deadline.setSeconds(deadline.getSeconds() + 10);  // 10 second timeout

client.GetUser({ id: 'user-123' }, { deadline }, (err, response) => {
  if (err) {
    if (err.code === grpc.status.DEADLINE_EXCEEDED) {
      console.error('Request timed out');
    } else {
      console.error('gRPC error:', err.code, err.message);
    }
    return;
  }
  console.log(response);
});

// With async/await using util.promisify
const { promisify } = require('util');
const getUser = promisify(client.GetUser.bind(client));

try {
  const deadline = new Date(Date.now() + 10000);  // 10 seconds from now
  const user = await getUser({ id: 'user-123' }, { deadline });
} catch (err) {
  if (err.code === grpc.status.DEADLINE_EXCEEDED) {
    // Handle timeout
  }
}

Go — use context with timeout:

import (
    "context"
    "time"
)

// Set timeout via context
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()

response, err := client.GetUser(ctx, &pb.GetUserRequest{Id: "user-123"})
if err != nil {
    st, ok := status.FromError(err)
    if ok && st.Code() == codes.DeadlineExceeded {
        log.Println("Request timed out")
    }
    return nil, err
}

Python:

import grpc

channel = grpc.insecure_channel('localhost:50051')
stub = pb2_grpc.MyServiceStub(channel)

try:
    # timeout in seconds
    response = stub.GetUser(pb2.GetUserRequest(id='user-123'), timeout=10)
except grpc.RpcError as e:
    if e.code() == grpc.StatusCode.DEADLINE_EXCEEDED:
        print("Request timed out")
    elif e.code() == grpc.StatusCode.UNAVAILABLE:
        print("Server unavailable:", e.details())
    else:
        print(f"gRPC error {e.code()}: {e.details()}")

Fix 3: Return Proper Status Codes from the Server

Servers should return meaningful gRPC status codes instead of generic errors:

// Go server — return proper status codes
import (
    "google.golang.org/grpc/codes"
    "google.golang.org/grpc/status"
)

func (s *server) GetUser(ctx context.Context, req *pb.GetUserRequest) (*pb.User, error) {
    user, err := s.db.GetUser(ctx, req.Id)

    if err == sql.ErrNoRows {
        // Resource not found
        return nil, status.Errorf(codes.NotFound, "user %s not found", req.Id)
    }
    if err != nil {
        // Internal server error
        return nil, status.Errorf(codes.Internal, "failed to get user: %v", err)
    }

    if req.Id == "" {
        // Invalid argument
        return nil, status.Error(codes.InvalidArgument, "user ID is required")
    }

    // Check auth from context
    if !isAuthorized(ctx) {
        return nil, status.Error(codes.PermissionDenied, "not authorized")
    }

    return userToProto(user), nil
}

gRPC status code reference:

Fix 4: Fix UNIMPLEMENTED Errors

UNIMPLEMENTED means the server doesn’t recognize the method. Common causes:

# 1. Check proto files are in sync
# Client and server must use the same .proto file version
# Regenerate stubs from the latest .proto:

# Node.js
npx grpc_tools_node_protoc \
  --js_out=import_style=commonjs,binary:. \
  --grpc_out=grpc_js:. \
  service.proto

# Go
protoc --go_out=. --go-grpc_out=. service.proto

# Python
python -m grpc_tools.protoc \
  -I. \
  --python_out=. \
  --grpc_python_out=. \
  service.proto

// Go server — ensure all methods are implemented
// If proto defines GetUser and ListUsers, you MUST implement both
// or embed the Unimplemented stub

type server struct {
    pb.UnimplementedMyServiceServer  // ← This provides default "UNIMPLEMENTED" responses
}

// Only implement what you need — unimplemented methods return UNIMPLEMENTED automatically
func (s *server) GetUser(ctx context.Context, req *pb.GetUserRequest) (*pb.User, error) {
    // ... implementation
}
// ListUsers will return UNIMPLEMENTED (from the embedded struct)

// Node.js server — implement ALL methods from the proto
const server = new grpc.Server();

server.addService(proto.MyService.service, {
  // Each method in the proto must have a matching key here
  getUser: (call, callback) => { ... },
  listUsers: (call, callback) => { ... },  // Don't miss any
  createUser: (call, callback) => { ... },
});

Fix 5: Use Interceptors for Consistent Error Handling

Interceptors (middleware) centralize error logging and retry logic:

// Node.js client interceptor — retry on UNAVAILABLE
const retryInterceptor = (options, nextCall) => {
  let savedMetadata;
  let savedSendMessage;
  let savedCallback;

  const requester = {
    start(metadata, listener, next) {
      savedMetadata = metadata;
      const newListener = {
        onReceiveMessage(message, next) { next(message); },
        onReceiveStatus(status, next) {
          if (
            status.code === grpc.status.UNAVAILABLE &&
            (options.retries || 0) < 3
          ) {
            // Retry
            const newCall = nextCall({ ...options, retries: (options.retries || 0) + 1 });
            newCall.start(savedMetadata, listener);
            savedSendMessage && newCall.sendMessage(savedSendMessage);
            newCall.halfClose();
          } else {
            next(status);
          }
        },
        onReceiveMetadata(metadata, next) { next(metadata); },
      };
      next(metadata, newListener);
    },
    sendMessage(message, next) {
      savedSendMessage = message;
      next(message);
    },
    halfClose(next) { next(); },
    cancel(message, next) { next(); },
  };

  return new grpc.InterceptingCall(nextCall(options), requester);
};

const client = new proto.MyService(
  'localhost:50051',
  grpc.credentials.createInsecure(),
  { interceptors: [retryInterceptor] }
);

Go server interceptor — log all errors:

import "google.golang.org/grpc"

func loggingInterceptor(
    ctx context.Context,
    req interface{},
    info *grpc.UnaryServerInfo,
    handler grpc.UnaryHandler,
) (interface{}, error) {
    resp, err := handler(ctx, req)
    if err != nil {
        st, _ := status.FromError(err)
        log.Printf("gRPC error: method=%s code=%s message=%s",
            info.FullMethod, st.Code(), st.Message())
    }
    return resp, err
}

server := grpc.NewServer(
    grpc.UnaryInterceptor(loggingInterceptor),
)

Fix 6: Configure gRPC for Load Balancers

Standard HTTP/1.1 load balancers (AWS ALB, nginx without gRPC module) break gRPC:

# nginx — enable gRPC proxying
server {
    listen 443 ssl http2;  # http2 is required for gRPC

    location / {
        grpc_pass grpc://backend:50051;

        # For TLS backend
        # grpc_pass grpcs://backend:50051;
    }
}

upstream backend {
    server backend-1:50051;
    server backend-2:50051;
    server backend-3:50051;
}

AWS ALB — use gRPC as the target group protocol:

# Terraform — ALB target group for gRPC
resource "aws_lb_target_group" "grpc" {
  name        = "grpc-targets"
  port        = 50051
  protocol    = "HTTP"
  target_type = "ip"

  protocol_version = "GRPC"  # ← This enables gRPC health checks and routing

  health_check {
    protocol = "HTTP"
    path     = "/grpc.health.v1.Health/Check"
    matcher  = "0"  # gRPC OK status code
  }
}

Client-side load balancing (no proxy needed):

// Go — use DNS resolver for client-side LB
conn, err := grpc.NewClient(
    "dns:///api.example.com:50051",  // DNS resolves to multiple IPs
    grpc.WithTransportCredentials(creds),
    grpc.WithDefaultServiceConfig(`{"loadBalancingPolicy":"round_robin"}`),
)

Still Not Working?

gRPC reflection for debugging — enable server reflection to inspect the server without having the .proto file. Use grpcurl to test:

# Install grpcurl
brew install grpcurl  # macOS

# List services
grpcurl -plaintext localhost:50051 list

# Describe a service
grpcurl -plaintext localhost:50051 describe MyPackage.MyService

# Call a method
grpcurl -plaintext -d '{"id": "user-123"}' localhost:50051 MyPackage.MyService/GetUser

RST_STREAM errors — the server reset the stream. This often means the server crashed or the connection was terminated by an upstream proxy. Check server logs and any intermediate load balancers for their HTTP timeout settings. gRPC long-running streams often need the LB timeout raised to hours.

Keep-alive settings — in Kubernetes, pods behind a service may have TCP connections closed by the network layer. Configure gRPC keep-alive:

conn, err := grpc.NewClient("...",
    grpc.WithKeepaliveParams(keepalive.ClientParameters{
        Time:                10 * time.Second,
        Timeout:             time.Second,
        PermitWithoutStream: true,
    }),
)

For related API issues, see Fix: GraphQL 400 Bad Request and Fix: axios Network Error.