Building Production Microservices: API Gateway, Resilience & Service Discovery with HazelJS

The Problem: Microservices Are Hard

Building microservices isn't just about splitting a monolith into smaller pieces. The real challenge is everything between the services — routing, load balancing, resilience, versioning, and deployment. Most teams end up stitching together a dozen libraries, writing glue code, and hoping it all holds together under production load.

HazelJS solves this with three purpose-built packages that work seamlessly together: @hazeljs/gateway for API routing and traffic management, @hazeljs/resilience for fault tolerance, and @hazeljs/discovery for service registration and load balancing.

@hazeljs/gateway — The API Gateway

The gateway package provides a full-featured API gateway that sits in front of your microservices. It handles routing, version management, canary deployments, traffic mirroring, and per-route resilience — all from a single, declarative configuration.

Three Ways to Define Routes

HazelJS gives you flexibility in how you define your gateway. Choose whichever approach fits your team:

1. Decorator-Based (Recommended)

Clean, type-safe, and self-documenting:

import {
  GatewayServer, Gateway, Route, ServiceRoute,
  VersionRoute, Canary, TrafficPolicy,
  GatewayCircuitBreaker, GatewayRateLimit,
} from '@hazeljs/gateway';

@Gateway({
  discovery: { cacheEnabled: true, cacheTTL: 15_000 },
  resilience: {
    defaultCircuitBreaker: { failureThreshold: 5, resetTimeout: 15_000 },
    defaultRetry: { maxAttempts: 2, backoff: 'exponential', baseDelay: 500 },
    defaultTimeout: 10_000,
  },
  metrics: { enabled: true },
  middleware: { cors: true, logging: true, requestId: true },
})
class ApiGateway {
  @Route({ path: '/api/users/**', methods: ['GET', 'POST', 'PUT', 'DELETE'] })
  @ServiceRoute({
    serviceName: 'user-service',
    loadBalancingStrategy: 'round-robin',
    stripPrefix: '/api/users',
  })
  @VersionRoute({
    strategy: 'header',
    header: 'X-API-Version',
    defaultVersion: 'v1',
    routes: {
      v1: { weight: 70, allowExplicit: true, filter: { tags: ['v1'] } },
      v2: { weight: 30, allowExplicit: true, filter: { tags: ['v2'] } },
    },
  })
  userRoutes: any;

  @Route({ path: '/api/orders/**', methods: ['GET', 'POST'] })
  @ServiceRoute({ serviceName: 'order-service', stripPrefix: '/api/orders' })
  @Canary({
    stable: { version: '1.0.0', weight: 90, filter: { tags: ['stable'] } },
    canary: { version: '2.0.0', weight: 10, filter: { tags: ['canary'] } },
    promotion: {
      strategy: 'error-rate',
      errorThreshold: 5,
      evaluationWindow: '30s',
      autoPromote: true,
      autoRollback: true,
      steps: [10, 25, 50, 75, 100],
      stepInterval: '15s',
    },
  })
  orderRoutes: any;

  @Route({ path: '/api/payments/**', methods: ['GET', 'POST'] })
  @ServiceRoute({ serviceName: 'payment-service', stripPrefix: '/api/payments' })
  @GatewayCircuitBreaker({ failureThreshold: 3, resetTimeout: 15_000 })
  @GatewayRateLimit({ strategy: 'token-bucket', max: 50, window: 60_000 })
  paymentRoutes: any;
}

// Create and start
const gateway = GatewayServer.fromClass(ApiGateway, backend);
gateway.startCanaries();

2. Config-Driven

Perfect for environment-variable driven deployments:

const gateway = GatewayServer.fromConfig({
  discovery: { cacheEnabled: true, cacheTTL: 15_000 },
  resilience: {
    defaultCircuitBreaker: { failureThreshold: 5, resetTimeout: 10_000 },
    defaultTimeout: 10_000,
  },
  routes: [
    {
      path: '/api/users/**',
      serviceName: 'user-service',
      versionRoute: {
        strategy: 'header',
        header: 'X-API-Version',
        defaultVersion: 'v1',
        routes: {
          v1: { weight: 80, filter: { tags: ['v1'] } },
          v2: { weight: 20, filter: { tags: ['v2'] } },
        },
      },
    },
    {
      path: '/api/payments/**',
      serviceName: 'payment-service',
      circuitBreaker: { failureThreshold: 3, resetTimeout: 15_000 },
      rateLimit: { strategy: 'token-bucket', max: 100, window: 60_000 },
    },
  ],
}, backend);

3. Programmatic

Add routes dynamically at runtime:

const gateway = GatewayServer.fromConfig({ routes: [] }, backend);

// Add routes on-the-fly
gateway.addRoute({
  path: '/api/users/**',
  serviceName: 'user-service',
  circuitBreaker: { failureThreshold: 3, resetTimeout: 10_000 },
});

gateway.addRoute({
  path: '/api/products/**',
  serviceName: 'product-service',
  rateLimit: { strategy: 'token-bucket', max: 20, window: 60_000 },
});

Version Routing

Route API requests to different service versions using headers, URI prefixes, query parameters, or weighted random selection:

import { VersionRouter } from '@hazeljs/gateway';

const router = new VersionRouter({
  strategy: 'header',        // Also: 'uri', 'query'
  header: 'X-API-Version',
  defaultVersion: 'v1',
  routes: {
    v1: { weight: 70, allowExplicit: true },
    v2: { weight: 25, allowExplicit: true },
    v3: { weight: 5, allowExplicit: true },
  },
});

// Explicit header: X-API-Version: v2
const result = router.resolve({
  method: 'GET', path: '/api/users',
  headers: { 'x-api-version': 'v2' },
});
// => { version: 'v2', resolvedBy: 'header' }

// No header — uses weighted random selection
const weighted = router.resolve({
  method: 'GET', path: '/api/users', headers: {},
});
// => { version: 'v1', resolvedBy: 'weight' } (70% chance)

Canary Deployments

Gradually shift traffic from stable to canary with automatic promotion or rollback based on real-time metrics:

import { CanaryEngine } from '@hazeljs/gateway';

const canary = new CanaryEngine({
  stable: { version: '1.0.0', weight: 90 },
  canary: { version: '2.0.0', weight: 10 },
  promotion: {
    strategy: 'error-rate',
    errorThreshold: 5,              // Rollback if >5% errors
    evaluationWindow: '30s',
    autoPromote: true,
    autoRollback: true,
    steps: [10, 25, 50, 75, 100],   // Progressive traffic shift
    stepInterval: '15s',
    minRequests: 10,
  },
});

canary.on('canary:promote', () => console.log('Canary promoted!'));
canary.on('canary:rollback', () => console.log('Canary rolled back!'));
canary.start();

// Manual controls
canary.pause();    // Freeze during incidents
canary.resume();   // Resume progression
canary.promote();  // Force promote
canary.rollback(); // Force rollback

Traffic Mirroring

Shadow traffic to analytics or testing services without affecting the main request flow:

@Route({ path: '/api/products/**', methods: ['GET'] })
@ServiceRoute({ serviceName: 'product-service' })
@TrafficPolicy({
  mirror: {
    service: 'analytics-service',
    percentage: 50,          // Mirror 50% of traffic
    waitForResponse: false,  // Fire-and-forget
  },
  timeout: 5_000,
  retry: { maxAttempts: 2, backoff: 'fixed', baseDelay: 500 },
})
productRoutes: any;

@hazeljs/resilience — Fault Tolerance

Production services fail. Networks drop, databases slow down, third-party APIs go offline. The resilience package gives you battle-tested patterns to handle all of it gracefully.

Circuit Breaker

Prevents cascading failures by stopping requests to unhealthy services. Supports count-based and time-based sliding windows, custom failure predicates, and fallbacks:

import { CircuitBreaker, CircuitBreakerRegistry } from '@hazeljs/resilience';

const breaker = new CircuitBreaker({
  failureThreshold: 3,
  successThreshold: 2,
  resetTimeout: 10_000,
  timeout: 30_000,
  slidingWindow: { type: 'time', size: 60_000 },
  failurePredicate: (error) => {
    // Don't count 404s as failures
    return !(error instanceof Error && error.message.includes('not found'));
  },
  onStateChange: (from, to) => {
    console.log(`Circuit: ${from} → ${to}`);
  },
});

// Execute with protection
const result = await breaker.execute(async () => {
  return await paymentService.charge(amount);
});

// Global registry for managing all breakers
const payment = CircuitBreakerRegistry.getOrCreate('payment-service', {
  failureThreshold: 3,
  resetTimeout: 15_000,
});

Retry Policy

Three backoff strategies with jitter to prevent thundering herd:

import { RetryPolicy } from '@hazeljs/resilience';

const retry = new RetryPolicy({
  maxAttempts: 3,
  backoff: 'exponential',  // Also: 'fixed', 'linear'
  baseDelay: 1_000,
  maxDelay: 30_000,
  jitter: true,            // Prevents thundering herd
  retryPredicate: (error) => {
    // Only retry transient errors
    return error.message.includes('timeout') ||
           error.message.includes('503');
  },
  onRetry: (error, attempt) => {
    console.log(`Retry ${attempt}: ${error.message}`);
  },
});

const result = await retry.execute(async () => {
  return await externalApi.fetchData();
});

Bulkhead, Rate Limiter & Timeout

import { Bulkhead, RateLimiter, withTimeout } from '@hazeljs/resilience';

// Bulkhead — isolate resource pools
const bulkhead = new Bulkhead({
  maxConcurrent: 10,    // Max 10 concurrent calls
  maxQueue: 20,         // Queue up to 20 more
  queueTimeout: 5_000,  // 5s queue wait timeout
});

// Rate Limiter — token bucket or sliding window
const limiter = new RateLimiter({
  strategy: 'token-bucket',  // Allows bursts
  max: 100,                  // 100 requests
  window: 60_000,            // per minute
});

// Timeout — reject slow operations
const result = await withTimeout(
  async () => databaseQuery(),
  5_000,
  'Database query exceeded 5s limit'
);

Decorator-Based Resilience

Stack multiple resilience patterns on a single method with clean decorators. The execution order is outermost to innermost:

import {
  WithCircuitBreaker, WithRetry, WithTimeout,
  WithBulkhead, WithRateLimit, Fallback,
} from '@hazeljs/resilience';

class PaymentService {
  @WithCircuitBreaker({ failureThreshold: 3, resetTimeout: 15_000 })
  @WithRetry({ maxAttempts: 2, backoff: 'exponential', baseDelay: 500 })
  @WithTimeout(5_000)
  @WithBulkhead({ maxConcurrent: 5, maxQueue: 10 })
  async processPayment(amount: number): Promise<string> {
    return await paymentGateway.charge(amount);
  }

  @Fallback('processPayment')
  async processPaymentFallback(amount: number): Promise<string> {
    return await offlineQueue.enqueue({ amount });
  }
}

// Execution: CircuitBreaker → Retry → Timeout → Bulkhead → method
// If circuit opens, Fallback method is called automatically

Metrics Collection

Track success rates, failure rates, and latency percentiles (p50, p95, p99) across all your services:

import { MetricsCollector, MetricsRegistry } from '@hazeljs/resilience';

// Per-service metrics
const collector = MetricsRegistry.getOrCreate('payment-service');
collector.recordSuccess(45);   // 45ms latency
collector.recordFailure(500);  // 500ms failed call

const snapshot = collector.getSnapshot();
// { totalCalls, successCalls, failureCalls, failureRate,
//   averageResponseTime, p50, p95, p99, min, max }

@hazeljs/discovery — Service Discovery

Services need to find each other. The discovery package handles registration, health checks, caching, and load balancing with support for Memory, Redis, Consul, and Kubernetes backends.

Service Registration

import {
  ServiceRegistry, DiscoveryClient, ServiceClient,
  MemoryRegistryBackend,
} from '@hazeljs/discovery';

// Register a service instance
const registry = new ServiceRegistry({
  name: 'user-service',
  host: 'localhost',
  port: 3001,
  healthCheckPath: '/health',
  healthCheckInterval: 30_000,
  metadata: { version: '2.0.0', weight: 80 },
  zone: 'us-east-1',
  tags: ['api', 'users', 'v2'],
}, backend);

await registry.register();  // Registers + starts health checks

6 Load Balancing Strategies

// Discovery client with caching
const client = new DiscoveryClient({
  cacheEnabled: true,
  cacheTTL: 30_000,
  refreshInterval: 15_000,
}, backend);

// Get one instance with load balancing
const instance = await client.getInstance('user-service', 'round-robin');

// Filter by zone, tags, metadata, or status
const filtered = await client.getInstances('user-service', {
  zone: 'us-east-1',
  tags: ['v2'],
  metadata: { version: '2.0.0' },
});

ServiceClient — HTTP with Auto-Discovery

// HTTP client that automatically discovers service instances
const userClient = new ServiceClient(discoveryClient, {
  serviceName: 'user-service',
  loadBalancingStrategy: 'round-robin',
  timeout: 5_000,
  retries: 3,
  retryDelay: 1_000,
});

// Automatic discovery + load balancing + smart retries
const users = await userClient.get('/users');
const user = await userClient.post('/users', { name: 'John' });
await userClient.put('/users/1', { name: 'Jane' });
await userClient.delete('/users/1');

Multiple Backends

// In-memory (development)
import { MemoryRegistryBackend } from '@hazeljs/discovery';
const backend = new MemoryRegistryBackend();

// Redis (production — distributed)
import { RedisRegistryBackend } from '@hazeljs/discovery';
const backend = new RedisRegistryBackend(redisClient, {
  keyPrefix: 'hazeljs:discovery:',
  ttl: 90,
});

// Consul (enterprise)
import { ConsulRegistryBackend } from '@hazeljs/discovery';
const backend = new ConsulRegistryBackend(consulClient, {
  ttl: '30s',
});

// Kubernetes (cloud-native — read-only discovery)
import { KubernetesRegistryBackend } from '@hazeljs/discovery';
const backend = new KubernetesRegistryBackend(kubeConfig, {
  namespace: 'production',
  labelSelector: 'app.kubernetes.io/managed-by=hazeljs',
});

Putting It All Together

Here's what a complete microservices architecture looks like when you combine all three packages. This is the exact pattern used in the hazeljs-gateway-starter project:

import { GatewayServer, Gateway, Route, ServiceRoute,
  VersionRoute, Canary, GatewayCircuitBreaker } from '@hazeljs/gateway';
import { ServiceRegistry, DiscoveryClient } from '@hazeljs/discovery';
import { CircuitBreaker, RetryPolicy, MetricsRegistry } from '@hazeljs/resilience';

// 1. Register services with discovery
const backend = new MemoryRegistryBackend();

const userRegistry = new ServiceRegistry({
  name: 'user-service', port: 3001,
  zone: 'us-east-1', tags: ['api', 'v1'],
  metadata: { version: '1.0.0' },
}, backend);
await userRegistry.register();

// 2. Define gateway with decorators
@Gateway({
  discovery: { cacheEnabled: true, cacheTTL: 10_000 },
  resilience: { defaultTimeout: 10_000 },
  metrics: { enabled: true },
})
class ProductionGateway {
  @Route({ path: '/api/users/**' })
  @ServiceRoute({ serviceName: 'user-service', stripPrefix: '/api/users' })
  @VersionRoute({
    strategy: 'header',
    defaultVersion: 'v1',
    routes: {
      v1: { weight: 70, filter: { tags: ['v1'] } },
      v2: { weight: 30, filter: { tags: ['v2'] } },
    },
  })
  userRoutes: any;

  @Route({ path: '/api/orders/**' })
  @ServiceRoute({ serviceName: 'order-service' })
  @Canary({
    stable: { version: '1.0.0', weight: 90 },
    canary: { version: '2.0.0', weight: 10 },
    promotion: {
      strategy: 'error-rate', errorThreshold: 5,
      autoPromote: true, autoRollback: true,
      steps: [10, 25, 50, 75, 100], stepInterval: '15s',
    },
  })
  @GatewayCircuitBreaker({ failureThreshold: 3 })
  orderRoutes: any;
}

// 3. Start the gateway
const gateway = GatewayServer.fromClass(ProductionGateway, backend);
gateway.startCanaries();

// 4. Monitor everything
gateway.on('canary:promote', (e) => alert(`Canary promoted: ${e.route}`));
gateway.on('circuit:open', (e) => alert(`Circuit opened: ${e.route}`));

const metrics = gateway.getMetrics().getSnapshot();
console.log(`Total calls: ${metrics.aggregated.totalCalls}`);
console.log(`Failure rate: ${metrics.aggregated.failureRate}`);

The Starter Project

We've published a complete hazeljs-gateway-starter application with 18 runnable demos that cover every feature of all three packages. Each demo is self-contained and thoroughly documented.

# Clone and run
cd hazeljs-gateway-starter
npm install

# Interactive demo menu
npm start

# Run specific demos
npm run demo:gateway:decorator       # Decorator-based gateway
npm run demo:resilience:circuit-breaker  # Circuit breaker patterns
npm run demo:discovery:load-balancing    # All 6 strategies
npm run demo:scenario:full-stack     # Complete integration

                    ┌──────────────────────────────────┐
                    │          API Gateway             │
                    │  Routes → Versions → Canary      │
                    │  Circuit Breaker + Rate Limit     │
                    └──────────────┬───────────────────┘
                                   │
                    ┌──────────────┴───────────────────┐
                    │       Service Discovery           │
                    │  Cache → Load Balancer → Backend  │
                    └──────────────┬───────────────────┘
                                   │
          ┌─────────┬──────────┬───┴────┬──────────┐
          │ User    │ Order    │Product │ Payment  │
          │ Service │ Service  │Service │ Service  │
          └─────────┴──────────┴────────┴──────────┘

What's Next

The gateway, resilience, and discovery packages are available now in HazelJS v0.2.0-beta. We're actively working on:

• gRPC support — native gRPC proxying in the gateway
• OpenTelemetry integration — distributed tracing across all services
• Dashboard — real-time visualization of gateway metrics, canary status, and circuit breaker states
• etcd backend — for distributed service discovery

Get Started

npm install @hazeljs/gateway @hazeljs/resilience @hazeljs/discovery

Check out the hazeljs-gateway-starter project for a complete, runnable example of everything covered in this post. Every feature, every pattern, every configuration option — all demonstrated with clear, documented code.