Hadrian is experimental alpha software. Do not use in production.
Hadrian
Deployment

Advanced Deployments

Production-grade deployment configurations with Vault, observability, Keycloak, and high availability

This guide covers advanced Docker Compose configurations for production-grade deployments, including secret management, observability, enterprise authentication, load balancing, and high availability.

Available Configurations

ConfigurationUse CaseComponents
vault.ymlSecret managementPostgreSQL, Redis, HashiCorp Vault
observability.ymlFull monitoring stackOTEL, Prometheus, Grafana, Jaeger, Loki
keycloak.ymlEnterprise authenticationKeycloak, OAuth2 Proxy
traefik.ymlLoad-balanced gatewayTraefik, 3× Gateway instances
postgres-ha.ymlDatabase high availabilityPostgreSQL Primary + 2 Replicas, PgBouncer
redis-cluster.ymlCache high availability6-node Redis Cluster
dlq.ymlAsync processing & retriesRabbitMQ or NATS JetStream
production.ymlFull production referenceAll components combined

Secret Management with Vault

Uses HashiCorp Vault for secure secret storage with Vault Agent for automatic secret injection.

Setup

# 1. Start Vault
docker compose -f docker-compose.vault.yml up vault -d

# 2. Initialize Vault (save the unseal keys and root token!)
docker exec hadrian-vault vault operator init

# 3. Unseal Vault (repeat with 3 different keys)
docker exec hadrian-vault vault operator unseal <key-1>
docker exec hadrian-vault vault operator unseal <key-2>
docker exec hadrian-vault vault operator unseal <key-3>

# 4. Login and configure
export VAULT_ADDR=http://localhost:8200
vault login <root-token>

# 5. Enable KV secrets engine and store secrets
vault secrets enable -path=secret kv-v2
vault kv put secret/gateway \
  openrouter_api_key="sk-or-..." \
  anthropic_api_key="sk-ant-..." \
  openai_api_key="sk-..."

# 6. Create AppRole for gateway (see Vault docs)
# 7. Start all services
docker compose -f docker-compose.vault.yml up -d

Save the unseal keys and root token securely! You'll need them to unseal Vault after restarts. For production, consider using auto-unseal with a cloud KMS.

Observability Stack

Full monitoring, tracing, and logging stack with OpenTelemetry.

Setup

docker compose -f docker-compose.observability.yml up -d

Endpoints

ServiceURLCredentials
Gatewayhttp://localhost:8080-
Grafanahttp://localhost:3001admin/admin
Prometheushttp://localhost:9090-
Jaeger UIhttp://localhost:16686-
Alertmanagerhttp://localhost:9093-

Features

  • OpenTelemetry Collector: Unified telemetry pipeline for metrics, traces, and logs
  • Prometheus: Metrics collection with pre-configured alerting rules
  • Grafana: Dashboards with auto-provisioned datasources
  • Jaeger: Distributed tracing for request flow visualization
  • Loki + Promtail: Log aggregation from all containers
  • Alertmanager: Alert routing to Slack, PagerDuty, or email

Alerting Configuration

Edit config/alertmanager.yml to configure alert destinations:

receivers:
  - name: "slack"
    slack_configs:
      - api_url: "https://hooks.slack.com/services/..."
        channel: "#alerts"
  - name: "pagerduty"
    pagerduty_configs:
      - service_key: "<integration-key>"

Enterprise Authentication with Keycloak

Enterprise identity management with OIDC/OAuth2.

Setup

docker compose -f docker-compose.keycloak.yml up -d

Endpoints

ServiceURLCredentials
Gatewayhttp://localhost:8080-
Keycloak Adminhttp://localhost:8080admin/admin
OIDC Discoveryhttp://localhost:8080/realms/hadrian/.well-known/openid-configuration-

Configuration Steps

  1. Access Keycloak admin console
  2. Create realm "hadrian"
  3. Create client "hadrian" with confidential access type
  4. Configure client credentials in .env:
OIDC_CLIENT_ID=hadrian
OIDC_CLIENT_SECRET=<client-secret-from-keycloak>

OAuth2 Proxy (Optional)

Add OAuth2 Proxy for browser-based authentication:

docker compose -f docker-compose.keycloak.yml --profile oauth2-proxy up -d
# Access gateway through OAuth2 Proxy at http://localhost:4180

Load Balancing with Traefik

Reverse proxy with automatic HTTPS, load balancing, and rate limiting.

Setup

docker compose -f docker-compose.traefik.yml up -d

Endpoints

ServiceURL
Gateway (HTTPS)https://gateway.localhost
Traefik Dashboardhttp://localhost:8080

Features

  • 3 gateway instances with round-robin load balancing
  • Automatic HTTPS with Let's Encrypt (configure for production)
  • Rate limiting: 100 req/s with 50 burst
  • Security headers: HSTS, XSS protection, content-type sniffing prevention
  • Sticky sessions for consistent routing
  • Health checks with automatic failover

Production HTTPS

For production deployments with real SSL certificates:

  1. Update environment variables:

    DOMAIN=gateway.example.com
ACME_EMAIL=admin@example.com

  2. Uncomment Let's Encrypt configuration in the compose file

  3. Ensure ports 80/443 are publicly accessible for ACME challenges

Database High Availability

PostgreSQL with streaming replication for read scaling and failover.

Setup

docker compose -f docker-compose.postgres-ha.yml up -d

Connection Endpoints

ConnectionHostPort
Primary (writes)pgbouncer-primary6432
Replicas (reads)pgbouncer-replica6433

Architecture

  • Primary PostgreSQL: Handles all write operations
  • 2 Read Replicas: Streaming replication for read scaling
  • PgBouncer: Connection pooling for both primary and replicas
  • HAProxy (optional): Load balancing across replicas

Gateway Configuration

Configure read/write splitting in hadrian.toml:

[database]
url = "postgres://gateway:pass@pgbouncer-primary:6432/gateway"
read_url = "postgres://gateway:pass@pgbouncer-replica:6432/gateway"

Read replicas are eventually consistent. Use the primary connection for operations that require immediate consistency.

Cache High Availability

6-node Redis Cluster for high availability caching.

Setup

# Start Redis nodes
docker compose -f docker-compose.redis-cluster.yml up -d

# Initialize cluster (one-time)
docker compose -f docker-compose.redis-cluster.yml --profile init up redis-cluster-init

Cluster Topology

NodeRole
redis-1, redis-2, redis-3Masters
redis-4, redis-5, redis-6Replicas

The cluster provides automatic failover if a master node goes down.

Redis Insight UI (Optional)

docker compose -f docker-compose.redis-cluster.yml --profile ui up -d redis-insight
# Access at http://localhost:8001

Async Processing with Message Queues

RabbitMQ for reliable async operations with dead letter handling.

Setup with RabbitMQ

docker compose -f docker-compose.dlq.yml up -d

Endpoints

ServiceURLCredentials
Gatewayhttp://localhost:8080-
RabbitMQ Managementhttp://localhost:15672guest/guest

Pre-configured Queues

QueuePurpose
usage.trackingAsync usage tracking with DLQ
webhooksWebhook delivery with priority and DLQ
audit.logAudit log stream

Alternative: NATS JetStream

For a lighter-weight message queue:

docker compose -f docker-compose.dlq.yml --profile nats up -d
# NATS monitoring: http://localhost:8222

Full Production Reference

Comprehensive production setup combining all components.

Setup

# Configure all environment variables
cp ../.env.example .env
# Edit .env with production values

# Start everything
docker compose -f docker-compose.production.yml up -d

Included Components

  • Traefik with automatic HTTPS
  • 3 load-balanced gateway instances
  • PostgreSQL with read replica and PgBouncer
  • 3-node Redis cluster
  • RabbitMQ for async processing
  • Keycloak for authentication
  • Full observability stack (OTEL, Prometheus, Grafana, Jaeger, Loki)

Network Isolation

NetworkServices
frontendTraefik, Grafana, Keycloak (public-facing)
backendDatabases, Redis, RabbitMQ (internal only)

Configuration Files

The deploy/config/ directory contains configuration for all services:

config/
├── alertmanager.yml          # Alert routing (Slack, PagerDuty, email)
├── grafana/
│   └── provisioning/
│       ├── dashboards/       # Dashboard auto-provisioning
│       └── datasources/      # Datasource configuration
├── loki.yaml                 # Log aggregation settings
├── nats.conf                 # NATS JetStream configuration
├── otel-collector.yaml       # OpenTelemetry pipeline
├── prometheus.yml            # Prometheus scrape config
├── prometheus-alerts.yml     # Alerting rules
├── promtail.yaml             # Log collection from Docker
├── rabbitmq/
│   ├── definitions.json      # Queue/exchange topology
│   └── rabbitmq.conf         # RabbitMQ settings
├── redis-cluster.conf        # Redis cluster settings
├── traefik/
│   └── dynamic/              # Dynamic Traefik configuration
├── vault/
│   └── config.hcl            # Vault server configuration
└── vault-agent/
    ├── agent.hcl             # Vault Agent configuration
    └── templates/            # Secret templates

Environment Variables

Key variables for advanced deployments:

VariableDescriptionRequired For
OPENROUTER_API_KEYOpenRouter API keyAll
POSTGRES_PASSWORDPostgreSQL passwordpostgres, postgres-ha, production
RABBITMQ_PASSWORDRabbitMQ passworddlq, production
KEYCLOAK_ADMIN_PASSWORDKeycloak admin passwordkeycloak, production
OIDC_CLIENT_SECRETOIDC client secretkeycloak, production
GRAFANA_PASSWORDGrafana admin passwordobservability, production
DOMAINYour domain nametraefik, production
ACME_EMAILLet's Encrypt emailtraefik, production

Scaling

Horizontal Scaling (Gateway)

# Scale to 5 gateway instances
docker compose -f docker-compose.traefik.yml up -d --scale gateway=5

Vertical Scaling (Resources)

Add resource limits to any service:

services:
  gateway:
    deploy:
      resources:
        limits:
          cpus: "2"
          memory: 2G
        reservations:
          cpus: "0.5"
          memory: 512M

Health Checks

All services include health checks. Monitor status:

# Check all service health
docker compose -f docker-compose.postgres.yml ps

# View health check details
docker inspect --format='{{json .State.Health}}' hadrian | jq

Troubleshooting

View Logs

# All services
docker compose -f docker-compose.postgres.yml logs -f

# Specific service
docker compose -f docker-compose.postgres.yml logs -f gateway

# Last 100 lines
docker compose -f docker-compose.postgres.yml logs --tail=100 gateway

Common Issues

Gateway can't connect to database:

# Check database is healthy
docker compose -f docker-compose.postgres.yml ps postgres
# Check network connectivity
docker exec hadrian ping postgres

Redis cluster not forming:

# Check cluster status
docker exec hadrian-redis-1 redis-cli cluster info
# Re-run cluster init
docker compose -f docker-compose.redis-cluster.yml --profile init up redis-cluster-init

Vault sealed after restart:

# Unseal with your keys
docker exec hadrian-vault vault operator unseal <key>

Production Checklist

  • Configure real domain name and SSL certificates
  • Set strong passwords for all services
  • Enable Vault with proper unsealing strategy (auto-unseal recommended)
  • Configure alerting destinations (Slack, PagerDuty, email)
  • Set up database backups
  • Configure log retention policies
  • Review and adjust resource limits
  • Set up external monitoring (uptime checks)
  • Configure firewall rules (only expose necessary ports)
  • Enable audit logging

Next Steps

Docker Deployment

Deploy Hadrian Gateway using Docker and Docker Compose

Security

Security model and best practices for Hadrian Gateway

On this page

Available ConfigurationsSecret Management with VaultSetupObservability StackSetupEndpointsFeaturesAlerting ConfigurationEnterprise Authentication with KeycloakSetupEndpointsConfiguration StepsOAuth2 Proxy (Optional)Load Balancing with TraefikSetupEndpointsFeaturesProduction HTTPSDatabase High AvailabilitySetupConnection EndpointsArchitectureGateway ConfigurationCache High AvailabilitySetupCluster TopologyRedis Insight UI (Optional)Async Processing with Message QueuesSetup with RabbitMQEndpointsPre-configured QueuesAlternative: NATS JetStreamFull Production ReferenceSetupIncluded ComponentsNetwork IsolationConfiguration FilesEnvironment VariablesScalingHorizontal Scaling (Gateway)Vertical Scaling (Resources)Health ChecksTroubleshootingView LogsCommon IssuesProduction ChecklistNext Steps