Observability at the Edge: OpenTelemetry & Grafana Alloy Setup

Instrumenting edge applications shouldn't require complex or heavy local setups. Deploying a standard OpenTelemetry (OTel) agent with Grafana Alloy and running a local LGTM (Loki, Grafana, Tempo, Mimir/Prometheus) stack in Docker Compose gives you instant, enterprise-grade diagnostics.

1. The LGTM Local Stack

To visualize your traces, logs, and metrics, you need a local aggregation layer. The Docker Compose setup below spins up the core components of the Grafana observability stack:

• Grafana: The visualization dashboard.
• Prometheus: Stores metrics exported by the clients.
• Loki: Aggregates application and transport logs.
• Tempo: Traces application logic flow across network boundaries.

# docker-compose.lgtm.yml
version: "3.8"

services:
  # 1. Grafana for visualization
  grafana:
    image: grafana/grafana:10.4.1
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    volumes:
      - grafana-storage:/var/lib/grafana

  # 2. Prometheus for metrics storage
  prometheus:
    image: prom/prometheus:v2.51.0
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--enable-feature=otlp-write-receiver"
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

  # 3. Loki for log aggregation
  loki:
    image: grafana/loki:2.9.8
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml

  # 4. Tempo for trace storage
  tempo:
    image: grafana/tempo:2.4.1
    command: [ "-config.file=/etc/tempo.yaml" ]
    ports:
      - "3200:3200" # tempo HTTP
      - "4317:4317" # otlp grpc
      - "4318:4318" # otlp http
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml

volumes:
  grafana-storage:
      

2. Configuring the Alloy Collector

Grafana Alloy acts as a lightweight telemetry pipeline receiver. It accepts standard OTLP (OpenTelemetry Protocol) payloads on your local network, filters them, batches them, and routes them to Loki, Tempo, and Prometheus:

# config.alloy
// 1. Listen for OTLP gRPC and HTTP payloads
otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
  }
  http {
    endpoint = "0.0.0.0:4318"
  }

  output {
    metrics = [otelcol.processor.batch.default.input]
    traces  = [otelcol.processor.batch.default.input]
    logs    = [otelcol.processor.batch.default.input]
  }
}

// 2. Batch telemetry events to optimize flow
otelcol.processor.batch "default" {
  output {
    metrics = [otelcol.exporter.prometheus.local.input]
    traces  = [otelcol.exporter.otlp.tempo.input]
    logs    = [otelcol.exporter.loki.local.input]
  }
}

// 3. Export Metrics to Prometheus
otelcol.exporter.prometheus "local" {
  forward_to = [prometheus.remote_write.local.receiver]
}

prometheus.remote_write "local" {
  endpoint {
    url = "http://prometheus:9090/api/v1/write"
  }
}

// 4. Export Traces to Tempo
otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo:4317"
    tls {
      insecure = true
    }
  }
}

// 5. Export Logs to Loki
otelcol.exporter.loki "local" {
  forward_to = [loki.write.local.receiver]
}

loki.write "local" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}
      

3. Instrumenting Edge Clients

Once the LGTM collector and Alloy pipeline are active, instrumenting your application involves pointing the standard OpenTelemetry SDK exporters to the Alloy endpoint:

• Traces Endpoint: http://localhost:4318/v1/traces
• Logs Endpoint: http://localhost:4318/v1/logs

In modern mobile client SDKs, these are typically initialized during launch by specifying the resource details and attaching the batch span processors. You can now use the Grafana explorer to trace client-side interactions down to the millisecond, isolating network bottlenecks instantly.