github

开源监控方案：Prometheus + Grafana搭建指南

📅 2026/7/31 👁️ 1 阅读 ✏️ 2026/7/31 更新

开源监控方案：Prometheus + Grafana搭建指南

在云原生时代，系统监控是保障服务稳定运行的基石。Prometheus + Grafana 已成为事实上的开源监控标准方案，被 Kubernetes、CNCF 生态广泛采用。本文将带你从零搭建一套完整的监控体系。

监控架构概览

一套完整的监控系统包含以下组件：

text
[被监控服务] -> [Exporter] -> [Prometheus] -> [Alertmanager] -> [通知渠道]
                                   |
                                   v
                               [Grafana] -> [可视化Dashboard]

  • Prometheus：时序数据库 + 采集引擎，负责数据存储和查询

  • Exporter：数据采集器，从各服务中暴露指标

  • Grafana：可视化面板，展示监控数据

  • Alertmanager：告警管理，处理告警规则和通知

    • Docker Compose 一键部署

    yaml
    # docker-compose.yml
version: "3.8"

networks:
  monitoring:
    driver: bridge

volumes:
  prometheus_data:
  grafana_data:

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports: ["9090:9090"]
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=30d"
      - "--web.enable-lifecycle"
    networks: [monitoring]
    restart: always

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports: ["3000:3000"]
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin123
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
    networks: [monitoring]
    restart: always

  alertmanager:
    image: prom/alertmanager:latest
    container_name: alertmanager
    ports: ["9093:9093"]
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    networks: [monitoring]
    restart: always

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    ports: ["9100:9100"]
    pid: host
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - "--path.procfs=/host/proc"
      - "--path.sysfs=/host/sys"
      - "--path.rootfs=/rootfs"
    networks: [monitoring]
    restart: always

    Prometheus 配置

    核心配置文件

    yaml
    # prometheus.yml
global:
  scrape_interval: 15s          # 采集间隔
  evaluation_interval: 15s      # 规则评估间隔
  scrape_timeout: 10s

# 告警规则文件
rule_files:
  - "rules/*.yml"

# 告警管理器
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

# 采集目标配置
scrape_configs:
  # Prometheus 自身监控
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  # 主机节点监控
  - job_name: "node-exporter"
    static_configs:
      - targets: ["node-exporter:9100"]
        labels:
          env: "production"

  # 应用服务监控
  - job_name: "my-app"
    metrics_path: /metrics
    static_configs:
      - targets: ["app:8080"]
        labels:
          service: "api-server"

  # Docker 容器监控
  - job_name: "cadvisor"
    static_configs:
      - targets: ["cadvisor:8080"]

    常用 Exporter

    | Exporter | 监控对象 | 默认端口 |
    |----------|---------|---------|
    | node_exporter | 主机(CPU/内存/磁盘) | 9100 |
    | mysqld_exporter | MySQL | 9104 |
    | postgres_exporter | PostgreSQL | 9187 |
    | redis_exporter | Redis | 9121 |
    | nginx_exporter | Nginx | 9113 |
    | blackbox_exporter | HTTP/TCP探测 | 9115 |

    告警规则配置

    定义告警规则

    yaml
    # rules/alerts.yml
groups:
  - name: node_alerts
    rules:
      # CPU 使用率过高
      - alert: HighCpuUsage
        expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "CPU使用率过高 ({{ $labels.instance }})"
          description: "CPU使用率: {{ $value }}%，超过80%已持续5分钟"

      # 内存不足
      - alert: HighMemoryUsage
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 85
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "内存使用率过高 ({{ $labels.instance }})"
          description: "内存使用率: {{ $value }}%"

      # 磁盘空间不足
      - alert: DiskSpaceLow
        expr: (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) * 100 > 90
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "磁盘空间不足 ({{ $labels.instance }} {{ $labels.mountpoint }})"
          description: "磁盘使用率: {{ $value }}%"

      # 服务宕机
      - alert: ServiceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "服务不可达 ({{ $labels.job }})"
          description: "{{ $labels.instance }} 已离线超过1分钟"

    Alertmanager 通知配置

    yaml
    # alertmanager.yml
global:
  resolve_timeout: 5m

# 告警路由
route:
  group_by: ["alertname", "severity"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: "default"
  routes:
    - match:
        severity: critical
      receiver: "critical-alerts"
    - match:
        severity: warning
      receiver: "warning-alerts"

receivers:
  - name: "default"
    webhook_configs:
      - url: "http://your-webhook/notify"

  - name: "critical-alerts"
    webhook_configs:
      - url: "https://hooks.slack.com/services/your-webhook"
        send_resolved: true

  - name: "warning-alerts"
    webhook_configs:
      - url: "https://hooks.slack.com/services/your-webhook"

    Grafana Dashboard 配置

    添加数据源

    在 Grafana 中添加 Prometheus 数据源：

    text
    URL: http://prometheus:9090
Access: Server (default)
Scrape interval: 15s

    常用 PromQL 查询

    promql
    # CPU 使用率
100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# 内存使用率
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

# 磁盘使用率
(1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100

# 网络流量
rate(node_network_receive_bytes_total{device!="lo"}[5m])

# HTTP 请求QPS
rate(http_requests_total[5m])

# 请求延迟P99
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

    导入社区 Dashboard

    Grafana 社区有大量现成的 Dashboard 模板：

  • Node Exporter Full (ID: 1860)：主机完整监控面板

  • Docker Monitoring (ID: 179)：Docker容器监控

  • MySQL Overview (ID: 7362)：MySQL监控

  • Redis Dashboard (ID: 11835)：Redis监控

    • 在 Grafana 中：Dashboards -> Import -> 输入 Dashboard ID -> 选择数据源。

    应用自定义指标

    Python 应用埋点

    python
    from prometheus_client import Counter, Histogram, Gauge, generate_latest, CONTENT_TYPE_LATEST
from flask import Flask, Response

app = Flask(__name__)

# 定义指标
REQUEST_COUNT = Counter(
    "http_requests_total",
    "Total HTTP requests",
    ["method", "endpoint", "status"]
)

REQUEST_LATENCY = Histogram(
    "http_request_duration_seconds",
    "HTTP request latency",
    ["endpoint"],
    buckets=[0.01, 0.05, 0.1, 0.5, 1, 5]
)

ACTIVE_USERS = Gauge(
    "active_users",
    "Number of active users"
)

@app.before_request
def before_request():
    import time
    request.start_time = time.time()

@app.after_request
def after_request(response):
    REQUEST_COUNT.labels(
        method=request.method,
        endpoint=request.path,
        status=response.status_code
    ).inc()
    
    REQUEST_LATENCY.labels(endpoint=request.path).observe(
        time.time() - request.start_time
    )
    return response

@app.route("/metrics")
def metrics():
    return Response(generate_latest(), mimetype=CONTENT_TYPE_LATEST)

# 使用示例
ACTIVE_USERS.set(42)

    Java (Spring Boot) 埋点

    xml
    <!-- pom.xml -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

    yaml
    # application.yml
management:
  endpoints:
    web:
      exposure:
        include: health,info,prometheus
  metrics:
    tags:
      application: my-service

    监控最佳实践

    1. 黄金信号监控

    Google SRE 提出的四大黄金信号：

  • 延迟：请求处理时间

  • 流量：请求QPS

  • 错误：错误率

  • 饱和度：资源使用率

    • 2. 告警分级

    yaml
    # 告警分级标准
critical:  # 立即处理
  - 服务完全不可用
  - 数据丢失风险
  - 磁盘使用率 > 95%

warning:   # 工作时间内处理
  - 性能下降
  - 资源使用率偏高
  - 磁盘使用率 > 85%

info:      # 仅记录不通知
  - 配置变更
  - 部署完成

    3. 数据保留策略

    yaml
    # Prometheus 数据保留
--storage.tsdb.retention.time=30d    # 保留30天
--storage.tsdb.retention.size=50GB   # 最大50GB

# 长期存储推荐使用 Thanos 或 VictoriaMetrics

    总结

    Prometheus + Grafana 组合为开发者提供了一套强大、灵活、完全开源的监控方案。通过本文的指南，你已经可以搭建起包含主机监控、应用监控、告警通知和可视化面板的完整监控体系。监控不是一次性工作，而是一个持续优化的过程——随着系统演进，不断调整指标、优化告警规则、完善 Dashboard，才能真正发挥监控的价值。记住，好的监控应该做到"在用户发现问题之前发现问题"，这才是运维的最高境界。

    💬 评论区 (0)

    0/500

    暂无评论，快来抢沙发吧！