Prometheus监控Linux

Linux 客户端安装docker 

export DOWNLOAD_URL="http://mirrors.163.com/docker-ce"
curl -fsSl https://get.docker.com/ |sh
apt install docker-compose -y

创建node-exporter

mkdir -p /data/docker-compose
cd /data/docker-compose
 
cat >docker-compose.yaml <<'EOF'
version: '3.3'
networks:
  monitoring:
    driver: bridge
 
services:
  node_exporter:
    image: prom/node-exporter:v1.5.0
    container_name: node-exporter
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command: 
      - '--path.procfs=/host/proc' 
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc|rootfs/var/lib/docker)($$|/)'
    networks:
      - monitoring
    ports:
      - '9100:9100'
EOF

启动docker

docker-compose up -d

浏览器验证

http://ip:9100/metrics

 prometheus节点操作

配置prometheeus/prometheus.yaml

  - job_name: 'node-exporter'
    scrape_interval: 15s
    static_configs:
    - targets: ['node_exporter:9100']
      labels:
        instance: Prometheus服务器 
    - targets: ['10.19.1.206:9100']
      labels:
        instance: 10.19.1.206服务器 
    - targets: ['10.19.1.220:9100']
      labels:
        instance: 10.19.1.220服务器

配置加载
curl -X POST http://localhost:9090/-/reload
检查容器状态
docker ps -a
docker logs -f node-exporter

CPU采集
node_cpu_seconds_total
node_cpu_seconds_total{ instance=”10.19.1.220服务器”}
node_load1
node_load5
node_load15

内存采集
node_memory_MemTotal_bytes
node_memory_MemAvailable_bytes (free+buffer+cache)
node_memory_MemFree_bytes
node_memory_SwapFree_bytes
node_memory_SwapTotal_bytes

磁盘采集
node_disk_

文件系统采集
node_filesystem_

网络采集
node_network_
node_network_transmit_drop_total
增加触发器配置

cat >> prometheus/alert.yml <<'EOF'
- name: node-exporter
  rules:
  - alerts: HostOutOfMemory
    expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10
	for: 2m
	labels:
	  severity: warning
	annotations:
	  summary: "主机内存不足，实例:{{ $labels.instance }}"
	  description: "内存可用率<10%,当前值:{{ $value }}"	  
  - alerts: HostMemoryUnderMemoryPressure
    expr: rate(node_vmstat_pgmajfault[1m]) > 1000
	for: 2m
	labels:
	  severity: warning
	annotations:
	  summary: "主机压力不足，实例:{{ $labels.instance }}"
	  description: "节点内存压力大，重大页面错误率高,当前值:{{ $value }}"   
  - alerts: HostUnusualNetworkThroughputIn
    expr: sum by (instance) (rate(node_network_receive_bytes_total[2m])) / 1024 / 1024 > 100
	for: 5m
	labels: 
	  severiry: warning
	annotations: 
	  summary: "异常流出网络吞吐量，实例: {{ $labels.instance }}"
	  description: "网络流入量 > 100 MB/s,当前值: {{ $value }}"  
  - alerts: HostUnusualDiskReadRate
    expr: sum by (instance) (rate(node_disk_read_bytes_total[2m])) / 1024 /1024 > 50
	for: 5m
	labels:
	  severity: warning
	annotations:
	  summary: "异常流出网络吞吐量，实例:{{ $labels.instance }}"
	  description: "网络流出流量 > 100 MB/s,当前值: {{ $value }}"
EOF

1. 检查配置
2. docker exec -it prometheus promtool check config /etc/prometheus/prometheus.yml

grafana展示node-exporter的数据