k8s的部署管理以及prometheus相关监控

安装运行prometheus和grafana
编辑prometheus配置文件，增加采集prometheus自身数据的采集规则
grafana上导入prometheus相关图表的dashboard

前面三步主要根据https://blog.csdn.net/shnu_cdk/article/details/132182858?spm=1001.2014.3001.5506

用Go编写一个prometheus exporter，包含prometheus四种指标类型

这一部分直接查看官方文档使用说明。

package main

import (
	"fmt"
	"math/rand"
	"net/http"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
	// 创建并注册指标
	counter := prometheus.NewCounter(prometheus.CounterOpts{
		Name: "my_counter",
		Help: "A counter metric",
	})
	prometheus.MustRegister(counter)

	gauge := prometheus.NewGauge(prometheus.GaugeOpts{
		Name: "my_gauge",
		Help: "A gauge metric",
	})
	prometheus.MustRegister(gauge)

	histogram := prometheus.NewHistogram(prometheus.HistogramOpts{
		Name: "my_histogram",
		Help: "A histogram metric",
	})
	prometheus.MustRegister(histogram)

	summary := prometheus.NewSummary(prometheus.SummaryOpts{
		Name: "my_summary",
		Help: "A summary metric",
	})
	prometheus.MustRegister(summary)

	// 定期更新指标值
	go func() {
		for {
			counter.Inc()
			gauge.Set(rand.Float64() * 100)
			histogram.Observe(rand.Float64() * 100)
			summary.Observe(rand.Float64() * 100)
			time.Sleep(time.Second)
		}
	}()

	// 启动HTTP服务，暴露指标
	http.Handle("/metrics", promhttp.Handler())
	fmt.Println("Exporter is running on http://localhost:8080/metrics")
	http.ListenAndServe(":8080", nil)
}

prometheus能够正常采集第四步中的exporter，并在grafana上展示
在yml配置文件上配置相对应的端口，这一部分和配置一个node差不多：

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]

  
  - job_name: 'export'
    
    static_configs:
      - targets: ['192.168.60.177:8080']
        labels:
          instance: export

了解alertmanager，知道alertmanager的webhook推送告警方式
主要是要在prometheus上配置相应的端口

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['192.168.60.169:9093']

然后在altermanager里面配置相应的路由树

route:
  group_by: ['alertname']
  group_wait: 1s
  group_interval: 1s
  repeat_interval: 1h
  receiver: 'web.hook'
receivers:
  - name: 'web.hook'
    webhook_configs:
      - url: 'http://192.168.60.188:8093/demo'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']