外部 prometheus监控k8s集群资源（pod、CPU、service、namespace、deployment等）

news/2024/5/19 1:32:36 标签: prometheus, kubernetes, 容器

prometheus监控k8s集群资源

  • 一，通过CADvisior 监控pod的资源状态
    • 1.1 授权外边用户可以访问prometheus接口。
    • 1.2 获取token保存
    • 1.3 配置prometheus.yml 启动并查看状态
    • 1.4 Grafana 导入仪表盘
  • 二，通过kube-state-metrics 监控k8s资源状态
    • 2.1 部署 kube-state-metrics
    • 2.2 配置prometheus.yml
    • 2.3 Grafana 导入仪表盘
    • 2.4 Grafana没有数据，添加路由转发

二进制安装的prometheus，监控k8s集群信息。

监控指标实现方式举例
Pod资源利用率cAdvisor容器CPU、内存利用率
K8s资源状态kube-state-metricscontroller控制器、Node、Namespace、Pod、ReplicaSet、service等

一，通过CADvisior 监控pod的资源状态

prometheus_12">1.1 授权外边用户可以访问prometheus接口。

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "extensions"
  resources:
    - ingresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: kube-system

kubectl apply -f rbac.yaml

1.2 获取token保存

kubectl get secrets -n kube-system |grep prometheus #查看toekn name
name：prometheus-token-vgxhc

kubectl describe secret prometheus-token-vgxhc -n kube-system > token.k8s
#kubectl get secrets -n kube-system -o yaml prometheus-token-vgxhc |grep token
scp token.k8s prometheus #拷贝到prometheus服务器prometheus的目录下

我的token放在 /opt/monitor/prometheus/token.k8s

prometheusyml__87">1.3 配置prometheus.yml 启动并查看状态

vim prometheus.yml

  - job_name: kubernetes-nodes-cadvisor
    metrics_path: /metrics
    scheme: https
    kubernetes_sd_configs:
    - role: node
      api_server: https://172.18.0.0:6443
      bearer_token_file: /opt/monitor/prometheus/token.k8s 
      tls_config:
        insecure_skip_verify: true
    bearer_token_file: /opt/monitor/prometheus/token.k8s 
    tls_config:
      insecure_skip_verify: true
    relabel_configs:
    # 将标签(.*)作为新标签名，原有值不变
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.*)
    # 修改NodeIP:10250为APIServerIP:6443
    - action: replace
      regex: (.*)
      source_labels: ["__address__"]
      target_label: __address__
      replacement: 172.18.0.0:6443
    # 实际访问指标接口 https://NodeIP:10250/metrics/cadvisor 这个接口只能APISERVER访问，故此重新标记标签使用APISERVER代理访问
    - action: replace
      source_labels: [__meta_kubernetes_node_name]
      target_label: __metrics_path__
      regex: (.*)
      replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor 


./promtool check config prometheus.yml 
重启prometheus 或 kill -HUP PrometheusPid

在prometheus的target页面查看
http://172.18.0.0:9090
在这里插入图片描述

1.4 Grafana 导入仪表盘

导入3119 仪表盘
在这里插入图片描述
在这里插入图片描述完成pod资源监控

二，通过kube-state-metrics 监控k8s资源状态

2.1 部署 kube-state-metrics

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups: [""]
  resources:
  - configmaps
  - secrets
  - nodes
  - pods
  - services
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  - endpoints
  verbs: ["list", "watch"]
- apiGroups: ["apps"]
  resources:
  - statefulsets
  - daemonsets
  - deployments
  - replicasets
  verbs: ["list", "watch"]
- apiGroups: ["batch"]
  resources:
  - cronjobs
  - jobs
  verbs: ["list", "watch"]
- apiGroups: ["autoscaling"]
  resources:
  - horizontalpodautoscalers
  verbs: ["list", "watch"]
- apiGroups: ["networking.k8s.io", "extensions"]
  resources:
  - ingresses 
  verbs: ["list", "watch"]
- apiGroups: ["storage.k8s.io"]
  resources:
  - storageclasses 
  verbs: ["list", "watch"]
- apiGroups: ["certificates.k8s.io"]
  resources:
  - certificatesigningrequests
  verbs: ["list", "watch"]
- apiGroups: ["policy"]
  resources:
  - poddisruptionbudgets 
  verbs: ["list", "watch"]



---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kube-state-metrics-resizer
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups: [""]
  resources:
  - pods
  verbs: ["get"]
- apiGroups: ["extensions","apps"]
  resources:
  - deployments
  resourceNames: ["kube-state-metrics"]
  verbs: ["get", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1 
kind: ClusterRoleBinding
metadata:
  name: kube-state-metrics
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kube-state-metrics-resizer
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: kube-system

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    k8s-app: kube-state-metrics
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    version: v1.3.0
spec:
  selector:
    matchLabels:
      k8s-app: kube-state-metrics
      version: v1.3.0
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: kube-state-metrics
        version: v1.3.0
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      priorityClassName: system-cluster-critical
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
        image: harbor.cpit.com.cn/monitor/kube-state-metrics:v1.8.0
        ports:
        - name: http-metrics
          containerPort: 8080
        - name: telemetry
          containerPort: 8081
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 5
      - name: addon-resizer
        image: harbor.cpit.com.cn/monitor/addon-resizer:1.8.6
        resources:
          limits:
            cpu: 1000m
            memory: 500Mi
          requests:
            cpu: 1000m
            memory: 500Mi
        env:
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        volumeMounts:
          - name: config-volume
            mountPath: /etc/config
        command:
          - /pod_nanny
          - --config-dir=/etc/config
          - --container=kube-state-metrics
          - --cpu=100m
          - --extra-cpu=1m
          - --memory=100Mi
          - --extra-memory=2Mi
          - --threshold=5
          - --deployment=kube-state-metrics
      volumes:
        - name: config-volume
          configMap:
            name: kube-state-metrics-config
---
# Config map for resource configuration.
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-state-metrics-config
  namespace: kube-system
  labels:
    k8s-app: kube-state-metrics
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
data:
  NannyConfiguration: |-
    apiVersion: nannyconfig/v1alpha1
    kind: NannyConfiguration

---

apiVersion: v1
kind: Service
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "kube-state-metrics"
  annotations:
    prometheus.io/scrape: 'true'
spec:
  ports:
  - name: http-metrics
    port: 8080
    targetPort: http-metrics
    protocol: TCP
  - name: telemetry
    port: 8081
    targetPort: telemetry
    protocol: TCP
  selector:
    k8s-app: kube-state-metrics

部署

kubectl apply -f kube-state-metrics.yaml
kubectl get pods -n kube-system

在这里插入图片描述
pod的正常运行

prometheusyml_384">2.2 配置prometheus.yml


  - job_name: kubernetes-service-endpoints
    kubernetes_sd_configs:
    - role: endpoints
      api_server: https://192.168.0.0:6443
      bearer_token_file: /opt/monitor/prometheus/token.k8s
      tls_config:
        insecure_skip_verify: true
    bearer_token_file: /opt/monitor/prometheus/token.k8s
    tls_config:
      insecure_skip_verify: true
     Service没配置注解prometheus.io/scrape的不采集
    relabel_configs:
    - action: keep
      regex: true
      source_labels:
      - __meta_kubernetes_service_annotation_prometheus_io_scrape
  重命名采集目标协议
    - action: replace
      regex: (https?)
      source_labels:
      - __meta_kubernetes_service_annotation_prometheus_io_scheme
      target_label: __scheme__
  重命名采集目标指标URL路径
    - action: replace
      regex: (.+)
      source_labels:
      - __meta_kubernetes_service_annotation_prometheus_io_path
      target_label: __metrics_path__
  重命名采集目标地址
    - action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      source_labels:
      - __address__
      - __meta_kubernetes_service_annotation_prometheus_io_port
      target_label: __address__
  将K8s标签(.*)作为新标签名，原有值不变
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
  生成命名空间标签
    - action: replace
      source_labels:
      - __meta_kubernetes_namespace
      target_label: kubernetes_namespace
  生成Service名称标签
    - action: replace
      source_labels:
      - __meta_kubernetes_service_name
      target_label: kubernetes_service_name


./promtool check config prometheus.yml 
重启prometheus 或 kill -HUP PrometheusPid

在prometheus的target页面查看
http://172.18.0.0:9090
在这里插入图片描述

2.3 Grafana 导入仪表盘

Grafana导入k8s集群资源对象监控仪表盘 6417

在这里插入图片描述
完成k8s集群资源对象监控仪表盘监控

2.4 Grafana没有数据，添加路由转发

ip route
ip route add 172.40.0.0/16 via 172.18.2.30 dev eth0
ip route

#172.40.1.208：kube-state-metrics pod 集群内部ip
#172.18.2.30：k8s master 节点ip

然后在查看Grafana仪表盘。


http://www.niftyadmin.cn/n/5207006.html

相关文章

VM CentOS7安装ffmpeg

VM CentOS7安装ffmpeg

项目中涉及给视频添加水印，使用到了ffmpeg，windows系统可直接使用，Linux需要手动编译完成ffmpeg后才可正常使用。 配置yum源: 备份原repo文件 cd /etc/yum.repos.d/mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.r…
阅读更多...
可以远程控制电脑桌面的软件有哪些？

可以远程控制电脑桌面的软件有哪些？

随着电脑办公的普及，人们对于远程控制电脑的需求也越来越大。远程控制电脑技术能够让用户在不同地点的电脑之间进行操作和访问，能够提高工作效率。可以远程控制电脑桌面的软件有哪些？ 1. 远程监控电脑软件 需要安装在被控制端电脑&#xff…
阅读更多...
机器学习之危险品车辆目标检测

机器学习之危险品车辆目标检测

危险品的运输涉及从离开仓库到由车辆运输到目的地的风险。监控事故、车辆运动动态以及车辆通过特定区域的频率对于监督车辆运输危险品的过程至关重要。 在线工具推荐： 三维数字孪生场景工具 - GLTF/GLB在线编辑器 - Three.js AI自动纹理化开发 - YOLO 虚幻合成数…
阅读更多...
万宾科技智能井盖传感器效果，特点有哪些？

万宾科技智能井盖传感器效果，特点有哪些？

现在城市发展越来越好，对基础设施的改造越来越多，比如修路搭桥、整改生态等都是为民服务的好工程。平时走在路上我们享受着平整的路面，井然有序的交通也为我们带来很大的方便。但是一个又一个的井盖看起来无关紧要，实际上如果路上…
阅读更多...
PC弹框点击确认，数据新增成功，关闭弹框

PC弹框点击确认，数据新增成功，关闭弹框

<el-dialog width"500px" :visible.sync"showDialog" title"新增角色"><el-form-item><el-row type"flex" justify"center"><el-col :span"12"><el-button size"mini" type…
阅读更多...
Raptor安装

Raptor安装

Raptor官网:https://raptor.martincarlisle.com/ 进入官网后&#xff0c;下拉找到 Download RAPTOR&#xff0c;windows系统的选择Windows Users 下载完成后打开&#xff0c;选择“next” 修改一下路径&#xff0c;不要放到C: 继续next 完结撒花
阅读更多...
斯坦福大学引入FlashFFTConv来优化机器学习中长序列的FFT卷积

斯坦福大学引入FlashFFTConv来优化机器学习中长序列的FFT卷积

斯坦福大学的FlashFFTConv优化了扩展序列的快速傅里叶变换(FFT)卷积。该方法引入Monarch分解&#xff0c;在FLOP和I/O成本之间取得平衡&#xff0c;提高模型质量和效率。并且优于PyTorch和FlashAttention-v2。它可以处理更长的序列&#xff0c;并在人工智能应用程序中打开新的可…
阅读更多...
主内存与工作内存数据交换八个步骤：

主内存与工作内存数据交换八个步骤：

一&#xff1a;锁定&#xff1a;把主内存内一个变量标识为一条线程独占的状态&#xff1b; 二&#xff1a;解锁&#xff1a;把一个处于锁定状态的变量释放出来。释放后可被其他线程锁定&#xff1b; 三&#xff1a;读取&#xff1a;把一个变量的值从主内存传输到工作内存中&a…
阅读更多...
最新文章

Copyright @ 2022~2023