文章目录
- 1.思路
- 2.部署准备
- 3.应用jar包修改配置和导入tw嵌入式的依赖（参考）
- 4.Prometheus部署
- 5.Prometheus配置
- 6.安装和配置Grafana
1.思路
Tongweb嵌入式最终是把依赖打入到java应用（也就是jar包里），然后启动jar包进行测试，所以我们需要考虑的是，启动jar包的时候，怎么获取到监控数据。
根据Prometheus+Grafana 监控Tongweb7来看，监控数据是通过jmx_prometheus_javaagent-0.19.0.jar和config.yaml这两个关键文件来获取的，但是这里用的是企业版的部署方式（类似于部署了tomcat然后在tomcat的webapps里部署应用的方式一样）。
所以我们可以通过启动参数的形式，启动应用jar包的同时，使用jmx_prometheus_javaagent来配置JMX指标的暴露端口为38061，并指定配置文件为config.yaml，也就是这样的指令来启动jar包，从而方便Prometheus拉取监控数据：
java -javaagent:jmx_prometheus_javaagent-0.19.0.jar=39081:config.yaml -jar spring-boot-accesslog-2.x-0.0.1.jar
其中config.yaml是相关配置文件，jmx_prometheus_javaagent-0.19.0.jar是拉取监控数据用的jar包，spring-boot-accesslog-2.x-0.0.1.jar是测试用的jar包（已经导入tongweb嵌入式的依赖并打包了），39081是绑定的监听端口（可以自己设置一个没有占用的端口）。
之后访问服务器ip:39081/metrics,如下图所示就说明获取成功了：
获取成功之后，再部署prometheus和grafana，修改对应配置，配置grafana的数据源和仪表盘即可。
2.部署准备
本次参考：Prometheus监控Tongweb容器
1.使用虚拟机ip：192.168.10.51（tongweb），192.168.10.52（Prometheus），192.168.10.8(Grafana ),其中51服务器确保9060，39081端口没有被占用,52服务器确保9090端口没有被占用，8服务器确保3000端口没有被占用，三台虚拟机均关闭防火墙。
2.使用的jar包：jmx_prometheus_javaagent-0.19.0.jar:jmx_prometheus_javaagent-0.19.0.jar下载链接
3.使用的jdk环境：jdk1.8(需配置jdk环境变量)
4.下载Grafana使用的仪表盘json文件：
TongWeb 面板，使用JMX Exporter采集数据
也可以新建一个json文件，复制粘贴以下内容：
{
"__inputs": [
{
"name": "DS_PROMETHEUS",
"label": "Prometheus",
"description": "",
"type": "datasource",
"pluginId": "prometheus",
"pluginName": "Prometheus"
}
],
"__elements": {},
"__requires": [
{
"type": "panel",
"id": "gauge",
"name": "Gauge",
"version": ""
},
{
"type": "grafana",
"id": "grafana",
"name": "Grafana",
"version": "10.0.1"
},
{
"type": "datasource",
"id": "prometheus",
"name": "Prometheus",
"version": "1.0.0"
},
{
"type": "panel",
"id": "stat",
"name": "Stat",
"version": ""
},
{
"type": "panel",
"id": "table",
"name": "Table",
"version": ""
},
{
"type": "panel",
"id": "timeseries",
"name": "Time series",
"version": ""
}
],
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "datasource",
"uid": "grafana"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"target": {
"limit": 100,
"matchAny": false,
"tags": [],
"type": "dashboard"
},
"type": "dashboard"
}
]
},
"description": "TongWeb 面板，使用JMX Exporter采集数据",
"editable": true,
"fiscalYearStartMonth": 0,
"gnetId": 19156,
"graphTooltip": 0,
"id": null,
"links": [
{
"asDropdown": false,
"icon": "external link",
"includeVars": false,
"keepTime": false,
"tags": [],
"targetBlank": true,
"title": "更新面板",
"tooltip": "",
"type": "link",
"url": "https://grafana.com/grafana/dashboards/19156"
}
],
"liveNow": false,
"panels": [
{
"collapsed": false,
"datasource": {
"type": "prometheus",
"uid": "fb65406e-39df-40f2-966c-9987082bdcfe"
},
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"id": 31,
"panels": [],
"repeat": "protocol",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "fb65406e-39df-40f2-966c-9987082bdcfe"
},
"refId": "A"
}
],
"title": "监控信息",
"type": "row"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "被监控服务是否处于运行状态，即该服务是否可达。当UP指标的值为1时，表示该服务正在运行，并且可以被访问，当UP指标的值为0时，则表示该服务不可达或处于停止状态。",
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [
{
"options": {
"0": {
"color": "red",
"index": 1,
"text": "未连通"
},
"1": {
"color": "green",
"index": 0,
"text": "连通"
}
},
"type": "value"
},
{
"options": {
"match": "null",
"result": {
"color": "red",
"index": 2,
"text": "未连通"
}
},
"type": "special"
}
],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 4,
"x": 0,
"y": 1
},
"id": 33,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"text": {},
"textMode": "auto"
},
"pluginVersion": "10.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": false,
"expr": "up{instance=~\"$instance\"}",
"instant": true,
"interval": "",
"legendFormat": "状态",
"refId": "A"
}
],
"title": "运行状态",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"description": "",
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"decimals": 2,
"mappings": [
{
"options": {
"match": "null",
"result": {
"text": "N/A"
}
},
"type": "special"
}
],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
}
]
},
"unit": "percent"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 5,
"x": 4,
"y": 1
},
"id": 17,
"links": [],
"maxDataPoints": 100,
"options": {
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"mean"
],
"fields": "",
"values": false
},
"showThresholdLabels": false,
"showThresholdMarkers": true
},
"pluginVersion": "10.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": false,
"expr": "jvm_memory_bytes_used{area='heap', instance=\"$instance\"}/jvm_memory_bytes_max{area='heap', instance=\"$instance\"}*100",
"format": "time_series",
"instant": true,
"interval": "",
"intervalFactor": 1,
"legendFormat": "",
"refId": "A"
}
],
"title": "JVM内存使用率",
"type": "gauge"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [
{
"options": {
"match": "null",
"result": {
"text": "N/A"
}
},
"type": "special"
}
],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "none"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 5,
"x": 9,
"y": 1
},
"id": 20,
"links": [],
"maxDataPoints": 100,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"mean"
],
"fields": "",
"values": false
},
"textMode": "auto"
},
"pluginVersion": "10.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": false,
"expr": "sum(tongweb_manager_sessioncounter{instance=\"$instance\"})",
"format": "time_series",
"instant": true,
"interval": "",
"intervalFactor": 1,
"legendFormat": "",
"refId": "A"
}
],
"title": "会话总个数",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [
{
"options": {
"match": "null",
"result": {
"text": "N/A"
}
},
"type": "special"
}
],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "none"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 5,
"x": 14,
"y": 1
},
"id": 21,
"links": [],
"maxDataPoints": 100,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"mean"
],
"fields": "",
"values": false
},
"textMode": "auto"
},
"pluginVersion": "10.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": false,
"expr": "sum(tongweb_manager_expiredsessions{instance=\"$instance\"}) ",
"format": "time_series",
"instant": true,
"interval": "",
"intervalFactor": 1,
"legendFormat": "",
"refId": "A"
}
],
"title": "会话过期数",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [
{
"options": {
"match": "null",
"result": {
"text": "N/A"
}
},
"type": "special"
}
],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "none"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 5,
"x": 19,
"y": 1
},
"id": 19,
"links": [],
"maxDataPoints": 100,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"mean"
],
"fields": "",
"values": false
},
"textMode": "auto"
},
"pluginVersion": "10.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": false,
"expr": "sum(tongweb_manager_sessioncounter{ instance=\"$instance\"} - tongweb_manager_expiredsessions{instance=\"$instance\"})",
"format": "time_series",
"instant": true,
"interval": "",
"intervalFactor": 1,
"legendFormat": "",
"refId": "A"
}
],
"title": "当前会话计数",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"description": "",
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": false
},
"mappings": [
{
"options": {
"match": "null",
"result": {
"text": "N/A"
}
},
"type": "special"
}
],
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "none"
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "线程池使用率*"
},
"properties": [
{
"id": "color",
"value": {
"mode": "continuous-GrYlRd"
}
},
{
"id": "custom.cellOptions",
"value": {
"mode": "gradient",
"type": "gauge"
}
},
{
"id": "unit",
"value": "percent"
},
{
"id": "decimals",
"value": 1
},
{
"id": "min",
"value": -1
},
{
"id": "thresholds",
"value": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
{
"id": "max",
"value": 100
}
]
},
{
"matcher": {
"id": "byName",
"options": "http通道"
},
"properties": [
{
"id": "custom.width",
"value": 200
}
]
},
{
"matcher": {
"id": "byName",
"options": "最大线程数"
},
"properties": [
{
"id": "custom.width",
"value": 90
}
]
},
{
"matcher": {
"id": "byName",
"options": "初始线程数"
},
"properties": [
{
"id": "custom.width",
"value": 90
}
]
},
{
"matcher": {
"id": "byName",
"options": "线程优先级"
},
"properties": [
{
"id": "custom.width",
"value": 90
}
]
},
{
"matcher": {
"id": "byName",
"options": "等待队列"
},
"properties": [
{
"id": "custom.width",
"value": 90
}
]
}
]
},
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 5
},
"id": 14,
"links": [],
"maxDataPoints": 100,
"options": {
"cellHeight": "sm",
"footer": {
"countRows": false,
"fields": "",
"reducer": [
"sum"
],
"show": false
},
"showHeader": true,
"sortBy": [
{
"desc": true,
"displayName": "线程池使用率"
}
]
},
"pluginVersion": "10.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": false,
"expr": "tongweb_threadpool_maxthreads{ instance=\"$instance\"}",
"format": "table",
"hide": false,
"instant": true,
"interval": "",
"intervalFactor": 1,
"legendFormat": "{{name}}-最大线程数",
"range": false,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"editorMode": "code",
"exemplar": false,
"expr": "tongweb_threadpool_minsparethreads{instance=\"$instance\"}",
"format": "table",
"hide": false,
"instant": true,
"legendFormat": "{{name}}-使用线程数",
"range": false,
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"editorMode": "code",
"exemplar": false,
"expr": "tongweb_threadpool_connectioncount{ instance=\"$instance\"}",
"format": "table",
"hide": false,
"instant": true,
"legendFormat": "{{name}}",
"range": false,
"refId": "C"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"editorMode": "code",
"exemplar": false,
"expr": "tongweb_threadpool_connectioncount{ instance=\"$instance\"}/tongweb_threadpool_maxthreads{ instance=\"$instance\"}*100",
"format": "table",
"hide": false,
"instant": true,
"range": false,
"refId": "D"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"editorMode": "code",
"exemplar": false,
"expr": "tongweb_threadpool_threadpriority{instance=\"$instance\"}",
"format": "table",
"hide": false,
"instant": true,
"range": false,
"refId": "E"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"editorMode": "code",
"exemplar": false,
"expr": "tongweb_threadpool_backlog{instance=\"$instance\"}",
"format": "table",
"hide": false,
"instant": true,
"range": false,
"refId": "F"
}
],
"title": "线程池信息",
"transformations": [
{
"id": "filterFieldsByName",
"options": {
"include": {
"pattern": "/^Value #[A-Z]|^name$/"
}
}
},
{
"id": "merge",
"options": {}
},
{
"id": "organize",
"options": {
"excludeByName": {},
"indexByName": {},
"renameByName": {
"Value #A": "最大线程数",
"Value #B": "初始线程数",
"Value #C": "正在执行任务的线程数*",
"Value #D": "线程池使用率*",
"Value #E": "线程优先级",
"Value #F": "等待队列",
"name": "http通道"
}
}
},
{
"id": "groupBy",
"options": {
"fields": {
"Value #E": {
"aggregations": [],
"operation": "groupby"
},
"http通道": {
"aggregations": [],
"operation": "groupby"
},
"初始线程数": {
"aggregations": [
"lastNotNull"
],
"operation": "groupby"
},
"最大线程数": {
"aggregations": [],
"operation": "groupby"
},
"正在执行任务的线程数": {
"aggregations": [],
"operation": "groupby"
},
"正在执行任务的线程数*": {
"aggregations": [],
"operation": "groupby"
},
"等待队列": {
"aggregations": [],
"operation": "groupby"
},
"线程优先级": {
"aggregations": [],
"operation": "groupby"
},
"线程池使用率": {
"aggregations": [],
"operation": "groupby"
},
"线程池使用率*": {
"aggregations": [],
"operation": "groupby"
}
}
}
}
],
"type": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 6,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 1,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "percent"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 5
},
"id": 40,
"options": {
"legend": {
"calcs": [],
"displayMode": "table",
"placement": "right",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "tongweb_threadpool_connectioncount{ instance=\"$instance\"}/tongweb_threadpool_maxthreads{ instance=\"$instance\"}*100",
"instant": false,
"legendFormat": "{{name}}",
"range": true,
"refId": "A"
}
],
"title": "线程池使用率",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"description": "",
"fieldConfig": {
"defaults": {
"color": {
"mode": "continuous-BlPu"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 14,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineStyle": {
"fill": "solid"
},
"lineWidth": 1,
"pointSize": 1,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 14
},
"id": 39,
"links": [],
"maxDataPoints": 100,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "10.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": false,
"expr": "jvm_memory_bytes_used{area='heap', instance=\"$instance\"}/jvm_memory_bytes_max{area='heap', instance=\"$instance\"}*100",
"format": "time_series",
"instant": false,
"interval": "",
"intervalFactor": 1,
"legendFormat": "内存使用率",
"range": true,
"refId": "A"
}
],
"title": "JVM内存使用率",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 6,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 1,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 14
},
"id": 41,
"options": {
"legend": {
"calcs": [],
"displayMode": "table",
"placement": "right",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "increase(tongweb_requestprocessor_errorcount{instance=\"$instance\"}[5m])",
"instant": false,
"legendFormat": "{{worker}}-{{name}}",
"range": true,
"refId": "A"
}
],
"title": "近5分钟错误增长数",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"description": "",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 8,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 1,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [
{
"options": {
"match": "null",
"result": {
"text": "N/A"
}
},
"type": "special"
}
],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "ms"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 23
},
"id": 35,
"links": [],
"maxDataPoints": 100,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "right",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "10.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": false,
"expr": "sum(increase(tongweb_requestprocessor_processingtime{instance=\"$instance\"}[5m])) by (worker) / (count(tongweb_requestprocessor_processingtime{instance=\"$instance\"}) by (worker)+0.01) ",
"format": "time_series",
"instant": false,
"interval": "",
"intervalFactor": 1,
"legendFormat": "{{worker}}",
"range": true,
"refId": "A"
}
],
"title": "平均响应时长",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"description": "",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 11,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 1,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"decimals": 2,
"mappings": [
{
"options": {
"match": "null",
"result": {
"text": "N/A"
}
},
"type": "special"
}
],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
}
]
},
"unit": "percent"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 23
},
"id": 37,
"links": [],
"maxDataPoints": 100,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "10.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": false,
"expr": "jvm_memory_pool_bytes_used{ instance=\"$instance\", pool!=\"Metaspace\"}/jvm_memory_pool_bytes_max{ instance=\"$instance\", pool!=\"Metaspace\"}*100",
"format": "time_series",
"instant": false,
"interval": "",
"intervalFactor": 1,
"legendFormat": "{{pool}}",
"range": true,
"refId": "A"
}
],
"title": "JVM内存池使用率",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"description": "",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"decimals": 0,
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "none"
},
"overrides": [
{
"matcher": {
"id": "byValue",
"options": {
"op": "gte",
"reducer": "allIsZero",
"value": 0
}
},
"properties": [
{
"id": "custom.hideFrom",
"value": {
"legend": true,
"tooltip": true,
"viz": false
}
}
]
}
]
},
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 32
},
"id": 23,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "table",
"placement": "right",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "10.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": false,
"expr": "topk(5, tongweb_globalrequestprocessor_requestcount{ instance=\"$instance\"})",
"format": "time_series",
"instant": false,
"interval": "",
"intervalFactor": 1,
"legendFormat": "{{name}} ",
"range": true,
"refId": "A"
}
],
"title": "请求计数TOP5",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"description": "",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"decimals": 0,
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "short"
},
"overrides": [
{
"matcher": {
"id": "byValue",
"options": {
"op": "gte",
"reducer": "allIsZero",
"value": 0
}
},
"properties": [
{
"id": "custom.hideFrom",
"value": {
"legend": true,
"tooltip": true,
"viz": false
}
}
]
}
]
},
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 32
},
"id": 24,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "table",
"placement": "right",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "10.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": true,
"expr": "topk(5,tongweb_requestprocessor_errorcount{ instance=\"$instance\"})",
"format": "time_series",
"interval": "",
"intervalFactor": 1,
"legendFormat": "{{worker}} ",
"range": true,
"refId": "A"
}
],
"title": "错误计数TOP5",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"description": "",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "decbytes"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 41
},
"id": 2,
"links": [],
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "right",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "10.0.1",
"repeatDirection": "h",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": true,
"expr": "increase(tongweb_globalrequestprocessor_bytesreceived{ instance=\"$instance\"}[$__interval])",
"format": "time_series",
"interval": "60s",
"intervalFactor": 1,
"legendFormat": "Received {{name}}",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": true,
"expr": "increase(tongweb_globalrequestprocessor_bytessent{ instance=\"$instance\"}[$__interval])",
"format": "time_series",
"interval": "60s",
"intervalFactor": 1,
"legendFormat": "Sent {{name}}",
"range": true,
"refId": "B"
}
],
"title": "发送/接收流量",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"description": "tomcat_requestcount_total是Tomcat服务器的一个请求计数器，它记录了所有已处理的请求总数。 在Tomcat服务器中，每个客户端请求都会被处理为一个请求，在处理请求期间，服务器会从连接中读取数据、解析数据、执行应用逻辑，并返回响应。每当Tomcat服务器成功处理一个请求时，其tomcat_requestcount_total指标就会相应地增加。监控和分析该指标可以帮助管理员和开发人员更好地了解Tomcat服务器的整体负载情况和活动水平，对于性能优化和故障排除都具有一定的价值。此外，该指标也可以通过与其他指标如tomcat_bytesreceived_total和tomcat_bytessent_total等结合使用，来监测网络流量和请求响应时间等性能指标，以进一步优化Tomcat服务器的配置和性能。",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"decimals": 0,
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 41
},
"id": 4,
"links": [],
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max",
"min"
],
"displayMode": "table",
"placement": "right",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "10.0.1",
"repeatDirection": "h",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": true,
"expr": "increase(tongweb_globalrequestprocessor_requestcount{ instance=\"$instance\"}[5m])",
"format": "time_series",
"interval": "60s",
"intervalFactor": 1,
"legendFormat": "Request {{name}}",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": true,
"expr": "increase(tongweb_globalrequestprocessor_errorcount{ instance=\"$instance\"}[5m])",
"format": "time_series",
"interval": "60s",
"intervalFactor": 1,
"legendFormat": "Error {{name}}",
"range": true,
"refId": "B"
}
],
"title": "请求计数/错误计数",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "$datasource"
},
"description": "",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"decimals": 0,
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "none"
},
"overrides": [
{
"__systemRef": "hideSeriesFrom",
"matcher": {
"id": "byNames",
"options": {
"mode": "exclude",
"names": [
"New"
],
"prefix": "All except:",
"readOnly": true
}
},
"properties": [
{
"id": "custom.hideFrom",
"value": {
"legend": false,
"tooltip": false,
"viz": true
}
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 50
},
"id": 5,
"links": [],
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max",
"min"
],
"displayMode": "table",
"placement": "right",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "10.0.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": true,
"expr": "sum(increase(tongweb_manager_sessioncounter{ instance=\"$instance\"}[5m])) by (instance)",
"format": "time_series",
"hide": false,
"interval": "60s",
"intervalFactor": 1,
"legendFormat": "New",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "sum(increase(tongweb_manager_expiredsessions{ instance=\"$instance\"}[5m])) by (instance)",
"format": "time_series",
"hide": false,
"interval": "60s",
"intervalFactor": 1,
"legendFormat": "Expired",
"range": true,
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "sum(increase(tongweb_manager_rejectedsessions{ instance=\"$instance\"}[5m])) by (instance)",
"format": "time_series",
"hide": false,
"interval": "60s",
"intervalFactor": 1,
"legendFormat": "Rejected",
"range": true,
"refId": "C"
}
],
"title": "会话计数统计",
"type": "timeseries"
}
],
"refresh": "5s",
"schemaVersion": 38,
"style": "dark",
"tags": [
"Tomcat8"
],
"templating": {
"list": [
{
"current": {
"selected": false,
"text": "Prometheus",
"value": "Prometheus"
},
"hide": 0,
"includeAll": false,
"multi": false,
"name": "datasource",
"options": [],
"query": "prometheus",
"queryValue": "",
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"type": "datasource"
},
{
"allValue": ".*",
"current": {},
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"definition": "label_values(jvm_info{},instance)",
"hide": 0,
"includeAll": false,
"label": "Instance",
"multi": false,
"name": "instance",
"options": [],
"query": {
"query": "label_values(jvm_info{},instance)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 1,
"tagValuesQuery": "",
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},
"time": {
"from": "now-15m",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
"timezone": "",
"title": "TongWeb_dashboard",
"uid": "tongweb",
"version": 71,
"weekStart": ""
}
也可以自行下载跟tomcat8.5相关的json文件仪表盘，这个自行百度。
5.tw7的安装包和授权，其中tw7使用的安装包为TongWeb7.0.4.9_M3_Enterprise_Linux.tar.gz。
3.应用jar包修改配置和导入tw嵌入式的依赖（参考）
本次示范用的是TongWeb7.0.E.6_P5下的spring-boot-accesslog-2.x：
请在本地开发环境配置好mavernhome，配置教程如下：
maven环境变量配置（超详细！）
配置后，window下，使用 installAll-{n} 脚本，解压 TongWeb 嵌入式版资源包，将 lib 目录中的 jar 包安装到本地 maven 仓库。
配置以下依赖到pom文件里（这一步可以参考手册：TongWeb V7.0 嵌入式版_JavaEE标准容器用户指南，记得去掉tomcat的依赖）：
<!--TongWeb 嵌入式版提供的与springboot集成的starter。 -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-websocket</artifactId>
</dependency>
<dependency>
<groupId>com.tongweb.springboot</groupId>
<artifactId>tongweb-spring-boot-websocket-2.x</artifactId>
<version>7.0.E.6_P5</version>
</dependency>
<dependency>
<groupId>com.tongweb.springboot</groupId>
<artifactId>tongweb-spring-boot-starter-2.x</artifactId>
<version>7.0.E.6_P5</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- jsp支持-->
<dependency>
<groupId>com.tongweb</groupId>
<artifactId>tongweb-jsp</artifactId>
<version>7.0.E.6_P5</version>
</dependency>
刷新maven，然后看看tongweb的依赖是否已经导入到项目里
授权文件放入到resource目录下，propeities配置文件配置：
server.tongweb.license.path=classpath:license.dat
如果用的不是propeities文件，配置参考如下：
server:
tongweb:
license:
path: classpath:license.dat
之后进行打包，打包看看有没有报错，有的话看看报错信息，如果是依赖冲突，可以使用mavenhelp，参考：
maven解决包冲突简单方式(插件maven helper | maven指令)
打包后最好在本地跑一次看看有没有问题，没有的话和已经下载好的jmx.xxxjar放入到服务器，下面是我放入服务器的目录截图：
自己新建一个config.yaml,以下是配置内容：
wercaseOutputLabelNames: true
lowercaseOutputName: true
whitelistObjectNames: ["java.lang:type=OperatingSystem", "TONGWEB:*","config:*"]
blacklistObjectNames: []
rules:
- pattern: ".*"
执行指令启动应用
java -javaagent:jmx_prometheus_javaagent-0.19.0.jar=39081:config.yaml -jar spring-boot-accesslog-2.x-0.0.1.jar
访问服务器ip:39081/metrics
这一部分在第一节已经有详细说明。
4.Prometheus部署
参考：一篇文章搞定Prometheus普罗米修斯监控系统的部署
5.Prometheus配置
配置prometheus.yml：
配置参考（注意缩进，192.168.10.51和39081分别对应的是tongweb这边的端口和ip）：
static_configs:
- targets: ["localhost:9090"]
- job_name: tongweb
static_configs:
- targets: ['192.168.10.51:39081']
labels:
instance: 192.168.10.51
下面是我本地的配置文件，自行删除不需要的部分：
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
scrape_configs:
- job_name: "prometheus"
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ["localhost:9090"]
- job_name: 'Prometheus19'
metrics_path: /metrics
static_configs:
- targets:
- rds
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 192.168.10.19:9098
- job_name: 'Prometheus20'
metrics_path: /metrics
static_configs:
- targets:
- rds
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 192.168.10.20:9098
- job_name: "tongweb"
static_configs:
- targets: ['192.168.10.51:39081']
labels:
instance: 192.168.10.51
修改完启动Prometheus
cd /usr/local/promethues/
./prometheus --config.file="/usr/local/promethues/prometheus.yml" &
ps ef |grep promethues
访问服务器ip:9090/targets?search=,点击status，target，就可以看到我们配置的信息：
6.安装和配置Grafana
Grafana下载地址
上图的话，安装后用 systemctl start grafana-server 启动，会方便一些。
或者用这个也行，这个的话后面启动要用脚本启动（本文使用这种方式）。
解压一下安装包
tar -vxf grafana-enterprise-10.3.1.linux-amd64.tar.gz
启动：
cd 到安装目录bin目录，执行./grafana-server,然后访问服务器ip:3000
注意：默认账号密码为：
账号：admin
密码：admin
如果你想卸载，可以关闭服务（直接pkill grafana-server也行），直接删除安装目录，然后删除对应配置
```xml
sudo rm -rf /var/lib/grafana
sudo rm -rf /var/log/grafana
sudo rm -rf /etc/grafana
sudo rm -rf /etc/grafana/grafana.ini
sudo rm -rf /var/lib/grafana/grafana.db
通常情况下，Grafana 服务的配置文件位于 /etc/grafana/grafana.ini，日志通常存储在 /var/log/grafana，而数据库默认位于 /var/lib/grafana/grafana.db。您可能需要查看这些文件以进行故障排查或者进行配置更改。记得在修改配置文件之后重启 Grafana 服务使更改生效。
Grafana只用于展示数据，需要配置Prometheus数据源。
写你的prometheus的url，带上端口号9090：
点击下方的sava&test，有下图的提示说明成功：
然后回到home，找到Create your first dashboard：
点击import a dashboard：
上传之前下载的json文件
上传后选择一下prometheus：
然后点击import，仪表盘就出现了：
