K8S?prometheus?operator監(jiān)控工作原理介紹
更新時間：2022年11月16日 14:56:55 作者：ans
這篇文章主要為大家介紹了K8S?prometheus?operator監(jiān)控工作原理介紹，有需要的朋友可以借鑒參考下，希望能夠有所幫助，祝大家多多進步，早日升職加薪
1. Prometheus Operator介紹
介紹文章：http://m.dhdzp.com/article/267693.htm
2016年年末，CoreOs引入了Operator 模式，并發(fā)布了Prometheus Operator 作為Operator模式的工作示例。Prometheus Operator自動創(chuàng)建和管理Prometheus監(jiān)控實例。
Prometheus Operator的任務(wù)是使得在Kubernetes運行Prometheus僅可能容易，同時保留可配置性以及使Kubernetes配置原生。
Prometheus Operator使我們的生活更容易——部署和維護。
2. 它如何工作
為了理解這個問題，我們首先需要了解Prometheus Operator得工作原理。
Prometheus Operator架構(gòu)圖.
我們成功部署 Prometheus Operator后可以看到一個新的CRDs(Custom Resource Defination):
- Prometheus，定義一個期望的
Prometheus deployment。 - ServiceMonitor，聲明式指定應(yīng)該如何監(jiān)控服務(wù)組；
Operator根據(jù)定義自動創(chuàng)建Prometheusscrape配置。 - Alertmanager，定義期望的
Alertmanager deployment。
當服務(wù)新版本更新時，將會常見一個新Pod。Prometheus監(jiān)控k8s API，因此當它檢測到這種變化時，它將為這個新服務(wù)(pod)創(chuàng)建一組新的配置。
3. ServiceMonitor
Prometheus Operator使用一個CRD，叫做 ServiceMonitor 將配置抽象到目標。
下面是個ServiceMonitor的示例：
apiVersion: monitoring.coreos.com/v1alpha1
kind: ServiceMonitor
metadata:
name: frontend
labels:
tier: frontend
spec:
selector:
matchLabels:
tier: frontend
endpoints:
- port: web # 指定exporter端口，這里指定的是endpoint的名稱
interval: 10s # 刷新間隔時間
這僅僅是定義一組服務(wù)應(yīng)該如何被監(jiān)控?，F(xiàn)在我們需要定義一個包含了該ServiceMonitor的Prometheus實例到其配置：
apiVersion: monitoring.coreos.com/v1alpha1
kind: Prometheus
metadata:
name: prometheus-frontend
labels:
prometheus: frontend
spec:
version: v1.3.0
#定義應(yīng)包括標簽為“tier=frontend”的所有ServiceMonitor 到服務(wù)器的配置中
serviceMonitors:
- selector:
matchLabels:
tier: frontend
現(xiàn)在Prometheus將會監(jiān)控每個帶有tier: frontend label的服務(wù)。
4. helm安裝
先決條件：
- 部署了
Helm
準備好動手操作：
helm repo add coreos https://s3-eu-west-1.amazonaws.com/coreos-charts/stable/ helm install coreos/prometheus-operator --name prometheus-operator --namespace monitoring
到目前為止，我們已經(jīng)在我們的集群中安裝了Prometheus Operator的TPR。
現(xiàn)在我們來部署Prometheus，Alertmanager和Grafana。
TIP: 當我使用一個龐大的Helm Charts時，我更傾向于創(chuàng)建一個獨立的value.yaml文件將包含我所有自定義的變更。這么做使我和同事為后期的變化和修改更容易。
helm install coreos/kube-prometheus --name kube-prometheus \
-f my_changes/prometheus.yaml \
-f my_changes/grafana.yaml \
-f my_changes/alertmanager.yaml
檢查一切是否運行正常
kubectl -n monitoring get po NAME READY STATUS RESTARTS AGE alertmanager-kube-prometheus-0 2/2 Running 0 1h kube-prometheus-exporter-kube-state-68dbb4f7c9-tr6rp 2/2 Running 0 1h kube-prometheus-exporter-node-bqcj4 1/1 Running 0 1h kube-prometheus-exporter-node-jmcq2 1/1 Running 0 1h kube-prometheus-exporter-node-qnzsn 1/1 Running 0 1h kube-prometheus-exporter-node-v4wn8 1/1 Running 0 1h kube-prometheus-exporter-node-x5226 1/1 Running 0 1h kube-prometheus-exporter-node-z996c 1/1 Running 0 1h kube-prometheus-grafana-54c96ffc77-tjl6g 2/2 Running 0 1h prometheus-kube-prometheus-0 2/2 Running 0 1h prometheus-operator-1591343780-5vb5q 1/1 Running 0 1h
訪問下Prometheus UI看一下Targets頁面：
kubectl -n monitoring port-forward prometheus-kube-prometheus-0 9090 Forwarding from 127.0.0.1:9090 -> 9090
瀏覽器展示如下：
5. yaml文件安裝
此安裝方法本人親測有效，用到的yaml文件都打包好了。解壓之后直接kubectl apply即可用。會自動監(jiān)控當前集群的所有node節(jié)點和pod。只需更改yaml文件中需要用到的鏡像。我這里都推到了公司公網(wǎng)harbor倉庫。部分鏡像已經(jīng)打成tar包。直接docker load -i即可用。
kube-state.tar.gz
webhook-dingtalk.tar.gz
prometheus-adapter.tar.gz
5.1 安裝
#軟件包集成了node Exporter alertmanager grafana prometheus ingress 所有服務(wù)的配置，只需解壓到K8S master中。 [root@lecode-k8s-master monitor]# ll total 1820 -rw-r--r-- 1 root root 875 Mar 11 2022 alertmanager-alertmanager.yaml -rw-r--r-- 1 root root 515 Mar 11 2022 alertmanager-podDisruptionBudget.yaml -rw-r--r-- 1 root root 4337 Mar 11 2022 alertmanager-prometheusRule.yaml -rw-r--r-- 1 root root 1483 Mar 14 2022 alertmanager-secret.yaml -rw-r--r-- 1 root root 301 Mar 11 2022 alertmanager-serviceAccount.yaml -rw-r--r-- 1 root root 540 Mar 11 2022 alertmanager-serviceMonitor.yaml -rw-r--r-- 1 root root 614 Mar 11 2022 alertmanager-service.yaml drwxr-x--- 2 root root 4096 Oct 25 13:49 backsvc #這里是grafana的service配置。nodeport模式。用于外部訪問。選擇使用 -rw-r--r-- 1 root root 278 Mar 11 2022 blackbox-exporter-clusterRoleBinding.yaml -rw-r--r-- 1 root root 287 Mar 11 2022 blackbox-exporter-clusterRole.yaml -rw-r--r-- 1 root root 1392 Mar 11 2022 blackbox-exporter-configuration.yaml -rw-r--r-- 1 root root 3081 Mar 11 2022 blackbox-exporter-deployment.yaml -rw-r--r-- 1 root root 96 Mar 11 2022 blackbox-exporter-serviceAccount.yaml -rw-r--r-- 1 root root 680 Mar 11 2022 blackbox-exporter-serviceMonitor.yaml -rw-r--r-- 1 root root 540 Mar 11 2022 blackbox-exporter-service.yaml -rw-r--r-- 1 root root 2521 Oct 25 13:36 dingtalk-dep.yaml -rw-r--r-- 1 root root 721 Mar 11 2022 grafana-dashboardDatasources.yaml -rw-r--r-- 1 root root 1448347 Mar 11 2022 grafana-dashboardDefinitions.yaml -rw-r--r-- 1 root root 625 Mar 11 2022 grafana-dashboardSources.yaml -rw-r--r-- 1 root root 8098 Mar 11 2022 grafana-deployment.yaml -rw-r--r-- 1 root root 86 Mar 11 2022 grafana-serviceAccount.yaml -rw-r--r-- 1 root root 398 Mar 11 2022 grafana-serviceMonitor.yaml -rw-r--r-- 1 root root 468 Mar 30 2022 grafana-service.yaml drwxr-xr-x 2 root root 4096 Oct 25 13:32 ingress #這里ingress資源也是可以直接用，可以把Prometheus和grafana服務(wù)暴露在外部。 -rw-r--r-- 1 root root 2639 Mar 14 2022 kube-prometheus-prometheusRule.yaml -rw-r--r-- 1 root root 3380 Mar 14 2022 kube-prometheus-prometheusRule.yamlbak -rw-r--r-- 1 root root 63531 Mar 11 2022 kubernetes-prometheusRule.yaml -rw-r--r-- 1 root root 6912 Mar 11 2022 kubernetes-serviceMonitorApiserver.yaml -rw-r--r-- 1 root root 425 Mar 11 2022 kubernetes-serviceMonitorCoreDNS.yaml -rw-r--r-- 1 root root 6431 Mar 11 2022 kubernetes-serviceMonitorKubeControllerManager.yaml -rw-r--r-- 1 root root 7629 Mar 11 2022 kubernetes-serviceMonitorKubelet.yaml -rw-r--r-- 1 root root 530 Mar 11 2022 kubernetes-serviceMonitorKubeScheduler.yaml -rw-r--r-- 1 root root 464 Mar 11 2022 kube-state-metrics-clusterRoleBinding.yaml -rw-r--r-- 1 root root 1712 Mar 11 2022 kube-state-metrics-clusterRole.yaml -rw-r--r-- 1 root root 2934 Oct 25 13:40 kube-state-metrics-deployment.yaml -rw-r--r-- 1 root root 3082 Mar 11 2022 kube-state-metrics-prometheusRule.yaml -rw-r--r-- 1 root root 280 Mar 11 2022 kube-state-metrics-serviceAccount.yaml -rw-r--r-- 1 root root 1011 Mar 11 2022 kube-state-metrics-serviceMonitor.yaml -rw-r--r-- 1 root root 580 Mar 11 2022 kube-state-metrics-service.yaml -rw-r--r-- 1 root root 444 Mar 11 2022 node-exporter-clusterRoleBinding.yaml -rw-r--r-- 1 root root 461 Mar 11 2022 node-exporter-clusterRole.yaml -rw-r--r-- 1 root root 3047 Mar 11 2022 node-exporter-daemonset.yaml -rw-r--r-- 1 root root 14356 Apr 11 2022 node-exporter-prometheusRule.yaml -rw-r--r-- 1 root root 270 Mar 11 2022 node-exporter-serviceAccount.yaml -rw-r--r-- 1 root root 850 Mar 11 2022 node-exporter-serviceMonitor.yaml -rw-r--r-- 1 root root 492 Mar 11 2022 node-exporter-service.yaml -rw-r--r-- 1 root root 482 Mar 11 2022 prometheus-adapter-apiService.yaml -rw-r--r-- 1 root root 576 Mar 11 2022 prometheus-adapter-clusterRoleAggregatedMetricsReader.yaml -rw-r--r-- 1 root root 494 Mar 11 2022 prometheus-adapter-clusterRoleBindingDelegator.yaml -rw-r--r-- 1 root root 471 Mar 11 2022 prometheus-adapter-clusterRoleBinding.yaml -rw-r--r-- 1 root root 378 Mar 11 2022 prometheus-adapter-clusterRoleServerResources.yaml -rw-r--r-- 1 root root 409 Mar 11 2022 prometheus-adapter-clusterRole.yaml -rw-r--r-- 1 root root 2204 Mar 11 2022 prometheus-adapter-configMap.yaml -rw-r--r-- 1 root root 2530 Oct 25 13:39 prometheus-adapter-deployment.yaml -rw-r--r-- 1 root root 506 Mar 11 2022 prometheus-adapter-podDisruptionBudget.yaml -rw-r--r-- 1 root root 515 Mar 11 2022 prometheus-adapter-roleBindingAuthReader.yaml -rw-r--r-- 1 root root 287 Mar 11 2022 prometheus-adapter-serviceAccount.yaml -rw-r--r-- 1 root root 677 Mar 11 2022 prometheus-adapter-serviceMonitor.yaml -rw-r--r-- 1 root root 501 Mar 11 2022 prometheus-adapter-service.yaml -rw-r--r-- 1 root root 447 Mar 11 2022 prometheus-clusterRoleBinding.yaml -rw-r--r-- 1 root root 394 Mar 11 2022 prometheus-clusterRole.yaml -rw-r--r-- 1 root root 5000 Mar 11 2022 prometheus-operator-prometheusRule.yaml -rw-r--r-- 1 root root 715 Mar 11 2022 prometheus-operator-serviceMonitor.yaml -rw-r--r-- 1 root root 499 Mar 11 2022 prometheus-podDisruptionBudget.yaml -rw-r--r-- 1 root root 14021 Mar 11 2022 prometheus-prometheusRule.yaml -rw-r--r-- 1 root root 1184 Mar 11 2022 prometheus-prometheus.yaml -rw-r--r-- 1 root root 471 Mar 11 2022 prometheus-roleBindingConfig.yaml -rw-r--r-- 1 root root 1547 Mar 11 2022 prometheus-roleBindingSpecificNamespaces.yaml -rw-r--r-- 1 root root 366 Mar 11 2022 prometheus-roleConfig.yaml -rw-r--r-- 1 root root 2047 Mar 11 2022 prometheus-roleSpecificNamespaces.yaml -rw-r--r-- 1 root root 271 Mar 11 2022 prometheus-serviceAccount.yaml -rw-r--r-- 1 root root 531 Mar 11 2022 prometheus-serviceMonitor.yaml -rw-r--r-- 1 root root 558 Mar 11 2022 prometheus-service.yaml drw-r--r-- 2 root root 4096 Oct 24 12:31 setup #先apply setup目錄中的yaml文件。然后apply一級目錄下的yaml文件。backsvc中的grafana的service資源清單。根據(jù)情況調(diào)整為nodeport或ClusterIP。K8S集群會自動在每臺K8S節(jié)點部署node-exporter并收集數(shù)據(jù)。登錄grafana后初始賬號密碼為admin admin。添加dashboard即可監(jiān)控K8S集群 [root@lecode-k8s-master monitor]# cd setup/ [root@lecode-k8s-master setup]# kubectl apply -f . [root@lecode-k8s-master setup]# cd .. [root@lecode-k8s-master monitor]# kubectl apply -f . [root@lecode-k8s-master monitor]# kubectl get po -n monitoring NAME READY STATUS RESTARTS AGE alertmanager-main-0 2/2 Running 0 74m alertmanager-main-1 2/2 Running 0 74m alertmanager-main-2 2/2 Running 0 74m blackbox-exporter-6798fb5bb4-d9m7m 3/3 Running 0 74m grafana-64668d8465-x7x9z 1/1 Running 0 74m kube-state-metrics-569d89897b-hlqxj 3/3 Running 0 57m node-exporter-6vqxg 2/2 Running 0 74m node-exporter-7dxh6 2/2 Running 0 74m node-exporter-9j5xk 2/2 Running 0 74m node-exporter-ftrmn 2/2 Running 0 74m node-exporter-qszkn 2/2 Running 0 74m node-exporter-wjkgj 2/2 Running 0 74m prometheus-adapter-5dd78c75c6-h2jf7 1/1 Running 0 58m prometheus-adapter-5dd78c75c6-qpwzv 1/1 Running 0 58m prometheus-k8s-0 2/2 Running 0 74m prometheus-k8s-1 2/2 Running 0 74m prometheus-operator-75d9b475d9-mmzgs 2/2 Running 0 80m webhook-dingtalk-6ffc94b49-z9z6l 1/1 Running 0 61m [root@lecode-k8s-master backsvc]# kubectl get svc -n monitoring NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-main NodePort 10.98.35.93 <none> 9093:30093/TCP 72m alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 72m blackbox-exporter ClusterIP 10.109.10.110 <none> 9115/TCP,19115/TCP 72m grafana NodePort 10.110.48.214 <none> 3000:30300/TCP 72m kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 72m node-exporter ClusterIP None <none> 9100/TCP 72m prometheus-adapter ClusterIP 10.97.23.176 <none> 443/TCP 72m prometheus-k8s ClusterIP 10.100.92.254 <none> 9090/TCP 72m prometheus-operated ClusterIP None <none> 9090/TCP 72m prometheus-operator ClusterIP None <none> 8443/TCP 78m webhook-dingtalk ClusterIP 10.100.131.63 <none> 80/TCP 72m
5.2 訪問服務(wù)
暴露服務(wù)三種方法：用service資源的nodeport模式，或者用k8s的ingress暴露服務(wù)或者本地nginx代理。本地的nginx代理模式
這里我grafana用的是nodeport模式。Prometheus用的是nginx代理。
附上nginx配置文件
[root@lecode-k8s-master setup]# cat /usr/local/nginx/conf/4-layer-conf.d/lecode-prometheus-operator.conf
#代理prometheus內(nèi)置Dashboard UI
upstream prometheus-dashboard {
server 10.100.92.254:9090; #這里ip為prometheus-k8s svc資源的ip
}
server {
listen 9090;
proxy_pass prometheus-dashboard;
}
#代理grafana
upstream grafana {
server 10.1.82.89:3000; #這里ip為grafana svc資源的ip
}
server {
listen 3000;
proxy_pass grafana;
}
訪問Prometheus targets
5.3 接入grafana
訪問grafana（默認密碼是admin admin）
去grafana官網(wǎng)下載對應(yīng)dashboard 地址：https://grafana.com/grafana/dashboards/
6. 監(jiān)控集群外的服務(wù)
6.1 exporter安裝
在對應(yīng)服務(wù)的本地安裝對應(yīng)的exporter用于收集數(shù)據(jù)（這里以mysql為例）
#下載對應(yīng)服務(wù)的exporter
#插件下載地址：https://www.modb.pro/db/216588
#插件下載地址：https://prometheus.io/download/
#下載完成后解壓mysqld_exporter-0.13.0.linux-amd64.tar.gz
#配置mysql-exporter
在root路徑下，創(chuàng)建.my.cnf文件。內(nèi)容如下:
[root@lecode-test-001 ~]# cat /root/.my.cnf
[client]
user=mysql_monitor
password=Mysql@123
#創(chuàng)建mysql 用戶并授權(quán)
CREATE USER ‘mysql_monitor'@‘localhost' IDENTIFIED BY ‘Mysql@123' WITH MAX_USER_CONNECTIONS 3;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON . TO ‘mysql_monitor'@‘localhost';
FLUSH PRIVILEGES;
EXIT
#啟動mysqld_exporter
[root@lecode-test-001 mysql-exporter]# nohup mysqld_exporter &
#找到對應(yīng)的端口
[root@lecode-test-001 mysql-exporter]# tail -f nohup.out
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:277 msg="Starting msqyld_exporter" version="(version=0.13.0, branch=HEAD, revision=ad2847c7fa67b9debafccd5a08bacb12fc9031f1)"
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:278 msg="Build context" (gogo1.16.4,userroot@e2043849cb1f,date20210531-07:30:16)=(MISSING)
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=global_status
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=global_variables
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=slave_status
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=info_schema.innodb_cmp
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=info_schema.innodb_cmpmem
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=info_schema.query_response_time
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:303 msg="Listening on address" address=:9104 #這是exporter的端口
level=info ts=2022-10-25T09:26:54.464Z caller=tls_config.go:191 msg="TLS is disabled." http2=false
#檢查端口
[root@lecode-test-001 mysql-exporter]# ss -lntup |grep 9104
tcp LISTEN 0 128 :::9104 :::* users:(("mysqld_exporter",pid=26115,fd=3))
6.2 K8S配置
創(chuàng)建endpoint資源關(guān)聯(lián)對應(yīng)服務(wù)主機的exporter端口。綁定service資源，通過ServiceMonitor資源添加Prometheus targets，
1）官方格式
kubectl -n monitoring get prometheus kube-prometheus -o yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
labels:
app: prometheus
chart: prometheus-0.0.14
heritage: Tiller
prometheus: kube-prometheus
release: kube-prometheus
name: kube-prometheus
namespace: monitoring
spec:
...
baseImage: quay.io/prometheus/prometheus
serviceMonitorSelector:
matchLabels:
prometheus: kube-prometheus
#接下來就是按照格式創(chuàng)建對應(yīng)的ServiceMonitor資源
通過ep資源把外部服務(wù)關(guān)聯(lián)到K8S內(nèi)部，綁定對應(yīng)的svc資源。在由serviceMonitor綁定對應(yīng)的svc資源把數(shù)據(jù)提交給Prometheus，serviceMonitor通過標簽選擇器關(guān)聯(lián)service，而service只需要通過端口關(guān)聯(lián)ep，這里的標簽和端口一定要注意一致.
2) 創(chuàng)建資源
[root@lecode-k8s-master monitor]# cat mysql.yaml
apiVersion: v1
kind: Endpoints
metadata:
name: mysql-test
namespace: monitoring
subsets:
- addresses:
- ip: 192.168.1.17 # ip為安裝應(yīng)用服務(wù)器的ip
ports:
- name: mysql
port: 9104 #export的端口
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: mysql
app.kubernetes.io/name: mysql
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: mysql-test
namespace: monitoring
spec:
clusterIP: None
clusterIPs:
- None
ports:
- name: mysql
port: 9104
protocol: TCP
sessionAffinity: None
type: ClusterIP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app.kubernetes.io/component: mysql
app.kubernetes.io/name: mysql
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: mysql-test
namespace: monitoring
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
interval: 30s
port: mysql
tlsConfig:
insecureSkipVerify: true
selector:
matchLabels:
app.kubernetes.io/component: mysql
app.kubernetes.io/name: mysql
app.kubernetes.io/part-of: kube-prometheus
#創(chuàng)建
[root@lecode-k8s-master monitor]# kubectl apply -f mysql.yaml
endpoints/mysql-test created
service/mysql-test created
servicemonitor.monitoring.coreos.com/mysql-test created
#檢查
[root@lecode-k8s-master monitor]# kubectl get -f mysql.yaml
NAME ENDPOINTS AGE
endpoints/mysql-test 192.168.1.17:9104 10m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mysql-test ClusterIP None <none> 9104/TCP 10m
NAME AGE
servicemonitor.monitoring.coreos.com/mysql-test 10m
6.3 接入grafana
7 監(jiān)控集群外的服務(wù)-redis
7.1 exporter安裝
#部署redis-exporter #插件下載地址：https://www.modb.pro/db/216588 [root@lecode-test-001 ~]# tar xf redis_exporter-v1.3.2.linux-amd64.tar.gz [root@lecode-test-001 ~]# ll drwxr-xr-x 2 root root 4096 Nov 6 2019 redis_exporter-v1.3.2.linux-amd64 -rw-r--r-- 1 root root 3376155 Oct 27 10:26 redis_exporter-v1.3.2.linux-amd64.tar.gz [root@lecode-test-001 ~]# mv redis_exporter-v1.3.2.linux-amd64 redis_exporter [root@lecode-test-001 ~]# cd redis_exporter [root@lecode-test-001 redis_exporter]# ll total 8488 -rw-r--r-- 1 root root 1063 Nov 6 2019 LICENSE -rw-r--r-- 1 root root 10284 Nov 6 2019 README.md -rwxr-xr-x 1 root root 8675328 Nov 6 2019 redis_exporter [root@lecode-test-001 redis_exporter]# nohup ./redis_exporter -redis.addr 192.168.1.17:6379 -redis.password 'Redislecodetest@shuli123' & [1] 4564 [root@lecode-test-001 redis_exporter]# nohup: ignoring input and appending output to a€?nohup.outa€? [root@lecode-test-001 redis_exporter]# tail -f nohup.out time="2022-10-27T10:26:48+08:00" level=info msg="Redis Metrics Exporter v1.3.2 build date: 2019-11-06-02:25:20 sha1: 175a69f33e8267e0a0ba47caab488db5e83a592e Go: go1.13.4 GOOS: linux GOARCH: amd64" time="2022-10-27T10:26:48+08:00" level=info msg="Providing metrics at :9121/metrics" #端口為9121
7.2 K8S配置
#創(chuàng)建redis-serviceMonitor資源
[root@lecode-k8s-master monitor]# cat redis.yaml
apiVersion: v1
kind: Endpoints
metadata:
name: redis-test
namespace: monitoring
subsets:
- addresses:
- ip: 192.168.1.17 # ip為安裝應(yīng)用服務(wù)器的ip
ports:
- name: redis
port: 9121 #exporter端口
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: redis
app.kubernetes.io/name: redis
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: redis-test
namespace: monitoring
spec:
clusterIP: None
clusterIPs:
- None
ports:
- name: redis
port: 9121
protocol: TCP
sessionAffinity: None
type: ClusterIP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app.kubernetes.io/component: redis
app.kubernetes.io/name: redis
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: redis-test
namespace: monitoring
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
interval: 30s
port: redis
tlsConfig:
insecureSkipVerify: true
selector:
matchLabels:
app.kubernetes.io/component: redis
app.kubernetes.io/name: redis
app.kubernetes.io/part-of: kube-prometheus
#創(chuàng)建資源
[root@lecode-k8s-master monitor]# kubectl apply -f redis.yaml
endpoints/redis-test created
service/redis-test created
servicemonitor.monitoring.coreos.com/redis-test created
[root@lecode-k8s-master monitor]# kubectl get ep,svc,serviceMonitor -n monitoring |grep redis
endpoints/redis-test 192.168.1.17:9121 6m2s
service/redis-test ClusterIP None <none> 9121/TCP 6m2s
servicemonitor.monitoring.coreos.com/redis-test 6m2s
7.3 Prometheus檢查
7.4 接入grafana
dashboard ID： 11835
8. 監(jiān)控集群外的服務(wù)-kafka
8.1 安裝exporter
#插件下載地址：https://www.modb.pro/db/216588 [root@lecode-test-001 ~]# tar xf kafka_exporter-1.4.2.linux-amd64.tar.gz [root@lecode-test-001 ~]# ll drwxrwxr-x 2 2000 2000 4096 Sep 16 2021 kafka_exporter-1.4.2.linux-amd64 -rw-r--r-- 1 root root 8499720 Oct 27 15:30 kafka_exporter-1.4.2.linux-amd64.tar.gz [root@lecode-test-001 ~]# mv kafka_exporter-1.4.2.linux-amd64 kafka_exporter [root@lecode-test-001 ~]# cd kafka_exporter [root@lecode-test-001 kafka_exporter]# ll total 17676 -rwxr-xr-x 1 2000 2000 18086208 Sep 16 2021 kafka_exporter -rw-rw-r-- 1 2000 2000 11357 Sep 16 2021 LICENSE [root@lecode-test-001 kafka_exporter]# nohup ./kafka_exporter --kafka.server=192.168.1.17:9092 & [1] 20777 [root@lecode-test-001 kafka_exporter]# nohup: ignoring input and appending output to ‘nohup.out' [root@lecode-test-001 kafka_exporter]# tail -f nohup.out I1027 15:32:38.904075 20777 kafka_exporter.go:769] Starting kafka_exporter (version=1.4.2, branch=HEAD, revision=0d5d4ac4ba63948748cc2c53b35ed95c310cd6f2) I1027 15:32:38.905515 20777 kafka_exporter.go:929] Listening on HTTP :9308 #exporter端口為9308
8.2 K8S配置
[root@lecode-k8s-master monitor]# cat kafka.yaml
apiVersion: v1
kind: Endpoints
metadata:
name: kafka-test
namespace: monitoring
subsets:
- addresses:
- ip: 192.168.1.17 # ip為安裝應(yīng)用服務(wù)器的ip
ports:
- name: kafka
port: 9308 #export的端口
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: kafka
app.kubernetes.io/name: kafka
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: kafka-test
namespace: monitoring
spec:
clusterIP: None
clusterIPs:
- None
ports:
- name: kafka
port: 9308
protocol: TCP
sessionAffinity: None
type: ClusterIP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app.kubernetes.io/component: kafka
app.kubernetes.io/name: kafka
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: kafka-test
namespace: monitoring
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
interval: 30s
port: kafka
tlsConfig:
insecureSkipVerify: true
selector:
matchLabels:
app.kubernetes.io/component: kafka
app.kubernetes.io/name: kafka
app.kubernetes.io/part-of: kube-prometheus
#創(chuàng)建
[root@lecode-k8s-master monitor]# kubectl apply -f kafka.yaml
endpoints/kafka-test created
service/kafka-test created
servicemonitor.monitoring.coreos.com/kafka-test created
[root@lecode-k8s-master monitor]# kubectl get -f kafka.yaml
NAME ENDPOINTS AGE
endpoints/kafka-test 192.168.1.17:9308 8m49s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kafka-test ClusterIP None <none> 9308/TCP 8m49s
NAME AGE
servicemonitor.monitoring.coreos.com/kafka-test 8m48s
8.3 Prometheus檢查
8.4 接入grafana
dashboard ID：7589
9. 監(jiān)控集群外的服務(wù)-zookeeper
9.1 安裝exporter
exporter下載地址：https://github.com/carlpett/zookeeper_exporter/releases/download/v1.0.2/zookeeper_exporter [root@lecode-test-001 zookeeper_exporter]# nohup ./zookeeper_exporter -zookeeper 192.168.1.17:2181 -bind-addr :9143 & [2] 8310 [root@lecode-test-001 zookeeper_exporter]# nohup: ignoring input and appending output to ‘nohup.out' [root@lecode-test-001 zookeeper_exporter]# tail -f nohup.out time="2022-10-27T15:58:27+08:00" level=info msg="zookeeper_exporter, version v1.0.2 (branch: HEAD, revision: d6e929223f6b3bf5ff25dd0340e8194cbd4d04fc)\n build user: @bd731f434d23\n build date: 2018-05-01T20:40:14+0000\n go version: go1.10.1" time="2022-10-27T15:58:27+08:00" level=info msg="Starting zookeeper_exporter" time="2022-10-27T15:58:27+08:00" level=info msg="Starting metric http endpoint on :9143" #exporter端口為9143
9.2 K8S配置
[root@lecode-k8s-master monitor]# cat zookeeper.yaml
apiVersion: v1
kind: Endpoints
metadata:
name: zookeeper-test
namespace: monitoring
subsets:
- addresses:
- ip: 192.168.1.17
ports:
- name: zookeeper
port: 9143
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: zookeeper
app.kubernetes.io/name: zookeeper
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: zookeeper-test
namespace: monitoring
spec:
clusterIP: None
clusterIPs:
- None
ports:
- name: zookeeper
port: 9143
protocol: TCP
sessionAffinity: None
type: ClusterIP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app.kubernetes.io/component: zookeeper
app.kubernetes.io/name: zookeeper
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: zookeeper-test
namespace: monitoring
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
interval: 30s
port: zookeeper
tlsConfig:
insecureSkipVerify: true
selector:
matchLabels:
app.kubernetes.io/component: zookeeper
app.kubernetes.io/name: zookeeper
app.kubernetes.io/part-of: kube-prometheus
#創(chuàng)建
[root@lecode-k8s-master monitor]# kubectl apply -f zookeeper.yaml
endpoints/zookeeper-test created
service/zookeeper-test created
servicemonitor.monitoring.coreos.com/zookeeper-test created
[root@lecode-k8s-master monitor]# kubectl get -f zookeeper.yaml
NAME ENDPOINTS AGE
endpoints/zookeeper-test 192.168.1.17:9143 9m55s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/zookeeper-test ClusterIP None <none> 9143/TCP 9m55s
NAME AGE
servicemonitor.monitoring.coreos.com/zookeeper-test 9m55s
9.3 Prometheus檢查
9.4 接入grafana
dashboard ID：15026
以上就是K8S prometheus operator監(jiān)控工作原理介紹的詳細內(nèi)容，更多關(guān)于K8S prometheus operator監(jiān)控的資料請關(guān)注腳本之家其它相關(guān)文章！
相關(guān)文章
這篇文章主要為大家詳細介紹了虛擬化技術(shù)原理，如何安裝與管理虛擬機virt-manger，virsh等,具有一定的參考價值，感興趣的小伙伴們可以參考一下2016-10-10
這篇文章主要介紹了詳解consul的安裝和配置，小編覺得挺不錯的，現(xiàn)在分享給大家，也給大家做個參考。一起跟隨小編過來看看吧2018-06-06
k8s?pod和service網(wǎng)絡(luò)暴露詳解
這篇文章主要介紹了借助iptables的路由轉(zhuǎn)發(fā)功能,打通k8s集群內(nèi)的pod和service網(wǎng)絡(luò),與外部網(wǎng)絡(luò)聯(lián)通,有需要的朋友可以借鑒參考下,希望能夠有所幫助,祝大家多多進步,早日升職加薪2023-11-11
k8s?Service?實現(xiàn)服務(wù)發(fā)現(xiàn)和負載均衡
這篇文章主要為大家介紹了k8s?Service?實現(xiàn)服務(wù)發(fā)現(xiàn)和負載均衡的工作原理及使用方式詳解，有需要的朋友可以借鑒參考下，希望能夠有所幫助，祝大家多多進步，早日升職加薪2023-04-04
這篇文章主要介紹了阿里云ECS排查CPU數(shù)據(jù)分析，有需要的朋友可以借鑒參考下，希望能夠有所幫助，祝大家多多進步早日升職加薪2022-04-04
一文詳解基于Kubescape進行Kubernetes安全加固
這篇文章主要為大家介紹了基于Kubescape進行Kubernetes安全加固詳解，有需要的朋友可以借鑒參考下，希望能夠有所幫助，祝大家多多進步，早日升職加薪2023-02-02
這篇文章主要介紹了k8s使用rbd作為存儲的操作方法,本文通過實例代碼給大家介紹的非常詳細,感興趣的朋友跟隨小編一起看看吧2024-06-06
Kubernetes教程之Windows?HostProcess?運行容器化負載
這篇文章主要介紹了Kubernetes?Windows?HostProcess?運行容器化負載,本篇內(nèi)容還是比較多的，總共包含了?Windows?HostProcess的創(chuàng)建、為?Windows?Pod?和容器配置?GMSA?和?Windows?的?Pod?和容器配置?RunAsUserName三大功能模塊，需要的朋友可以參考下2022-07-07
這篇文章主要給大家介紹了關(guān)于kubernetes(k8s)中Yaml文件詳解的相關(guān)資料,YAML是專門用來寫配置文件的語言,非常簡潔和強大,使用比json更方便,需要的朋友可以參考下2023-12-12
kubernetes部署dashboard及應(yīng)用小結(jié)
Dashboard?是基于網(wǎng)頁的?Kubernetes?用戶界面,可以對?Deployment?實現(xiàn)彈性伸縮、發(fā)起滾動升級、重啟?Pod?或者使用向?qū)?chuàng)建新的應(yīng)用,這篇文章主要介紹了kubernetes部署dashboard,需要的朋友可以參考下2024-06-06