kubernetes集群搭建-Prometheus+Grafana监控

mac2024-04-02 34

背景

首先，开始搭建Prometheus+Grafana监控前，默认你已经部署了kubernetes集群，这里的部署方式全部都已经容器化部署

至于kubernetes集群如何搭建部署，本人写了篇文档，详细内容可以参考 (可以无脑按照步骤一步步操作，就能成功搭建出K8S集群)：

地址： https://blog.csdn.net/wt334502157/article/details/83992120

主要该文档中也可以查看一下拉取镜像等问题，已经将所有yaml打成压缩包放在文档中，如果审核通过则可以下载了，审核不通过就只能一个一个yaml复制了。

URL: https://download.csdn.net/download/wt334502157/11945579

---开始部署---

[root@master ~]# docker pull registry.cn-shanghai.aliyuncs.com/wangting/node-exporter:latest [root@master ~]# docker pull registry.cn-shanghai.aliyuncs.com/wangting/prometheus:v2.0.0 [root@master ~]# docker pull registry.cn-shanghai.aliyuncs.com/wangting/grafana:4.2.0

docker tag imageID xxx/xxx:tag

因为是从我私人仓库拉下来的，所以pull下来以后需要做一下更名，换成yaml格式里的名字，对应image名如下，可以先尝试直接下载原镜像，能下载就直接pull，不用这么麻烦。

prom/node-exporter prom/prometheus:v2.0.0 grafana/grafana:4.2.0

部署前先下载镜像，提前做好准备工作

部署node-exporter；先从简单的下手，我们知道以往用zabbix的时候，被监控的各个节点主机都会部署一个agent，同理，我们在K8S集群部署监控系统时，策略是在每个node节点上部署一个node-exporter；并且以守护进程DaemonSet的方式部署，这样加节点删节点，就无需再做相应的监控配置

在所有master和node节点先拉取镜像

node-exporter.yaml

--- apiVersion: apps/v1 kind: DaemonSet metadata: name: node-exporter namespace: kube-system labels: k8s-app: node-exporter spec: selector: matchLabels: k8s-app: node-exporter template: metadata: labels: k8s-app: node-exporter spec: containers: - image: prom/node-exporter name: node-exporter ports: - containerPort: 9100 protocol: TCP name: http --- apiVersion: v1 kind: Service metadata: labels: k8s-app: node-exporter name: node-exporter namespace: kube-system spec: ports: - name: http port: 9100 nodePort: 31672 protocol: TCP type: NodePort selector: k8s-app: node-exporter [root@master k8s-prometheus-grafana]# kubectl apply -f node-exporter.yaml

部署prometheus组件

[root@master prometheus]# ls configmap.yaml prometheus.deploy.yml prometheus.svc.yml rbac-setup.yaml

kubectl apply -f configmap.yaml

apiVersion: v1 kind: ConfigMap metadata: name: prometheus-config namespace: kube-system data: prometheus.yml: | global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'kubernetes-apiservers' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: default;kubernetes;https - job_name: 'kubernetes-nodes' kubernetes_sd_configs: - role: node scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - target_label: __address__ replacement: kubernetes.default.svc:443 - source_labels: [__meta_kubernetes_node_name] regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/${1}/proxy/metrics - job_name: 'kubernetes-cadvisor' kubernetes_sd_configs: - role: node scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - target_label: __address__ replacement: kubernetes.default.svc:443 - source_labels: [__meta_kubernetes_node_name] regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor - job_name: 'kubernetes-service-endpoints' kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] action: replace target_label: __scheme__ regex: (https?) - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] action: replace target_label: __address__ regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 - action: labelmap regex: __meta_kubernetes_service_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_service_name] action: replace target_label: kubernetes_name - job_name: 'kubernetes-services' kubernetes_sd_configs: - role: service metrics_path: /probe params: module: [http_2xx] relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe] action: keep regex: true - source_labels: [__address__] target_label: __param_target - target_label: __address__ replacement: blackbox-exporter.example.com:9115 - source_labels: [__param_target] target_label: instance - action: labelmap regex: __meta_kubernetes_service_label_(.+) - source_labels: [__meta_kubernetes_namespace] target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_service_name] target_label: kubernetes_name - job_name: 'kubernetes-ingresses' kubernetes_sd_configs: - role: ingress relabel_configs: - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe] action: keep regex: true - source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path] regex: (.+);(.+);(.+) replacement: ${1}://${2}${3} target_label: __param_target - target_label: __address__ replacement: blackbox-exporter.example.com:9115 - source_labels: [__param_target] target_label: instance - action: labelmap regex: __meta_kubernetes_ingress_label_(.+) - source_labels: [__meta_kubernetes_namespace] target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_ingress_name] target_label: kubernetes_name - job_name: 'kubernetes-pods' kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label: __address__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name

kubectl apply -f prometheus.deploy.yml

--- apiVersion: apps/v1 kind: Deployment metadata: labels: name: prometheus-deployment name: prometheus namespace: kube-system spec: replicas: 1 selector: matchLabels: app: prometheus template: metadata: labels: app: prometheus spec: containers: - image: prom/prometheus:v2.0.0 name: prometheus command: - "/bin/prometheus" args: - "--config.file=/etc/prometheus/prometheus.yml" - "--storage.tsdb.path=/prometheus" - "--storage.tsdb.retention=24h" ports: - containerPort: 9090 protocol: TCP volumeMounts: - mountPath: "/prometheus" name: data - mountPath: "/etc/prometheus" name: config-volume resources: requests: cpu: 100m memory: 100Mi limits: cpu: 500m memory: 2500Mi serviceAccountName: prometheus volumes: - name: data emptyDir: {} - name: config-volume configMap: name: prometheus-config

kubectl apply -f prometheus.svc.yml

--- kind: Service apiVersion: v1 metadata: labels: app: prometheus name: prometheus namespace: kube-system spec: type: NodePort ports: - port: 9090 targetPort: 9090 nodePort: 30003 selector: app: prometheus

kubectl apply -f rbac-setup.yaml

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: prometheus rules: - apiGroups: [""] resources: - nodes - nodes/proxy - services - endpoints - pods verbs: ["get", "list", "watch"] - apiGroups: - extensions resources: - ingresses verbs: ["get", "list", "watch"] - nonResourceURLs: ["/metrics"] verbs: ["get"] --- apiVersion: v1 kind: ServiceAccount metadata: name: prometheus namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: prometheus roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: prometheus subjects: - kind: ServiceAccount name: prometheus namespace: kube-system

部署grafana组件

kubectl apply -f grafana-deploy.yaml

apiVersion: apps/v1 kind: Deployment metadata: name: grafana-core namespace: kube-system labels: app: grafana component: core spec: selector: matchLabels: app: grafana component: core replicas: 1 template: metadata: labels: app: grafana component: core spec: containers: - image: grafana/grafana:4.2.0 name: grafana-core imagePullPolicy: IfNotPresent # env: resources: # keep request = limit to keep this container in guaranteed class limits: cpu: 100m memory: 100Mi requests: cpu: 100m memory: 100Mi env: # The following env variables set up basic auth twith the default admin user and admin password. - name: GF_AUTH_BASIC_ENABLED value: "true" - name: GF_AUTH_ANONYMOUS_ENABLED value: "false" # - name: GF_AUTH_ANONYMOUS_ORG_ROLE # value: Admin # does not really work, because of template variables in exported dashboards: # - name: GF_DASHBOARDS_JSON_ENABLED # value: "true" readinessProbe: httpGet: path: /login port: 3000 # initialDelaySeconds: 30 # timeoutSeconds: 1 volumeMounts: - name: grafana-persistent-storage mountPath: /var volumes: - name: grafana-persistent-storage emptyDir: {}

kubectl apply -f grafana-ing.yaml

apiVersion: extensions/v1beta1 kind: Ingress metadata: name: grafana namespace: kube-system spec: rules: - host: k8s.grafana http: paths: - path: / backend: serviceName: grafana servicePort: 3000

kubectl apply -f grafana-svc.yaml

apiVersion: v1 kind: Service metadata: name: grafana namespace: kube-system labels: app: grafana component: core spec: type: NodePort ports: - port: 3000 selector: app: grafana component: core

本人至此，部署工作已经完毕了

[root@master k8s-prometheus-grafana]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-5644d7b6d9-xdqzj 1/1 Running 3 2d23h coredns-5644d7b6d9-xkgt8 1/1 Running 4 2d23h etcd-master 1/1 Running 3 2d23h grafana-core-7789756d87-mv6nc 1/1 Running 0 130m kube-apiserver-master 1/1 Running 3 2d23h kube-controller-manager-master 1/1 Running 3 2d23h kube-flannel-ds-amd64-4rl55 1/1 Running 2 2d23h kube-flannel-ds-amd64-9nn2n 1/1 Running 2 2d23h kube-flannel-ds-amd64-gpgdq 1/1 Running 4 2d23h kube-flannel-ds-amd64-s6rwb 1/1 Running 2 2d23h kube-proxy-7cp2p 1/1 Running 2 2d23h kube-proxy-bbc26 1/1 Running 2 2d23h kube-proxy-c5qdp 1/1 Running 3 2d23h kube-proxy-wlrkc 1/1 Running 4 2d23h kube-scheduler-master 1/1 Running 3 2d23h node-exporter-2ngvh 1/1 Running 0 138m node-exporter-7xk4d 1/1 Running 0 138m node-exporter-qcqnw 1/1 Running 0 138m prometheus-759d85775b-cbgvz 1/1 Running 0 135m [root@master k8s-prometheus-grafana]# [root@master k8s-prometheus-grafana]# kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE grafana NodePort 10.96.221.83 <none> 3000:32150/TCP 130m kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 2d23h node-exporter NodePort 10.110.76.58 <none> 9100:31672/TCP 143m prometheus NodePort 10.103.153.35 <none> 9090:30003/TCP 135m

查询一下SVC，metrics--port=31672； prometheus--port=30003； grafana--port=32150

然后你的云主机对应有个公网IP

用公网IP+端口的形式访问

例如：

http://47.100.52.15:31672/metrics # metrics-node-exporter

http://47.100.52.15::30003/graph # prometheus

http://47.100.52.15:32150 # grafana

如果你有域名，做个nginx代理，例如：这样通过

grafana.xxxxx.com 来访问监控，就方便好看了。

root@wangting:/etc/nginx# vim nginx.conf ------- server { listen 80; #端口 server_name grafana.xxxxxx.com; #服务名 charset utf-8; # 避免中文乱码 autoindex on; location / { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_pass http://172.19.xxx.xxx:32150; } } -------

最新回复(0)