部署条件

• 一套稳定运行的kubernetes集群
• 一个或多个集群外部的exporter
• 一个可用的Prometheus Operator 服务

部署环境

kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.10", GitCommit:"7e54d50d3012cf3389e43b096ba35300f36e0817", GitTreeState:"clean", BuildDate:"2022-08-17T18:32:54Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.10", GitCommit:"7e54d50d3012cf3389e43b096ba35300f36e0817", GitTreeState:"clean", BuildDate:"2022-08-17T18:26:59Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}

------------分割线------------

/ # consul --version
Consul v1.16.0
Revision 192df66a
Build Date 2023-06-26T20:07:11Z
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

架构图

部署consul

apiVersion: v1
kind: ConfigMap
metadata:
  name: consul-config
  namespace: kubesphere-monitoring-system
  labels:
    app: consul
data:
  consul.hcl: |+
    log_level = "INFO"
    data_dir = "/consul/data"
    client_addr = "0.0.0.0"
    ui_config {
      enabled = true
    }
    ports = {
      grpc = -1
      https = -1
      dns = -1
      grpc_tls = -1
      serf_wan = -1
    }
    peering {
      enabled = false
    }
    connect {
      enabled = false
    }
    acl = {
      enabled = true
      default_policy = "deny"
      enable_token_persistence = true
      tokens {
        initial_management = "f2569af0-20ed-4d2a-ae33-a325578db5d6"
        agent = "f2569af0-20ed-4d2a-ae33-a325578db5d6"
      }
    }
    server = true
    bootstrap_expect = 1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: consul
  namespace: kubesphere-monitoring-system
  labels:
    app: consul
spec:
  replicas: 1
  selector:
    matchLabels:
      app: consul
  template:
    metadata:
      labels:
        app: consul
    spec:
      volumes:
        - name: consul-config
          configMap:
            name: consul-config
      containers:
        - name: consul
          image: 'hashicorp/consul:1.16'
          args:
            - agent
          ports:
            - name: http
              containerPort: 8500
              protocol: TCP
          env:
            - name: TZ
              value: Asia/Shanghai
          volumeMounts:
            - name: consul-data
              mountPath: /consul/data
            - name: consul-config
              mountPath: /consul/config/consul.hcl
              subPath: consul.hcl
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: consul-data
        namespace: kubesphere-monitoring-system
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: csi-rbd-sc
        volumeMode: Filesystem
  serviceName: consul

---
apiVersion: v1
kind: Service
metadata:
  name: consul
  namespace: kubesphere-monitoring-system
  labels:
    app: consul
spec:
  ports:
    - name: http
      protocol: TCP
      port: 8500
      targetPort: 8500
      nodePort: 30110
  selector:
    app: consul
  type: NodePort

注册集群外部exporter

1、注册服务

curl --header "X-Consul-Token: f2569af0-20ed-4d2a-ae33-a325578db5d6" \
      --request PUT \
      --data '{
          "ID": "test-kafka-exporter",
          "Name": "kafka-exporter",
          "Address": "10.84.3.45",
          "Port": 9308,
          "Tags": ["linux", "prome"],
          "Checks": [
              {
                  "HTTP": "http://10.84.3.45:9308/metrics",
                  "Interval": "5s"
              }
          ]
      }' \
      http://10.233.35.208:8500/v1/agent/service/register

2、检查是否注册成功

访问 集群任意IP+30110端口

https://IP:30110

输入 consul 配置文件中的token 进行登录

看到consul Dashboard 中 services中 kafka_exporter 为绿色且对勾 是为注册成功

3、删除服务

curl --header "X-Consul-Token: f2569af0-20ed-4d2a-ae33-a325578db5d6" \
     --request PUT \
     http://10.233.35.208:8500/v1/agent/service/deregister/test-kafka-exporter

接入Prometheus-Operator

1、创建additionalScrapeConfigs配置文件

cat > prometheus-additional.yaml << EOF
- job_name: 'consul-services'
  consul_sd_configs:
    - server: 'consul.kubesphere-monitoring-system.svc.cluster.local:8500'
      token: 'f2569af0-20ed-4d2a-ae33-a325578db5d6'
  relabel_configs:
    - source_labels: [__meta_consul_service]
      target_label: job
    - source_labels: [__meta_consul_service]
      action: drop
      regex: consul
    - source_labels: [__meta_consul_service]
      target_label: service
    - source_labels: [__meta_consul_tags]
      target_label: consul_tags
EOF

2、根据additionalScrapeConfigs创建secret

kubectl -n kubesphere-monitoring-system create secret generic additional-config --from-file=prometheus-additional.yaml

3、修改Prometheus-Operator的prometheus配置,添加以下三行配置信息

spec: 
  additionalScrapeConfigs:
    key: prometheus-additional.yaml
    name: additional-config

4、验证是否配置成功