[kubernetes]-filebeat以sidecar模式收集pod日志

本是古典 何须时尚 2022-12-01 11:58 286阅读 0赞

目的: 收集测试环境日志到kibana

定义日志收集相关配置的一个通配configmap

test-filebeat-config.yaml

  1. #定义日志收集相关配置的一个通配configmap
  2. ---
  3. apiVersion: v1
  4. kind: ConfigMap
  5. metadata:
  6.   name: test-filebeat-config
  7.   namespace: test
  8.   labels:
  9.     config: test-filebeat-config
  10. data:
  11.   filebeat.yml: |-
  12.     filebeat.prospectors:
  13.     - type: log
  14.       paths:
  15.         - /data/tomcat-haozhuo-ec/json.info.*.log
  16.       json.keys_under_root: true
  17.       json.overwrite_keys: true
  18.       tags: ["info","ec"]
  20.     - type: log
  21.       enabled: true
  22.       paths:
  23.        - /data/tomcat-haozhuo-ec/json.error.*.log
  24.       json.keys_under_root: true
  25.       json.overwrite_keys: true
  26.       tags: ["error","ec"]
  27.     - type: log
  28.       paths:
  29.         - /data/tomcat-haozhuo-basic/json.info.*.log
  30.       json.keys_under_root: true
  31.       json.overwrite_keys: true
  32.       tags: ["info","basic"]
  34.     - type: log
  35.       enabled: true
  36.       paths:
  37.        - /data/tomcat-haozhuo-basic/json.error.*.log
  38.       json.keys_under_root: true
  39.       json.overwrite_keys: true
  40.       tags: ["error","basic"]
  41.     - type: log
  42.       paths:
  43.         - /data/tomcat-haozhuo-deep/json.info.*.log
  44.       json.keys_under_root: true
  45.       json.overwrite_keys: true
  46.       tags: ["info","deep"]
  48.     - type: log
  49.       enabled: true
  50.       paths:
  51.        - /data/tomcat-haozhuo-deep/json.error.*.log
  52.       json.keys_under_root: true
  53.       json.overwrite_keys: true
  54.       tags: ["error","deep"]
  55.     - type: log
  56.       paths:
  57.         - /data/tomcat-haozhuo-video/json.info.*.log
  58.       json.keys_under_root: true
  59.       json.overwrite_keys: true
  60.       tags: ["info","video"]
  62.     - type: log
  63.       enabled: true
  64.       paths:
  65.        - /data/tomcat-haozhuo-video/json.error.*.log
  66.       json.keys_under_root: true
  67.       json.overwrite_keys: true
  68.       tags: ["error","video"]
  69.     - type: log
  70.       paths:
  71.         - /data/tomcat-haozhuo-wx/json.info.*.log
  72.       json.keys_under_root: true
  73.       json.overwrite_keys: true
  74.       tags: ["info","wx"]
  76.     - type: log
  77.       enabled: true
  78.       paths:
  79.        - /data/tomcat-haozhuo-wx/json.error.*.log
  80.       json.keys_under_root: true
  81.       json.overwrite_keys: true
  82.       tags: ["error","wx"]
  83.     - type: log
  84.       paths:
  85.         - /data/tomcat-haozhuo-system/json.info.*.log
  86.       json.keys_under_root: true
  87.       json.overwrite_keys: true
  88.       tags: ["info","system"]
  90.     - type: log
  91.       enabled: true
  92.       paths:
  93.        - /data/tomcat-haozhuo-system/json.error.*.log
  94.       json.keys_under_root: true
  95.       json.overwrite_keys: true
  96.       tags: ["error","system"]
  97.     - type: log
  98.       paths:
  99.         - /data/tomcat-haozhuo-shoppingmall/json.info.*.log
  100.       json.keys_under_root: true
  101.       json.overwrite_keys: true
  102.       tags: ["info","shoppingmall"]
  104.     - type: log
  105.       enabled: true
  106.       paths:
  107.        - /data/tomcat-haozhuo-shoppingmall/json.error.*.log
  108.       json.keys_under_root: true
  109.       json.overwrite_keys: true
  110.       tags: ["error","shoppingmall"]
  111.     - type: log
  112.       paths:
  113.         - /data/tomcat-haozhuo-reportContent/json.info.*.log
  114.       json.keys_under_root: true
  115.       json.overwrite_keys: true
  116.       tags: ["info","reportContent"]
  118.     - type: log
  119.       enabled: true
  120.       paths:
  121.        - /data/tomcat-haozhuo-reportContent/json.error.*.log
  122.       json.keys_under_root: true
  123.       json.overwrite_keys: true
  124.       tags: ["error","reportContent"]
  125.     - type: log
  126.       paths:
  127.         - /data/tomcat-haozhuo-point/json.info.*.log
  128.       json.keys_under_root: true
  129.       json.overwrite_keys: true
  130.       tags: ["info","point"]
  132.     - type: log
  133.       enabled: true
  134.       paths:
  135.        - /data/tomcat-haozhuo-point/json.error.*.log
  136.       json.keys_under_root: true
  137.       json.overwrite_keys: true
  138.       tags: ["error","point"]
  139.     - type: log
  140.       paths:
  141.         - /data/tomcat-haozhuo-openapi/json.info.*.log
  142.       json.keys_under_root: true
  143.       json.overwrite_keys: true
  144.       tags: ["info","openapi"]
  146.     - type: log
  147.       enabled: true
  148.       paths:
  149.        - /data/tomcat-haozhuo-openapi/json.error.*.log
  150.       json.keys_under_root: true
  151.       json.overwrite_keys: true
  152.       tags: ["error","openapi"]
  153.     - type: log
  154.       paths:
  155.         - /data/tomcat-haozhuo-job/json.info.*.log
  156.       json.keys_under_root: true
  157.       json.overwrite_keys: true
  158.       tags: ["info","job"]
  160.     - type: log
  161.       enabled: true
  162.       paths:
  163.        - /data/tomcat-haozhuo-job/json.error.*.log
  164.       json.keys_under_root: true
  165.       json.overwrite_keys: true
  166.       tags: ["error","job"]
  167.     - type: log
  168.       paths:
  169.         - /data/tomcat-haozhuo-info/json.info.*.log
  170.       json.keys_under_root: true
  171.       json.overwrite_keys: true
  172.       tags: ["info","info"]
  174.     - type: log
  175.       enabled: true
  176.       paths:
  177.        - /data/tomcat-haozhuo-info/json.error.*.log
  178.       json.keys_under_root: true
  179.       json.overwrite_keys: true
  180.       tags: ["error","info"]
  181.     - type: log
  182.       paths:
  183.         - /data/tomcat-haozhuo-hm/json.info.*.log
  184.       json.keys_under_root: true
  185.       json.overwrite_keys: true
  186.       tags: ["info","hm"]
  187.     - type: log
  188.       enabled: true
  189.       paths:
  190.        - /data/tomcat-haozhuo-hm/json.error.*.log
  191.       json.keys_under_root: true
  192.       json.overwrite_keys: true
  193.       tags: ["error","hm"]
  194.     - type: log
  195.       paths:
  196.         - /data/tomcat-haozhuo-health/json.info.*.log
  197.       json.keys_under_root: true
  198.       json.overwrite_keys: true
  199.       tags: ["info","health"]
  200.     - type: log
  201.       enabled: true
  202.       paths:
  203.        - /data/tomcat-haozhuo-health/json.error.*.log
  204.       json.keys_under_root: true
  205.       json.overwrite_keys: true
  206.       tags: ["error","health"]
  207.     - type: log
  208.       paths:
  209.         - /data/tomcat-haozhuo-family/json.info.*.log
  210.       json.keys_under_root: true
  211.       json.overwrite_keys: true
  212.       tags: ["info","family"]
  213.     - type: log
  214.       enabled: true
  215.       paths:
  216.        - /data/tomcat-haozhuo-family/json.error.*.log
  217.       json.keys_under_root: true
  218.       json.overwrite_keys: true
  219.       tags: ["error","family"]
  220.     - type: log
  221.       paths:
  222.         - /data/tomcat-haozhuo-disc/json.info.*.log
  223.       json.keys_under_root: true
  224.       json.overwrite_keys: true
  225.       tags: ["info","disc"]
  226.     - type: log
  227.       enabled: true
  228.       paths:
  229.        - /data/tomcat-haozhuo-disc/json.error.*.log
  230.       json.keys_under_root: true
  231.       json.overwrite_keys: true
  232.       tags: ["error","disc"]
  233.     - type: log
  234.       paths:
  235.         - /data/tomcat-haozhuo-check/json.info.*.log
  236.       json.keys_under_root: true
  237.       json.overwrite_keys: true
  238.       tags: ["info","check"]
  239.     - type: log
  240.       enabled: true
  241.       paths:
  242.        - /data/tomcat-haozhuo-check/json.error.*.log
  243.       json.keys_under_root: true
  244.       json.overwrite_keys: true
  245.       tags: ["error","check"]
  246.     - type: log
  247.       paths:
  248.         - /data/tomcat-haozhuo-card/json.info.*.log
  249.       json.keys_under_root: true
  250.       json.overwrite_keys: true
  251.       tags: ["info","card"]
  252.     - type: log
  253.       enabled: true
  254.       paths:
  255.        - /data/tomcat-haozhuo-card/json.error.*.log
  256.       json.keys_under_root: true
  257.       json.overwrite_keys: true
  258.       tags: ["error","card"]
  259.     - type: log
  260.       paths:
  261.         - /data/tomcat-haozhuo-appServer/json.info.*.log
  262.       json.keys_under_root: true
  263.       json.overwrite_keys: true
  264.       tags: ["info","appServer"]
  265.     - type: log
  266.       enabled: true
  267.       paths:
  268.        - /data/tomcat-haozhuo-appServer/json.error.*.log
  269.       json.keys_under_root: true
  270.       json.overwrite_keys: true
  271.       tags: ["error","appServer"]
  272.     - type: log
  273.       paths:
  274.         - /data/tomcat-haozhuo-admin/json.info.*.log
  275.       json.keys_under_root: true
  276.       json.overwrite_keys: true
  277.       tags: ["info","admin"]
  278.     - type: log
  279.       enabled: true
  280.       paths:
  281.        - /data/tomcat-haozhuo-admin/json.error.*.log
  282.       json.keys_under_root: true
  283.       json.overwrite_keys: true
  284.       tags: ["error","admin"]
  285.     - type: log
  286.       paths:
  287.         - /data/tomcat-haozhuo-account/json.info.*.log
  288.       json.keys_under_root: true
  289.       json.overwrite_keys: true
  290.       tags: ["info","account"]
  291.     - type: log
  292.       enabled: true
  293.       paths:
  294.        - /data/tomcat-haozhuo-account/json.error.*.log
  295.       json.keys_under_root: true
  296.       json.overwrite_keys: true
  297.       tags: ["error","account"]
  298.     - type: log
  299.       paths:
  300.         - /data/tomcat-haozhuo-hmp/json.info.*.log
  301.       json.keys_under_root: true
  302.       json.overwrite_keys: true
  303.       tags: ["info","hmp"]
  304.     - type: log
  305.       enabled: true
  306.       paths:
  307.        - /data/tomcat-haozhuo-hmp/json.error.*.log
  308.       json.keys_under_root: true
  309.       json.overwrite_keys: true
  310.       tags: ["error","hmp"]
  311.     - type: log
  312.       paths:
  313.         - /data/tomcat-haozhuo-search/json.info.*.log
  314.       json.keys_under_root: true
  315.       json.overwrite_keys: true
  316.       tags: ["info","search"]
  318.     - type: log
  319.       enabled: true
  320.       paths:
  321.        - /data/tomcat-haozhuo-search/json.error.*.log
  322.       json.keys_under_root: true
  323.       json.overwrite_keys: true
  324.       tags: ["error","search"]
  325.     - type: log
  326.       paths:
  327.         - /data/tomcat-haozhuo-service_center/info.*.log
  328.       json.keys_under_root: true
  329.       json.overwrite_keys: true
  330.       tags: ["info","service_center"]
  331.     - type: log
  332.       enabled: true
  333.       paths:
  334.        - /data/tomcat-haozhuo-service_center/error.*.log
  335.       json.keys_under_root: true
  336.       json.overwrite_keys: true
  337.       tags: ["error","service_center"]
  338.     - type: log
  339.       paths:
  340.         - /data/tomcat-haozhuo-appoint_center/info.*.log
  341.       json.keys_under_root: true
  342.       json.overwrite_keys: true
  343.       tags: ["info","appoint_center"]
  344.     - type: log
  345.       enabled: true
  346.       paths:
  347.        - /data/tomcat-haozhuo-appoint_center/error.*.log
  348.       json.keys_under_root: true
  349.       json.overwrite_keys: true
  350.       tags: ["error","appoint_center"]
  351.     filebeat.config.modules:
  352.       path: ${ path.config}/modules.d/*.yml
  353.       reload.enabled: false
  354.     setup.template.enabled: false
  355.     setup.template.settings:
  356.       index.number_of_shards: 3
  357.     setup.template.name: "filebeat"
  358.     setup.template.fields: "fields.yml"
  359.     setup.template.overwrite: false
  360.     setup.kibana:
  361.     output.elasticsearch:
  362.      hosts: ["192.168.1.165:9200"]
  363.      index: "jsonlog-%{+yyyy.MM.dd}"

创建一个statefulset部署应用服务。filebeat以sidecar形式一起部署 StatefulSet.yaml

  1. apiVersion: apps/v1
  2. kind: StatefulSet
  3. metadata:
  4. # deployment名字 和svc 和ingress绑定没关系
  5.   name: haozhuo-account-dp
  6.   namespace: test
  7. spec:
  8.   serviceName: haozhuo-account
  9.   replicas: 1
  10.   # 在定义模板的时候必须定义labels,因为Deployment.spec.selector是必须字段,而他又必须和template.labels对应
  11.   selector:
  12.     matchLabels:
  13.       app: haozhuo-account
  14.   # template里面定义的内容会应用到下面所有的副本集里面(例如depolyment下的pod),在template.spec.containers里面不能定义labels标签。可以kubectl get pods --show-labels查看
  15.   template:
  16.     metadata:
  17.       labels:
  18.         app: haozhuo-account
  19.         env: test
  20.     spec:
  21.       containers:
  22. # 20200822
  24.       - image: registry.cn-shanghai.aliyuncs.com/yjk-datag/filebeat:6 # 提前下载下来到私有镜像库的镜像(官方的可能会被墙)
  25.         name: filebeat
  26.         args: [
  27.           "-c", "/srv/filebeat/filebeat.yml",
  28.           "-e",
  29.         ]
  30.         env:
  31.         - name: POD_IP
  32.           valueFrom:
  33.             fieldRef:
  34.               apiVersion: v1
  35.               fieldPath: status.podIP
  36.         - name: pod_name
  37.           valueFrom:
  38.             fieldRef:
  39.               apiVersion: v1
  40.               fieldPath: metadata.name
  41.         - name: POD_IP
  42.           valueFrom:
  43.             fieldRef:
  44.               apiVersion: v1
  45.               fieldPath: status.podIP
  46.         - name: pod_name
  47.           valueFrom:
  48.             fieldRef:
  49.               apiVersion: v1
  50.               fieldPath: metadata.name
  51.        # securityContext:
  52.        # runAsUser: 0
  53.         resources:
  54.           limits:
  55.             memory: 200Mi
  56.           requests:
  57.             cpu: 200m
  58.             memory: 200Mi
  59.         volumeMounts:
  60.         - name: config               # 将configmap的内容放到容器本地目录
  61.           mountPath: /srv/filebeat/
  62.         - name: data
  63.           mountPath: /usr/share/filebeat/data
  64.         - name: logdata       # 同一个pod内的两个应用共享目录logdata, 一个写一个读
  65.           mountPath: /data/tomcat-haozhuo-account/
  69. # 20200822
  73.       # containers名字 和svc 和ingress绑定没关系
  74.       - name: haozhuo-account
  75.         image: registry.cn-shanghai.aliyuncs.com/yjk-datag/k8s-haozhuo-account:v28
  76.         #image: registry.cn-shanghai.aliyuncs.com/yjk-datag/haozhuo-video:vtest06
  77.         env:
  78.         - name: DUBBO_IP_TO_REGISTRY
  79.           valueFrom:
  80.             fieldRef:
  81.               fieldPath: status.hostIP
  82.         - name: DUBBO_PORT_TO_REGISTRY
  83.           value: "20901"
  84.         resources:
  85.           limits:
  86.             cpu: 2000m
  87.             memory: 2048Mi
  88.           requests:
  89.             cpu: 1000m
  90.             memory: 1024Mi
  91.         ports:
  92.         - name: jacoco-port
  93.           containerPort: 6309
  94.         - name: liveness-port
  95.           containerPort: 8080
  96. # 使用了 hostPort 的容器只能调度到端口不冲突的 Node 上，除非有必要（比如运行一些系统级的 daemon 服务），不建议使用端口映射功能。
  97. # 跟 hostPort 一样的原因，避免使用 hostNetwork。
  98. # hostPorts:
  99. # - min: 8000
  100. # max: 8080
  101.           hostPort: 8801
  102.         - name: readiness-port
  103.           containerPort: 20901
  104.           hostPort: 20901
  105. # 后续需要jacoco端口也暴露一下
  107.         livenessProbe:
  108.           tcpSocket:
  109.             port: liveness-port
  110.           initialDelaySeconds: 20
  111.           periodSeconds: 3
  113.         readinessProbe:
  114.           tcpSocket:
  115.             port: readiness-port
  116.           initialDelaySeconds: 30
  117.           periodSeconds: 10
  120. # 20200822
  122.         volumeMounts:
  123.         - name: logdata       # 指定挂在目录到logdata 同一个pod内的两个应用共享目录logdata, 一个写一个读
  124.         # 我这里要挂载应用容器haozhuo-account中/srv/tomcat/tomcat-haozhuo-account/yjklogs/里面的*.log到filebeat容器的/data/tomcat-haozhuo-account/下
  125.           mountPath: /srv/tomcat/tomcat-haozhuo-account/yjklogs/
  126.       volumes:
  127.       - name: data
  128.         emptyDir: { }
  129.       - name: logdata         #定义logdata为EmptyDir类型挂载目录
  130.         emptyDir: { }
  131.       - name: config
  132.         configMap:
  133.           name: test-filebeat-config  #使用前面定义的configmap
  134.           items:
  135.           - key: filebeat.yml
  136.             path: filebeat.yml
  138. # 20200822
  140.       imagePullSecrets:
  141.       - name: aliyun-ns-test

导入并上传镜像

  1. docker load < filebeat6.tar.gz
  3. docker tag prima/filebeat:6 registry.cn-shanghai.aliyuncs.com/yjk-datag/filebeat:6
  5. docker push registry.cn-shanghai.aliyuncs.com/yjk-datag/filebeat:6

format_png

应用yaml

  1. kubectl apply -f StatefulSet.yaml

等到容器启动完成

进入容器需要 -c 指定容器name

  1. # 进入应用容器
  2. kubectl exec -it haozhuo-account-dp-0   -c haozhuo-account  /bin/bash -n test
  3. # 进入filebeat容器
  4. kubectl exec -it haozhuo-account-dp-0   -c filebeat  /bin/bash -n test
  6. # 如果不使用-c的话会进入sidecar容器

format_png 1

format_png 2

手动输出error到error日志或者info日志

  1. echo '{"@timestamp":"2020-08-22T15:59:10.014+08:00","@version":1,"message":"请求url:http://192.168.1.164:8030/dingtalk/order/saveOrderDataRecord 入参:","logger_name":"com.haozhuo.common.sys.http.HttpRequest","thread_name":"account-updateOrderBizData_Worker-1","level":"INFO","level_value":20000,"HOSTNAME":"haozhuo-account-dp-0.haozhuo-account.test.svc.cluster.local","tid":"6494f345-b18e-40ac-b4fc-baf851cab191"} ' >> /data/tomcat-haozhuo-account/json.info.2020-08-22.log

观察filebeat 发现采集成功

format_png 3

  1. kubectl logs -f  haozhuo-account-dp-0  filebeat -n test 
  2. kubectl logs -f  haozhuo-account-dp-0  haozhuo-account -n test

format_png 4

发表评论

表情：
评论列表 （有 0 条评论，286人围观）

还没有评论，来说两句吧...

相关阅读