Systemd Log Flood Run Docker Runtime X2drunc Moby Runc Mount

發現 /var/log/system 裡頭大量重複的 log:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


Jun 28 14:30:03 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.BHxFRm.mount: Deactivated successfully.
Jun 28 14:30:23 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.KWS2zz.mount: Deactivated successfully.
Jun 28 14:30:33 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.Vb7E7L.mount: Deactivated successfully.
Jun 28 14:30:53 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.1B05S3.mount: Deactivated successfully.
Jun 28 14:31:03 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.uUSsuO.mount: Deactivated successfully.
Jun 28 14:31:13 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.s3gTlw.mount: Deactivated successfully.
Jun 28 14:31:33 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.hDy9ZF.mount: Deactivated successfully.
Jun 28 14:31:43 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.YYXMI6.mount: Deactivated successfully.
Jun 28 14:32:03 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.5k0cuM.mount: Deactivated successfully.
Jun 28 14:32:13 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.QLn6f8.mount: Deactivated successfully.
Jun 28 14:32:23 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.EdwPh7.mount: Deactivated successfully.
Jun 28 14:32:33 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.gf4O7U.mount: Deactivated successfully.
Jun 28 14:33:03 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.gqYRNZ.mount: Deactivated successfully.
Jun 28 14:33:34 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.w4IFzY.mount: Deactivated successfully.
Jun 28 14:33:44 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.tK3r5m.mount: Deactivated successfully.
Jun 28 14:33:54 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.Uh0uVh.mount: Deactivated successfully.
Jun 28 14:34:14 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.y6wojs.mount: Deactivated successfully.
Jun 28 14:34:24 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.bYEpi5.mount: Deactivated successfully.
Jun 28 14:34:34 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.VKiHNk.mount: Deactivated successfully.
Jun 28 14:34:54 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.D2KiyQ.mount: Deactivated successfully.
Jun 28 14:35:24 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.wEaYsv.mount: Deactivated successfully.
Jun 28 14:35:44 container-clone systemd[1]: run-docker-runtime\x2drunc-moby-1f61c727cb8dac371d3e3a00274ad37599bbf550473c55481a8f1bcb8e9aef70-runc.8vjJQq.mount: Deactivated successfully.

經過一番調查後，關鍵情報在 docker/for-linux issues #679。

找到 container #1f61c727 下 docker inspect 後，確實發現有啟用 health check。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56


{

  ...

  "Health": {
      "Status": "healthy",
      "FailingStreak": 0,
      "Log": [
          {
              "Start": "2024-06-28T15:09:54.506723522+08:00",
              "End": "2024-06-28T15:09:54.549099001+08:00",
              "ExitCode": 0,
              "Output": "{\"cluster_id\":\"dae76c58-321c-4c55-981d-a6fcf2b94720\",\"node_id\":\"9e5ff5c9-e40d-4744-ad20-074bd7d25fd8\",\"version\":\"6.0.3+eb761c5\",\"tagline\":\"Manage your logs in the dark and have lasers going and make it look like you're from space!\"}"
          },
          {
              "Start": "2024-06-28T15:10:04.551047607+08:00",
              "End": "2024-06-28T15:10:04.593694092+08:00",
              "ExitCode": 0,
              "Output": "{\"cluster_id\":\"dae76c58-321c-4c55-981d-a6fcf2b94720\",\"node_id\":\"9e5ff5c9-e40d-4744-ad20-074bd7d25fd8\",\"version\":\"6.0.3+eb761c5\",\"tagline\":\"Manage your logs in the dark and have lasers going and make it look like you're from space!\"}"
          },
          {
              "Start": "2024-06-28T15:10:14.596051024+08:00",
              "End": "2024-06-28T15:10:14.641753851+08:00",
              "ExitCode": 0,
              "Output": "{\"cluster_id\":\"dae76c58-321c-4c55-981d-a6fcf2b94720\",\"node_id\":\"9e5ff5c9-e40d-4744-ad20-074bd7d25fd8\",\"version\":\"6.0.3+eb761c5\",\"tagline\":\"Manage your logs in the dark and have lasers going and make it look like you're from space!\"}"
          },
          {
              "Start": "2024-06-28T15:10:24.64357778+08:00",
              "End": "2024-06-28T15:10:24.685984565+08:00",
              "ExitCode": 0,
              "Output": "{\"cluster_id\":\"dae76c58-321c-4c55-981d-a6fcf2b94720\",\"node_id\":\"9e5ff5c9-e40d-4744-ad20-074bd7d25fd8\",\"version\":\"6.0.3+eb761c5\",\"tagline\":\"Manage your logs in the dark and have lasers going and make it look like you're from space!\"}"
          },
          {
              "Start": "2024-06-28T15:10:34.688235952+08:00",
              "End": "2024-06-28T15:10:34.730117272+08:00",
              "ExitCode": 0,
              "Output": "{\"cluster_id\":\"dae76c58-321c-4c55-981d-a6fcf2b94720\",\"node_id\":\"9e5ff5c9-e40d-4744-ad20-074bd7d25fd8\",\"version\":\"6.0.3+eb761c5\",\"tagline\":\"Manage your logs in the dark and have lasers going and make it look like you're from space!\"}"
          }
      ]
  }

  ...

  "Healthcheck": {
    "Test": [
        "CMD-SHELL",
        "/health_check.sh"
    ],
    "Interval": 10000000000,
    "Timeout": 2000000000,
    "Retries": 12
  },

  ...

}

目前沒有正確的的解決方式，緩解措施有兩種：

在 rsyslog 對符合特定條件的 log 進行排除
在 systemd 透過 LogLevelMax 選項控制 (從 v249 版開始支援)

rsyslog 的方法可能只解了一半，有人提到僅阻止了 rsyslog，但是 log 還是會寫到 journald，因此我採用的是 systemd 的方法。

稍微改寫後如下 3 個指令

1
2
3


sudo mkdir -p /etc/systemd/system/run-docker-.mount.d
echo "[Mount]\nLogLevelMax=notice" | sudo tee 10-silence.conf
sudo systemctl daemon-reload

暫時沒必要記錄這種 log，就先這樣做了。

Reference: systemd logs filled with mount unit entries if healtcheck is enabled · Issue #679 · docker/for-linux · GitHub