[2018-10-16T01:03:39.569+0000] at this point there should be 1 fluentd running in Running state
[2018-10-16T01:03:39.572+0000] NAME READY STATUS RESTARTS AGE
[2018-10-16T01:03:39.572+0000] logging-es-data-master-3wtdcqfo-1-7rbjd 2/2 Running 0 26m
[2018-10-16T01:03:39.572+0000] logging-es-ops-data-master-dde8kclb-1-n287x 2/2 Running 0 26m
[2018-10-16T01:03:39.572+0000] logging-fluentd-qtf2p 1/1 Running 0 5s
[2018-10-16T01:03:39.572+0000] logging-kibana-1-k9lvd 2/2 Running 0 28m
[2018-10-16T01:03:39.572+0000] logging-kibana-ops-1-p8xfj 2/2 Running 0 28m
[2018-10-16T01:03:39.572+0000] logging-mux-1-q6mnx 1/1 Running 0 27m
[2018-10-16T01:03:39.775+0000] node/ip-172-18-7-24.ec2.internal labeled
[2018-10-16T01:03:40.232+0000] at this point there should be 0 fluentd running
[2018-10-16T01:03:40.234+0000] NAME READY STATUS RESTARTS AGE
[2018-10-16T01:03:40.234+0000] logging-es-data-master-3wtdcqfo-1-7rbjd 2/2 Running 0 26m
[2018-10-16T01:03:40.234+0000] logging-es-ops-data-master-dde8kclb-1-n287x 2/2 Running 0 26m
[2018-10-16T01:03:40.234+0000] logging-fluentd-qtf2p 1/1 Terminating 0 6s
[2018-10-16T01:03:40.234+0000] logging-kibana-1-k9lvd 2/2 Running 0 28m
[2018-10-16T01:03:40.234+0000] logging-kibana-ops-1-p8xfj 2/2 Running 0 28m
[2018-10-16T01:03:40.234+0000] logging-mux-1-q6mnx 1/1 Running 0 27m
[2018-10-16T01:03:40.418+0000] daemonset.extensions/logging-fluentd updated
[2018-10-16T01:03:40.623+0000] node/ip-172-18-7-24.ec2.internal labeled
[2018-10-16T01:03:53.748+0000] Deleted archived journal /var/log/journal/4372f1e2f8c642d3a2f3ed11aa3fe654/system@e88eb39dfa5f41cba87692648482f60b-0000000000002765-0005784cff525eba.journal (8.0M).
[2018-10-16T01:03:53.748+0000] Deleted archived journal /var/log/journal/4372f1e2f8c642d3a2f3ed11aa3fe654/system@e88eb39dfa5f41cba87692648482f60b-00000000000028da-0005784d00c52673.journal (16.0M).
[2018-10-16T01:03:53.748+0000] Deleted archived journal /var/log/journal/4372f1e2f8c642d3a2f3ed11aa3fe654/user-1001@baf31c3443a346adb109a7a9a922f8c3-00000000000028d8-0005784d00c4866f.journal (8.0M).
[2018-10-16T01:03:53.748+0000] Deleted archived journal /var/log/journal/4372f1e2f8c642d3a2f3ed11aa3fe654/system@e88eb39dfa5f41cba87692648482f60b-0000000000005b03-0005784dbed2d1ee.journal (8.0M).
[2018-10-16T01:03:53.748+0000] Deleted archived journal /var/log/journal/4372f1e2f8c642d3a2f3ed11aa3fe654/user-1001@baf31c3443a346adb109a7a9a922f8c3-0000000000005ba2-0005784dbff95d73.journal (8.0M).
[2018-10-16T01:03:53.748+0000] Vacuuming done, freed 48.0M of archived journals on disk.
[2018-10-16T01:03:54.486+0000] deploymentconfig.apps.openshift.io/logging-mux scaled
[2018-10-16T01:03:54.960+0000] logging-mux-1-q6mnx 1/1 Terminating 0 27m
[2018-10-16T01:03:55.243+0000] NAME REVISION DESIRED CURRENT TRIGGERED BY
[2018-10-16T01:03:55.243+0000] logging-es-data-master-3wtdcqfo 1 1 1
[2018-10-16T01:03:55.243+0000] logging-es-ops-data-master-dde8kclb 1 1 1
[2018-10-16T01:03:55.243+0000] logging-kibana 1 1 1 config
[2018-10-16T01:03:55.243+0000] logging-kibana-ops 1 1 1 config
[2018-10-16T01:03:55.243+0000] logging-mux 1 0 0 config
[2018-10-16T01:03:55.460+0000] deploymentconfig.apps.openshift.io/logging-mux updated
[2018-10-16T01:03:55.753+0000] logging-mux-1-q6mnx 1/1 Terminating 0 27m
[2018-10-16T01:03:55.753+0000] logging-mux-2-deploy 0/1 ContainerCreating 0 0s
[2018-10-16T01:03:56.001+0000] NAME REVISION DESIRED CURRENT TRIGGERED BY
[2018-10-16T01:03:56.001+0000] logging-es-data-master-3wtdcqfo 1 1 1
[2018-10-16T01:03:56.001+0000] logging-es-ops-data-master-dde8kclb 1 1 1
[2018-10-16T01:03:56.001+0000] logging-kibana 1 1 1 config
[2018-10-16T01:03:56.001+0000] logging-kibana-ops 1 1 1 config
[2018-10-16T01:03:56.001+0000] logging-mux 2 0 0 config
[2018-10-16T01:03:56.197+0000] deploymentconfig.apps.openshift.io/logging-mux scaled
[2018-10-16T01:03:56.453+0000] logging-mux-1-q6mnx 1/1 Terminating 0 27m
[2018-10-16T01:03:56.453+0000] logging-mux-2-deploy 0/1 ContainerCreating 0 1s
[2018-10-16T01:03:56.639+0000] NAME REVISION DESIRED CURRENT TRIGGERED BY
[2018-10-16T01:03:56.639+0000] logging-es-data-master-3wtdcqfo 1 1 1
[2018-10-16T01:03:56.639+0000] logging-es-ops-data-master-dde8kclb 1 1 1
[2018-10-16T01:03:56.639+0000] logging-kibana 1 1 1 config
[2018-10-16T01:03:56.639+0000] logging-kibana-ops 1 1 1 config
[2018-10-16T01:03:56.639+0000] logging-mux 2 1 0 config
[2018-10-16T01:04:03.261+0000] Test 1, expecting generate_syslog_config.rb to have created configuration file logging-mux-2-kmm2b
[2018-10-16T01:04:03.554+0000] deploymentconfig.apps.openshift.io/logging-mux scaled
[2018-10-16T01:04:04.056+0000] deploymentconfig.apps.openshift.io/logging-mux updated
[2018-10-16T01:04:04.258+0000] deploymentconfig.apps.openshift.io/logging-mux scaled
[2018-10-16T01:04:13.855+0000] Test 2, expecting generate_syslog_config.rb to not create a configuration file logging-mux-3-gbmnw
[2018-10-16T01:04:14.100+0000] deploymentconfig.apps.openshift.io/logging-mux scaled
[2018-10-16T01:04:14.577+0000] deploymentconfig.apps.openshift.io/logging-mux updated
[2018-10-16T01:04:14.776+0000] deploymentconfig.apps.openshift.io/logging-mux scaled
[2018-10-16T01:04:25.022+0000] Test 3, expecting generate_syslog_config.rb to generate multiple stores logging-mux-4-crjql
[2018-10-16T01:04:25.290+0000] deploymentconfig.apps.openshift.io/logging-mux scaled
[2018-10-16T01:04:25.795+0000] deploymentconfig.apps.openshift.io/logging-mux updated
[2018-10-16T01:04:26.005+0000] deploymentconfig.apps.openshift.io/logging-mux scaled
[2018-10-16T01:04:36.102+0000] Test 4, making sure tag_key=message does not cause remote-syslog plugin crash logging-mux-5-gm7kn
[2018-10-16T01:04:36.362+0000] deploymentconfig.apps.openshift.io/logging-mux scaled
[2018-10-16T01:04:36.991+0000] deploymentconfig.apps.openshift.io/logging-mux updated
[2018-10-16T01:04:37.196+0000] deploymentconfig.apps.openshift.io/logging-mux scaled
[2018-10-16T01:04:47.645+0000] Test 5, making sure tag_key=bogus does not cause remote-syslog plugin crash logging-mux-6-n4l9x
[2018-10-16T01:04:47.648+0000] iptables ACCEPT 601
[2018-10-16T01:04:47.676+0000] Chain INPUT (policy ACCEPT)
[2018-10-16T01:04:47.676+0000] target prot opt source destination
[2018-10-16T01:04:47.676+0000] KUBE-EXTERNAL-SERVICES all -- anywhere anywhere ctstate NEW /* kubernetes externally-visible service portals */
[2018-10-16T01:04:47.676+0000] KUBE-NODEPORT-NON-LOCAL all -- anywhere anywhere /* Ensure that non-local NodePort traffic can flow */
[2018-10-16T01:04:47.676+0000] OPENSHIFT-FIREWALL-ALLOW all -- anywhere anywhere /* firewall overrides */
[2018-10-16T01:04:47.676+0000] KUBE-FIREWALL all -- anywhere anywhere
[2018-10-16T01:04:47.676+0000] ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
[2018-10-16T01:04:47.676+0000] ACCEPT icmp -- anywhere anywhere
[2018-10-16T01:04:47.676+0000] ACCEPT all -- anywhere anywhere
[2018-10-16T01:04:47.676+0000] ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh
[2018-10-16T01:04:47.676+0000] OS_FIREWALL_ALLOW all -- anywhere anywhere
[2018-10-16T01:04:47.676+0000] REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
[2018-10-16T01:04:47.676+0000] ACCEPT tcp -- anywhere anywhere tcp dpt:syslog-conn
[2018-10-16T01:04:47.676+0000]
[2018-10-16T01:04:47.676+0000] Chain FORWARD (policy ACCEPT)
[2018-10-16T01:04:47.676+0000] target prot opt source destination
[2018-10-16T01:04:47.676+0000] KUBE-FORWARD all -- anywhere anywhere /* kubernetes forwarding rules */
[2018-10-16T01:04:47.676+0000] OPENSHIFT-ADMIN-OUTPUT-RULES all -- anywhere anywhere /* administrator overrides */
[2018-10-16T01:04:47.676+0000] OPENSHIFT-FIREWALL-FORWARD all -- anywhere anywhere /* firewall overrides */
[2018-10-16T01:04:47.676+0000] DOCKER-ISOLATION all -- anywhere anywhere
[2018-10-16T01:04:47.676+0000] DOCKER all -- anywhere anywhere
[2018-10-16T01:04:47.676+0000] ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
[2018-10-16T01:04:47.676+0000] ACCEPT all -- anywhere anywhere
[2018-10-16T01:04:47.676+0000] ACCEPT all -- anywhere anywhere
[2018-10-16T01:04:47.676+0000] REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
[2018-10-16T01:04:47.676+0000]
[2018-10-16T01:04:47.676+0000] Chain OUTPUT (policy ACCEPT)
[2018-10-16T01:04:47.676+0000] target prot opt source destination
[2018-10-16T01:04:47.676+0000] KUBE-SERVICES all -- anywhere anywhere ctstate NEW /* kubernetes service portals */
[2018-10-16T01:04:47.676+0000] KUBE-FIREWALL all -- anywhere anywhere
[2018-10-16T01:04:47.676+0000]
[2018-10-16T01:04:47.676+0000] Chain DOCKER (1 references)
[2018-10-16T01:04:47.676+0000] target prot opt source destination
[2018-10-16T01:04:47.676+0000]
[2018-10-16T01:04:47.676+0000] Chain DOCKER-ISOLATION (1 references)
[2018-10-16T01:04:47.676+0000] target prot opt source destination
[2018-10-16T01:04:47.676+0000] RETURN all -- anywhere anywhere
[2018-10-16T01:04:47.676+0000]
[2018-10-16T01:04:47.676+0000] Chain KUBE-EXTERNAL-SERVICES (1 references)
[2018-10-16T01:04:47.676+0000] target prot opt source destination
[2018-10-16T01:04:47.676+0000]
[2018-10-16T01:04:47.676+0000] Chain KUBE-FIREWALL (2 references)
[2018-10-16T01:04:47.676+0000] target prot opt source destination
[2018-10-16T01:04:47.676+0000] DROP all -- anywhere anywhere /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
[2018-10-16T01:04:47.676+0000]
[2018-10-16T01:04:47.676+0000] Chain KUBE-FORWARD (1 references)
[2018-10-16T01:04:47.676+0000] target prot opt source destination
[2018-10-16T01:04:47.676+0000] ACCEPT all -- anywhere anywhere /* kubernetes forwarding rules */ mark match 0x1/0x1
[2018-10-16T01:04:47.676+0000] ACCEPT all -- ip-10-128-0-0.ec2.internal/14 anywhere /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
[2018-10-16T01:04:47.676+0000] ACCEPT all -- anywhere ip-10-128-0-0.ec2.internal/14 /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED
[2018-10-16T01:04:47.676+0000]
[2018-10-16T01:04:47.676+0000] Chain KUBE-NODEPORT-NON-LOCAL (1 references)
[2018-10-16T01:04:47.676+0000] target prot opt source destination
[2018-10-16T01:04:47.676+0000]
[2018-10-16T01:04:47.676+0000] Chain KUBE-SERVICES (1 references)
[2018-10-16T01:04:47.676+0000] target prot opt source destination
[2018-10-16T01:04:47.676+0000] REJECT tcp -- anywhere asb.openshift-ansible-service-broker.svc.cluster.local /* openshift-ansible-service-broker/asb:port-1337 has no endpoints */ tcp dpt:menandmice-dns reject-with icmp-port-unreachable
[2018-10-16T01:04:47.676+0000] REJECT tcp -- anywhere asb.openshift-ansible-service-broker.svc.cluster.local /* openshift-ansible-service-broker/asb:port-1338 has no endpoints */ tcp dpt:wmc-log-svc reject-with icmp-port-unreachable
[2018-10-16T01:04:47.676+0000]
[2018-10-16T01:04:47.676+0000] Chain OPENSHIFT-ADMIN-OUTPUT-RULES (1 references)
[2018-10-16T01:04:47.676+0000] target prot opt source destination
[2018-10-16T01:04:47.676+0000]
[2018-10-16T01:04:47.676+0000] Chain OPENSHIFT-FIREWALL-ALLOW (1 references)
[2018-10-16T01:04:47.676+0000] target prot opt source destination
[2018-10-16T01:04:47.676+0000] ACCEPT udp -- anywhere anywhere udp dpt:4789 /* VXLAN incoming */
[2018-10-16T01:04:47.676+0000] ACCEPT all -- anywhere anywhere /* from SDN to localhost */
[2018-10-16T01:04:47.676+0000] ACCEPT all -- anywhere anywhere /* from docker to localhost */
[2018-10-16T01:04:47.676+0000]
[2018-10-16T01:04:47.676+0000] Chain OPENSHIFT-FIREWALL-FORWARD (1 references)
[2018-10-16T01:04:47.676+0000] target prot opt source destination
[2018-10-16T01:04:47.676+0000] DROP all -- ip-10-128-0-0.ec2.internal/14 anywhere /* attempted resend after connection close */ ctstate INVALID
[2018-10-16T01:04:47.676+0000] ACCEPT all -- anywhere ip-10-128-0-0.ec2.internal/14 /* forward traffic from SDN */
[2018-10-16T01:04:47.676+0000] ACCEPT all -- ip-10-128-0-0.ec2.internal/14 anywhere /* forward traffic to SDN */
[2018-10-16T01:04:47.676+0000]
[2018-10-16T01:04:47.676+0000] Chain OS_FIREWALL_ALLOW (1 references)
[2018-10-16T01:04:47.676+0000] target prot opt source destination
[2018-10-16T01:04:47.676+0000] ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:10250
[2018-10-16T01:04:47.676+0000] ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:10256
[2018-10-16T01:04:47.676+0000] ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:http
[2018-10-16T01:04:47.676+0000] ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:https
[2018-10-16T01:04:47.676+0000] ACCEPT udp -- anywhere anywhere state NEW udp dpt:4789
[2018-10-16T01:04:47.676+0000] ACCEPT tcp -- anywhere anywhere state NEW tcp dpts:ndmps:32000
[2018-10-16T01:04:47.676+0000] ACCEPT udp -- anywhere anywhere state NEW udp dpts:30000:32000
[2018-10-16T01:04:47.676+0000] ACCEPT tcp -- anywhere anywhere state NEW tcp dpts:cslistener:ndmp
[2018-10-16T01:04:47.676+0000] ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:2379
[2018-10-16T01:04:47.676+0000] ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:2380
[2018-10-16T01:04:47.676+0000] ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:pcsync-https
[2018-10-16T01:04:47.676+0000] ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:pcsync-http
[2018-10-16T01:04:47.676+0000] ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:senomix02
[2018-10-16T01:04:47.676+0000] ACCEPT udp -- anywhere anywhere state NEW udp dpt:senomix02
[2018-10-16T01:04:47.789+0000] total 8
[2018-10-16T01:04:47.789+0000] -rw-r--r--. 1 root root 255 Feb 14 2018 21-cloudinit.conf
[2018-10-16T01:04:47.789+0000] -rw-r--r--. 1 root root 49 Feb 20 2018 listen.conf
[2018-10-16T01:04:47.830+0000] Before restarting rsyslog
[2018-10-16T01:04:47.841+0000] ● rsyslog.service - System Logging Service
[2018-10-16T01:04:47.841+0000] Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
[2018-10-16T01:04:47.841+0000] Active: active (running) since Tue 2018-10-16 00:14:08 UTC; 50min ago
[2018-10-16T01:04:47.841+0000] Docs: man:rsyslogd(8)
[2018-10-16T01:04:47.841+0000] http://www.rsyslog.com/doc/
[2018-10-16T01:04:47.841+0000] Main PID: 6152 (rsyslogd)
[2018-10-16T01:04:47.841+0000] Memory: 6.6M
[2018-10-16T01:04:47.841+0000] CGroup: /system.slice/rsyslog.service
[2018-10-16T01:04:47.841+0000] └─6152 /usr/sbin/rsyslogd -n
[2018-10-16T01:04:47.841+0000]
[2018-10-16T01:04:47.841+0000] Oct 16 00:55:49 ip-172-18-7-24.ec2.internal rsyslogd[6152]: imjournal: journal reloaded... [v8.24.0 try http://www.rsyslog.com/e/0 ]
[2018-10-16T01:04:47.841+0000] Oct 16 00:55:49 ip-172-18-7-24.ec2.internal rsyslogd[6152]: imjournal: journal reloaded... [v8.24.0 try http://www.rsyslog.com/e/0 ]
[2018-10-16T01:04:47.841+0000] Oct 16 01:03:53 ip-172-18-7-24.ec2.internal rsyslogd[6152]: imjournal: journal reloaded... [v8.24.0 try http://www.rsyslog.com/e/0 ]
[2018-10-16T01:04:47.841+0000] Oct 16 01:03:53 ip-172-18-7-24.ec2.internal rsyslogd[6152]: imjournal: journal reloaded... [v8.24.0 try http://www.rsyslog.com/e/0 ]
[2018-10-16T01:04:47.841+0000] Oct 16 01:03:53 ip-172-18-7-24.ec2.internal rsyslogd[6152]: imjournal: journal reloaded... [v8.24.0 try http://www.rsyslog.com/e/0 ]
[2018-10-16T01:04:47.841+0000] Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
[2018-10-16T01:04:51.920+0000] restorecon reset /var/log/messages context unconfined_u:object_r:var_log_t:s0->system_u:object_r:var_log_t:s0
[2018-10-16T01:04:52.027+0000] After restarted rsyslog
[2018-10-16T01:04:52.030+0000] ● rsyslog.service - System Logging Service
[2018-10-16T01:04:52.030+0000] Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
[2018-10-16T01:04:52.030+0000] Active: active (running) since Tue 2018-10-16 01:04:52 UTC; 35ms ago
[2018-10-16T01:04:52.030+0000] Docs: man:rsyslogd(8)
[2018-10-16T01:04:52.030+0000] http://www.rsyslog.com/doc/
[2018-10-16T01:04:52.030+0000] Main PID: 26790 (rsyslogd)
[2018-10-16T01:04:52.030+0000] Memory: 620.0K
[2018-10-16T01:04:52.030+0000] CGroup: /system.slice/rsyslog.service
[2018-10-16T01:04:52.030+0000] └─26790 /usr/sbin/rsyslogd -n
[2018-10-16T01:04:52.030+0000]
[2018-10-16T01:04:52.030+0000] Oct 16 01:04:51 ip-172-18-7-24.ec2.internal systemd[1]: Starting System Logging Service...
[2018-10-16T01:04:52.030+0000] Oct 16 01:04:52 ip-172-18-7-24.ec2.internal rsyslogd[26790]: [origin software="rsyslogd" swVersion="8.24.0" x-pid="26790" x-info="http://www.rsyslog.com"] start
[2018-10-16T01:04:52.030+0000] Oct 16 01:04:52 ip-172-18-7-24.ec2.internal systemd[1]: Started System Logging Service.
[2018-10-16T01:04:52.349+0000] deploymentconfig.apps.openshift.io/logging-mux scaled
[2018-10-16T01:04:52.820+0000] deploymentconfig.apps.openshift.io/logging-mux updated
[2018-10-16T01:04:53.040+0000] deploymentconfig.apps.openshift.io/logging-mux scaled
[2018-10-16T01:05:02.419+0000] node/ip-172-18-7-24.ec2.internal labeled
[2018-10-16T01:05:02.907+0000] daemonset.extensions/logging-fluentd updated
[2018-10-16T01:05:03.129+0000] node/ip-172-18-7-24.ec2.internal labeled
[2018-10-16T01:05:14.110+0000] ==> /etc/fluent/fluent.conf <==
[2018-10-16T01:05:14.110+0000] # This file is the fluentd configuration entrypoint. Edit with care.
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.110+0000] @include configs.d/openshift/system.conf
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.110+0000] # In each section below, pre- and post- includes don't include anything initially;
[2018-10-16T01:05:14.110+0000] # they exist to enable future additions to openshift conf as needed.
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.110+0000] ## sources
[2018-10-16T01:05:14.110+0000] ## ordered so that syslog always runs last...
[2018-10-16T01:05:14.110+0000] @include configs.d/openshift/input-pre-*.conf
[2018-10-16T01:05:14.110+0000] @include configs.d/dynamic/input-docker-*.conf
[2018-10-16T01:05:14.110+0000] @include configs.d/dynamic/input-syslog-*.conf
[2018-10-16T01:05:14.110+0000] @include configs.d/openshift/input-post-*.conf
[2018-10-16T01:05:14.110+0000] ##
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.110+0000] ==> /etc/fluent/configs.d/openshift/output-operations.conf <==
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.110+0000] @type copy
[2018-10-16T01:05:14.110+0000] @include output-es-ops-retry.conf
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.110+0000] @type copy
[2018-10-16T01:05:14.110+0000] @include ../dynamic/output-es-ops-config.conf
[2018-10-16T01:05:14.110+0000] @include ../dynamic/output-remote-syslog.conf
[2018-10-16T01:05:14.110+0000] @include ../user/output-ops-extra-*.conf
[2018-10-16T01:05:14.110+0000] @include ../user/secure-forward.conf
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.110+0000] ==> /etc/fluent/configs.d/openshift/output-applications.conf <==
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.110+0000] @type copy
[2018-10-16T01:05:14.110+0000] @include output-es-retry.conf
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.110+0000] @type copy
[2018-10-16T01:05:14.110+0000] @include output-es-config.conf
[2018-10-16T01:05:14.110+0000] @include ../dynamic/output-remote-syslog.conf
[2018-10-16T01:05:14.110+0000] @include ../user/output-extra-*.conf
[2018-10-16T01:05:14.110+0000] @include ../user/secure-forward.conf
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.110+0000] ==> /etc/fluent/configs.d/dynamic/output-remote-syslog.conf <==
[2018-10-16T01:05:14.110+0000] ## This file was generated by generate-syslog-config.rb
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.110+0000] @type syslog_buffered
[2018-10-16T01:05:14.110+0000] @id remote-syslog-input
[2018-10-16T01:05:14.110+0000] remote_syslog ip-172-18-7-24.ec2.internal
[2018-10-16T01:05:14.110+0000] port 601
[2018-10-16T01:05:14.110+0000] hostname logging-mux-7-px48g
[2018-10-16T01:05:14.110+0000] tag_key ident,systemd.u.SYSLOG_IDENTIFIER,local1.err
[2018-10-16T01:05:14.110+0000] facility local0
[2018-10-16T01:05:14.110+0000] severity info
[2018-10-16T01:05:14.110+0000] use_record true
[2018-10-16T01:05:14.110+0000]
[2018-10-16T01:05:14.401+0000] ping ip-172-18-7-24.ec2.internal from logging-mux-7-px48g
[2018-10-16T01:05:14.404+0000] PING ip-172-18-7-24.ec2.internal (172.18.7.24) 56(84) bytes of data.
[2018-10-16T01:05:14.404+0000] 64 bytes from ip-172-18-7-24.ec2.internal (172.18.7.24): icmp_seq=1 ttl=64 time=0.129 ms
[2018-10-16T01:05:14.404+0000] 64 bytes from ip-172-18-7-24.ec2.internal (172.18.7.24): icmp_seq=2 ttl=64 time=0.047 ms
[2018-10-16T01:05:14.404+0000] 64 bytes from ip-172-18-7-24.ec2.internal (172.18.7.24): icmp_seq=3 ttl=64 time=0.076 ms
[2018-10-16T01:05:14.404+0000]
[2018-10-16T01:05:14.404+0000] --- ip-172-18-7-24.ec2.internal ping statistics ---
[2018-10-16T01:05:14.404+0000] 3 packets transmitted, 3 received, 0% packet loss, time 2001ms
[2018-10-16T01:05:14.404+0000] rtt min/avg/max/mdev = 0.047/0.084/0.129/0.033 ms
[2018-10-16T01:05:16.672+0000] docker info
[2018-10-16T01:05:16.675+0000] Containers: 58
[2018-10-16T01:05:16.675+0000] Running: 46
[2018-10-16T01:05:16.675+0000] Paused: 0
[2018-10-16T01:05:16.675+0000] Stopped: 12
[2018-10-16T01:05:16.675+0000] Images: 42
[2018-10-16T01:05:16.675+0000] Server Version: 1.13.1
[2018-10-16T01:05:16.675+0000] Storage Driver: overlay2
[2018-10-16T01:05:16.675+0000] Backing Filesystem: xfs
[2018-10-16T01:05:16.675+0000] Supports d_type: true
[2018-10-16T01:05:16.675+0000] Native Overlay Diff: true
[2018-10-16T01:05:16.675+0000] Logging Driver: json-file
[2018-10-16T01:05:16.675+0000] Cgroup Driver: systemd
[2018-10-16T01:05:16.675+0000] Plugins:
[2018-10-16T01:05:16.675+0000] Volume: local
[2018-10-16T01:05:16.675+0000] Network: bridge host macvlan null overlay
[2018-10-16T01:05:16.675+0000] Authorization: rhel-push-plugin
[2018-10-16T01:05:16.675+0000] Swarm: inactive
[2018-10-16T01:05:16.675+0000] Runtimes: docker-runc runc
[2018-10-16T01:05:16.675+0000] Default Runtime: docker-runc
[2018-10-16T01:05:16.675+0000] Init Binary: /usr/libexec/docker/docker-init-current
[2018-10-16T01:05:16.675+0000] containerd version: (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
[2018-10-16T01:05:16.675+0000] runc version: e9c345b3f906d5dc5e8100b05ce37073a811c74a (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
[2018-10-16T01:05:16.675+0000] init version: 5b117de7f824f3d3825737cf09581645abbe35d4 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
[2018-10-16T01:05:16.675+0000] Security Options:
[2018-10-16T01:05:16.675+0000] seccomp
[2018-10-16T01:05:16.675+0000] Profile: /etc/docker/seccomp.json
[2018-10-16T01:05:16.675+0000] selinux
[2018-10-16T01:05:16.675+0000] Kernel Version: 3.10.0-862.3.3.el7.x86_64
[2018-10-16T01:05:16.675+0000] Operating System: Red Hat Enterprise Linux Server 7.5 (Maipo)
[2018-10-16T01:05:16.675+0000] OSType: linux
[2018-10-16T01:05:16.675+0000] Architecture: x86_64
[2018-10-16T01:05:16.675+0000] Number of Docker Hooks: 3
[2018-10-16T01:05:16.675+0000] CPUs: 4
[2018-10-16T01:05:16.675+0000] Total Memory: 15.51 GiB
[2018-10-16T01:05:16.675+0000] Name: ip-172-18-7-24.ec2.internal
[2018-10-16T01:05:16.675+0000] ID: 6WRR:FJU2:46WT:RR52:7YZX:NEJ2:L22M:D5KY:3JL4:DPKI:QS4L:JCJ6
[2018-10-16T01:05:16.675+0000] Docker Root Dir: /var/lib/docker
[2018-10-16T01:05:16.675+0000] Debug Mode (client): false
[2018-10-16T01:05:16.675+0000] Debug Mode (server): false
[2018-10-16T01:05:16.675+0000] Registry: https://registry.access.redhat.com/v1/
[2018-10-16T01:05:16.675+0000] Experimental: false
[2018-10-16T01:05:16.675+0000] Insecure Registries:
[2018-10-16T01:05:16.675+0000] ci.dev.openshift.redhat.com:5000
[2018-10-16T01:05:16.675+0000] 172.30.0.0/16
[2018-10-16T01:05:16.675+0000] 127.0.0.0/8
[2018-10-16T01:05:16.675+0000] Live Restore Enabled: false
[2018-10-16T01:05:16.675+0000] Registries: registry.access.redhat.com (secure), docker.io (secure)
[2018-10-16T01:05:18.152+0000] added es message 93c05be7ff3942b5a48e3e56180a3f92
[2018-10-16T01:05:18.155+0000] added es-ops message 1b0beed244a54e1a905ed4c01e16400f
[2018-10-16T01:05:29.164+0000] good - wait_for_fluentd_to_catch_up: found 1 record .operations.* for 'GET /93c05be7ff3942b5a48e3e56180a3f92 404 '
[2018-10-16T01:05:29.370+0000] 1
[2018-10-16T01:06:02.636+0000] 6,22,Oct 16 01:06:02,ip-172-18-7-24.ec2.internal,1b0beed244a54e1a905ed4c01e16400f:, 1b0beed244a54e1a905ed4c01e16400f
[2018-10-16T01:06:02.723+0000] 5,10,Oct 16 01:06:02,ip-172-18-7-24.ec2.internal,sudo:, origin : TTY=unknown ; PWD=/data/src/github.com/openshift/origin-aggregated-logging ; USER=root ; COMMAND=/bin/find /var/log/containers -name *.log -exec grep -b -n GET /93c05be7ff3942b5a48e3e56180a3f92 404 {} /dev/null ;
[2018-10-16T01:06:02.723+0000] 6,16,Oct 16 01:06:02,ip-172-18-7-24.ec2.internal,output_ops_tag:, namespace_name=openshift-logging, container_name=kibana, pod_name=logging-kibana-1-k9lvd, message=GET /93c05be7ff3942b5a48e3e56180a3f92 404 2ms - 9.0B
[2018-10-16T01:06:03.024+0000] node/ip-172-18-7-24.ec2.internal labeled
[2018-10-16T01:06:03.521+0000] daemonset.extensions/logging-fluentd updated
[2018-10-16T01:06:03.748+0000] node/ip-172-18-7-24.ec2.internal labeled
[2018-10-16T01:06:13.895+0000] deploymentconfig.apps.openshift.io/logging-mux scaled
[2018-10-16T01:06:14.391+0000] deploymentconfig.apps.openshift.io/logging-mux updated
[2018-10-16T01:06:14.606+0000] deploymentconfig.apps.openshift.io/logging-mux scaled
[2018-10-16T01:06:23.606+0000] Test 7, no tag_key logging-mux-8-mtmq7
[2018-10-16T01:06:24.708+0000] added es message febaffbe1b38455fae98516255c79fb9
[2018-10-16T01:06:24.711+0000] added es-ops message d4cf88814018466992128d553d0492d5
[2018-10-16T01:07:14.732+0000] good - wait_for_fluentd_to_catch_up: found 1 record .operations.* for 'GET /febaffbe1b38455fae98516255c79fb9 404 '
[2018-10-16T01:07:14.884+0000] 1
[2018-10-16T01:07:24.282+0000] 6,22,Oct 16 01:07:24,ip-172-18-7-24.ec2.internal,output_ops_tag:, d4cf88814018466992128d553d0492d5
[2018-10-16T01:07:24.358+0000] 5,10,Oct 16 01:07:24,ip-172-18-7-24.ec2.internal,output_ops_tag:, origin : TTY=unknown ; PWD=/data/src/github.com/openshift/origin-aggregated-logging ; USER=root ; COMMAND=/bin/find /var/log/containers -name *.log -exec grep -b -n GET /febaffbe1b38455fae98516255c79fb9 404 {} /dev/null ;
[2018-10-16T01:07:24.358+0000] 6,16,Oct 16 01:07:24,ip-172-18-7-24.ec2.internal,output_ops_tag:, namespace_name=openshift-logging, container_name=kibana, pod_name=logging-kibana-1-k9lvd, message=GET /febaffbe1b38455fae98516255c79fb9 404 1ms - 9.0B
[2018-10-16T01:07:24.806+0000] good - no NoMethodError in the no tag_key case
[2018-10-16T01:07:24.812+0000] node/ip-172-18-7-24.ec2.internal labeled
[2018-10-16T01:07:25.345+0000] daemonset.extensions "logging-fluentd" deleted
[2018-10-16T01:07:25.345+0000] daemonset.extensions/logging-fluentd replaced
[2018-10-16T01:07:26.684+0000] node/ip-172-18-7-24.ec2.internal labeled
[2018-10-16T01:07:30.468+0000] deploymentconfig.apps.openshift.io/logging-mux scaled
[2018-10-16T01:07:31.136+0000] Warning: oc apply should be used on resource created by either oc create --save-config or oc apply
[2018-10-16T01:07:31.136+0000] deploymentconfig.apps.openshift.io/logging-mux configured
[2018-10-16T01:07:31.872+0000] deploymentconfig.apps.openshift.io/logging-mux scaled