Fri, Mar 25 2022 3:08:42 pm | [2022/03/25 07:08:42] [debug] [outputes.0] HTTP Status=200 URI=/_bulk [2022/03/24 04:19:50] [debug] [out coro] cb_destroy coro_id=3 [ warn] [engine] failed to flush chunk #3220 - Github Describe the bug I have a pretty basic setup where I'm trying to write to a Cassandra backend and Loki just isn't creating any chunks. [2022/03/24 04:20:04] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [task] created task=0x7ff2f1839b20 id=6 OK Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=104048905 watch_fd=14 hi @yangtian9999 . This is the total record count of all unique chunks sent by this output. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:22 pm | [2022/03/25 07:08:22] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 has been assigned (recycled) Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [input chunk] update output instances with new chunk size diff=1083 TLS error: unexpected EOF Issue #6165 fluent/fluent-bit Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY [2022/03/22 03:57:48] [ warn] [engine] failed to flush chunk '1-1647920205.172447077.flb', retry in 912 seconds: task_id=12, input=tail.0 > output=es.0 (out_id=0) edit the value.yaml, change to es ip, 10.3.4.84 is the es ip address. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"N-Mmun8BI6SaBP9luqVq","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [2022/03/24 04:21:20] [debug] [input:tail:tail.0] inode=35326801 with offset=0 appended as /var/log/containers/hello-world-89knq_argo_main-f011b1f724e7c495af7d5b545d658efd4bff6ae88489a16581f492d744142807.log Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [input chunk] update output instances with new chunk size diff=650 Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [upstream] KA connection #35 to 10.3.4.84:9200 has been assigned (recycled) [2022/03/22 03:57:47] [ warn] [engine] failed to flush chunk '1-1647920812.174022994.flb', retry in 746 seconds: task_id=619, input=tail.0 > output=es.0 (out_id=0) Fluentd will wait to flush the buffered chunks for delayed events. But now loki still can't start and is writing a lot of following errors about structure. Fri, Mar 25 2022 3:08:44 pm | [2022/03/25 07:08:44] [debug] [outputes.0] task_id=17 assigned to thread #0 Fri, Mar 25 2022 3:08:31 pm | [2022/03/25 07:08:31] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=104048905 events: IN_ATTRIB Here's my config; schema_config: configs: - from: 2020-05-15 store: cassandra object_store: cassandra . Please refer this: [2022/03/24 04:19:38] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [retry] re-using retry for task_id=0 attempts=2 Fri, Mar 25 2022 3:08:31 pm | [2022/03/25 07:08:31] [ warn] [engine] failed to flush chunk '1-1648192110.850147571.flb', retry in 9 seconds: task_id=9, input=tail.0 > output=es.0 (out_id=0) [2022/03/24 04:19:50] [ warn] [engine] failed to flush chunk '1-1648095560.297175793.flb', retry in 9 seconds: task_id=2, input=tail.0 > output=es.0 (out_id=0) "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"1OMmun8BI6SaBP9liJWQ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [2022/03/24 04:20:00] [debug] [out coro] cb_destroy coro_id=4 ): k3s 1.19.8, use docker-ce backend, 20.10.12. How can I debug why Fluentd is not sending data to Elasticsearch? Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [input chunk] update output instances with new chunk size diff=862 Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [input chunk] update output instances with new chunk size diff=681 Fri, Mar 25 2022 3:08:23 pm | [2022/03/25 07:08:23] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [input chunk] update output instances with new chunk size diff=650 Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [out coro] cb_destroy coro_id=13 The script itself works great, however this container configuration didn't work for me, the problem for me was that I have loki's psp enabled (enforcing non-root execution) which cauesed crond to fail (it must elevate to run the job) I ended up using a simple while loop that wraps the script execution with a 1m sleep and that did the trick. fluentbit fails to communicate with fluentd - Stack Overflow Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. . Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [input chunk] update output instances with new chunk size diff=1085 Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [input chunk] update output instances with new chunk size diff=1182 [2022/03/24 04:19:38] [debug] [retry] re-using retry for task_id=1 attempts=2 Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [outputes.0] task_id=9 assigned to thread #1 Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [outputes.0] HTTP Status=200 URI=/_bulk "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"HeMmun8BI6SaBP9lh4vZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. [2022/03/24 04:20:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/local-path-provisioner-7ff9579c6-mcwsb_kube-system_local-path-provisioner-47a630b5c79ea227664d87ae336d6a7b80fdce7028230c6031175099461cd221.log, inode 444123 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. [2022/03/24 04:19:50] [debug] [retry] re-using retry for task_id=2 attempts=3 Fri, Mar 25 2022 3:08:47 pm | [2022/03/25 07:08:47] [ warn] [engine] failed to flush chunk '1-1648192099.641327100.flb', retry in 60 seconds: task_id=2, input=tail.0 > output=es.0 (out_id=0) To Reproduce. [2022/03/24 04:19:34] [debug] [http_client] not using http_proxy for header Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 has been assigned (recycled) Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [input chunk] update output instances with new chunk size diff=695 [2022/03/24 04:21:20] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/hello-world-89knq_argo_wait-a7f77229883282b7aebce253b8c371dd28e0606575ded307669b43b272d9a2f4.log, inode 1772851 Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [out coro] cb_destroy coro_id=10 Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY fluent-bit when trying to use OUTPUT gelf getting connection timeout "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"YeMnun8BI6SaBP9lLtm1","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:33 pm | [2022/03/25 07:08:33] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY When there are lots of messages in the request / chunk and the rejected message is at the end of the list then you never see the cause in the fluent-bit logs. Kubernetes? "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"2uMmun8BI6SaBP9liJWQ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamicall [2022/03/22 03:57:49] [ warn] [engine] failed to flush chunk '1-1647920934.181870214.flb', retry in 786 seconds: task_id=739, input=tail.0 > output=es.0 (out_id=0), use helm to install helm-charts-fluent-bit-0.19.19.