All Questions
63 questions
0
votes
1
answer
166
views
failed to write data into buffer by buffer overflow action=:throw_exception
I have deployed fluentd as kubernetes pod to upload the kubernetes log to AWS OpenSearch.
Initially for first few days it was working properly(sending log to AWS OpenSearch), after few days the latest ...
- 9
0
votes
2
answers
63
views
Troubleshooting Fluentd as a Log Aggregator: Connectivity Issues
I am having trouble sending API logs from one Fluentd server to another, and then to Elasticsearch. When ! tried sending logs directly from one Fluentd server to Elasticsearch, it worked fine. Now, I ...
- 33
0
votes
1
answer
172
views
Log parsing in fluentd
I deploy my Kubernetese projects, I do log management with EFK stack.
This is current log displayed in Kibana. Now I want this log string to be 'broken' into new tags. In this case:
fluentd version: ...
- 733
1
vote
1
answer
2k
views
FluentD elasticsearch Plugin @type elasticsearch_data_stream with dynamic datastreams
update-1:
I have made some progress on this, by defining a concrete data_stream_name in match. The only thing left is to figure out a way to do dynamic data stream. I am updating the code sample below ...
- 313
0
votes
1
answer
323
views
fluentd(td-agent) output file, Time is missing from the filename
I want to get help.
There is a message as below.
The message is being received and saved to a file.
I want to display the time in the file name, but it's failing.
Env
td-agent 4.3.1 fluentd 1.14.6
...
- 55
1
vote
0
answers
303
views
EFK stack - fluentd not sending logs to ElasticSearch
I am working on a EFK stack for log management in microservices.
Problem statement: Unable to send logs to ElasticSearch from Fluentd, though it was was working earlier with the same setup. ...
- 133
1
vote
2
answers
329
views
Nginx cannot connect fluentd in EFK stack
I am setting up a stack with an application consisting of nginx, redis, mysql, myapp. Nginx proxies requests to myapp. I want to send logs from nginx to EFK stack, but an error occurs when starting ...
-1
votes
1
answer
2k
views
Trying to add FluentD to my workflow but it fails to connect
I was looking through github bugs and noticed some similar but different things and I am really really confused as to how to implement FluentD with a Securely enabled Elasticsearch flow.
The Error I ...
- 10.7k
1
vote
0
answers
267
views
fluentd indices not adding to elasticsearch and kibana
I've deployed EFK stack in IBM Kuberentes cloud by following the step by step guide from this article. Every deployment is done successfully, all EFK stack are deployed fine, but I'm unable to find ...
- 101
1
vote
0
answers
1k
views
Fluentd unable to send logs to Elasticsearch
Images being used here are:
fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
docker.elastic.co/elasticsearch/elasticsearch:7.5.1
Fluentd is able to place the logs in file-test.log file ...
- 11
0
votes
1
answer
30
views
How to expand keys in flunetd with logs in sorted way (latest logs should be on top)
I'm using EFK stack.
My flunetd configuration is:
<parse>
@type multi_format
<pattern>
format json
time_key time
time_type string
time_format "%Y-%m-%...
- 1
5
votes
2
answers
19k
views
fluentd elasticsearch plugin - The client is unable to verify that the server is Elasticsearch
I want to send some nginx logs from fluentd to elasticsearch , however, fluentd is unable to start due to following error message:
The client is unable to verify that the server is Elasticsearch. ...
- 1,414
1
vote
1
answer
1k
views
Disable mapping for a specific field using an Index Template Elasticsearch 6.8
I have an EFK pipeline set up. Everyday a new index is created using the logstash-* prefix. Every time a new field is sent by Fluentd, the field is added to the index pattern logstash-*. I'm trying to ...
- 111
0
votes
1
answer
569
views
Build the EFK system used for simulating logging server on Docker
I want to simulate laravel logging to EFK system server
Base on this, I build up two container. One of laravel project's container. The ohter is EFK system container
but EFK's fluentd does not catch ...
- 88
7
votes
3
answers
7k
views
EFK system is build on docker but fluentd can't start up
I want to build the efk logger system by docker compose.
Everything is setup, only fluentd has problem.
fluentd docker container logs
2022-02-15 02:06:11 +0000 [info]: parsing config file is ...
- 88
0
votes
0
answers
3k
views
failed to write data into buffer by buffer overflow action=:block
Can someone help me about this issue
https://github.com/fluent/fluentd/issues/3626
- 91
0
votes
0
answers
760
views
FluentD, grep only specific logs events
I have logs, and I want to grep only the logs and send to Elasticsearch which contains 'error="400 - Rejected by Elasticsearch' and 'failed to parse field' and ignore the others.
log:2022-02-04 ...
- 91
0
votes
1
answer
11k
views
failed to flush the buffer in fluentd looging
I am getting these errors during ES logging using fluentd.
I'm using fluentd logging on k8s for application logging, we are handling 100M (around 400 tps) and getting this issue.
I'm using M6g.2xlarge(...
- 91
0
votes
0
answers
129
views
Fluentd Throwing error and not uploading data to Elastic
unexpected error error_class=Elasticsearch::UnsupportedProductError error="The client noticed that the server is not a supported distribution of Elasticsearch."
5
votes
1
answer
8k
views
Fluent Bit 1.8+ and MULTILINE_PARSER
My goal is to collect logs from Java (Spring Boot) applications running on Bare Kubernetes.
These logs are then translated into ES and visualized in Kibana.
For these purposes I deployed Fleunt Bit 1....
- 349
0
votes
4
answers
2k
views
How to connect fluentd to opendistro for elasticsearch
I have created a docker file :
FROM fluentd:v1.14.0-debian-1.0
USER root
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version&...
1
vote
0
answers
1k
views
How to remove the unwanted characters from fluentd logs
Currently I am sending my Kubernetes logs to cloud watch using Fluentd, but when I check the logs in cloudwatch, the logs are having extra unicode characters. I tried different ways to and regexp to ...
- 11
1
vote
1
answer
690
views
OpenShift Logging: How do I make Kibana group lines of a stack trace into one record?
Context
I have the Red Hat OpenShift Logging Operator installed in a cluster, explained on this page. This is basically an EFK stack (ElasticSearch, Fluentd, Kibana). Normally with Fluentd you can ...
- 142
3
votes
1
answer
1k
views
FluentD forward logs from kafka to another fluentD
I need to send my application logs into a FluentD which is part of an EFK service. so I tried to config another FluentD to do that.
my-fluent.conf:
<source>
@type kafka_group
consumer_group ...
- 1,125
4
votes
1
answer
7k
views
Fluentd - How to parse logs whose messages are JSON formatted parsed AND whose messages are in text; as is without getting lost due to parse error
I have certain log messages from certain services that are in JSON format; and then this fluentd filter is able to parse that properly. However with this; it discards all other logs from other ...
- 6,164
2
votes
1
answer
4k
views
How to parse a fluent-bit json $log.<field> key?
I have the following log to be parsed:
TID: [-1234] [] [2021-05-31 09:53:26,680] - Unique ID: Evento_Teste, Event: {"event":{"metaData":"blue"}}
And below the ...
- 2,857
1
vote
0
answers
1k
views
Fluentd JSON logs truncate/splitting after 16385 characters- How to concate?
I have deployed Bitnami EFK stack on K8s environment:
repository: bitnami/fluentd
tag: 1.12.1-debian-10-r0
Currently, one of the modules/applications inside my namespaces are configured to ...
- 523
0
votes
0
answers
692
views
EFK stack JSON log not being shown
I have deployed an EFK stack in a Kubernetes cluster.
I have configured it in a way where fluentd will fetch Nginx logs as well as PHP logs( both are in JSON format and both are one JSON log per line )...
- 452
2
votes
1
answer
5k
views
error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster
I am getting this error from the fluentd pods and they keep restarting. I am running this on kuberentes v1.17.9-eks-4c6976.
Not sure of what the cause is. Any help would be appreciated.
/usr/local/...
- 1,615
-1
votes
1
answer
516
views
EFK stack wit geoip data
Hello please can someone advise on following issue:
I have AKS kubernetes cluster and I am using EFK stack to get logs. Fluentd is running in pods.
I have nginx application pod for which I parse log ...
- 39
1
vote
1
answer
3k
views
Fluentd - Could not communicate to Elasticsearch, resetting connection and trying again. getaddrinfo: Name or service not known (SocketError)
In an EFK setup, the fluentd suddenly stopped sending to elasticsearch with the following errors in the logs:
2020-09-28 18:48:55 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting ...
- 1,961
0
votes
1
answer
122
views
Host journal logs no present in EFK Kubernetes stack
I'm using kube-fluentd-operator to aggregate logs using fluentd into Elasticsearch and query them in Kibana.
I can see my application (pods) logs inside the cluster.
However I cannot see the journal ...
- 21.9k
0
votes
1
answer
1k
views
kubernetes container_name got null in fluentdconfiguration
I try to get log from my application container and attach fluentd log agent as sidecar container in my project. And I want to get which log is coming from which application in my Kibana dashboard. ...
- 13.7k
1
vote
1
answer
2k
views
How i filter fluentD logs on kubernetes?
My kubernetes have liveness enable, and it log on application, like this:
kubectl logs -n example-namespace example-app node-app
::ffff:127.0.0.1 - - [17/Sep/2020:14:12:19 +0000] "GET /docs HTTP/...
0
votes
1
answer
4k
views
Parse multiline logs as a single event in Fluentd
We have EFK implemented on Openshift Container Platform version 4.3.
Issue:
Multiline logs such as Java Stack trace, SQL queries are not getting parsed as a single event in Fluentd and because of this ...
- 35
0
votes
0
answers
181
views
Fluentd. Why aren't custom logs delivered to Elasticsearch (EFK)?
For testing, I created a file in my home directory:
touch /home/testuser/test.log
I use td-agent to deliver logs to Elasticsearch (EFK).
This is my test configuration in td-agent.conf:
<source>
...
- 2,681
2
votes
0
answers
1k
views
Why Fluentd-elasticsearch cannot send logs to elasticsearch?
I have deployed EFK stack on Kubernetes and I get following error messages and looks like Fluentd cannot communicate with elasticsearch.
[warn]: [elasticsearch] Could not communicate to Elasticsearch, ...
- 523
-1
votes
1
answer
438
views
EKS EFK logging approach
I am trying to decide an approach for logs processing in a EKS cluster. Idea is to use EFK. We thought we can use fluentd to push the logs to elastic search. But most of the blogs uses fluentd to send ...
- 399
1
vote
2
answers
8k
views
Set fluentD elastic-search index dynamically
I'm trying to forward logs to elastic-search and got stuck with setting the index dynamically (by field in the input data).
My input data format is JSON and always have the key "es_idx". I wish to ...
- 429
0
votes
0
answers
120
views
Find the timestamp of last buffer flush in Fluentd
I am new to Fluentd (and the hole EFK stack). I have Fluentd which sends the logs to Elasticsearch (both in Kubernetes) and I am trying to find a way to find the time when the last (successful) flush ...
- 109
0
votes
0
answers
306
views
EFK - Have preconfigured filter by container that will appear in Kibana
I've got the EFK stack installed on kubernetes following this addon: https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch
What I want to achieve is having all ...
0
votes
0
answers
166
views
Need to restrict user in Kibana to access specific application logs in EFK
I would like to implement elastic search and Kibana instance as shared between multiple users, where multiple users can have their applications(services) and write their logs in the same of different ...
- 367
1
vote
1
answer
107
views
Openshift applications logs forward to a single file
how can I forward my applications logs in Openshift to a single file on my filesystem? It is possible with fluentd?
BR
- 17
2
votes
1
answer
561
views
Logs are not shown in order after shipped to Elasticsearch using Fluentd
We have an application deployed in Kubernetes and all applications are configured to log to stdout.We use fluentd DaemonSet to collect logs from /var/lib/docker/containers/ folders and ship them to an ...
2
votes
1
answer
2k
views
Using fluentd, I want to output only one key data from json data
I want to output the kubernetes log to a file.
but, I could only output it as json data.
I want to output only "message" part to file.
How do I choose "message" to print?
Which filter should I choose?...
- 55
0
votes
0
answers
65
views
Display Version, container name, timestamp in Kibana with EFK
I have a PHP app deployed with Dockerfile, using EFK stack (Elasticsearch, Fluent-d, and Kibana) and I'm using "Logger\FluentLogger"
plugin to send the log to the elasticsearch.
$logger = new ...
- 11
0
votes
1
answer
3k
views
Side Effects When Using fluentd logging driver with Docker
I am setting up an EFK stack which works as expected but came with a few drawbacks that I would like to work around.
If my fluentd instance is not running, other containers using that use the fluetnd ...
- 2,869
0
votes
1
answer
985
views
td-agent is unable to ship logs from file when the file contains a single multiline log
td-agent unable to ship logs from line when log file contains single multiline logs. The logs are not picked up by td-agent until a new line is added
Installed td-agent on a windows machine. ...
- 37
1
vote
2
answers
3k
views
fluentd config to parse auth.log
I'm using the following config to have fluentd read the auth.logs and send it to elastic search but i'm faced with an error saying pattern doesn't match and the logs are not pushed to ES.
I'm using ...
- 1,144
3
votes
1
answer
3k
views
How to install Fluentd plugins on k8s
I have set up EFK on Kubernetes, currently I have access only to logs from logstash but wondering how can I install some plugins for Fluentd in order to get some logs from eg. NGINX which I use as a ...
- 1,473