All Questions
236 questions
0
votes
2
answers
169
views
Can I setup multiple fluentd configurations in one fluentd installation?
I want to create multiple pipelines for datastreams that use fluentd. For example, if I have a data stream that comes from kafka, I want to use some filters and then finally send the data to elastic ...
- 19
0
votes
1
answer
166
views
failed to write data into buffer by buffer overflow action=:throw_exception
I have deployed fluentd as kubernetes pod to upload the kubernetes log to AWS OpenSearch.
Initially for first few days it was working properly(sending log to AWS OpenSearch), after few days the latest ...
- 9
0
votes
2
answers
453
views
Docker swarm and fluentd logging driver
I tried to record some logs for front-end nginx-based containers using fluentd docker logging driver, but failed.
I ended up with following configuration for fluentd (located in /tmp/fluentd/fluent....
- 143
1
vote
0
answers
33
views
Add specific log in Docker Container
I have a Java Application (Jboss server) and in the currently VM architecture I have a audit.log configured with log4j
<appender name="AUDIT" class="org.jboss.logging.appender....
- 149
0
votes
2
answers
63
views
Troubleshooting Fluentd as a Log Aggregator: Connectivity Issues
I am having trouble sending API logs from one Fluentd server to another, and then to Elasticsearch. When ! tried sending logs directly from one Fluentd server to Elasticsearch, it worked fine. Now, I ...
- 33
1
vote
0
answers
355
views
Parsing structured log from a container by google cloud ops agent
I have an application running inside a container inside a GCP vm, outputting log that looks like this: {"severity":"INFO","message":"Http request served","...
- 521
0
votes
0
answers
105
views
unable to serialize JSON type logs In fluentd(logging-operator)
this is my really log
{
"level": "info",
"time": "2024-03-28T10:34:44.345Z",
"req": {
"id": 6,
"method&...
- 1
1
vote
0
answers
49
views
Check with log strout received from fluentd forwarder using fluentd aggregator
In the k8s environment, fluentd is running in daemonset format, so the pod mounted the fluent.conf file as configmap. Below is the contents of the conf file. Collect logs starting with calico-node and ...
- 11
1
vote
0
answers
139
views
Filtering Logs from Fluentd Docker Container
I am currently trying to filter the output of logs for my docker containers when I run (logs -f). I need to filter the logs since the fluentd container is especially noisy with many errors like:
...
- 21
1
vote
1
answer
1k
views
fluentd vs prometheus vs elasticsearch, when to use what?
When to use fluentd vs prometheus vs elasticsearch?
There seems to be overlapping between these tools interms of the feature set they provide but I have seen projects using all these 3 tools in unison....
- 147
2
votes
1
answer
202
views
Parse string of key=values with escaped characters
Loki outputs the following log in a key-value format with the structure key1=value1 key2=value2:
level=info ts=2023-10-20T14:30:48.716410806Z caller=metrics.go:159 component=frontend org_id=fake ...
- 111
0
votes
0
answers
481
views
Fluent-bit - Splitting json
I'm sending logs to ES with fluentd. App logs are in JSON format. Here is one of them.
I am using ECS and Grafana.
{
"@timestamp": "2020-12-24T12:16:43.2784Z",
"level"...
- 355
1
vote
0
answers
83
views
Fluentbit Tag Modifications
I am using fluent-bit to send all of my K8S logs to Elasticsearch. Currently, I want to ship all of the logs for which I have included the string syslog. Then, if a log includes the following string: &...
- 31
1
vote
0
answers
1k
views
How to filter out fluentd logs to just logs and remove all metadata
I am trying to setup fluentd into my kubernetes cluster and I am able to push the logs. Only issue is it is pushing in json format with a lot of extra junk which I don't need.
Here is how the logs are ...
- 242
0
votes
1
answer
172
views
Log parsing in fluentd
I deploy my Kubernetese projects, I do log management with EFK stack.
This is current log displayed in Kibana. Now I want this log string to be 'broken' into new tags. In this case:
fluentd version: ...
- 733
0
votes
3
answers
1k
views
fluentbit excluding liveness and readiness
I am sending logs from my K8s cluster to elasticsearch using fluenbit.
When I look at kibana, I see a lot of entries for liveness and readiness from all of the cluster pods that don't apply to me. ...
- 31
0
votes
0
answers
559
views
fluentd How to concatenate long logs (>16K are split)
We occasionally have a log message greater than 16K that when scraped by fluentd and forwarded to Elasticsearch will appear as two separate documents. This often happens with Java exceptions/stack ...
0
votes
0
answers
818
views
Fluentd parsing nginx access log doesn't work
I want to parse nginx access logs and provide the data to prometheus. The problem is, fluentd (or rather td-agent) doesn't want to match my regex. The example is straight from docs, but modified with ...
- 94
2
votes
0
answers
310
views
fluentd S3 use part of log tag in bucket path
Is there a way to use first part of the tag in bucket path while uploading to aws s3 using fluentd?
I can use the entire tag like this:
<store>
@type s3
s3_bucket "#{ENV['S3_BUCKET']}&...
- 21
0
votes
0
answers
624
views
Troubleshooting Mapper Parsing Exception while logging to OpenSearch with Fluentd
While trying to send logs from Fluentd to OpenSearch, below errors are being observed. I do not have any explicit mapping types defined on OpenSearch, and it also appears that there isn't any mapping ...
- 11
0
votes
1
answer
1k
views
How do i use my parser as a multiline parser in fluentbit?
I am trying to parse the logs i get from my spring-boot application with fluentbit in a specific way.
parsers.conf
[PARSER]
Name springboot
Format regex
regex ^(?<time>[^ ]+)( )+(...
- 1
2
votes
0
answers
251
views
Fluentd buffer in multi-tenant scenarios
What are the best-practices when it comes to setting up the fluentd buffer for a multi-tenant-scenario?
I have used the fluent-operator to setup a multi-tenant fluentbit and fluentd logging solution, ...
- 221
1
vote
1
answer
1k
views
Can fluentbit listen for syslogs on multiple ports, transform logs depending on the port, and send to different destinations?
I'm new to fluentbit and my org is restructuring our log ingestion nodes. The ingestion nodes take in syslogs from multiple platforms. Each platform has 100s of individual devices and this continues ...
- 95
0
votes
1
answer
54
views
View contextual log option is not being show for event logs sent by fluentd
I am sending the application logs from fluentd to elasticsearch, I am not able to see the view contextual logs in elasticsearch UI and I have added the data format which is sent to elasticsearch from ...
- 41
1
vote
0
answers
121
views
How to convert JSON log to non JSON flat log using fluent bit?
I have an application log file, where each line is a JSON. A line looks like below.
{"timestamp":"2023-04-25T03:40:03.180Z","level":"DEBUG","message":&...
- 24k
0
votes
0
answers
71
views
EFK ibtegration with springboot dependency vulnerability issue
I'm trying to integrate my spring-boot application with the EFK Docker instance.
So when I was searching for solutions or guidance for that, I found that I had to use
Fluent Logger for Java dependency....
- 1,425
1
vote
0
answers
1k
views
Fluentd to OpenSearch: How to create dynamic indexes with K8s metadata?
I'm using the fluent-operator to deploy fluentbit and fluentd.
Fluentbit collects and enriches the logs with Kubernetes metadata, then forwards to Fluentd.
Fluentd ships the logs to AWS OpenSearch.
I ...
- 221
0
votes
0
answers
131
views
fluent-bit to GCP through VPN tunnel
I have an on-prem 4 nodes k8s cluster. There is VPN tunnel between mikrotik router and GCP subnet so GCP VM's and 4 on-prem nodes can communicate. I want to send logs from k8s cluster to one ...
- 63
3
votes
1
answer
1k
views
Flatten nested JSON using fluentd
I have a program that writes structured logs, and the following example applies:
{
"time": "time_val",
"log": "{
\"field1\": \"value1\&...
- 343
1
vote
0
answers
148
views
Is there a way for fluentd to read config from Azure Vault?
I have fluentd instance that I'm using to collect application logs and store them into mongodb using mongo-plugin .
It works well but the problem is that I have to put mongo credentials inside fluentd ...
- 742
0
votes
1
answer
726
views
Fluentd config to collect logs for each namespace separately
What would be the Fluentd configuration to collect logs and create a separate log file / folder path of each namespace separately ?
I want to use a Fluentd instance and have a configuration that would ...
- 997
2
votes
1
answer
6k
views
fluentbit connection to fluentd refused
The issue
I have been trying to use the fluent-operator to deploy fluentbit and fluentd in a multi-tenant scenario in EKS cluster.
The goal is to collect logs with fluentbit and then forward to ...
- 221
0
votes
1
answer
1k
views
td-agent.log doesn't have error logs but the logs are not still appearing in http endpoint
I am sending logs using fluentd to my coralogix account.
I configured everything and made my td-agent.service running properly and without error as shown in the td-agent.log. However, I can't still ...
0
votes
1
answer
402
views
Is there any way to implement zstd compression in fluentd s3 plugin?
We have the requirement to compress and send files to S3 with zstd compression. I am trying to implement zstandard compression plugin with fluentd s3 plugin so that the data that is sent to s3 using ...
1
vote
0
answers
214
views
Fluentd JSON string parsing with multiple data type in array
I am trying to set up a logging pipeline with Fluentd and elastic search. One of my log patterns looks like the following:
{
"key": "value",
"inputs": [
[
&...
- 13
1
vote
0
answers
630
views
Send Cloudwatch logs to Loki
I have a k8s cluster where I installed Loki to keep track of all the Pods' logs. I also have an AWS API gateway that sends API requests to a service in my cluster.
I also enable Cloudwatch logs for ...
- 59
0
votes
1
answer
1k
views
Fluentd doesn't parse json log from docker
I have some problems parsing json logs that were received from docker container. I know this question is probably a duplicate but none of the solutions found, including the documentation (https://docs....
- 51
0
votes
1
answer
183
views
How to use fluentd to forward sysmon logs?
Alright so I am confuse,I can forward application,security and system logs. But how do I forward sysmon logs located at Application and Services/Microsoft/Windows/Sysmon
I can't find any documentation....
- 13
0
votes
0
answers
154
views
Kubernetes Application Level Logging
What are the best available options for application-level logging from Kubernetes? I want to write logs from multiple instances of python applications to a single file in a shared location.
0
votes
1
answer
359
views
Fluentd: Change host field (Splunk HEC output)
I deployed the BanzaiCloud Logging Operator to send logs to Splunk using fluentd. Splunk interprets the source (host) as fluentd-0 but I want to have the $namespace.$pod as the host field. I tried to ...
- 109
1
vote
1
answer
836
views
How to let fluentd to collect logs from a container outside of k8s cluster?
I have an EFK (ElasticSearch, Fluentd, Kibana) being deployed in a Kubernetes cluster. I can get the logs from pods in the cluster.
However, I have a container which is outside of the cluster (at ...
- 153
0
votes
1
answer
1k
views
Fluentd | how to drop logs of specific container [duplicate]
we have Fluentd running on our multiple K8s clusters. and with Fluentd we are using Elasticsearch to store our logs from all remote K8s clusters.
there are a few applications for which we do not want ...
- 454
1
vote
0
answers
592
views
Fluentd incorrectly routing logs to its own STDOUT
I have a GKE cluster, in which I'm using Fluentd Kubernetes Daemonset v1.14.3 (docker image: fluent/fluentd-kubernetes-daemonset:v1.14.3-debian-gcs-1.1) to collect the logs from certain containers and ...
- 5,225
0
votes
0
answers
79
views
Push Docker container logs compute engine to GKE
Is there a way to send docker container logs running in GCE to a GKE service using efk stack?
- 1
0
votes
1
answer
1k
views
AWS Lambda - Fluentd instead of CloudWatch possible?
I want to move my logging for AWS Lambda from CloudWatch to Elasticsearch and Kibana, using Fluentd. I understand that I need to use Lambda outputs as an input to Fluentd, which I already have set up. ...
- 1,503
0
votes
1
answer
382
views
How to tell fluentd to filter logs based on "logger_name" field?
My code issues structured logs, which contain a logger_name field. These logs are later sent to some remote logging storage via fluentd. I'd like fluentd to filter logs coming from Kafka streams ...
- 1,412
0
votes
0
answers
164
views
Fluentd not forwarding logs to elastic search
I have deployed fluentd and elastic search in k8s. If I check the log of the fluentd pod it logs: The client is unable to verify that the server is Elasticsearch. Some functionality may not be ...
- 501
0
votes
0
answers
150
views
fluentd input everything that syslog has, before syslog starts
I have rsyslog on ubuntu configured to send everything to fluentd, and it works fine, but there are log messages in /var/log/syslog from before syslog.service starts and these log messages are not ...
- 6,011
0
votes
1
answer
1k
views
Fluentd on Kubernetes - Parse Nginx Access Log in Json
I currently have this nginx log output.
log_format json_logs escape=json
'{'
'"time_local":"$time_local",'
...
- 2,731
3
votes
1
answer
4k
views
How to get specific namespace for log fluentd
So I wanted to fetch logs from a specific namespace and send them to opensearch, so I didn't get a clue so I decided to try to make it like this:
apiVersion: v1
kind: ConfigMap
metadata:
labels:
...
- 33