1,322 questions
0
votes
1
answer
20
views
How send only records that meet some conditions in FluentD
What is a problem?
I want send to ElasticSearch just some record based in conditions.
This is my main conf.
<source>
bind 0.0.0.0
<parse>
expression /<SOME CUSTOM REGEXP>/
...
- 1,127
0
votes
0
answers
26
views
How to write multiline if-else statements inside <record> in Fluentd record_transformer?
I am trying to use the record_transformer filter plugin in Fluentd to modify log records. Specifically, I want to include a field where the value is determined by a multiline Ruby if-else statement. ...
0
votes
2
answers
47
views
Can I have single AWS Firelens sidecar container to fetch logs from multiple containers running in cluster?
I have ECS cluster configured on EC2 instances which has a set of services running in it. I've configured AWS Firelens sidecar container to route the ECS container logs to AWS Data firehose.
As per my ...
- 11
-1
votes
2
answers
54
views
mysql general log generates a lot of data so that it burdens the server [closed]
My IT team recently activated the MySQL general log to meet regulatory requirements. However, we’ve encountered significant performance issues:
After 20+ hours of activation, the general log ...
0
votes
1
answer
61
views
Apply fluentd configuration depending on environment variable
Given the following example configuration for fluentd, how could I enable or disable the configuration for Elasticsearch based on an environment variable ENABLE_ELASTICSEARCH?
fluentd will be started ...
- 12.8k
0
votes
0
answers
22
views
FluentD file type buffer path
I have my old fluentd config in which I have a <match> type copy then having two <scope> one is a type forward to a server with a <buffer> and the second is prometheus type for ...
- 41
0
votes
0
answers
19
views
Fluentd throwing error Net::ReadTimout when flushing buffer
we are setting up a fluentd aggregator on a VM to aggregate logs from OpenShift pods and sending the message to Splunk. We have the plugins installed.
However upon running, we encounter the following ...
- 2,009
0
votes
0
answers
27
views
Fluentd not inserting logs into MongoDB (using fluent-plugin-mongo)
I'm using Fluentd with the fluent-plugin-mongo plugin to read log files and insert them into MongoDB. The logs are being read and output to stdout as expected, but they are not being inserted into ...
- 751
0
votes
0
answers
31
views
opentelemetry collector, route events to different exporters
We use fluentd-hec to send openshift container logs to Splunk Enterprise and this works nicely. However fluentd-hec is not supported anymore and Splunk offers an alternative, their own distribution of ...
- 123
1
vote
1
answer
39
views
In Kibana dashboard, how do you transform a Field value that contains JSON (in fluentd Data View)?
Versions:
Elasticsearch / Kibana: 8.6.0
AWS EKS k8s: 1.24
k8s logging operator: ghcr.io/banzaicloud/logging-operator:3.17.10
Our logging stack is Elasticsearch => fluentd => Kibana.
We use a ...
- 1,467
1
vote
1
answer
21
views
Fluentd 1.16.3 on windows: how to enrich certain log entries without filtering out the rest?
I need to add a custom field to all log entries that contain string "myapp.database - Statement returned". All other logs that do not contain this string, should not be discarded/filtered ...
- 577
2
votes
1
answer
28
views
Fluentd Not able to format json to customized output
Below is the json output which we recieve in Fluentd from OTEL Collector
"tags": [
{
"key": "otel.library.name",
"vStr": "com....
0
votes
1
answer
152
views
fluentd output to s3 failed
I was trying to put logs produced by an application on docker-compose to my s3 bucket using fluentd, but got the following error
fluentd | 2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/...
- 11
0
votes
3
answers
145
views
Fluentd sending wrong time to cloudwatch
In fluentd, a source is syslog and the target is fluent-plugin-cloudwatch-logs plugin. Everything works, only the time is wrong.
<source>
@type syslog
<transport tcp>
</transport&...
- 2,998
0
votes
2
answers
169
views
Can I setup multiple fluentd configurations in one fluentd installation?
I want to create multiple pipelines for datastreams that use fluentd. For example, if I have a data stream that comes from kafka, I want to use some filters and then finally send the data to elastic ...
- 19
0
votes
1
answer
166
views
failed to write data into buffer by buffer overflow action=:throw_exception
I have deployed fluentd as kubernetes pod to upload the kubernetes log to AWS OpenSearch.
Initially for first few days it was working properly(sending log to AWS OpenSearch), after few days the latest ...
- 9
0
votes
2
answers
453
views
Docker swarm and fluentd logging driver
I tried to record some logs for front-end nginx-based containers using fluentd docker logging driver, but failed.
I ended up with following configuration for fluentd (located in /tmp/fluentd/fluent....
- 143
0
votes
0
answers
12
views
How to use the output of a command as input of another command in fluentd?
With fluentd, I want to use each line of the output of jps -lv command (that lists the running JVM on machine), as input argument of jcmd <PID> GC.heap_info (that gives info on heap usage).
I've ...
- 161
1
vote
0
answers
33
views
Fluentd plugin how to parse a value with a dynamic key?
Here is the input
message= [
{
"SNMPv2-MIB::sysUpTime.0"=>"338 days, 20:31:17.37",
"SNMPv2-MIB::snmpTrapOID.0"=>"IF-MIB::linkDown&...
- 11
0
votes
0
answers
204
views
Understanding Fluentd Splunk Forwarding Configs
I have set up Fluentd daemonsets to send logs to Splunk from AWS EKS clusters. Logs are being forwarded to HEC endpoint. I however see multiple events appear in a single output line on Splunk ...
- 183
1
vote
0
answers
186
views
Fluentd send logs to http
I have a fluentd setup that adds all logs into a new json object called log. I would like to post to http match that only the elements within the log object.
How can I do it?
Below my configuration:
&...
- 11
0
votes
0
answers
14
views
Fluentd - AWS managed opensearch service logging issue
I need configure FluentD to send logs from my AWS-EKS nodes to AWS managed OpenSearch dashboard.
FluentD as DaemonSet installed on each of my AWS-EKS node.
What is a problem?
Problem:-
FluentD unable ...
- 9
0
votes
0
answers
456
views
"buffer space has too many data" BufferOverflowError in fluent
This Fluentd service consumes from Kafka and stores data in OpenSearch.
The longest logs are about 32,700 bytes, while typical logs are around 10 to 15 MB.
<buffer>
chunk_limit_size 50m
...
- 1
0
votes
0
answers
90
views
Monitor fluentd.conf with exec plugin
I have fluentd deployed in K8s container. The below is fluentd configuration to monitor the fluentd.conf. If there is any change done to fluentd.conf file the reload.sh script will run which will ...
- 49
1
vote
0
answers
33
views
Add specific log in Docker Container
I have a Java Application (Jboss server) and in the currently VM architecture I have a audit.log configured with log4j
<appender name="AUDIT" class="org.jboss.logging.appender....
- 149
0
votes
2
answers
63
views
Troubleshooting Fluentd as a Log Aggregator: Connectivity Issues
I am having trouble sending API logs from one Fluentd server to another, and then to Elasticsearch. When ! tried sending logs directly from one Fluentd server to Elasticsearch, it worked fine. Now, I ...
- 33
0
votes
0
answers
120
views
Fluentd Geoip Plugin is not working in Amazon Linux 2023
While installing 'fluent-gem install fluent-plugin-geo' in Amazon Linux 2023, we get below errors. Is this a known issue already?
Note: Fluentd is installed using Repo (Script on website), and all ...
1
vote
0
answers
161
views
How to Enable Mutual TLS (mTLS) for Elasticsearch and Fluent Bit?
I'm currently working on securing communication between Elasticsearch and Fluent Bit in my Kubernetes environment. I want to implement Mutual TLS (mTLS) to enhance the security of the communication ...
0
votes
2
answers
63
views
How to match optional group in regular expression
I want a regular expression for using in fluentd for parsing nginx error logs.
The sample row is:
2024/04/15 09:06:29 [error] 3443790#3443790: *176070165 limiting requests, excess: 2.957 by zone "...
- 166
1
vote
0
answers
355
views
Parsing structured log from a container by google cloud ops agent
I have an application running inside a container inside a GCP vm, outputting log that looks like this: {"severity":"INFO","message":"Http request served","...
- 521
0
votes
1
answer
26
views
LDAP - 389ds - FluentD - Filter plugin - type record_transformer - can it have more than one <record> stanza entries?
Can FluentD 1.16.3/+ have more than one record entries inside filter plugin for type record_transformer?
I have the following configuration, but it seems like it's keeping ONLY the last record entry.
...
- 17.2k
0
votes
0
answers
257
views
FluentD / Fluent-Bit: Concatenate multiple lines of log files and generate one JSON record for all key-value from each line
FluentD 1.16.3 and Fluent-bit 1.8.11
I have the following lines in my container log file /var/log/containers/.log*
When a ldap user credentials fail (bad user/password)
024-03-28T16:09:23.048182266Z [...
- 17.2k
0
votes
0
answers
105
views
unable to serialize JSON type logs In fluentd(logging-operator)
this is my really log
{
"level": "info",
"time": "2024-03-28T10:34:44.345Z",
"req": {
"id": 6,
"method&...
- 1
0
votes
1
answer
375
views
Filter logs fluent-bit on regex parsing
I have a fluentbit running that scrapes json logs from a dir:
[PARSER]
Name json
Format json
Time_Key time
# Time_Format %llu
Time_Keep On
[PARSER]
Name ...
- 502
0
votes
0
answers
241
views
Fluentd - Opensearch: Logs not sending to Opensearch
I am new to Fluentd and Opensearch configuration.
As part of requirememt , I have to install Opensearch and Fluentd using helmcharts. Using Output plugin configuraton in fluentd, those logs has to ...
- 21
0
votes
1
answer
31
views
how to use fluentd to collect data from a service
I'm new to fluentd.
I have a Java-DropWizard.io service running in a docker environment.
I can ask this service for a bunch of metrics data in json format by curl http://ip:PORT/admin/metrics.
Is ...
- 3,870
0
votes
1
answer
383
views
Are the fluentbit filter plugin filtering step by step?
Are the fluentbit filters filtering step by step? How it works actually?
When the first filter eliminates some data, will the second filter make another elimination from the first filtered data?
Could ...
- 41
0
votes
0
answers
42
views
Azure App Service - unresponsiveness while connecting to Fluentd
I am in the process of pushing the logs
Application logs to 'Fluentd'
From 'Fluentd' to Dynatrace.
The application is Spring Boot application which is deployed as Azure App Service.
In the ...
- 339
0
votes
0
answers
78
views
Elasticsearch not accepting ILM-Config from Fluentd
I have this Output-Plugin-Config for fluentd:
<match logging.**>
@type elasticsearch_data_stream
host <<fluent-conf-host-elasticsearch>>
port <<fluent-conf-port-...
0
votes
0
answers
130
views
Can't use Fluentd with the "fluent-plugin-sql" plugin
I want to use fluentd to read data from a database in a dockerized environment.
In the container where I have installed fluentd I put this "fluent.conf" file:
<source>
@type sql
...
- 2,997
0
votes
0
answers
60
views
How throttling can be prevented across loki , fluentd and fluent-bit
I currently have a logging solution in Kubernetes as follows.
Fluent-bit deployed as daemon sets that collect logs from the nodes
After collecting logs, fluent-bit forwards data to two destinations
-...
- 11
0
votes
0
answers
167
views
export metrics with fluentd
I'm new to Fluentd and recently I've been writing a simple configuration for Fluentd. I send access_log from Nginx to Fluentd with a particular format. Then I parse it in Fluentd and expose metrics to ...
- 437
0
votes
1
answer
103
views
Fluentd installation failed with message in Windows
I'm doing testing Fluentd for collecting log files from Apache Tomcat in Windows OS.
So, I tried install with [Fluentd-Packages v5.x] and [Calyptia-Fluent v1.3.x] and failed with Windows Message that &...
- 19
0
votes
0
answers
324
views
Remove excess line breaks from s3 log files (fluent-bit s3 output plugin)
I am using fluent-bit s3 output plugin to upload Kubernetes pod logs to s3. I see excessive line breaks in s3 log files as below:
2024-01-24 10:03:34.510 [65b0e07526a14752251fdf7a2e309f58] INFO [Log] ...
- 162
0
votes
1
answer
401
views
illegal_argument_exception: index.lifecycle.rollover_alias [alias-efk-stack] does not point to index [efk-stack-pod-name]
Need help for setting up ilm policy for existing indices. I have created first the index Lifecycle Policy which will delete data older than 10 days with name "10-days-archival-policy". Then ...
1
vote
1
answer
89
views
Transform custom Docker logs into fluentd into elastic search
I will publish some Docker containers incorporating a common logging framework (written in golang). The logging format is a JSON format.
There is distinct data in this custom json logging format that ...
- 785
1
vote
1
answer
364
views
Use <source> inside <label @FLUENT_LOG> section in fluentd configuration
I want to change the format of fluentd own logs before sending on stdout. To achieve this, I have captured fluentd logs using label @FLUENT_LOG and then configured a filter to format the logs and then ...
- 49
1
vote
0
answers
49
views
Check with log strout received from fluentd forwarder using fluentd aggregator
In the k8s environment, fluentd is running in daemonset format, so the pod mounted the fluent.conf file as configmap. Below is the contents of the conf file. Collect logs starting with calico-node and ...
- 11
1
vote
1
answer
60
views
How could I get fluentd to retain a value from one event then add it to all the following events?
Is there a way to have fluentd retain a value from one event, then append that value to all events from another soruce, until it recieves another similiar event with a new value for that field?
For ...
- 11
0
votes
0
answers
69
views
How to get metrics for number logs processed in FluentD
Currently, we employ the following approach for sending logs to OpenSearch: receiving application logs in Gzip format stored in S3. We then input this S3 Gzip file(collection of logs) into FluentD, ...
- 17