All Questions
Tagged with fluentd fluent-bit
89 questions
1
vote
0
answers
161
views
How to Enable Mutual TLS (mTLS) for Elasticsearch and Fluent Bit?
I'm currently working on securing communication between Elasticsearch and Fluent Bit in my Kubernetes environment. I want to implement Mutual TLS (mTLS) to enhance the security of the communication ...
0
votes
0
answers
257
views
FluentD / Fluent-Bit: Concatenate multiple lines of log files and generate one JSON record for all key-value from each line
FluentD 1.16.3 and Fluent-bit 1.8.11
I have the following lines in my container log file /var/log/containers/.log*
When a ldap user credentials fail (bad user/password)
024-03-28T16:09:23.048182266Z [...
- 17.2k
0
votes
1
answer
376
views
Filter logs fluent-bit on regex parsing
I have a fluentbit running that scrapes json logs from a dir:
[PARSER]
Name json
Format json
Time_Key time
# Time_Format %llu
Time_Keep On
[PARSER]
Name ...
- 502
0
votes
1
answer
383
views
Are the fluentbit filter plugin filtering step by step?
Are the fluentbit filters filtering step by step? How it works actually?
When the first filter eliminates some data, will the second filter make another elimination from the first filtered data?
Could ...
- 41
0
votes
0
answers
60
views
How throttling can be prevented across loki , fluentd and fluent-bit
I currently have a logging solution in Kubernetes as follows.
Fluent-bit deployed as daemon sets that collect logs from the nodes
After collecting logs, fluent-bit forwards data to two destinations
-...
- 11
0
votes
0
answers
325
views
Remove excess line breaks from s3 log files (fluent-bit s3 output plugin)
I am using fluent-bit s3 output plugin to upload Kubernetes pod logs to s3. I see excessive line breaks in s3 log files as below:
2024-01-24 10:03:34.510 [65b0e07526a14752251fdf7a2e309f58] INFO [Log] ...
- 162
0
votes
0
answers
274
views
Fluentbit unable to export metrics from node_metrics_exporter input to kafka output
Configured fluentbit daemonset in kubernetes with following data in configmap given below
data:
custom_parsers.conf: |
[PARSER]
Name json
Format json
...
- 11
0
votes
0
answers
481
views
Fluent-bit - Splitting json
I'm sending logs to ES with fluentd. App logs are in JSON format. Here is one of them.
I am using ECS and Grafana.
{
"@timestamp": "2020-12-24T12:16:43.2784Z",
"level"...
- 355
0
votes
0
answers
674
views
How to send entire JSON payload to syslog through fluent bit
I am trying to send the input memory metrics from fluent-bit to a remote syslog server. The output for memory metrics, when printing to stdout, looks typically like this:
[0] memory: [1694350030....
- 1
1
vote
0
answers
83
views
Fluentbit Tag Modifications
I am using fluent-bit to send all of my K8S logs to Elasticsearch. Currently, I want to ship all of the logs for which I have included the string syslog. Then, if a log includes the following string: &...
- 31
1
vote
0
answers
1k
views
How to filter out fluentd logs to just logs and remove all metadata
I am trying to setup fluentd into my kubernetes cluster and I am able to push the logs. Only issue is it is pushing in json format with a lot of extra junk which I don't need.
Here is how the logs are ...
- 242
0
votes
1
answer
1k
views
Fluentd - turn log message string into searchable fields
I have been trying to transform application log messages into searchable fields in OpenSearch Dashboards(Kibana).
For example, this log entry:
2023-06-20T05:59:59.568967474Z stdout F {"level"...
- 221
0
votes
1
answer
1k
views
How do i use my parser as a multiline parser in fluentbit?
I am trying to parse the logs i get from my spring-boot application with fluentbit in a specific way.
parsers.conf
[PARSER]
Name springboot
Format regex
regex ^(?<time>[^ ]+)( )+(...
- 1
1
vote
1
answer
1k
views
Can fluentbit listen for syslogs on multiple ports, transform logs depending on the port, and send to different destinations?
I'm new to fluentbit and my org is restructuring our log ingestion nodes. The ingestion nodes take in syslogs from multiple platforms. Each platform has 100s of individual devices and this continues ...
- 95
1
vote
0
answers
121
views
How to convert JSON log to non JSON flat log using fluent bit?
I have an application log file, where each line is a JSON. A line looks like below.
{"timestamp":"2023-04-25T03:40:03.180Z","level":"DEBUG","message":&...
- 24k
1
vote
0
answers
1k
views
Fluentd to OpenSearch: How to create dynamic indexes with K8s metadata?
I'm using the fluent-operator to deploy fluentbit and fluentd.
Fluentbit collects and enriches the logs with Kubernetes metadata, then forwards to Fluentd.
Fluentd ships the logs to AWS OpenSearch.
I ...
- 221
0
votes
0
answers
131
views
fluent-bit to GCP through VPN tunnel
I have an on-prem 4 nodes k8s cluster. There is VPN tunnel between mikrotik router and GCP subnet so GCP VM's and 4 on-prem nodes can communicate. I want to send logs from k8s cluster to one ...
- 63
1
vote
1
answer
586
views
Rancher fluientd: Error response from daemon: Duplicate mount point: /var/lib/docker/containers
I have a problem with the rancher-logging-root-fluentbit daemon.
Some pods start correctly, others have this error.
Error: Error response from daemon: Duplicate mount point: /var/lib/docker/containers
...
- 54
5
votes
1
answer
4k
views
What's the purpose of `Time_Key` in Fluent Bit parsers?
I'm new to learning Fluent Bit, and I can't wrap my head around the benefit of specifying the Time_Key field in a parser.
An example from the documentation is below, but I don't know what the point of ...
- 53
2
votes
1
answer
2k
views
FluentBit S3 upload with container name as key in s3
My log file name in s3 looks like kube.var.log.containers.development-api-connect-green-58db8964cb-wrzg5_default_api-connect-fa7cafd99a1bbb8bca002c8ab5e3b2aefc774566bb7e9eb054054112f43f1e87.log/ here ...
- 305
2
votes
1
answer
6k
views
fluentbit connection to fluentd refused
The issue
I have been trying to use the fluent-operator to deploy fluentbit and fluentd in a multi-tenant scenario in EKS cluster.
The goal is to collect logs with fluentbit and then forward to ...
- 221
0
votes
1
answer
3k
views
How do I make fluent-bit forward docker logs?
I am trying to run a fluent-bit container according to https://www.velebit.ai/blog/tech-blog-collecting-logs-in-docker-clusters/. Note the ports section in the docker-compose.yml. I have td-agent ...
- 518
1
vote
0
answers
745
views
Fluent-bit parsing json in side of a string
I have a k8s logging stack set up like this:
fluent-bit => fluentd => elastic
I have a working set up with this config:
fluent-bit.conf
[SERVICE]
Daemon Off
Parsers_File custom_parsers.conf
...
- 19
0
votes
1
answer
418
views
Fluentbit pump local file to splunk
I am trying to pump local file to splunk using fluentbit. The Splunk is currently https and secure.
I kept encountering error message of unexpected EOF, I am not sure what have I done wrongly in the ...
- 499
2
votes
1
answer
2k
views
Fluentbit not sending EKS logs to S3
I've tried to send EKS logs to S3, but logs are not being pushed to S3. But, I can able to visualize logs in Elastic search. Below is the output section of my fb-configmap.conf file.
output-...
- 145
5
votes
1
answer
18k
views
Why I get the "failed to flush chunk" error in fluent-bit?
I am sending logs from my nestjs project to elastic search using fluent-bit. However, I get the following error periodically:
[2022/06/14 21:43:18] [ warn] [engine] failed to flush chunk '1-1654871535....
- 601
-1
votes
1
answer
948
views
Can't see logs coming from fluent forward receiver
I am trying to collect application level logs using fluent-bit, I want to listen these logs on otel-collector-contrib
I am generating dummy logs for testing using the command given below
docker run --...
- 6,466
0
votes
1
answer
996
views
GCP: Is there a way to have both the OPS Agent and the Legacy Agent installed?
My current set up uses the legacy logging agent (google-fluentd), so all my logging is configured for fluentd. I am supposed to switch to the OPS Agent, which uses fluent bit, therefore a different ...
- 317
1
vote
1
answer
328
views
Getting data of pod using binary
I'm trying to figure a way of building a configuration file/script that can help me retrieve the logs of the Kubernetes pod into elastic but using Binary fluent-bit only.
I was managed to retrieve the ...
- 273
0
votes
0
answers
1k
views
How to use fluentbit with Nestjs?
I have a fluentbit running in the Docker container 192.168.1.201:24224 which is connected to the elasticsearch.
Now, I am trying to connect my nestjs logger to the fluentbit:
logger.ts
import * as ...
- 601
1
vote
0
answers
1k
views
How to run Fluent-Bit in AWS-Lambda
I have built a more simple test image that works. The Dockerfile installs python, then Fluent-Bit, and then the CMD is a shell script that starts both processes.
Here is my Dockerfile
################...
1
vote
2
answers
2k
views
How to create a lua script for fluentbit throttle by specific key
I would like to throttle logs per kubernetes service - in other other words apply throttle filter for specific tags or keys.
Fluentd had a group_key attribute that enables throttling at a service ...
- 1,051
0
votes
1
answer
4k
views
Handling logs of huge volume with fluent-bit/fluentd
We have the following observability stack.
We are often challenged with huge influx of logs from certain apps running on ECS which causes the log aggregator to restart and eventually making ES ...
- 1,051
3
votes
2
answers
5k
views
Fluent Bit does not send logs from my EKS custom applications
I am using AWS Opensearch to retrieve the logs from all my Kubernetes applications.
I have the following pods: Kube-proxy, Fluent-bit, aws-node, aws-load-balancer-controller, and all my apps (around ...
- 1,705
2
votes
1
answer
492
views
How to encrypt fluentd SQL plugin password?
Because of security reasons, we can't keep SQL authentication in plain text, is there a way to hide or encrypt passwords?
I am getting bad documentation and bad support from the plugin site. ...
- 427
1
vote
2
answers
5k
views
"[error] [upstream] connection timed out after 10 seconds" failed when fluent-bit tries to communicate with fluentd in Kubernetes
I'm using fluent-bit to collect logs and pass it to fluentd for processing in a Kubernetes environment. Fluent-bit instances are controlled by DaemonSet and read logs from docker containers.
[...
- 4,801
1
vote
1
answer
3k
views
Parse python multiline traceback to one line with fluentbit
My python application writing logs to STDOUT and I am collecting the logs with fluentbit agent. My logs sample
Checking for future activity
Activity Time: 1636487814
Current Time: 1636490831
...
- 295
5
votes
1
answer
8k
views
Fluent Bit 1.8+ and MULTILINE_PARSER
My goal is to collect logs from Java (Spring Boot) applications running on Bare Kubernetes.
These logs are then translated into ES and visualized in Kibana.
For these purposes I deployed Fleunt Bit 1....
- 349
0
votes
1
answer
3k
views
Connection refused trying to use hostnames for "Forward" TCP from FluentBit into a FluentD instance in Kubernetes
I have a fluentbit deployed as a sidecar. This fluentbit has an output of type Forward that is suppose to send the logs to a FluentD deployed as a DaemonSet.
The implementation works when using the ...
- 629
1
vote
2
answers
414
views
Fluency with forward plugin: how to add kubernetes metadata to logs
Hey i have a question.
Im using logback-more-appenders(fluency plugin) to send logs to EFK stack (fluent-bit) which is working in kubernetes cluster, but it lacks kubernetes metadata ( like node/pod ...
- 28
0
votes
1
answer
2k
views
How to write fluent bit input logs to localhost syslog server
I'm working on collecting logs from docker containerized application. I'm able to bring the logs to stdout output plugin but when I am trying syslog output plugin then it is not writing on syslog ...
- 401
0
votes
0
answers
657
views
Append 2 fields for host value in Fluentbit running in k8s
Is it possible to append 2 fields as a value for host in fluentbit. The fluentbit runs as a Daemonset
[FILTER]
Name modify
Match *
Add host ${K8S_CLUSTER_NAME}
The above has the host with ...
- 2,012
1
vote
0
answers
904
views
Splunk Kubernetes example
I found this entry
Send Kubernetes pod's logs to Splunk
I was wondering if anyone had a front-to-back example (blog, video, etc.).
I'm used to Kubernetes, but not used to Splunk. I want to deploy ...
- 1,843
3
votes
1
answer
6k
views
Fluentd : Is there a way to add multiple tags in single match block
I have multiple source with different tags.
Im trying to add multiple tags inside single match block like this.
<source>
@type tail
@label @TESTLABEL
path /var/log/containers/app-one-*.log
...
- 777
2
votes
2
answers
744
views
Time between successive retries in Fluent Bit
There is a property called Retry_Limit but how to set time between successive retries?
[OUTPUT]
Name es
Host 192.168.5.20
Port 9200
Logstash_Format On
...
- 498
2
votes
1
answer
4k
views
How to parse a fluent-bit json $log.<field> key?
I have the following log to be parsed:
TID: [-1234] [] [2021-05-31 09:53:26,680] - Unique ID: Evento_Teste, Event: {"event":{"metaData":"blue"}}
And below the ...
- 2,857
0
votes
1
answer
3k
views
Fluent Bit not saving any data on filesystem
I am new to fluent bit and currently doing a POC. I tried multiple things but couldn't make Fluent Bit save any data to filesystem.
[SERVICE]
flush 1
daemon Off
log_level trace
...
- 498
0
votes
1
answer
2k
views
C# Serilog conditional logging if one fails?
We are in the analysis stage, right now we have an architecture where Serilog will write to a local file, and then there is a Fluentd that will pull the logs and dumps them to the elastic search. The ...
- 427
1
vote
1
answer
1k
views
Handling exception stack trace while writing serial no in json logs using Json template layout log4j2
I am using json template layout for writing json logs for my ecs service to s3 via kinesis firehose and firelens.
Following is the configuration for json template layout I am using -
{
"...
- 141
0
votes
0
answers
3k
views
How can I remove the key from fluent-bit output?
I am using the following input in fluent-bit configuration of Kubernetes ConfigMap (YAML):
[INPUT]
Name mem
Tag memory_usage
Interval_Sec 60
...
- 572