All Questions
27 questions
0
votes
2
answers
63
views
How to match optional group in regular expression
I want a regular expression for using in fluentd for parsing nginx error logs.
The sample row is:
2024/04/15 09:06:29 [error] 3443790#3443790: *176070165 limiting requests, excess: 2.957 by zone "...
- 166
0
votes
0
answers
818
views
Fluentd parsing nginx access log doesn't work
I want to parse nginx access logs and provide the data to prometheus. The problem is, fluentd (or rather td-agent) doesn't want to match my regex. The example is straight from docs, but modified with ...
- 94
0
votes
0
answers
316
views
Fluentd with nginx syslog
I have ngnix and fluentd in Docker.
My ngnix config use syslog:
access_log syslog:server=server_ip:5140,tag=nginx_access json;
error_log syslog:server=server_ip:5140,tag=nginx_error info;
If i check ...
- 541
0
votes
0
answers
720
views
Scrape nginx logs with fluentd and push the metrics to prometheus
I have a nginx k8s deployment. I am running fluentd as a sidecar inside my nginx pod.
The nginx config look like below:
data:
nginx.conf: |
events {
worker_connections 1024;
}
...
- 1,157
0
votes
1
answer
1k
views
Fluentd on Kubernetes - Parse Nginx Access Log in Json
I currently have this nginx log output.
log_format json_logs escape=json
'{'
'"time_local":"$time_local",'
...
- 2,731
1
vote
2
answers
329
views
Nginx cannot connect fluentd in EFK stack
I am setting up a stack with an application consisting of nginx, redis, mysql, myapp. Nginx proxies requests to myapp. I want to send logs from nginx to EFK stack, but an error occurs when starting ...
0
votes
0
answers
955
views
How to use nginx with fluentd in Docker Compose
I have a docker-compose file where it contains 4 containers -
flask app
nginx for reverse proxy
fluentd for log processing
mongodb for storing the logs
version: '3'
services:
fluentd:
...
- 3,783
1
vote
1
answer
498
views
Parsing error messges from nginx log to stackdriver with google-fluend
We trying to set up alerting by parsing nginx error, and we need only messages with [error] level. ([error] 27720#27720: *29765923 access forbidden by rule, client: blalbablab). Main goal is to not ...
- 558
-1
votes
1
answer
2k
views
Parse Nginx Ingress Access Log in FluentD Using Multi Format Parser (Regex)
I have a Nginx Ingress Controller in the K8S cluster that has the following log format (I took it from /etc/nginx/nginx.conf in the container):
log_format upstreaminfo '$remote_addr - $remote_user [$...
- 4,942
2
votes
1
answer
3k
views
Nginx json logs are incorrectly parsed by Fluentd in Elasticsearch (+ Kibana)
I have an nginx 1.16.1 running in docker in vanilla K8s (bare metal).
Logs are forwarded to stdout by ln -sf /dev/stdout /var/log/nginx/access.log & then are transferred to Elasticsearch docker....
- 374
1
vote
1
answer
3k
views
How to change time_format from time_iso8601 in fluentd and send to clickhouse?
There is nginx.conf
....
http {
....
log_format main '"$remote_addr" "$time_iso8601"';
access_log /var/log/nginx/access.log main;
.....
}
Logs access.log
"127.0.0.1" "2019-12-28T10:53:...
- 629
0
votes
0
answers
305
views
How to store request data of post requests on an nginx server?
I have a python application running on an gunicorn server which is behind a nginx server. A post request with JSON data hits the nginx server, which then forwards it to gunicorn server. There is a ...
- 755
0
votes
1
answer
969
views
Data flow from Nginx access log -> Rsyslog or Syslog -> Fluentd -> Kinesis
I am working on passing nginx access logs to Fluentd to aws kinesis to AWS S3 via kinesis firehose. nginx logs will be pushed to AWS Glacier during log rotation.
I am at the initial steps where I ...
- 932
2
votes
1
answer
4k
views
fluentd regex pattern not match error for NGINX logs
I've setup fluentd to parse nginx access logs since I've added some extra fields to the logs I cannot use the nginx format. My configuration is:
<source>
@type tail
@id ...
- 41
2
votes
1
answer
509
views
Optional named group in Ruby RegExp
I'm using a plugin for parsing nginx log entries sent to fluentd, and trying to change it so it would optionally support another field.
This is the current REGEXP used for parsing the nginx log entry ...
- 6,767
0
votes
1
answer
1k
views
use geoip with nginx logs in fluentd
I'm trying to use geoip with nginx logs in fluentd.
as for nginx, I'm using the following filter:
<filter *.nginx>
@type record_transformer
enable_ruby
<record>
log ${ if ( !...
- 824
2
votes
1
answer
435
views
logging nginx events from a docker container managed by kubernetes
Currently, to my understanding, kubernetes offers no logging solutions on it's own and it also does not allow one to specify the logging driver when using docker as the container technology due to ...
- 4,770
3
votes
1
answer
4k
views
Using fluentd in docker to get nginx logs
I have a scenario where nginx is running in one container and fluentd is in another container, i mapped nginx logs to var/logs/nginx directory, but i was unable to retrieve logs to elastic search ...
- 321
1
vote
1
answer
330
views
fluentd doesn't tag the events properly
I have a simple fluentd configuration:
<source>
type forward
</source>
<source>
type tail
path /var/log/nginx/access.log
pos_file /var/log/td-agent/nignx-access.log
tag ...
- 511
8
votes
4
answers
15k
views
Parse nginx ingress logs in fluentd
I'd like to parse ingress nginx logs using fluentd in Kubernetes. That was quite easy in Logstash, but I'm confused regarding fluentd syntax.
Right now I have the following rules:
<source>
...
- 733
8
votes
1
answer
12k
views
docker compose oci runtime error, executable file not found in $PATH
I'm following this post:
http://eric-price.net/blog/centralized-logging-docker-aws-elasticsearch
This is what my docker-compose.yml looks like :
version: "2"
services:
fluentd:
image: ...
- 511
0
votes
0
answers
80
views
Fluentd logging site activity options
I'm wanting to implement Fluentd to log everything from user actions to nginx logs.
We have 1 site that runs express (server side react)
users login
view content
like content
comment on content
...
- 15.8k
0
votes
1
answer
2k
views
Regex to correctly parse url from nginx logs
I'm using fluentd to pipe nginx logs to elasticsearch using the tail plugin. A regex is provided for parsing out the fields:
^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<...
4
votes
1
answer
2k
views
Nginx Access Log To Kinesis
I am looking for a scalable solution.
Nginx will server a pixel (1x1 gif) with a query string to an html page.
This query string will be in the nginx access logs.
I need to stream, or send this data ...
- 447
0
votes
2
answers
1k
views
td-agent praser nginx log cpu 100%
my english is not good ,sorry ...
these days ,i always test td-agent on AWS EC2(Amazon Linux AMI release 2014.03).
input log from nginx and output log to S3 is OK, this very good!
then i want praser ...
- 11
0
votes
1
answer
3k
views
fluentd and nginx access log format
how can I write the regex for the following nginx access log format in fluentd ?
log_format main '$remote_addr - $remote_user [$time_local] $request '
'"$status" $...
- 300
1
vote
2
answers
1k
views
fluentd logging nginx realtime
I am working on a project where I'm looking at using fluentd. I need to parse the nginx access log in real time, probably using the in_tail. But a question arises when log rotation occurs.
Given this ...
