All Questions
Tagged with open-telemetry kubernetes
46 questions
0
votes
0
answers
56
views
Add kubernetes pod labels to splunk logs using k8sattributes processor
How to add pod labels to splunk logs? Can't get the syntax right and the documentation doesn't seem to cover it. I'm using the otel-collector in daemonset mode, running in a kubernetes cluster and ...
- 13
0
votes
0
answers
63
views
How to prevent opentelemtry collector running as daemonset scrapping duplicate metrics
I have open telemetry collector running as daemonset in k8s cluster. The cluster has following Prometheus Receiver configuration.
config:
scrape_configs:
- job_name: 'otel-node-...
- 243
1
vote
2
answers
121
views
Can't add Prometheus as DataSource in Grafana, via Minikube
I am using MiniKube 1.31, I decided to follow the initials steps on OpenTelemetry Collector Operator applying the suggested CMD (kubectl apply -f https://github.com/open-telemetry/opentelemetry-...
- 123
0
votes
1
answer
69
views
Opentelemetry Collector (mode: daemonset) is failing when enabling the `k8sattributes` processor
So getting this error
failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:opentelemetry:oteld-collector" cannot list resource "pods" in API group "" at ...
- 943
0
votes
0
answers
403
views
Error in OpenTelemetry With Istio: sending queue is full
I'm using OpenTelemetry as a tracing collector in istio, but after running OpenTelemetry Collector for a while, the logs for istio-ingressgateway and the logs for Sidecar (istio-proxy) in the Pod keep ...
0
votes
0
answers
188
views
How to add labels to prometheus metrics from dotnet application sending opentelemetry to a collector
I have a dotnet container which I can't change the source code of running inside a kubernetes cluster as a deployment.
The application is sending its metrics to an otel collector configured in ...
- 472
-1
votes
2
answers
881
views
Opentelemetry cant connect to Prometheus on Kubernetes
I have a kubernetes cluster with apps in .net 7 installed . The cluster also has OpenTelemetry as a deployment and Prometheus server installed.
The opentelemetry has a values file as show below
mode: ...
- 1,004
0
votes
0
answers
168
views
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'processorMetrics' defined in class path resource
When I run my application in kubernetes cluster, I get the below exception and it does not come up.
2024-02-21 10:52:17,206 ERROR o.s.b.SpringApplication [main] Application run failed
org....
- 614
0
votes
0
answers
186
views
Opentelemetry Kubeletstats receiver - Unable to get network stats in Openshift 4
I need help on kubeletstats receiver.
I'm using the below configuration in opentelemetry collector which is running in openshift.
...
image: 'ghcr.io/open-telemetry/opentelemetry-collector-releases/...
- 11
1
vote
0
answers
117
views
OpenTelemetry: Traces Split Between Zipkin Replicas in Kubernetes Deployment
I'm using OpenTelemetry in my scala project for tracing and seeing traces in zipkin.
I'm creating tracer object as:
object OpenTelemetry {
val urlForZipkin ="http://zipkin:9411/api/v2/spans&...
- 86
-1
votes
1
answer
189
views
quarkus opentelemetry kuberbetes pod name
We are on Quarkus 2.13.x and are using the opentelemetry libraries to implement telemetry with some automatic and manual context propagation. It's working fine so far but i recently noticed one of the ...
- 139
1
vote
0
answers
424
views
k8attributes is not coming in opentelemetry generated metrics
Hi i have opentelemetry configurations in my EKS cluster here i am exporting matrics/traces and logs. In my traces k8attributes is coming and i can see on grafana with tempo.
enter image description ...
- 27
0
votes
1
answer
2k
views
Grafana Tempo doesn't show traces after sometime
hi i have setup grafana tempo with opentelemetry in my EKS cluster. at initial time tempo show traces but after 1 hr none trace show on grafana it show me 404 Not Found
here is my tempo configuration
...
- 27
0
votes
1
answer
1k
views
Having metrics from opentelemetry nodejs library scraped (or pushed)
I am using GrowthBook which has recently introduced OTEL.
Everything is running on my platform on Kubernetes
I see that the only thing that is done in that context is to import the corresponding @...
- 19.1k
2
votes
0
answers
591
views
Java Auto Instrumentation creating problems in the namespace
I am trying to install OpenTelemetry in our Staging Environment, the collector being in DaemonSet Mode.
My goal is to create a Java Instrumentation customer resource, which defines the configuration ...
1
vote
0
answers
412
views
How to exclude K8s resource attributes from traces created with opentelemetry?
I am using the Opentelemetry Operator on a K8s cluster to auto instrument my java application through the Instrumentation CR and send traces first to the Grafana Agent and then to the Tempo backend ...
0
votes
1
answer
2k
views
Most OpenTelemetry instrumentations are disabled in my Spring application
I have injected an OpenTelemetry JAVA agent in a Spring application in OpenShift via kubernetes operator, but most instrumentations are disabled and not generating traces.
OpenTelemetry Agent image: ...
- 9
-1
votes
1
answer
732
views
OTEL Collector deployment on a Kubernetes cluster
We are planning to implement OTEL with Kubernetes and I can't seem to find the correct answer when it comes to k8s.
The question is, do I need/should have the collector agent on each node or just have ...
- 325
0
votes
1
answer
796
views
Connection Refused error on Fluent-bit Opentelemetry INPUT
I am trying to send logs from Opentelemetry Collector Contrib (export) to Fluent-bit using the Opentelemetry INPUT plugin. But I get error in the OTel Contrib pod:
2023-06-30T14:17:07.770Z warn ...
- 1,574
1
vote
1
answer
1k
views
How can I configure a springboot java application to send traces to jaeger
I have a springboot application deployed in my kubernetes cluster. I would like to start collecting traces and send them to my jaeger (deployed with jaeger operator).
I tried to follow a couple of ...
- 43
0
votes
1
answer
1k
views
K8s Port forward into network namespace - Connection Refused
I'm running into an issue while hosting Open Telemetry Collector on K8s. We are using open telemetry helm-charts to deploy in an OKE cluster. The deployment succeeds however, the port-forward to the ...
- 571
0
votes
0
answers
865
views
I have some template related error during helm unit testing
I have a helm chart with the following repository structure. It is basically a nested helm chart. The other helm chart is 'otel-collector' and the inner helm chart is 'opentelemetry-collector'
├── ...
- 11
1
vote
1
answer
1k
views
Issue when using kafka in opentelemetry collector exporters running in openshift
I'm having kafka cluster deployed in openshift in different namespace also opentelemetry is delpoyed in another namespace.
This is my yaml file for opentelemetry
apiVersion: opentelemetry.io/v1alpha1
...
- 11
0
votes
0
answers
286
views
How to resolve conflicts between Boto3 in opentelemetry auto-instrument and Kubernetes applications?
I installed OpenTelemetry Operator and OpenTelemetry Collector by helm to implement auto-instrument application which deployed on Kubernetes. Implementation steps:
Install OpenTelemetry Operator helm ...
5
votes
1
answer
771
views
OpenTelemetry Operator - Not Able to Inject auto Instrumentation libraries
We are planning to use open telemetry as a standard for all our applications and we have installed opentelemetry-operator one of our Kubernetes cluster. otl collector and instrumentation is running ...
- 51
0
votes
1
answer
203
views
Kubernetes error: unknown field "otel" error when trying to setup Open Telemetry with Dapr
I'm following the Dapr documentation to have Dapr sending data to an Open Telemetry collector pod in kubernetes.
The first step is to configure Dapr to send distributed tracing data.
Tracing Docs
So I'...
0
votes
1
answer
965
views
Does an open-telemetry Metrics instrument need to persist in order to properly record histogram metrics?
Consider the following case:
One has a kubernetes cronjob that runs on a schedule, say once every 30 seconds. This cronjob spins up a new pod, runs a python script that captures some measurements, and ...
1
vote
1
answer
281
views
Pass an entire yaml from values to templates without filling empty fields with nulls
I am trying to pass given part of values.yaml into helm template:
receivers:
test1:
test2:
test3:
test4:
using function:
{{ .Values.receivers | toYaml | nindent 2}}
Code is ...
- 13
3
votes
1
answer
2k
views
Open-telemetry auto instrumentation does not work without sidecar
I work at a startup, and we recently migrated our workloads to use Kubernetes, specifically we are running inside a cluster in EKS (AWS).
I'm currently trying to implement a observability stack on our ...
1
vote
2
answers
3k
views
Opentelemetry collector gives "partialSuccess": {} response
I have an opentelemetry collector contrib (0.69.0) installed inside OpenShift container and I want to export my traces to Datadog. I have an http endpoint URL exposed for my Collector service and I ...
- 421
1
vote
1
answer
1k
views
OpenTelemetry Collector, Prometheus, and K8s namespaces
Given the below scenario (which I believe is quite standard) - how can the metrics know what namespaces they came from? They always seem to just refer to the monitoring namespace - which sortof makes ...
- 5,962
0
votes
1
answer
1k
views
How to deal with clock skew in a multi-node Kubernetes cluster?
The issue is that the logs for a specific request (identified by a traceId printed in each log line) are being collected by loki-stack in a Kubernetes cluster, but the logs are not being displayed in ...
- 5,654
2
votes
0
answers
155
views
Use OpenTelemetry to track Kubernetes pods/containers cpu, memory and health
Does opentelemetry provide auto instrumentation for Kubernetes pods/containers cpu, memory and health if I am using opentelemetry-collector-contrib?
- 2,007
0
votes
1
answer
215
views
How to debug/list all OpenTelemetry targets scraped into a k8s cluster
I'm configuring OpenTelemetry on a k8s cluster with the following configuration
- job_name: MYAPP
kubernetes_sd_configs:
- role: pod
metrics_path: /metrics
scrape_interval: 10s
...
- 7,797
3
votes
1
answer
917
views
no matches for kind "OpenTelemetryCollector" in version "opentelemetry.io/v1alpha1"
I tried instrumenting the open telemetry by deploying the following script.
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: my-instrumentation
spec:
exporter:
...
- 5,358
2
votes
1
answer
728
views
Is there a step by step instruction to enable tracing of kubernetes components?
I'm running Kubernetes version v1.24.0, trying to enable tracing in Kubernetes components and so I can instrument my custom controller to work with tracing.
However, the instruction https://kubernetes....
1
vote
1
answer
2k
views
Gateway vs Agent OpenTelemetry collector deployment on Kubernetes
there is 2 ways to deploy OpenTelemetry collector on Kubernetes
https://opentelemetry.io/docs/collector/deployment/
Agent and Gateway
my question is when deploying OpenTelemetry collector as a ...
- 11
0
votes
1
answer
577
views
How to inject custom tags into OpenTelemetry when using a side-car pattern
I am new to OpenTelemetry. I am trying to understand how to set up custom tags when using a side-car Observability pattern as DAPR.
My point is to find a way to add OpenTelemetry tags to a Trace in an ...
- 1,574
0
votes
2
answers
833
views
Could open-telemetry collector be used without instrumentation?
We are building application with argo workflows. We are wondering if we could just set up opentelemetry collector inside of our kubernetes cluster, and start using it as stdout exporter into elastic ...
4
votes
1
answer
12k
views
failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:monitoring" cannot list resource "pods" in API group "" at the cluster scope
Not sure what I am missing. Please, find below all the config scripts I have used
2022-07-21T07:26:56.903Z info service/collector.go:220 Starting otelcol... {"service": "my-...
- 840
1
vote
1
answer
5k
views
OpenTelemetry doesn't show metrics in Prometheus
I've setup OpenTelemetry in Kubernetes. Below is my config.
exporters:
logging: {}
extensions:
health_check: {}
memory_ballast: {}
processors:
batch: {}
memory_limiter:
check_interval: ...
- 528
1
vote
1
answer
4k
views
Open Telemetry python-FastAPI auto instrumentation in k8s not collecting data
So I am trying to instrument a FastAPI python server with Open Telemetry. I installed the dependencies needed through poetry:
opentelemetry-api = "^1.11.1"
opentelemetry-distro = {extras = [&...
- 162
0
votes
1
answer
241
views
OpenTelemetry Service Point not showing in Prometheus
I installed OpenTelemetry operator in Kubernets. Installed it using helm.
helm install my-opentelemetry-collector open-telemetry/opentelemetry-collector
Below is the ConfigMap of collector
...
- 528
5
votes
2
answers
20k
views
Unable to export traces to OpenTelemetry Collector on Kubernetes
I am using the opentelemetry-ruby otlp exporter for auto instrumentation:
https://github.com/open-telemetry/opentelemetry-ruby/tree/main/exporter/otlp
The otel collector was installed as a daemonset:
...
- 357
1
vote
2
answers
1k
views
How can I connect my Kubernetes pod to a remote Jaeger?
I am trying to connect my pod from Kubernetes (k8s) cluster to a remote Jaeger server. I've tested and it can work well if both of them are on the same machine. However, when I run my app on k8s, my ...
- 1,409
0
votes
1
answer
3k
views
Using OTEL_EXPORTER_OTLP_HEADERS in Open Telemetry Collector to avoid secrets in ConfigMap
I have a Kubernetes cluster with a pod running an instance of Open Telemetry Collector.
My .Net app inside Kubernetes exports traces to the Collector instance which in turn exports them to Elastic ...
- 1,389