2,102 questions
0
votes
1
answer
43
views
Not able to download the JDBC sink plugin in Confluent Kafka Connect
I am working on a project in which I need to pull data from kafka topics and push it into the mysql database. I have seen alot of tutorials on youtube but most of the tutorials are outdated or 4 years ...
0
votes
1
answer
65
views
Confluent Replicator not passing Headers into Converter#fromConnectData
tl;dr; how can I implement a Kafka Converter which uses headers?
(when using Confluent Replicator)
I have made a custom Kafka Connect Converter, and as I understand it, the toConnectData is used when ...
- 151
0
votes
1
answer
37
views
How to use multiple condition left joins in Confluent KSQL query?
I have a KSQL Table with multiple primary keys. I need to create a new stream and perform a LEFT JOIN with the initial Table. However, it appears KSQL does not handle multiple condition LEFT JOINs.
...
- 1
0
votes
1
answer
79
views
Zookeeper Confluent Kafka 'Error opening file zookeeper-gc.log' when Running Confluent
I'm building a Kafka Application on Windows Subsystem for Linux 2 (WSL2).
I'm using Zookeeper 3.5.8, confluent-6.1.0 and java openjdk version 17.0.12.
I successfully installed all the above platforms ...
- 815
0
votes
0
answers
51
views
Confluent cli gives error when trying to start producer with value format set to avro
Currently trying to set up a local development environment with Confluent. I've installed the latest version of Confluent on my machine (Macbook Air M2) but when trying to attempt to start a producer, ...
0
votes
2
answers
128
views
Issue setting Confluent Kafka JAVA_HOME path in .bashrc with Error - no such file or directory
I am trying to set a JAVA_HOME path for my Confluent and Kafka application. I'm using confluent-6.1.0, Java 15, and Windows OS (WSL).
The confluent and Java paths are set in a '.bashrc' file. When ...
- 815
0
votes
0
answers
58
views
Schema Registry 409 when issuing pull query against table with Struct Grouped
I have a simple test case below
CREATE STREAM _xtest2 (
ID INT KEY,
custid int,
zone string,
pid int
)
WITH (
KAFKA_TOPIC='xtest2',
PARTITIONS=2,
replicas=1,
FORMAT = 'AVRO'
);
CREATE ...
- 562
0
votes
1
answer
232
views
Confluent Kafka Topic Authorization Failed
I was using a .sh script to login into confluent Kafka using a username and password, then try to update the config of a topic. Here, am able to login into Kafka. However, getting error while doing ...
- 51
0
votes
0
answers
18
views
Why doesn't large select query return any value in ksql?
I have a fairly large query in ksqldb where all the different columns of the query do work, but when I combine them all together I don't get any result. I don't get any feedback telling me why there ...
- 23.4k
0
votes
1
answer
126
views
Not able to externally connect to confluent kafka cluster's brokers with confluent for kubernetes
I am trying to set up External connection with Confluent for Kubernetes. I've followed the documentation :
spec:
listeners:
external:
externalAccess:
type: loadBalancer
...
- 56
0
votes
0
answers
77
views
Confluent schema registry 7.6.0 taking too long to catchup with offsets during startup
I have deployed confluent schema registry version 7.6.0 on my on-premise Kubernetes cluster,and it is pointing to _schemas topic in Kafka cluster hosted to Confluent cloud on GCP.
The_schemas topic is ...
- 61
1
vote
0
answers
44
views
How to write data to Confluent Kafka with SchemaRegistry format on sparkstructured?
I am to trying write a batch data to kafka topic with schema registry in databricks using pyspark, i serialize the data with pyspark to_avro function and write it to the topic, but the consumers can’t ...
- 11
1
vote
0
answers
219
views
Async Api: Generate Java Client using Schema Registry and AVRO messages
I´m using Confluent Async Api Tool to export the topic / schema definitions to an async-api.yml file.
I want to generate a Spring Cloud Client using Schema Registry and AVRO generated classes from the ...
0
votes
1
answer
70
views
Issue with setting up KSQLDB locally with Remote Kafka Cluster
I'm trying to set up KSQLDB locally and connect it to remote kafka server running on upstash. But, after configuring the credentials, I'm not able to connect to the kafka server.
I installed KSQLDB ...
- 1
2
votes
0
answers
420
views
Security Manager is deprecated and will be removed in a future release
Ksql Db was starting absolutely fine up until now when I ran into this issue. I am aware that port on to which ksqldb starts shouldn't be occupied by any other process, so that's not an issue.
Anyway, ...
- 131
0
votes
0
answers
88
views
confluent cloud login using github actions
while trying to write a workflow in github actions to install confluent cli and also login using
confleunt login --no-browser it actual requires an external authentication from the browser by ...
- 1
0
votes
0
answers
28
views
Problem with Kafka setup in Kubernetes - Bootstrap broker disconnected [duplicate]
I've been following the steps to set up Confluent Platform with static host-based routing in Kubernetes. However, I'm encountering issues when trying to produce messages to a Kafka topic from an ...
0
votes
0
answers
549
views
Kafka Producer Error - Bootstrap Broker Disconnected
I'm encountering an issue while attempting to produce messages to a Kafka topic using the kafka-console-producer command. Despite configuring the connection details correctly, I face the following ...
1
vote
1
answer
350
views
How to login to confluent platform using confluent cli?
I have installed confluent platform and all the local services are up and running. The problem arises when I try to install a connector, like lets say Postgresql connector by issuing the following ...
- 131
0
votes
0
answers
165
views
Schema Validation with Pyflink and Kafka's Schema Registry
I was able register my expected schema for the Kafka topic for the records arriving using the below command:
curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data ...
- 605
0
votes
0
answers
128
views
Unable to Find Debezium Heartbeat Topic in Confluent UI/CLI with Kafka Connect Standalone on Kubernetes
I'm currently using the Confluent Kubernetes Operator for managing a Kafka cluster, along with a Kafka Connect standalone node. Within this setup, I've configured a Debezium PostgreSQL source ...
- 11
0
votes
0
answers
75
views
Kafka - Getting error - Error checking or creating metrics topic
Trying to install confluent platform 7.5.3 - Kafka on 6 nodes and see below messages in server.log
Getting error - Error checking or creating metrics topic (io.confluent.metrics.reporter....
- 120
0
votes
0
answers
37
views
Not seeing results in ksqldb query
in ksqldb-cli (Version: 7.1.7)
I defined a stream based on the docs:
https://docs.confluent.io/5.4.11/ksql/docs/developer-guide/create-a-stream.html
The topic contains JSON data but I use String ...
- 3,777
0
votes
0
answers
73
views
After updating to Confluent 7.5 brokers can no longer connect to Zookeeper
The goal is to update the Kafka cluster to version 7.6 in order to migrate to Karft.
A few key facts about the cluster:
No Docker / Podman
3 ZooKeeper (single VMs in vSphere)
3 brokers (individual ...
- 81
0
votes
1
answer
178
views
Confluent JDBC source connector issue with table whitelist option
We are using Confluent Kafka JDBC source connector to fetch data from AWS RDS(Oracle db).The connector is working for select * from table name but it does not work with table whitelist option.It is ...
- 87
0
votes
0
answers
128
views
Command "confluent local kafka start" is not working without internet connectivity
Command "confluent local kafka start" is not working without internet connectivity.
I have loaded the docker image in the local docker registry already.
I am behind a firewall, I can't ...
- 961
0
votes
0
answers
498
views
Produce to Confluent Kafka Topic with SASL_SSL and OAUTHBEARER mechanism using python client
I'm trying to produce one message to a topic in my RBAC-enabled and TLS-enabled Confluent Kafka cluster. My solution only works partially. I'm seeking help to successfully publish a message to the ...
- 498
0
votes
2
answers
384
views
Consume EventHub using Kafka from spark structured streaming job with Service Principal-Secret Auth
I am new working in this and I am building a spark structured streaming job where I am trying to consume EventHub events using Kafka,I followed a official documentation example where uses service ...
- 73
0
votes
0
answers
64
views
clickhouse, get headers and meta from AvroConfluent message
I have a kafka topic in the AvroConfluent format, I can take data from messages, but only from the fields that are in the format_avro_schema_registry_url, previously the data format was JSONEachRow ...
- 63
1
vote
0
answers
64
views
When sending values from SharePlex to Kafka and then back to Oracle, how should the sink connector be configured?
When changes from Oracle are sent to Kafka through SharePlex and the values are as follows, without primary keys. When attempting to send these values back from Kafka to Oracle, how should the JDBC ...
- 13
0
votes
1
answer
59
views
Kafka compression on Broker side
There are settings to enable compression per Producer and per Topic.
But i also heard about global setting - Broker side compression (for all topics), is it real ?
If so how to configure ? Any link ?
- 1,797
0
votes
0
answers
60
views
Confluent HDFS Sink connector error while connecting HDFS to Hive
Trying to push messages from Kafka to Hive by using confluent HDFS Sink connector. Successfully moved data from Kafka to HDFS, but while connecting Hive from HDFS is failing with below error,
"...
- 1
0
votes
0
answers
46
views
Docker, Kafka and .NET - Error to connect .NET application container in Kafka container
I'm trying to make a connection between my .NET container (consumer) and Kafka container, but I'm getting an error when the .NET consumer tries to connect to Kafka.
I've read a lot about listeners and ...
0
votes
1
answer
793
views
What is the difference between debezium vs confluent JdbcSinkConnector?
I tried to connect Kafka with Postgres through the sink connector, and I am a little bit confused. There are 2 types of the connectors: 1) io.debezium.connector.jdbc.JdbcSinkConnector
2)
io.confluent....
- 35
0
votes
0
answers
192
views
Issue with Quarkus and Kafka Avro Schemas not using self generated avro scheme
I'm currently encountering an issue with Quarkus while trying to connect to Kafka using Avro Schemas already added in the Confluent Avro Schemas under the name default.avro Here's a breakdown of the ...
0
votes
1
answer
570
views
Cluster linking in Apache Kafka
Can someone please confirm if Cluster Linking is available in Apache Kafka? Or its limited only to Confluent Kafka.
I have a requirement to link one VM based Confluent Platform cluster to another VM ...
- 407
1
vote
1
answer
604
views
Kafka in Docker using Confluent CLI doesn't work
I'm trying to set up a local Kafka cluster on my Mac (Intel-based, running macOS Sonoma 14.3.1) by following the Confluent developer guide. However, when I try to start the cluster, it doesn't do ...
- 3,521
0
votes
0
answers
163
views
mongo Kafka connector issue confluent-hub install - error java.net.ConnectException
Using below document to setup Mongo Kafka connector . Hit roadblock :
https://www.mongodb.com/docs/kafka-connector/v1.5/tutorials/tutorial-setup/#std-label-kafka-tutorials-docker-setup
when trying to ...
- 984
0
votes
1
answer
29
views
How to hide Confluent CLI tooltip
This is new, the tool tip to auto complete commands was a good to have but it doesn't even allow me to see the output of the command just entered.
I typed environment list but can't even see the list ...
0
votes
1
answer
452
views
Best Practice for deduplication using Flink SQL Query from raw_topic and create new topic with deduplicates(Confluent)
I have a question regrading Flink SQL via Confluent. I have .NET Windows Service app that produces that data to the Kafka topic and the application runs 24 hours and it will have massive amount of ...
0
votes
1
answer
384
views
Flink AvroDeserilized Records defaulting to GenericType
I am trying to read data from a kafka topic which has an avro schema. I have generated POJO from my avro schema which i am using it to deserialize it. At least that what i understood from reading ...
- 1
0
votes
0
answers
245
views
Kafka Confluent - Issue with Audit Log Configuration
I am using Kafka Confluent platform 7.5.0 on-premises, hosted on a Linux Ubuntu 22.04.3 LTS VM.
My goal is to capture all PRODUCE events and write them to the topic confluent-audit-log-events_produce.
...
- 77
0
votes
1
answer
2k
views
Kafka in KRaft mode keeps asking about Zookeeper
I have this docker-compose configuration:
kafka:
image: confluentinc/cp-kafka:latest
container_name: 'virl-kafka'
restart: always
ports:
- "9092:9092"
environment:...
- 99.4k
0
votes
1
answer
295
views
Single Confluent Kafka cluster in different host machine
I want to mount a Confluent Kafka Cluster in Kraft mode, in which, each broker is containerized in a Docker container. Each of this broker should be placed on different host machine. I want that two ...
- 31
0
votes
1
answer
309
views
Kafka Raft SASL_SSL client failing to initialize
I am struggling to get Kafka Raft running in my environment. I have tried multiple security protocols SASL_SSL, PLAINTEXT, SASL_PLAINTEXT -- The brokers cannot seem to get connected.
We do have a SELF ...
- 107
0
votes
0
answers
28
views
After enabling ACL's, replication factor errrors in cluster
I set up a single node cluster on openshift with SASL_SSL and ACL's. Until I enabled ACL's everything worked fine, the moment I configured ACL this is what I see:
For a "normal" user with ...
- 56
0
votes
0
answers
138
views
Problem with TimestampConverter from Confluent
I'm new with Kafk and I'm struggling dealing with timestamp field within my JDBC sink connector.
Making long things short, I'm using Docker to test a source PostgresSQL connector to copy the data of ...
- 43
1
vote
0
answers
271
views
Kafka Connect Confluent JDBC Sink Lag Issue
I am running a CDC pipeline with a PostgreSQL source (Debezium Source Connector) and a PostgreSQL target using the Confluent JDBC Sink Connector (version 10.6.4) to read data from Kafka. The issue I ...
0
votes
2
answers
827
views
How do I set the target table name for my sink JDBC connector?
I have a JDBC sink connector retrieving data from a Postgres source connector, and I need to copy the data from my "parametros" source table to my "parametros_sistema" target table....
- 43
0
votes
1
answer
55
views
Unable to get Kafka working due to address issue
I'm trying to follow a cloud guru course on kafka but when I setup the server and try and run it I get this error in the kafka server.log file:
WARN Cannot open channel to 2 at election address zoo2/...
- 2,919