All Questions
Tagged with confluent-platform java
126 questions
0
votes
1
answer
65
views
Confluent Replicator not passing Headers into Converter#fromConnectData
tl;dr; how can I implement a Kafka Converter which uses headers?
(when using Confluent Replicator)
I have made a custom Kafka Connect Converter, and as I understand it, the toConnectData is used when ...
- 151
0
votes
1
answer
79
views
Zookeeper Confluent Kafka 'Error opening file zookeeper-gc.log' when Running Confluent
I'm building a Kafka Application on Windows Subsystem for Linux 2 (WSL2).
I'm using Zookeeper 3.5.8, confluent-6.1.0 and java openjdk version 17.0.12.
I successfully installed all the above platforms ...
- 815
0
votes
2
answers
128
views
Issue setting Confluent Kafka JAVA_HOME path in .bashrc with Error - no such file or directory
I am trying to set a JAVA_HOME path for my Confluent and Kafka application. I'm using confluent-6.1.0, Java 15, and Windows OS (WSL).
The confluent and Java paths are set in a '.bashrc' file. When ...
- 815
0
votes
0
answers
138
views
Problem with TimestampConverter from Confluent
I'm new with Kafk and I'm struggling dealing with timestamp field within my JDBC sink connector.
Making long things short, I'm using Docker to test a source PostgresSQL connector to copy the data of ...
- 43
0
votes
2
answers
827
views
How do I set the target table name for my sink JDBC connector?
I have a JDBC sink connector retrieving data from a Postgres source connector, and I need to copy the data from my "parametros" source table to my "parametros_sistema" target table....
- 43
0
votes
0
answers
39
views
(io.confluent.connect.avro.AvroDataConfig:370) Exception in thread "main" java.lang.NoClassDefFoundError: org/hamcrest/Matcher
I am running ksql-tests via command line and I get this error. I tried adding jar manually to classpath before running tests, but still shows this error.
How can I run ksql-test-runner via command ...
- 2,468
1
vote
0
answers
1k
views
kafka producer throwing error when sending an Avro record to topic:io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: null
I am trying to create a topic and produce an avro record to a kafka topic, but I am facing an issue while send the record to the producer. I am getting an error.
org.apache.kafka.common.errors....
0
votes
1
answer
2k
views
Kafka Streams: how to avoid scheduler punctuator to be suppressed before completion
I have a Kafka stream application that needs to process some data into a StateStore every x minutes.
I have created a transformer receiving data and storing them into the StateStore. The Transformer ...
- 327
1
vote
1
answer
397
views
Creating test data from Confluent Control Center JSON representation
I'm trying to write some unit tests for Kafka Streams and have a number of quite complex schemas that I need to incorporate into my tests.
Instead of just creating objects from scratch each time, I ...
- 1,428
1
vote
1
answer
1k
views
Kafka streams joins two specific Avro objects
Basic task
I have 2 identical streams in Kafka in Avro format. I'm trying to do basic left join with those 2 streams.
Keys
For keys in both topics I'm using timestamp rounded to milliseconds since ...
- 15.1k
0
votes
0
answers
897
views
java.lang.VerifyError: Bad return type when using org.springframework.kafka.support.serializer.JsonDeserializer()
I would like to try on the Synchronous Kafka on https://dzone.com/articles/synchronous-kafka-using-spring-request-reply-1
However, I am hitting java.lang.VerifyError: Bad return type when it's loading ...
- 1,903
0
votes
2
answers
667
views
ksqlDB java client: not able to deserialize data from client.executeStatement()
I am trying to query a KSQL Stream on confluent cloud using Java ksqlDb-api-client:0.24.0 using the below code
ClientOptions options = ClientOptions.create()
.setHost(...
- 1,476
3
votes
1
answer
4k
views
Confluent Kafka schema not found error from producer when auto register is off
I have a schema registry, kafka, zookeeper with a producer running locally. Problem is that when I manually register the schema and turn off auto register on the producer, I get the following ...
- 144
0
votes
0
answers
282
views
Trying to understand Apache Kafka as a microservice. Still confused
I am learning about Kafka, and I am curious as to how a Kafka client should exist in a Microservices architecture I want Kafka to keep a log of important information, and enable automatic reaction to ...
- 109
0
votes
1
answer
579
views
Parsing Avro messages in flink , giving null pointer exception if field is nullable in Avro Schema
I need to parse the messages from confluent Kafka stored in Avro. But while applying filter it is giving null pointer exception, without filter i was able to write back into kafka but while applying ...
- 776
0
votes
2
answers
826
views
how can I aggregate the values from an array of micrometer counters
I’m collecting a series of metrics for a kafka stream application, the issue I have is I’d like a consolidated value for the meters of a specific name. To make this a little clearer, these metrics are ...
0
votes
0
answers
3k
views
Json message validation on confluent kafka with schema registry
I am sending a json message to a kafka topic using spring boot kafka with java. The message has a json schema published on schema registry.
When the producer sends the message, how do I get the json ...
- 1,387
3
votes
1
answer
2k
views
Kafka Streams KTable foreign key join not working as expected
I'm trying to have a simple foreign key join in Kafka Streams similar to many articles (like this for one: https://www.confluent.io/blog/data-enrichment-with-kafka-streams-foreign-key-joins/).
When I ...
- 93
2
votes
1
answer
898
views
Importing json data into postgres using kafka jdbc sink connector
I'm new to kafka.
My requirement is to load json data from kafka topic to postgres table.
Below is the table structure
CREATE TABLE dup_emp (
emp_id integer PRIMARY KEY,
emp_name text,
...
- 161
1
vote
2
answers
5k
views
Why does an Avro field that was string now require avro.java.string type?
In Avro IDL I have a Message record defined as follows:
record Message{
MessageId id;
array<string> dataField;
}
I am using this record in another record with a null union:
record ...
- 413
1
vote
1
answer
662
views
Custom Kafka connector build fails in kafka-connect-maven-plugin
I am developing a custom Kafka Connect connector. I build the project with Maven using
io.confluent:kafka-connect-maven-plugin:0.12.0 (latest and greatest version of the plugin, as far as I know), ...
- 545
4
votes
1
answer
3k
views
Kafka Streams groupByKey not working due to RocksDB in Mac m1
I'm getting the below rocksdb error when I try to use groupByKey feature of kafka streams in a function. whereas a simple consumer function works fine.
Env: confluent v1.30.0 (confluent running on ...
0
votes
1
answer
494
views
what is the best approach to listen and process messages concurrently in kafka?
I am working on a Kafka application to process requests from topic-request-A and call microservice and publish the response to topic-response-A.
I have 100 microservices to be called, so I have 100 ...
- 1
1
vote
1
answer
911
views
KsqlClientException: Received 401 response from server: Unauthorized. Error code: 40100
Trying to connect to Confluent hosted KSQL db.
String username = API_KEY;
String password = API_SECRET;
String CONFLUENT_CLOUD_HOST = "pksqlc-xxxxx.us-east-2.aws.confluent....
- 454
0
votes
2
answers
1k
views
kafka JDBC source connector not able to pick up postgress table
i have a standalone kafka runing on windows and a JDBC source connector connected with postgress db which is available on my machine. when i try to run the connector it shows a warning that no tasks ...
- 273
0
votes
1
answer
270
views
Is Confluent required for JDBC source and sink connectors?
Do I need to use Confluent specifically for Kafka JDBC source and sink connectors, or can I use default apache Kafka implementation? I tried to start connect-standalone with a JDBC source config as a ...
- 273
0
votes
1
answer
1k
views
Spring boot TopicRecordNameStrategy usage consumer reading a recordType relavent to it
I am trying an example to publish event to kafka via two different API's. One takes Create events and other takes update events of same topic users.
Both these API's publishes messages to same kafka ...
- 57
0
votes
1
answer
2k
views
Spring Kafka: How does kafkaTemplate executeInTransaction method play with Consumer's read_committed isolation level
I am seeing that the isolation.level=read_committed consumer property ensures that only committed messages will be read by the consumer. I am trying to understand what is meant by a committed message ...
- 2,762
0
votes
0
answers
2k
views
Kafka Java client Timed out waiting for a node assignment
I have installed Confluent Platform on a Centos7 VM. It works fine from within the VM's guest OS.
Now I want to connect to Kafka through a Java client from the Host OS which is windows 10. However, ...
- 1,774
1
vote
1
answer
642
views
Strategy to choose when doing Serialization and Deserialization using spring-kafka library
I need to Serialize or Deserialize any type of Java Object may be Integer/ String or <T> or User or Account in my project. There might be more than 1 type I am not sure which one to use while ...
- 161
1
vote
1
answer
422
views
Issue with Kafka Connect with Solace Source Connector and JDBC SInk Connector
I want to use kafka connect solace source connector with avro converter to push messages to kafka topic. Once messages published on kafka topic, I want to use kafka connect jdbc sink connector with ...
0
votes
0
answers
417
views
kafka connect failed to start (Caused by: java.lang.SecurityException)
Trying to install confluent platform in my macOS (11.2.3)
But the connect can not start. I using java 8 update 281.
Followed the suggestions of this stackoverflow answer but no luck.
Starting ...
- 667
0
votes
1
answer
403
views
Spring Boot : REST API Endpoint to start & stop SSL enabled KAFKA Consumer : asking for authentication
I have the following REST API Endpoint to start & stop SSL enabled KAFKA Consumer however it is asking for authentication.
@RestController
@RequestMapping(value = "/kafka/consumer")
...
- 516
0
votes
0
answers
438
views
Kafka - Topics, Consumer Groups And Consumer Replicas
I'm new to Kafka and would like to know the best approach for configuring the topics, partitions, consumer groups and consumer app replicas.
Working on an existing setup, the configuration handed down ...
- 197
0
votes
1
answer
900
views
kafka streams serilaization error: java.lang.String cannot be cast to [Byte], issue in using json data
I am new to kafka streams and i face an issue when i try to do aggregations on json data.below is my streams code, i have copied the code and the sample input below and the error , i am using kafka ...
-1
votes
1
answer
1k
views
Kafka Streams 2.6, Partition Assignor and Rebalancing Strategy
In my current Kafka version which is 2.6, i am using Streams API and i have a question. When i start a stream, it writes Streams,Admin,Consumer and Produces configs. I noticed something strange that ...
- 27
1
vote
0
answers
267
views
Kafka how to prevent using old records when joining multiple streams
i'm fairly new to kafka and trying to join bunch of streams on a unique key. I have two different streams that i get from listening on two different topics. I repartition and join them by their ...
- 213
5
votes
1
answer
705
views
Confluent Cloud - Spring Boot Consumer REST Endpoint?
I am trying to build a Java Spring Boot application that would post & get the messages from Confluent Cloud Kafka.
I followed the article for publishing a Kafka message into Confluent Cloud and it ...
- 516
0
votes
0
answers
1k
views
Update the Apache Kafka Spring boot solution to connect with Confluent Cloud Kafka?
I have a Java Spring Boot application that connects with Apache Kafka
KafkaController.java
package com.demo.controllers;
import com.demo.engine.Producer;
import org.springframework.beans.factory....
- 516
2
votes
1
answer
882
views
Teradata and Apache Kafka using Kafka connect
Apologies if my question is all over the place but I am new to Apache Kafka and currently I am trying to import data from a Teradata table (on a VM) into Kafka using my Windows OS. So far I've done ...
- 913
3
votes
0
answers
235
views
Unable to send message to kafka topic from neo4j stream
I'm trying to send messages from neo4j stream to kafka topic.
this is my docker-compose configuration for neo4j
version: '3'
services:
neo4j:
image: neo4j:3.5.8-enterprise
hostname: klinks
...
- 2,307
1
vote
0
answers
594
views
Confluent Connect, Debezium, java.lang.OutOfMemoryError
We are hosting a cluster of Confluent Connect v5.01 containers in Amazon EKS to run several instances of the Debezium for SQL Server connector v0.9.5. Sometimes when we reconfigure one of these ...
- 4,208
0
votes
1
answer
697
views
Not able to fetch records from SAP HANA using Kafka Connect
I am new to Kafka Connect and I am trying to replicate/fetch data from SAP S/4 HANA and persist it on HDFS using Kafka Connect. Till now I have tried quite a lot of things by following these links:
...
0
votes
1
answer
2k
views
Could not transfer artifact io.confluent:kafka-connect-storage-common-parent:pom:6.0.0-SNAPSHOT from/to confluent (${confluent.maven.repo})
I am trying Kafka connect for the first time and I want to connect SAP S/4 HANA to Hive. I have created the SAP S/4 source Kafka connector using this:
https://github.com/SAP/kafka-connect-sap
But, I ...
4
votes
2
answers
10k
views
Kafka Avro deserializer is not able to deserialize the Kafka message of a specific Avro record
I am trying to deserialize Avro messgaes that are in Kafka to POJOs generated from Avro schema. I am using KafkaAvroDeserializer for this conversion.
I am able to see the GenericRecord in the ...
- 382
1
vote
1
answer
2k
views
How to read Kafka message header from a kstream apllication
Is it possible to read Kafka message header which I set in my kafkaProducer application from kstream application? My KafkaProducer looks like this; I have set header in my message
public class ...
- 139
0
votes
1
answer
460
views
CONFLUENT -- Java Error --- Error:(16,114) java: illegal escape character [duplicate]
I am having trouble with Java escape characters. I have these from Confluent ccloud configs example.
but does not work . both give me the same error message Error:(16,114) java: illegal escape ...
2
votes
1
answer
687
views
Kafka - non-ZooKeeper-based consumers Vs ZooKeeper based consumers
While I am going through the documentation, I come across the following phrase
https://kafka.apache.org/20/documentation.html#basic_ops_consumer_lag
bin/kafka-consumer-groups.sh --bootstrap-...
- 2,037
3
votes
3
answers
3k
views
Can kafka partitions be spread across multiple kafka cluster nodes?
My application has a list of kafka cluster nodes specified in the spring.kafka.bootstrap-servers property and listens to topics on all these nodes.
If I were to create a topic on one of these nodes, ...
- 358