186 questions
0
votes
1
answer
31
views
Kafka MM2 Replication duplicated
I have two kafka clusters and a kafka connect instance, use mm2 to replicate messages, but I face data duplication.
source Kafka cluster: xx.xx.xx.184:21001,xx.xx.xx.184:21002,xx.xx.xx.184:21003
...
- 11
0
votes
0
answers
15
views
Need connector/client-id information while getting MM2 kafka.connect.mirror based metrics
I am trying to get "ReplicationLatencyAvg" for MM2 replication based on kafka-connect framework. In my MM2 cluster, multiple connectors are created to support multiple source -> 1 ...
0
votes
1
answer
28
views
Kafka mirror maker 2 copy topic as avro not bytes
i have a kafka cluster in the US and i would like to replicate a topic onto an EU cluster. these 2 cluster share the same schema registry. i snapshot a table and can see the messages as avro on the NA ...
- 67
0
votes
1
answer
34
views
Kafka MirrorMaker2 cannot connect to kafka node with TLS
I have two k8s setups on which kafka clusters are running. On my origin setup no TLS or authentication is required. My target setup has both SCRAM-SHA-512 and TLS setup.
The config of my ...
0
votes
0
answers
24
views
Preserving offsets in Kafka migrations with MirrorMaker
When migrating kafka clusters with MirrorMaker, I've found that offsets are not preserved. That is a problem because it means after the migration some stateful services that I have won't be able to ...
- 480
2
votes
0
answers
75
views
kafka MirrorMaker2 - stop offset sync when consumer group idle on source cluster
I'm figuring out a way to migrate a Kafka cluster from v.2.3.0 (with Zookeper) to a new one v.3.7.1 (with Kraft). For the migration we can stop consumer/producers but without message loss. For this ...
- 21
0
votes
0
answers
22
views
Kafka MirrorMaker2 not replicating consumer group offset
When I create a new topic in source cluster with few messages, the new topic will be replicated successfully to target cluster in few seconds. Then I create a consumer group in source, consume 1 ...
0
votes
1
answer
257
views
Kafka - MirrorMaker 2.0 production deployment
Question: How to Configure Docker for Kafka MirrorMaker 2.0 to Replicate Messages Between Two Clusters?
I have two Kafka clusters: a source cluster and a destination cluster, each with 3 brokers. I ...
0
votes
1
answer
100
views
Minimal MirrorSourceConnector configurations to replicate data to another Kafka instance
I am trying to replicate topics from one Kafka cluster to another using Mirror Maker.
I am using docker compose to create the following resources:
networks:
local_kafka:
name: local_kafka
...
- 306
1
vote
1
answer
179
views
Kafka Migration with MM2 and Flink: How to Handle Offset Changes and Savepoints?
I'm currently migrating a Kafka cluster using MirrorMaker 2 (MM2). My plan is to switch my Flink Kafka connector to the new cluster once the migration is complete.
Ideally, I want Flink to resume ...
- 13
0
votes
1
answer
135
views
How to replicate blank topics in Mirror maker 2
As a part of migration from one AWS MSK (v2.2.1) to another AWS MSK (v3.5.1), we are planning to make use of mirror maker 2 to replicate topics and topic events. However i was able to setup the ...
0
votes
1
answer
34
views
Is there any way to config the number partion topic checkpoint of mirrormaker2
I use mirrormaker 2 to sync kafka A-> B. I want to increase performance consunmer group synchrony by increasing partition of topic checkpoint. Is there any way to config the number partion topic ...
0
votes
0
answers
304
views
How to connect mirror maker 2 runing on kafka connect to kafka with SASL_PLAINTEXT authentication
I want to configure syncing message from Cluster A to Cluster B kafka. I am using kafka mirror maker 2 running on kafka connect. When I configure hearbeat connector, I got the error as follow:
[2024-...
0
votes
1
answer
66
views
Migrate data from Confluent Kafka to Bitnami Kafka
We have a very old version of Confluent Kafka running on our kubernetes cluster, cp-kafka:5.4.1 and we are now moving to the bitnami kafka latest version. How can I migrate all my data from my old ...
- 71
0
votes
0
answers
52
views
Apache Kafka Mirror Maker 2 not wanting to work
I have setup a mirror maker 2 instance up in my vmware but when I go to run the program it does not seem to do anything. I followed the guidance from the Readme and KIP-382 to get this setup. My ...
- 1
0
votes
0
answers
67
views
Apache Kafka Mirror Maker 2 - The selected topic is not replicated
Can you help me solve the following problem?
The data is not replicated to the other server.
There are two servers: 192.168.150.151 and 192.168.150.152. Kafka runs in a docker container in both.
The ...
- 1
0
votes
0
answers
38
views
How can MirrorMaker2 start mirroring from the next message after changing destinations
I have MM2 replication enabled from ClusterA->ClusterB, and in case ClusterB goes down I need to start MM2 replication to another secondary ClusterC. i.e. ClusterA->ClusterC.
Similar to this ...
0
votes
0
answers
56
views
Kafka mirror maker: Data in some topics are not replicated
Using mirror maker to replicate data across two geo -redundant sites.
I am not talking about internal topics.
But, data in some of the external topics that are designed to be replicated are not ...
- 598
0
votes
0
answers
46
views
Can Kafka Mirror Maker failover between 2 Kafka clusters?
When using MirrorMaker for a forwarding flow like A->B, B->C what are the
options for handling failure of B so that C could then mirror from A?
From what I can see MirrorMaker has no concept of ...
- 29.1k
0
votes
1
answer
88
views
Can we perform a live migration between two different Kafka version from onprem to cloud using Kafka Mirror?
We have a use case to perform kafka migration lively, as per our understanding its better to do sync across the same Kafka version using Kafka mirror and then do the upgrade
have any one performed ...
- 3
0
votes
0
answers
95
views
Mirrormaker: Not authorized to access topics
I've installed the redpanda on 2 EKS clusters thosr are in us-east-1 and us-east-2 regions. I've enabled the replication from us-east-2 cluster to us-east-1 cluster. I've configured kafka mirrormaker ...
0
votes
0
answers
107
views
Mirror Maker is failing with ConnectRestException
We are using Apache Kafka Mirror Maker 2.0 to replicate the data/records from One DC to another DC. We have Mirror Maker cluster of 3 nodes. We don't have much load in terms of messages per seconds, ...
- 162
0
votes
1
answer
186
views
kafka mirrormaker setup to replicate data between 2 redpanda cluster
I've deployed the redpanda in 2 EKS cluster and redpanda broker pods are up & running. One EKS cluster is in us-east-1 region and one is in us-east-2 region.
Now I need to replicate the data ...
0
votes
1
answer
178
views
AWS MSK Connect MirrorMaker2 - Failed to Flush
Using MirrorMaker2 on MSK Connect (2.7.1) the source connector is throwing the following error:
[Worker-0d8c5a576b5ef6e99] [2023-12-22 16:01:00,771] ERROR [msk-dev-conv-mm2-sourceconnector|task-1|...
- 1,493
0
votes
1
answer
166
views
where are the kafka mirrormaker2 consumer groups
I have added a mirror maker 2 source connector to my connect cluster:
{
"name": "test_mirror",
"config": {
"connector.class": "org.apache.kafka....
- 20.2k
0
votes
2
answers
341
views
Kafka Mirror Maker snappy compression
I am trying to configure a mirror maker between two kafkas and the data mirroring works fine for topics without compression. When mirroring a topic with a snappy compression I get the following ...
- 342
0
votes
1
answer
348
views
How does migrated kafka consumer behave after repartitioning while cluster migration?
I have couple of conceptual doubts on the behaviour of consumers.
Cluster migration gives great opportunity while repartition topics which are over partitioned or under partitioned in current cluster.
...
- 21
0
votes
1
answer
409
views
MM2 offset synchronization when consumer group not created on target due to non activity
I have secondary cluster, that till disaster has no consumers.
while DR this cluster need be consumed by same consumer groups as on Primary and it requried to consume with same offset
if to set sync....
- 5
0
votes
0
answers
182
views
Mm2 on kafka connect stops replicating topic after DisconnectException
I have 3 kafka-connect nodes, running mm2 tasks.
Those task replicate 4 topics from remote uk and us clusters into single uk kafka cluster.
Everything works fine most of the time, but at some point I ...
- 939
1
vote
1
answer
387
views
Kafka Mirror to Apache Kafka from AWS MKS Cluster
I'm confused about the architecture of mirroring a topic among an AWS MKS cluster (source) to another Apache Kafka cluster (target).
The source cluster also uses AWS Glue Schema registry, so I need to ...
- 41
0
votes
2
answers
557
views
Strimzi mirrormaker2 consumer offset not getting translated to target cluster
We're using Strimzi MirrorMaker2 to replicate some topics from one kafka cluster to another kafka cluster (unidirectional) for DR purpose.
As a poc, I have created a sample fast producer producing to ...
- 2,243
0
votes
1
answer
694
views
Mirror Maker 2.0 performance tuning optimisation
I am using Mirror Maker 2.0 to replicate the data from one cluster to another cluster. I have done the setup and it is working fine with few topics.
I am facing issues when we are trying to increase ...
- 162
0
votes
1
answer
263
views
Copy data from one Kafka cluster to another in different Kubernetes cluster
I have two Kubernetes clusters located on two different networks.
On each k8s cluster, a Kafka cluster is deployed using Apache Kafka packaged by Bitnami.
I would like to copy data from one Kafka ...
- 145
0
votes
0
answers
429
views
MirrorMaker 2 - Event Hub connection terminated during authentication
The goal
We are trying to mirror a topic on a source Kafka (open source variant) to Azure Event Hubs using MirrorMaker 2. I have reviewed the documentation of both Kafka and Event Hub. MirrorMaker 2 ...
0
votes
0
answers
308
views
How can I verify all the messages in a source topic are replicated to a target (remote) topic in Kafka using MirrorMaker2
Looking for a simplw way to verify all messages in a source topic are replicated to a remote topic using MM2, specifically topics receiving high throughput/volume.
Here's my simple deployment,
...
- 135
0
votes
1
answer
1k
views
Distributed Mode in Dedicated MirrorMaker 2 cluster - not balancing tasks evenly
I've been trying to set a cluster in Kubernetes with nodes based on MirrorMaker, using Kafka version 3.5.1.
The setting I used is based on what is described in this KPI, and relates to the usage of ...
- 471
1
vote
1
answer
615
views
Kafka MirrorMaker 2 re-creating the deleted source topic in source for active-passive mode
I have a MM2 service running between 2 Kafka clusters in the Active-Passive mode. Upon deleting a topic from the source, MM2 is re-creating the topic again in the source node. Is is the expected ...
- 61
0
votes
1
answer
188
views
MirrorMaker 2 Avro format
it is necessary to transfer topics in avro format using MirrorMaker 2. is it possible to override
'key.converter'= 'org.apache.kafka.connect.converters.ByteArrayConverter',
'value.converter'=...
- 141
4
votes
1
answer
975
views
Kafka MirrorMaker 2 logging WARN message of TopicExistsException
I'm running MirrorMaker 2 for replicating the data between 2 Kafka clusters. After starting the service, and waiting for a while, the connect.logs contain multiple warning messages for each individual ...
- 61
1
vote
1
answer
579
views
Kafka MirrorMaker2 not replicating Kafka internal topics starting with "__"
I am trying to use MirrorMaker 2 to replicate the Kafka topics, but it doesn't replicate the topics starting with "__" (Kafka internal topics). What configs should I need to add in my MM2 ...
- 61
0
votes
0
answers
358
views
Kafka MM2 Offset Translation
I'm using the Confluent Platform Community Edition (7.4) as a new cluster and I wish to replicate data from my old 2.3 Apache Kafka cluster.
I setup Kafka Connect on the new side and tried to setup ...
- 93
1
vote
1
answer
613
views
Kafka connect mirror maker latency issue from cluster to cluster
I am having 2 Kafka clusters with 2 nodes each with 2 zookeepers for each cluster. They are located in the EU and the US physically. I have successfully configured the mirror maker with the below ...
0
votes
1
answer
639
views
How to Migrate Kafka Cluster with No Downtime Using MM2?
I need to migrate our production Kafka cluster to a new cluster with more capacity, and have been researching how to do this with no downtime.
This video has been helpful, but from my understanding of ...
1
vote
0
answers
438
views
MirrorMaker2 syncs some but not all consumer groups defined by pattern
We are running MirrorMaker2 with a SourceConnector and a CheckpointConnector via Strimzi and try to sync topics from our integration cluster to our dev cluster. Of 43 consumer groups, only 22 are ...
1
vote
1
answer
193
views
MirrorMaker2: There are no in-flight requests for node 0
We have set up mm2 to replicate data from an on-premise-kafka-cluster to azure eventhubs (one direction). (We have already done that in the past without any problems with two on-premise-kafka-clusters....
- 61
1
vote
1
answer
64
views
how to push data from kafka cluster east[KCL _1] to kafka cluster north and if KCL_1 fails switch to kafka cluster west[KCL_2] using MirrorMaker2
Scenario:
Two Data centers East DC(data center) and West DC both have same data stream and serve as backup of each other.
North DC needs continuous data either from East or West DC.
During my search ...
- 11
0
votes
1
answer
67
views
How do I produce an event to the topic in cluster2 from a consumer (cluster1) after some transition in the data?
I am just a beginner in Kafka. I need to consume kafka events from 2 different topics (A and A1) in PROD cluster and using the event payload I need to hit one REST api. The result from the rest api ...
- 1
0
votes
0
answers
506
views
Kafka MirrorMaker 2 - Rename Topic AND retain consumer offsets
I am looking to rename a large amount of topics (so we can create a virtual namespace for a second environment) i.e. production_topic1/development_topic1 etc.. However I also need to retain the ...
0
votes
1
answer
244
views
Dynamic Registration/Un-Registration of topics - leading to offset issues
We are trying to leverage Dynamic Kafka Topic Listening (based on approach https://www.geeksforgeeks.org/spring-boot-start-stop-a-kafka-listener-dynamically/) to handle disaster recovery of the Kafka ...
- 51
0
votes
1
answer
156
views
Does MirrorMaker2 replicates offset that are stored in zookeeper storage?
I am having Kafka consumer (zookeeper based) with client version 0.8.1.1 which uses zookeeper as offset storage (consuming from Kafka cluster 1.0.1). Now, I want to replicate the messages and offsets (...
- 3,463