23 questions
0
votes
0
answers
70
views
Payload compression at Debezium Server. "Debezium Server - Pub/Sub Pipeline"
We have a setup where we are using Debezium to read the logs from MYSQL, sending them across to Pub/Sub, and then transferring them to BigQuery via BQ Subscriptions. This pipeline is working fine but ...
- 166
0
votes
0
answers
37
views
Debezium with custom Driver for mysql 5.1
I'm new to debezium CDC and I tested using debezium-server to stream data from mysql 5.5 to pubsub sink. I know the version of mysql is very old and not officially supported based on debezium ...
- 11
0
votes
1
answer
86
views
Debezium server job performance issue
I am running debezium server (without Kafka) on linux server and it is having performance issue. There is lag of around 12-14hrs from reading data from Oracle to source to sending it to Azure Eventhub....
0
votes
1
answer
219
views
debezium standalone server with http sink
we are running Debezium standalone server to capture the changes from postgresql database, and when a change is capture, we want it to send to a http endpoint. It works perfectly fine, however I can't ...
- 557
0
votes
0
answers
33
views
Debezium have any tools to rescan table?
I need to ensure that I have all actual data in destination table. I need to rescan table at least one time per day. How to write all rows from table in kafka from debezium for one table?
Example: I ...
- 19
0
votes
1
answer
337
views
Debezium Server with NATS JetStream No Responders error
I tried to setup CDC with NATS JetStream. I get the following error:
Error Publishing: 503 No Responders Available For Request
Tell me where did I go wrong?
Debezium full log: https://logpaste.com/...
- 9,413
0
votes
0
answers
114
views
Debezium SQL Server is throwing OutOfMemory Error after a restart
We are using debezium SQL Server Connector v2.6.1.Final & Kafka Connect v3.7.0 to send data from SQL Server database to Kinesis 1.14.8 and from there to our downstream
So the initial snapshot we ...
0
votes
0
answers
149
views
Debezium Server - GCP PubSub - Add a field from outbox table to the header
I'm working with Debezium server and an outbox table to publish a message to PubSub. I've added a filter in the subscription level of pubsub topic to filter out the messages. In order to work this ...
- 1
0
votes
0
answers
117
views
How to use a unique sorting key for Debezium with GCP Pub/Sub
I am using Debezium to capture changes for all tables in my database.
I am rerouting every event into a single GCP Pub/Sub topic so that my worker can consume those messages in the same order they ...
- 38
1
vote
0
answers
29
views
Is there any parameter for debezium connector to do not change numeric type?
I want to prevent debezium connector and sink also to do not change numeric type from numeric to double precision between two postgres databases.
- 11
2
votes
0
answers
460
views
Debezium UI for nats jet stream
We are using NATS Jetstream for streaming messages from debezium connector. I want to create UI for debezium to see changes.
I have seen in the documentation about this
link, here they have mentioned ...
- 21
0
votes
1
answer
480
views
java.util.concurrent.ExecutionException: org.apache.kafka.connect.errors.ConnectException: java.nio.file.NoSuchFileException: \tmp\connect.offsets
I am running kafka and debezium PostgreSQL connector locally on Windows. everything else is good, topics are also stored. just getting this error:
java.util.concurrent.ExecutionException: org.apache....
- 138
1
vote
1
answer
494
views
Track a particular table using Debezium
I am using Debezium embedded oracle connector for spring to track changes in my tables. Is there a property/way to track a particular table from a schema.
My schema has a lot of tables and I do not ...
- 11
0
votes
0
answers
180
views
Debezium server fffset file storage claim or cleanup
So, for non-kafka deployment, we specify offset storage file with debezium.source.offset.storage.file.filename as per documentation, how does the storage is reclaimed or cleanup happens?
I am planning ...
- 3,033
0
votes
2
answers
974
views
Debezium 2.0 confluent AvroConverter is not working
Since debezium 2.0 has decided to remove the io.confluent.connect.avro.AvroConverter I tried to inject it directly but it keep giving me following error:
Here is my simplified deployment:
apiVersion: ...
- 1,208
0
votes
0
answers
163
views
Debezium Server source connector for GCP cloud SQL Server without Kafka
I am trying to use Debezium server (Standalone without Kafka) to load data (incremental) from GCP Cloud SQL Server to PubSub and then BigQuery.
I am running a test on my laptop to see if that's doable....
- 650
0
votes
0
answers
349
views
Debezium sql connector status api is not showing exception
we are trying to upgrade debezium-connector-sqlserver-1.0.0. to debezium-connector-sqlserver-1.9.5..After upgrade ,the debezium connector status api always shows running status ,even if connector has ...
- 31
1
vote
0
answers
166
views
Debezium Server MongoDB connector Enabling the Signalling doesn't work
I run the Debezium Server with source MongoDB and sink Kinesis on k8s. After enabling the signalling by setting
DEBEZIUM_SOURCE_SIGNAL_DATA_COLLECTION="test.debezium_signaling"
...
- 836
0
votes
0
answers
415
views
Task becomes UNASSIGNED for Debezium MySQL source connector
I am using debezium 1.9. I created a connector using below config
{
"name": "user_management_db-connector-5",
"config": {
"connector.class": &...
- 839
1
vote
2
answers
2k
views
Debezium server error: java.lang.OutOfMemoryError: Java heap space
Debezium server: v 1.9.0.Final
MongoDB Atlas: v 4.2.20
Running on AWS ECS with Fargate w/ 1GB CPU & 4GB MEMORY
Overview:
Debezium starts an initial snapshot and it sends some data to kinesis but ...
- 1,546
1
vote
2
answers
4k
views
Debezium Postgres Connector "After applying the include/exclude list filters, no changes will be captured"
I am using Debezium (a Kafka Connector) to capture Postgres database changes and I am getting an error from Debezium. Does anyone know what the error below means and perhaps offer a suggestion to fix ...
- 7,171
0
votes
1
answer
384
views
Debezium server configuration for Redis client
I'm running Debezium Server that streams into Redis. I want to pass "socket timeout" and "connection timeout" configurations to Jedis (the redis client that debezium is using), and ...
- 436
2
votes
1
answer
815
views
Debezium server docker cannot access AWS credentials
My goal is to use Debezium for real-time CDC from MongoDB to AWS Kinesis. I am using the official Debezium server 1.8 docker image. I'm able to run the image locally and I've configured the settings ...
- 1,546