75 questions
0
votes
0
answers
27
views
Catching CompletionException from kotlin function
My ultimate goal is to find the number of partitions for a Pulsar topic (Getting number of partitions of a Pulsar topic: could not be parsed into a proper Uri, missing scheme) but in the process of an ...
- 3,321
0
votes
0
answers
19
views
Getting number of partitions of a Pulsar topic: could not be parsed into a proper Uri, missing scheme
I'm unable to get the number of partitions from a topic via the PulsarAdmin API:
val admin: PulsarAdmin =
PulsarAdmin
.builder()
....
- 3,321
0
votes
0
answers
11
views
Flink does not subscribe to all topics if parallelism is more than 1
I am working on a Flink job that reads data from two different Pulsar topics using the Flink Pulsar connector. The job works as expected when the parallelism is set to 1. However, when I increase the ...
1
vote
0
answers
20
views
Pulsar Reader seek for topics with partitions
TLDR; How is
void seek(Function<String,Object> function)
throws PulsarClientException
in the pulsar reader API expected to be used?
-- More info
I'd like to read messages from a pulsar topic ...
- 3,321
0
votes
1
answer
28
views
How to config Message Filtering of Debezium connector for PostgreSQL?
Debezium provides the filter single message transform (SMT).
https://debezium.io/documentation/reference/stable/transformations/filtering.html#set-up-filter
If I want to add the following ...
0
votes
0
answers
14
views
Pulsar MoP - MQTT - Port 1883 Connection Refused - When Both Pulsar and Client are in separate Docker containers
I have pulsar in one container and docker in another container.
I am using docker desktop.
Scenario 1: Mosquitto Broker in one docker container and Mosquitto Client in another.
This works fine and I ...
0
votes
0
answers
30
views
In Pulsar, is there a good way to override the publish time property for messages for testing purposes?
I have logic that reads past messages from a pulsar topic and sends them with the wait times between the sent messages matching the original timestamp differences.
For example:
MessageA sent at ...
- 3,321
0
votes
0
answers
27
views
springboot 3.3.1 user pulsar plugin @PulsarListener with KeySharedPolicySticky policy question
I try to use @PulsarListener springboot 3.3.1 Component to use SubscriptionType.Key_Shared to consumer data with two different consumer service, need to consumer with with different key please see ...
0
votes
1
answer
95
views
Pulsar auto delete policy
I changed the configuration in pulsar's configuration file broker.conf to
brokerDeleteInactiveTopicsEnabled: true
brokerDeleteInactiveTopicsMode: delete_when_no_subscriptions
...
0
votes
0
answers
21
views
Python Pulsar Message to InfluxDB v2 Sink-Connector
i am trying to send a pulsar message with a python producer which should be processed by an InfluxDB v2 Sink-Connector. Especially the message content / payload in combination with the used producer ...
- 1
1
vote
0
answers
79
views
Can pulsar consumer subscribe to partitioned topic with partition N by regexp?
I have simple pulsar consumer
pulsar_client.subscribe( topic=re.compile('persistent://public/default/some_topic_[a-zA-Z0-9_\-]+1$'), subscription_name=consumer_group, consumer_type=ConsumerType....
0
votes
0
answers
14
views
Need way to remove Consume throughput received logger
I am using logstash-pulsar-input plugin in which i can see jar org.apache.pulsar:pulsar-client, 2.11.0 being used and while checking on that found ConsumerStatsRecorderImpl class is printing logs of ...
0
votes
0
answers
22
views
PulsarIO.read() failing with AutoValue_PulsarSourceDescriptor not found
What happened?
I have configured PulsarIO plugin via Beam to read messages from Pulsar as below:
PCollection<PulsarMessage> pCollectionAll = p.apply("ReadPulsarMessage", PulsarIO
...
0
votes
1
answer
164
views
Apache Pulsar Node js app - consumer.receive function has blocked the whole functionality
I am reading messages continuously. I want to close my client and consumer if no message is there to read.
The below code is working fine if there are messages to read but after reading all the ...
- 490
0
votes
0
answers
118
views
Apache Pulsar: Error Checking/Getting Partition Metadata while creating producer
When attempting to create a topic on Apache Pulsar during testing using Node.js, the following errors occurred:
[[::1]:53665 -> [::1]:6650] Failed partition-metadata lookup req_id: 10 error: ...
- 53
0
votes
1
answer
144
views
Pulsar backlog message count always returned as zero
I want to know the number of messages in a pulsar partitioned topic that isn't picked up by any of the consumers yet. My application is in python so I am using the REST APIs. I tried both /admin/v2/...
- 269
0
votes
1
answer
128
views
Issue with Pulsar GeoReplication
I have followed the steps and completed the Geo replication setup as per the pulsar documentation
https://pulsar.apache.org/docs/3.1.x/administration-geo/
All the geo replication related setup was ...
2
votes
1
answer
103
views
How to deploy pulsarra.rar in Jboss?
I'm trying to deploy and configure DataStax Apache Pulsar JMS Client Resource Adapter in JBoss but I'm getting the following error:
2023-08-28 11:21:30,684 INFO [org.jboss.weld.deployer] (MSC service ...
- 623
0
votes
0
answers
196
views
Flink SQL connector for Pulsar deserialization issue with Avro schema
When using Apache Flink SQL connector for Pulsar (link), I consistently get the Avro deserialization problem when trying to read from a Flink table that is backed by a Pulsar topic (having an Avro ...
0
votes
2
answers
303
views
Pulsar client numIoThreads and numListenerThreads per consumer instance or for all consumer instances
Let's say I have the following pulsar config:
pulsar:
topicX:
client:
config:
numIoThreads: 3
numListenerThreads: 15
consumer:
config:
receiverQueueSize: ...
- 276
1
vote
1
answer
442
views
Spring boot org.apache.pulsar.client.api.Message.getValue() seems to lock any line of code it's included in. How do I stop it from doing this?
I am using spring boot to connect to a Pulsar message broker but I cannot access the message payload no matter what class I use in the call to:
//...
void listen(<T> message)
//...
I'm using ...
- 55
0
votes
1
answer
162
views
Camel-Pulsar How to add Transaction in Route
We have been using apache camel with JMS to route and process messages. We are now considering switching from JMS to using Pulsar. I have been experimenting with the camel-pulsar component but can't ...
- 1
1
vote
0
answers
327
views
Pulsar does a infinite redelivery of message when maxredelivery count is breached for DLQ
I have simple setup where in i have pulasr consumer subscribing to a topic with DLQ of maxredelivery count= 2 on shared subscription type.
So when the message is consumed we send the message via ...
- 11
1
vote
1
answer
699
views
Pulsar producer send_async() with callback function acknowledging the sent message
I have a use case where messages from an input_topic gets consumed and sent to a list of topics. I'm using producers[i].send_async(msg, callback=callback) where callback = lambda res, msg: consumer....
- 13
1
vote
1
answer
185
views
How to run Push-Queries with Apache Pulsar?
I love the push queries (called continuous queries in Apache Flink) of Apache ksqlDB. https://developer.confluent.io/learn-kafka/ksqldb/push-queries-and-pull-queries/
It allows to get notified via ...
- 2,058
2
votes
1
answer
345
views
Make sure messages goes to a specific consumer
I have multiple consumers subscribed to the same Pulsar topic. How do I make sure certain messages go to specific consumers?
The closest thing I understand is the key-shared consumer type. However, ...
- 23
0
votes
1
answer
877
views
How to send JsonArray data in apache-pulsar-client?
I am a beginner who just started developing pulsar-client with spring boot.
First of all, I learned the basics through pulsar doc and git, but I was stuck testing batch transmission of messages from ...
- 45
1
vote
0
answers
260
views
pulsar-admin sink localrun failure: ClientCnx Failed to authenticate the client. (tls auth enabled)
command:
bin/pulsar-admin sinks localrun -a connectors/pulsar-io-mongo-2.10.2.nar --tenant public --namespace default --inputs up-20wt --name mongo-sink --sink-config-file work/config.json --...
- 11
3
votes
1
answer
151
views
Excluding consumer of a topic by sender in Pulsar
Intro
We're developing a system to support multiple real-time messages (chat) and updates (activity notifications).
That is, user A can receive via Web Socket messages for :
receiving new chat ...
- 2,470
3
votes
0
answers
323
views
Is there any Pulsar UI where messages can be updated/produced?
The only UI I found was Pulsar Manager , which don't even allow to see the messages in a topic.
By any chance is there a Pulsar UI with better features where messages can be manipulated?
- 41
1
vote
0
answers
434
views
How to move specific messages from one topic to another on demand in Pulsar (using pulsar-admin)?
I have a requirement of moving specific messages from DLQ to RETRY queue. I want to use pulsar-admin for this task.
I have message-id for all the messages.
~note: don't want to copy and produce ...
- 41
1
vote
1
answer
391
views
java, apache pulsar Topic partition
If I have a topic called var test = "test"
and I create a bean:
@Bean(name = "test")
public Producer<Test> testProducer(PulsarClient pulsarClient){
return pulsarClient....
1
vote
1
answer
265
views
Not able to connect to PulsarIO using Apache Beam Java sdk
while executing below code to connect to apache pulsar using apache beam PulsarIO in java sdk. Getting below error while adding pulsar client in beam pipeline.
Beam version 2.40, 2.41
javaSE 1.8
...
- 21
0
votes
1
answer
418
views
PulsarClient.builder() extremely long error logger
I am trying to make a pulsar client written in java, however I get an error with the logger(slf4j). I thought that it was because my PropertyConfigurator was badly made, but even if I use the ...
2
votes
3
answers
784
views
Apache pulsar / Pulsar Manager
I have installed apache pulsar using helm chart in AWS EKS,
Reference : Apache Pulsar Helm Deployment on Kubernetes
also enabled pulsar manager and deployed pulsar manager UI, all pods are up and ...
- 21
0
votes
0
answers
380
views
make_shared<string> segment fault
when I use make_shared, the program will coredump, even the line never been called. when I remove make_shared it works fine.
#include <pulsar/Client.h>
using namespace pulsar;
using namespace ...
0
votes
2
answers
775
views
Unexpect error 'Pulsar error: ServiceUnitNotReady' when try to connect to pulsar
Hello I was debuging a pulsar code, and I was working perfectly, but one time after a re-execution of the code suddenly show a error, of when try to connect:
Loading .env environment variables...
0000-...
- 5,232
0
votes
1
answer
995
views
how to remove the pulsar message from queue for a topic after message is consumed by a consumer application?
i have application which consumes topic from producer application using pulsar. i need a logic or some configuration to remove the messages from queue for a topic once it is consumed by my consumer ...
0
votes
1
answer
945
views
Pulsar Consumer is not consuming fast enough
I have a pulsar client and consumer code as below.
PulsarClient client = PulsarClient.builder()
.serviceUrl(pulsarServerUrl).enableTlsHostnameVerification(false)
....
-3
votes
1
answer
1k
views
How to insert to clickhouse asynchronously with callback function in golang?
Now I want to insert some data from kafka to clickhouse. Is there any way to sense failure after asynchronous insertion failure because of ack?
- 143
1
vote
0
answers
25
views
Is exclusive type more resource efficient than shared type in Pulsar?
I found that it will cost lost of memory when using shared type of subscription; So will exclusive mode save more resources?
pulsarConsumer, err := pulsarClient.Subscribe(pulsar.ConsumerOptions{
...
- 143
1
vote
1
answer
240
views
Pulsar with source connector Rabbitmq error
I am trying to use as source the RabbitMQ for Apache Pulsar.
I run locally the binaries of Pulsar and I use the offical RabbitMQ connector. The RabbitMQ is running in server. For few moments can I see ...
- 386
1
vote
1
answer
446
views
How can I read all topic?
When I create the consumer
consumer = pulsar.Client(
PULSAR_URL,
authentication=AuthenticationOauth2(params)
).subscribe(
topic=PULSAR_TOPIC,
...
- 5,232
1
vote
0
answers
178
views
Pulsar Manager is not deployed when following the pulsar kubernetes deployment guide
According to this guide, pulsar manager should be installed when you install the helm chart. When I follow the steps, pulsar manager does not show up as a pod or service. I am deploying to an Azure ...
- 11
1
vote
0
answers
306
views
Deploy Apache Pulsar cluster, on Azure (terraform)
I need to create a Pulsar cluster in a completely automatic way. First, the machines are created, then the software is installed. I am looking for such a solution, but the examples available on the ...
- 11
1
vote
1
answer
96
views
Apache Pulsar on hybrid cloud
I want to use Apache Pulsar on a hybrid cloud where I want to deploy Apache Pulsar cluster on cloud.My subscriber will be on cloud and consumer will be deployed on premises. Is it possible to have ...
- 11
0
votes
1
answer
246
views
Debezium io with pulsar
I want to understand how pulsar uses debezium io connect for CDC.
While creating the source using pulsar-admin source create, how can I pass broker url and authentication params or client. Similar to ...
0
votes
1
answer
365
views
pulsar compression/decompression cycle
I am curious about how compression works in pulsar. from the public doc, it states "You can compress messages published by producers during transportation"
does it mean client compress the ...
- 600
1
vote
1
answer
703
views
Replay / Reset messages in Apache Pulsar
Is it possible to replay old messages (within retention period)? There is no documentation about it. I would like to do this based on a timestamp or period.
A hint in the right direction would be much ...
- 682
1
vote
1
answer
1k
views
Pulsar function fails to deserialize message because of wrong schema type (JSON instead of AVRO)
When running Pulsar in docker as standalone, we are facing this weird issue when deserializing the message in the specific case.
We are using version 2.7.1.
We have a script creating topics and ...
- 200