177 questions
0
votes
0
answers
47
views
Tracing kafka message event in lmax disruptor ring buffer sequence
Below is the code for kafka message consumed and published within lmax disruptor ring buffer, now i need to implement tracing for each event may goes through multiple sequence. Before publishing the ...
- 1
0
votes
0
answers
21
views
Tracing in LMAX disrupter processing
I am looking for options to be able to fetch and track logs in the flow of handlers processing in LMAX ring buffer, i.e, for an particular message posted into ring buffer, get all its respective logs ...
1
vote
0
answers
110
views
Multi producer multi consumer ring buffer implementation using disruptor pattern
I was trying to implement disruptor pattern of multi-producer and multi-consumer ring buffer whose implementation are described here (for consumer) and here (for producer). I am observing ...
- 13
0
votes
1
answer
78
views
How to wait for all Disruptor messages to be consumed in a test case
I'm trying to unit test an application which publishes/consumes messages using the Disruptor ring buffer. In the test I would like to
Configure the Disruptor and it's EventHandlers
Publish messages ...
- 27.8k
0
votes
2
answers
138
views
Determining if threads are actually doing any work in Java
We have an application which uses Lmax Disruptor. We originally configured the Disruptor by
this.disruptor = new Disruptor<>(
GenericEvent::new,
...
- 2,148
0
votes
1
answer
583
views
LMAX disruptor busy spin vs ConcurrentLinkedQueue
When studying the LMAX disruptor I noticed that it uses the ring buffer model using the busy spin model with the CAS (compare-and-swap) to avoid locks like those that exist in a blocking queue, but ...
0
votes
0
answers
134
views
go-disruptor ringBuffer visibility discuss
In go-disruptor project.
I would like to know how to guarantee that writing and reading of ringBuffer is thread safe. Because I found that the writing and reading of the ringBuffer are not locked, and ...
- 11
0
votes
0
answers
316
views
How to instantiate LMAX Disruptor for sequential processing
Version 3 of the LMAX Disruptor deprecated the constructor that takes an Executor. So the following gives a compile warning:
Disruptor(eventFactory, bufferSize, Executors.newSingleThreadExecutor())
...
- 131
0
votes
1
answer
74
views
How to convert ReentrantReadWriteLock logic to LMAX Disruptor with barriers
I have a shared collection an ArrayList and also i use a ReentrantReadWriteLock lock to secure the entering on a critical area from different threads. My threads are three a writer,read,delete thread. ...
- 319
0
votes
1
answer
174
views
Parallel Producers in Disruptor (Ring buffer) (used in log4j2)
So I understand the purpose of the LMX Disruptor algorithm (used in log4j2). My question is why can’t producers just write to multiple slots in the ring buffer in parallel. Why do we need a write ...
0
votes
0
answers
175
views
Java/Kotlin parallel processing like LMAX disruptor
imagine that we have a data processing pipeline, consists of 3 steps
val finalValue=fn3(fn2(fn1(originalValue)))
however fn2 is cpu intensive, thus we want to parallelize it
/->...
- 115
0
votes
0
answers
185
views
Limit number of active threads in LMAX Disruptor(RingBuffer)
I have multiple consumers of different types that can be run in parallel but I don't want all of them to run in parallel at the same time because I don't have that many CPU cores and performance would ...
- 3,161
0
votes
1
answer
522
views
Using lmax Disruptor (3.0) in java to process millions of documents
I have the following use-case:
When my service starts, it may need to deal with millions of documents in as short of a burst as possible. There will be three sources of data.
I have set up the ...
- 2,148
2
votes
1
answer
839
views
AttributeError: Can't get attribute 'journalerReader' on <module '__mp_main__
I tried to implement Lmax in python .I tried to handle data in 4 processes
import disruptor
import multiprocessing
import random
if __name__ == '__main__':
cb = disruptor.CircularBuffer(5)
...
- 7,261
2
votes
0
answers
330
views
Business logic imp in LMAX architecture in python
I am trying to implement disruptor in Lmax architecture .
As you know in lmax architecture we have a ring buffer for creating a queue to process data .
Here you can see the structure of that :
I have ...
- 7,261
1
vote
0
answers
150
views
Prioritize messages in disruptor
Generally the events in my disruptor need to be handle sequentially. However, we sometimes receive high priority events which needs to be prioritize to be handled first. Is there such a functionality ...
- 1,571
0
votes
1
answer
199
views
Run task at start of disruptor handler thread
I am using the LMAX disruptor and it works really well.
I would like to execute some core affinity code (that needs to run ON THE DISRUPTOR HANDLER's THREAD), at the begginning of the disruptor's ...
0
votes
1
answer
248
views
LMAX Disruptor remainingCapacity equals 0, even before some time waiting
I added data to Disruptor by calling tryPublishEvent function.
After I wait 40 second and tried to check unprocessedDataCount by following calculation:
long ringBufferUnProcessedCount = disruptor....
- 156
1
vote
1
answer
968
views
How is LMAX Disruptor garbage collector friendly
I am trying to understand how lmax disruptor is GC friendly. I see that the event objects on the ring are re-used, however, the "data" that is "set" on these objects are on the ...
- 2,306
0
votes
1
answer
480
views
Disruptor behavior - Drain full buffer before consuming new data
I have the following scenario.
We were load testing our application in Cloud on K8s. Our inbound messages are coming over Kafka and we are writing back to Kafka. Our architecture is such that where ...
- 574
2
votes
1
answer
676
views
Using LMAX Disruptor for event handling reduces performance compared to Netty Handlers
I have written a Netty HTTP Server.
I am figuring out right way to handle Http requests
Options: (1. Netty Handlers 2. LMAX Disruptor).
I have read it somewhere that LMAX Disruptor are good for ...
- 461
1
vote
1
answer
7k
views
Log4j2 asynchronous logging is getting stuck when there is heavy logging from application
We are facing an issue with one of our applications where the peak load is causing the log4j2 to hang and the application doesn't repond.
log4j2.xml
We have a CustomAppender, AppMessageAppender ...
- 23
0
votes
1
answer
430
views
Using Hazelcast map with LMAX disruptor - performance impact
We have a requirement to run Active-Active instances of an order manager application for elasticity. Within our team, Hazelcast is the preferred distributed cache for sharing state across elastic ...
- 409
3
votes
1
answer
1k
views
LMAX Disruptor in Python
Is there already an implementation (or at least binding to) Python3?
I can see that an original implementation is in Java on Github, but I do not see anything for Python3.
- 1,983
1
vote
0
answers
296
views
LMAX Disruptor - sequential & parallel database/service calls on event handling
I a have a workflow (different tasks) that process data from Kafka/Redis streams. Some tasks are sequential & some parallel - on the high leve tasks fall into, verification -> computation -> ...
- 2,902
2
votes
1
answer
517
views
Why lmax disruptor architecture use 2 disruptor?
In " lmax disruptor architecture design" it is showing that , they are taking input and enqueing it in input disruptor and there are multiple event handlers like journaling ,un-marshalling , ...
- 106
0
votes
1
answer
785
views
Why does the Disruptor hold lots of data when the producer is much faster than the consumer?
I'm learning about the LMAX Disruptor and have a problem: When I have a very large ring buffer, like 1024, and my producer is much faster than my consumer, the ring buffer will hold lots of data, but ...
- 575
-2
votes
1
answer
999
views
How to make a Java NIO (Non blocking IO) based TCP server using Disruptor?
I'm trying to implement a JAVA NIO based TCP server using Disruptor.
Java NIO works in a non-blocking fashion. All the new connections first hit the ServerAccept Socket. Then using the key (which is ...
- 1,568
0
votes
0
answers
79
views
How to debug disruptor performance?
so I have code:
private void schedule(....) {
long s = m_ringBuffer.next();
.....
m_logger.info("1 [{}]", s);
m_ringBuffer.publish(s);
.....
}
...
- 393
0
votes
1
answer
92
views
Is there a way to judge whether all the events have been consumed?
Is there a way to judge whether all the events have been consumed? Similar to the BlockingQueue.isEmpty() method.
I want to add some business logic when consumers are idle.
Looking forward to your ...
- 1
1
vote
2
answers
749
views
Pollable disruptor
I did quite a bit of research, but I can not find the solution yet. I would like to find a pollable ring buffer, similar to the LMAX disruptor one. I have many publishers and one reader.
The problem ...
- 713
1
vote
1
answer
2k
views
Apache camel onCompletion call back doesn't trigger after route is completed
I have the following route setup in camel. I have posted the complete code here.
After the ProcessorTwo is invoked, i am expecting the ProcessorOnComplete to be invoked, but it is not at all triggered....
- 708
0
votes
0
answers
82
views
Disruptor Consumer check available sequence
I'm using Disruptor for my multi-thread app and now I have a use case where for my consumer of ringbuffer, I don't want it to wait. I just want to check if there's sequence available to be consumed. ...
- 393
-1
votes
1
answer
345
views
LMAX's disruptor. How to define multi EventHandlerGroup?
I want to handle different event with multi-EventHandleGroup?I can't find example with that.thanks...
it's the code:
executorService = Executors.newFixedThreadPool(threadPoolSize);
disruptor = ...
- 1
1
vote
1
answer
172
views
LMAX Distruptor Partition and join batch
So currently I have a Executor implementation with blocking queue and the implementation specific is like, I have list of items per request and I divide them into partitions each partition is then ...
- 196
0
votes
0
answers
315
views
Is the lmax disruptor a good architecture choice where there are several consumers all of which perform the same task and rely on some local cache?
My use case is this -
Several events are produced and dispatched onto a threadpool.
Each event is processed separately and the output of the processing isn't needed for some reconciliation later. (So,...
0
votes
1
answer
891
views
LMAX Disruptor Producer Incorrectly Wraparound + Overwrite Before Consumer Completes Read
I am recently introduced to the LMAX Disruptor and decided to give it a try. Thanks to the developers, the setup was quick and hassle free. But I think I am running into an issue if someone can help ...
- 11
-1
votes
1
answer
481
views
how to fulfill the ringbuffer with data from my hashmap lmax disruptor
Actually, after reading and using a simple example of disruptive I can't find how to where should I fulfill my the ring buffer with data from my hashmap already fulfilled with data from ...
0
votes
1
answer
172
views
Difference between complex event processing and ring buffer
What is the difference between complex event processing and ring buffer disruptor?
Both are used to receive data with low latency and less utilisation of system resource.
Of course CEP with esper ...
- 1,264
1
vote
1
answer
399
views
how to push list to lmax disruptor in the eventclass
I need to store and get a huge data from mongodb so Im asked to receive and store data using lmax disruptor I have passed few days looking to a simple tutorial on the lmax github account, but I didn ...
0
votes
1
answer
395
views
Can we use single thread to consume multiple disruptor?
I have different data coming from different source (multicast, tcp).
I'm intending to use different Disruptor to manage each type of data individually while I would like to consume all these data by ...
- 81
0
votes
1
answer
400
views
LMAX Disruptor - Maintain order of events
I have an application that loads time series data from various files. The application opens one thread per file to load the data in parallel. The records in the files are ordered but I need to deliver ...
- 1
1
vote
1
answer
171
views
how to check whether an event is already in disruptor ringbuffer
My disruptor/ringbuffer application has multiple producers and multiple workers. An proceducer puts an event to the RingBuffer, if the event is not in the RingBuffer. For an event, a worker query ...
- 2,057
4
votes
1
answer
1k
views
Disruptor vs. Reactive architecture for middle-frequency trading system
I'm trying to choose an appropriate architecture for a middle-frequency trading system I'm working on. Currently, I receive messages from Web Socket or Rest and process them right there. Sometimes it ...
- 25.3k
1
vote
0
answers
284
views
How to handle logging in a multiThread java application?
I Have a Multi-threaded java application running in google compute engine instance, this application might reach up to 10k threads, each one doing a task might reach 30 min execution time.
Everything ...
- 503
1
vote
2
answers
1k
views
Disruptor Journaling Implementation
In Martin Fowler's write-up of the LMAX-disruptor architecture, he says:
The journaler's job is to store all the events in a durable form, so
that they can be replayed should anything go wrong. ...
- 68.5k
2
votes
1
answer
1k
views
Why my Disruptor program don't take full advantage of the ringbuffer
Disruptor github address is: https://github.com/LMAX-Exchange/disruptor
I've a simple test for it as below:
public class DisruptorMain {
@SuppressWarnings({ "rawtypes", "unchecked" })
public ...
- 257
2
votes
1
answer
549
views
.NET Disruptor async patterns
I am using Disruptor-net in a C# application. I'm having some trouble understanding how to do async operations in the disruptor pattern.
Assuming I have a few event handlers, and the last one in the ...
- 8,212
3
votes
1
answer
3k
views
Relationship between kafka and disruptor
I'm starting reading about kafka and found some similarities with Disruptor farmework.
Is there any relationship between kafka/Disruptor,can we consider kafka as an implementation of Disruptor?..
Can ...
- 943
3
votes
2
answers
3k
views
Why is disruptor slower with smaller ring buffer?
Following the Disruptor Getting Started Guide, I've built a minimal disruptor with a single producer and single consumer.
Producer
import com.lmax.disruptor.RingBuffer;
public class ...
- 16.2k