238 questions
2
votes
1
answer
51
views
BlockingCollection.Add in a loop with await taking considerable more time than expected [duplicate]
Here is the simplest repro I could make illustrating the issue. This is a Console application that creates a bounded BlockingCollection<int>, starts a background thread that consumes items from ...
- 879
2
votes
2
answers
313
views
BlockingCollection<T> throws unexpected InvalidOperationException
I was trying out BlockingCollection<T> (as a Queue) in .NET8 and sometimes I end up with the exception:
"System.InvalidOperationException: The collection argument is empty and has been ...
- 31
0
votes
1
answer
121
views
Thread-safety issues with concurrent message deserialization using BlockingCollection in C#
I have the following setup
internal class Program
{
public BlockingCollection<ArraySegment<byte>> MessageQueue { get; set; } = [];
static void Main(string[] args)
{
...
- 33
0
votes
1
answer
159
views
LongRunning Task behaviour
I have a class that contains a BlockingCollection<Task> that is used to queue up a number of tasks to run in order (FIFO).
The queue itself is started on a separate long running task.
public ...
- 879
1
vote
1
answer
203
views
Tasks stopping their work partway through without throwing any exceptions
I'm developing a fairly basic "Worker" application to run on a server, picking up jobs from a SQL server, queuing them in a Blocking Collection and spawning a configured number of Tasks to ...
- 31
2
votes
1
answer
94
views
Cannot Add Tuple to BlockingCollection: Error CS1503 in C#
I'm working on a project that involves object detection with real-time video in C#. As part of the project, I use a BlockingCollection to manage a list of bounding boxes for detected objects, and the ...
- 191
1
vote
0
answers
277
views
Producer Consumer with PriorityQueue
I need to implement the Producer Consumer pattern with priorities and I'm searching the best option in terms of robust and performance.
These are my requirements
Multiple Threads can enqueue elements
...
- 734
4
votes
1
answer
3k
views
Producer Consumer Queue best practices and performance
I'm building a producer consummer queue in C# and I was reading for searching the best method in terms of robust and performance.
For years I was using always BlockingCollection but I have discovered ...
- 734
1
vote
1
answer
259
views
Parallel.ForEach on a BlockingCollection causes steady increase of threads
I observed a strange behavior while experimenting with the Parallel.ForEach method and the BlockingCollection<T> class. Apparently calling the two lines below on a separate thread, is enough to ...
- 43.1k
0
votes
1
answer
633
views
Is ConcurrentDictionary always add item by order in C#?
I tried BlockingCollection and ConcurrentDictionary. It continues by adding BlockingCollection items as soon as the thread is completed, but ConcurrentDictionary is always adding thread order. Is the ...
- 104
1
vote
0
answers
61
views
Blocking Collection GetConsumingEnumerable stops consuming increasing the size of the collection
I have a BlockingCollection to which a string is added every 25ms. There is a thread(ProducerConsumer()) for consuming the collection. This is running on a windows service. The GetConsumingEnumerable ...
1
vote
1
answer
130
views
Cannot retrieve chunks of data from BlockingCollection<T>
I often find myself in a situation where I do want to stream data in chunks rather then one by one. Usually I do this when I need to do some I/O based operation like database inserts where I want to ...
- 708
0
votes
1
answer
120
views
Net 6 ConsoleApp multiple BlockingCollection<T> huge CPU consumption
I have a Net 6 Console app where I use several BlockingCollections to process files that are dropped in a folder. I watch the folder using Net's FileWatcher().
In the Created event, I use a Channel to ...
- 693
0
votes
1
answer
199
views
How to implement a sorted buffer?
I need to traverse a collection of disjoint folders; each folder is associated to a visited time configurated somewhere in the folder.
I then sort the folders, and process the one with the earliest ...
- 19
1
vote
1
answer
124
views
How to invoke a consumer method as soon as BlockingCollection got populated?
Background:
By reading so many sources I understood BlockingCollection<T> is designed to get rid of the requirement of checking if new data is available in the shared collection between threads. ...
- 636
0
votes
0
answers
71
views
Having a huge chunk of memory on initializing an array of blocking collection
When I initialize a collection of byte array it resulted in a high memory consumption of 12gb memory.
BlockingCollection<byte[]>[] bc = new BlockingCollection<byte[]>();
- 9
1
vote
2
answers
562
views
What thread-safe collection use to cache messages
I'm working on project with following workflow :
Background service consumme messages from Rabbitmq's queue
Background service use background task queue like this and here to process task paralleling
...
- 423
0
votes
2
answers
356
views
C# BlockingCollection loop on IsCompleted instead of GetConsumingEnumerable
In an application, many actions need to be logged in the database. but this logging process should not slow down the request. So they should be done in an asynchronous queue or something.
This is my ...
- 440
2
votes
3
answers
312
views
How to implement the BlockingCollection.TakeFromAny equivalent for Channels?
I am trying to implement an asynchronous method that takes an array of ChannelReader<T>s, and takes a value from any of the channels that has an item available. It is a method with similar ...
- 43.1k
0
votes
2
answers
63
views
Dequeue a collection to write to disk until no more items left in collection
So, I have a list of file shares.
I then need to obtain ALL folders in these file shares. This is all the "easy" stuff I have done.
Ultimately after some logic, I am adding an object into a ...
- 5,812
1
vote
2
answers
162
views
How to cancel Actions already in a BlockingCollection<Action>
I have a simple class that enqueues actions to execute:
BlockingCollection<Action> blockingCollectionOfEvents = new BlockingCollection<Action>();
System.Threading.Thread workerThread;
...
- 860
0
votes
1
answer
815
views
Best way to implement consumer queue that you can remove items from sequentially (.net 6)
new poster here so I hope this makes sense ...
I need to create a collection that I can remove items from in sequence (basically stock market time series data).
The data producer is multi-threaded and ...
- 13
0
votes
1
answer
396
views
BlockingCollection stops taking after few iterations
In its bare-bone, my producer/consumer pattern reads as the following.
public class Consumer<T>
{
Task consumer;
BlockingCollection<T> buffer;
public Consumer()
{
...
- 3,292
2
votes
1
answer
205
views
How to copy BlockingCollection and edit the new one without editing the origin
I want to copy a BlockingCollection and edit the copy.
(dataModelCollection is the copy of DataModelListRaw)
When I do this:
BlockingCollection<DataModel> dataModelCollection = DataModelListRaw;
...
- 58
2
votes
2
answers
1k
views
BlockingCollection alternatives?
We are using BlockingCollection to implement producer-consumer pattern in a real-time application:
BlockingCollection<T> collection = new BlockingCollection<T>();
CancellationTokenSource ...
- 163
1
vote
1
answer
958
views
BlockingCollection where the consumers are also producers
I have a bunch of requests to process, and during the processing of those requests, more "sub-requests" can be generated and added to the same blocking collection. The consumers add sub-...
- 544
1
vote
1
answer
425
views
BlockingCollection bounded capacity performance degradation
I've got this C# process (.net 5.0) that reads from a zip file, deserializes the json to an object, and then transforms the json objects to DataTables for storage into a Sql Server database. After a ...
- 697
2
votes
1
answer
270
views
How to notify failure in producer-consumer pattern using BlockingCollection?
I'm trying to create a lifetime process that batches incoming messages for DB bulk insert.
The new message is coming in 1 at a time, in an irregular interval.
My solution to this would be something ...
- 35
1
vote
1
answer
3k
views
async use with FileSystemWatcher in Net 5
I'm trying to create an application in Net 5 that watches a folder and any time files are being dropped in the folder, it should run a certain set of tasks (getting some info from the files, copy them ...
- 693
0
votes
1
answer
126
views
Saving images via producer-consumer pattern using BlockingCollection
I'm facing a producer-consumer problem: I have a camera that sends images very quickly and I have to save them to disk. The images are in the form of ushort[]. The camera always overrides the same ...
- 1
1
vote
1
answer
853
views
Search for a particular element in BlockingCollection
I have a BlockingCollection:
private readonly BlockingCollection<ImageKeys> myCompletedImages;
I have a method that adds items to the BlockingCollection:
public void addItem(ImageKey ...
- 292
1
vote
3
answers
2k
views
BlockingCollection<T> in a BackgroundService causes high CPU usage
I have a .NET BackgroundService for managing notifications by using a BlockingCollection<Notification>.
My implementation is cause high CPU usage, even though there is not that much work to be ...
- 1,390
0
votes
1
answer
583
views
await completion of BlockingCollection [closed]
For a multithreaded application I want to await until a BlockingCollection is completed and empty (IsCompleted = true). I implemented the below and this seems to be working.
Since it's multithreading ...
- 529
2
votes
1
answer
2k
views
C# RabbitMQ pool/queue approach
I'm trying to create a queue/pool of rabbit mq channels for reuse, which they recommend to do this, but don't provide any approach of how to actually do this!
I have this working but ideally I don't ...
- 2,919
1
vote
0
answers
30
views
Why not use TryTakeFromAny instead of TakeFromAny [duplicate]
So I am just learning about Pipelines, and was wondering when you would have a case where you WANT to use TakeFromAny instead of TryTakeFromAny. TryTakeFromAny seems to be a lot safer to use, rather ...
- 13
0
votes
0
answers
282
views
MemoryCache Update List in Thread Safe Manner
I'm trying to fully understand the implications of thread safety in maintaining a BlockingCollection inside a MemoryCache
The scenario: log recent users that request any page from a site. Use an ...
- 7,782
0
votes
1
answer
2k
views
Asp.net core Web API parallel call , batch processing with some data lost
Created one web API in asp.net core 3.x which is responsible to save data in to Azure service bus queue and that data we will process for reporting.
API load is too high so we decided to save data in-...
- 1,569
2
votes
1
answer
87
views
BlockingCollection Paged to Disk
I have read about BigQueue and the like to be a durable disk bound queue.
Is there anything for c# that is not durable but "pages" queue overflow to disk.
I have incoming byte[] after ...
- 5,960
0
votes
2
answers
565
views
Are multiple consumers in BlockingCollection handled concurrently?
I have a BlockingCollection which I write to from one thread and I read from another.
The producer thread takes items received from a server and adds them to the BlockingCollection, while the reading ...
- 2,655
-1
votes
1
answer
1k
views
Get count of current active Tasks spawned by a multithreaded application [duplicate]
Hi I am a student intern with little to no C# experience who got put into a situation to take over a windows service that uses TaskCompletionSource and BlockingCollection to implement multithreading. ...
- 639
1
vote
1
answer
158
views
Does the BlockingCollection.AddToAny Method offer any performance advantages over adding to a single BlockingCollection?
As far as I know, adding to a BlockingCollection doesn't block, o AddToAny that I can think of, is if some of them called CompleteAdding().
Is this the only use for AddToAny? And once they all call ...
- 2,655
0
votes
1
answer
113
views
Strange behavior of starting multiple tasks that block [duplicate]
Here is a test method that starts (without awaiting) 100 Tasks that each call GetConsumingEnumerable on a BlockingCollection. (Update: the behavior described below is not specific to this method; it ...
- 5,805
0
votes
0
answers
74
views
C# WebAPI Thread Aborting when Sharing Data Access Logic
I'm getting the following error on my C# Web API: "Exception thrown: 'System.Threading.ThreadAbortException' in System.Data.dll
Thread was being aborted". I have a long running process on ...
- 123
-1
votes
1
answer
380
views
Set a maximum processing of items in a BlockingCollection
I have a blockingcollection of downloads and I would like to have a maximum number of concurrent download I can do, aka a maximum number of concurrent await downloadService.download(release), but the ...
- 37
0
votes
0
answers
57
views
Will BlockingCollection or TPL will be suitable for multiple producer-consumer scenario
Will only BlockingCollection or TPL with BlockingCollection will be suitable for this case:
program will gather data from multiple socket streams (will act as client). Get Packet from socket, forward ...
- 27
-1
votes
1
answer
53
views
Multiple consumers update result array inconsistently
I have a large file, each row can be process separately, so I launch one reader, and multiple parsers.
The each parser will write result back to a result holder array for further process.
I found if I ...
- 2,016
0
votes
0
answers
459
views
Why use while(true) with .take with a blockingcollection?
I have been looking into using blockingcollection. I found this article very helpful
Although, he is using "tasks" for multi threading purposes, I don't think I need this. But anyways, the ...
- 179
0
votes
1
answer
168
views
VB.net - 5 threads reading/writing to the same variable
I have an app that uses 5 concurrent threads to perform tasks. The threads will need to read from a list of items and pick the next available one. As soon as they've done so, they need to add one to ...
- 341
2
votes
1
answer
167
views
Timer vs Thread vs RegisteredWaitHandle for consuming from BlockingCollection
I have a service which talks to around 50 devices over Ethernet using TCP. These devices push data at various speeds (ranging from 50ms to 1500ms). Hence I use BlockingCollection (per device) for ...
- 827
0
votes
1
answer
282
views
Take method of BlockingCollection slower?
I was using BlockingCollection to queue data received over network and process it. I was consuming from the blockingcollection at every 100 ms in a timer. But I found that consumer was slow and length ...
- 827