477 questions
0
votes
0
answers
42
views
Completable Future with Executor for handling Exceptions only
There is a list of objects List<Foo> fooDetails, which I will use to create Bar request corresponding to each element of fooDetails from database and then do a rest template call.
public void ...
- 115
0
votes
1
answer
78
views
Wrapping Stream.parallel() by Executor service in Java
I'm studying Java Core on course DMDEV in Udemy and YouTube. On Level 2 in lessons Concurrent I probably saw mistake in code the tutor was made. But my question to him was still without response. So ...
- 1
1
vote
0
answers
25
views
How Can I delete Comet Executor.exe from AppData\Romaing\Comet Executor?
My friend's 11yrs old son visited at home with family and let him use my desktop. This generation kids are genius with computer, and installed cloud based game program called Comet Executor. It hidden ...
- 11
0
votes
0
answers
66
views
How does Application Master, Spark Driver and Spark Context work together?
After reading over this topic through multiple articles and answers, some ideas are still hazy. My understanding of spark execution process is like this.
When a spark job is submitted an application ...
0
votes
0
answers
22
views
Android Thread InputStream from web page
I have an old app in Android that just downloads the content of a web page and sends it to a Bluetooth printer. It was working fine but with the new Google restrictions to Android apps it's just ...
0
votes
0
answers
35
views
Is there a known issue using a ignite semaphore with many concurrent requests?
We tried to use the igite semaphore to control access to resources in a cluster. At low concurrency viz. < 10 threads the semaphore seems to work as documented. However when we have 100's of ...
- 21
0
votes
1
answer
36
views
How to share data between ExecutorService.execute() from another class with ExecutorService while networking. Android
For context, I am making an app which communicates with TCP server hosted currently on my local pc. I am using SSLSocket for secure connection. All communications are ok. I have made a special class ...
0
votes
0
answers
26
views
Efficiency metric in Apache Spark Monitoring API
I got the following from an API call for monitoring Apache Spark:
"diagnostic_details": {
"data": {
"stages":[...],
"executors":{"...
- 11
1
vote
1
answer
235
views
What executor does co_await boost::asio::this_coro::executor produce?
The documentation of boost::asio::this_coro::executor states that it is an
Awaitable object that returns the executor of the current coroutine.
To me this seems somewhat vague as soon as multiple ...
- 1,462
2
votes
1
answer
420
views
Boost Asio: Executors in C++20 coroutines
While experimenting with boost::asio::awaitable and Executors, I keep observing some rather confusing behaviour that I would like to understand better
Preparation
Please take a look at the follwing ...
- 1,462
-1
votes
1
answer
103
views
Make a `Callable` of my `Runnable` in Java
I have some tasks defined as Runnable objects. I want to invoke those tasks by using the invokeAll or invokeAny methods on ExecutorService.
The problem is that those methods take a collection of ...
- 336k
0
votes
1
answer
57
views
Java POST Issue, how to check return from server?
How to check if is true or false ? Using executor but success seems not to change its value.. there is any trick to make it in a 'easy' way ? Im able to reach inside the try catch, but seems that its ...
- 35
0
votes
0
answers
146
views
Does having large number of parquet files causes memory overhead while reading using Spark?
I have a Spark job and every time I do a historical fix, my job fails because of memory limit exceed. While doing historical fix, I create a table in my temporary database and read it (for the first ...
- 1
0
votes
1
answer
151
views
Weird race-condition in java ThreadPoolExecutor
I`m trying to make an api-checking app that must send requests asynchronously using the pool of bots while termination date is not reached.
On startup creates a final List<Bot> is created and ...
0
votes
0
answers
83
views
Managing executors with fragments
I'm developing an app that performs tests on mobile network. The app has different Fragment (Home, Dashboard and Notification).
The Home fragment is able to start a periodic test using the Executors ...
1
vote
1
answer
702
views
Mocking not working with ExecutorService.submit()
While working with Executor Service mocking is not working. Following is the code.
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util....
- 113
-2
votes
1
answer
181
views
How to concurrently fire multiple list of Objects independent of each list using Multi threading in Java
I have a unique requirement for which i am unable to find relevant references online.
We usually get huge map of employee objects from a data warehouse application every weekend, where for every ...
- 51
0
votes
2
answers
431
views
Spark - Only driver run tasks
I'm trying to run GroupByTest using Spark standalone mode.
I've run it successfully on one machine, when I try to run it on a different machine it works but it's seems like Driver is the only instance ...
- 337
0
votes
1
answer
117
views
How to calculate Spark driver and executor memory in local machine?
I am a beginner with spark, generally in some executions with a java.lang.OutOfMemoryError: Java heap space is raised:
java.lang.OutOfMemoryError: Java heap space at java.base/java.nio.HeapByteBuffer.&...
- 13
1
vote
0
answers
328
views
aiogram. Remove executor messages from console
Using python3 aiogram library for a telegram bot. Is there a way to remove the messages from console generated by the executor? messages like Goodbye! or Updates were skipped successfully. if you ...
- 5,694
1
vote
2
answers
779
views
Do Spark's stages within a job run in parallel?
Do Spark's stages within a job run in parallel?
I know that within a job in Spark, multiple stages could be running in parallel, but when I checked, it seems like the executors are doing a context ...
- 11
-1
votes
1
answer
687
views
How to get spark.executor.memory size?
I am using Spark 2.3.2.3.1.0.0-78.
I tried to use
spark_session.sparkContext._conf.get('spark.executor.memory')
but I only received None.
How can I get spark.executor.memory's value?
- 101
1
vote
1
answer
77
views
Measure wait time of threadpool tasks in python
I'm running a threadpool executor to which I submit some tasks.
executor = ThreadPoolExecutor(thread_name_prefix='OMS.oms_thread_', max_workers=16)
task = executor.submit(method_to_run, args)
I know ...
- 1,283
0
votes
1
answer
866
views
EMR on EC2: How does Spark decide on the number of executors it needs to spawn?
I am trying to reason around Spark's default behavior here. Here's my scenario:
I am submitting a Spark job on a cluster with one master and 10 core/slave nodes.
These core nodes have been randomly ...
- 790
0
votes
0
answers
62
views
Spring ThreadPoolTaskExecutor response time increases a lot
I have configured Spring ThreadPoolTaskExecutor, trying to have in mind the average load of 1,1k rpm.
@Qualifier("executor")
public Executor asyncExecutor() {
this....
1
vote
0
answers
132
views
Define specific spark executors per worker on a Spark cluster of 3 worker nodes
I have a Spark cluster of 3 servers (1 worker per server = 3 workers). The resources are very much the same across servers (70 cores, 386GB of RAM each).
I also have an application that I spark-submit,...
0
votes
0
answers
463
views
Executor.submit() returns null
I am working in a multithreaded environment with many methods to be called using Executors. The code works fine even for large data earlier, but it threw a null pointer exception once.
Here is the ...
- 31
3
votes
1
answer
3k
views
Gitlab runner configuration on windows
i am trying to use gitlab runner installed on my windows machine . But pipeline execution fails with error:
ERROR: Job failed (system failure): prepare environment: failed to start process: exec: &...
- 65
4
votes
0
answers
458
views
Executor Service - create two single thread executors or use a thread pool of 2?
I have a task that runs in an endless while loop to read from a queue of numbers and checks if there is a sequence gap. Then, the main thread listens to UDP multicast and if it receives a packet, it ...
- 1,105
5
votes
1
answer
11k
views
passing multiple arguments to a thread pool executor
I am writing a code to run pool executor and use a function with two arguments.
args=[(0,users[0]),(1,users[1]),(2,users[2]),(3,users[3]),(4,users[4]),(5,users[5]),(6,users[6])]
if __name__ ==...
0
votes
1
answer
2k
views
How to get number of executors from SparkSession in pyspark?
spark = SparkSession.builder.getOrCreate()
spark.sparkContext.getConf().get('spark.executor.instances')
# Result: None
spark.conf.get('spark.executor.instances')
# Result: java.util....
- 743
2
votes
0
answers
465
views
airflow : executor could not be loaded on apache airflow localhost
i have install apache airflow on localhost using ubuntu. the executor can't be loaded, this is the traceback :
[2022-12-20 22:11:13,927] {manager.py:343} WARNING - Ending without manager process.
[...
- 99
0
votes
1
answer
278
views
CosmosDB Java V4 SDK executeBulkOperations - "Main sink completion failed. EmitResult: FAIL_TERMINATED"
If I run a simple set of document updates on the CosmosDB container using
container.executeBulkOperations(bulkOperations) I'm getting strange suspicious log messages from the underlying Cosmos DB Java ...
- 61
1
vote
2
answers
563
views
With Mockito Trying to verify of a method call, inside which Executor.execute method is called
this is the method call which should be verified, whether is called.
Mockito.verify(messageHandler).handleMessage(message);
and this method is called inside prepareContext() method, which is called ...
- 458
0
votes
1
answer
29
views
Multiprocessing using Functions Held in Variables as the Called Function
If I have this:
import concurrent.futures
def SetParameters():
return { "processVal" : 'q',
"keys" : { "key1" : { "fn" : a,
...
0
votes
3
answers
2k
views
Limit on number of threads in Thread pool in Java executors framework?
I have been working on Spring Boot application where I have limited memory for JVM upto 2 GB.
At controller, user sends some request which is handled by my Executors thread pool.
It is a light weight ...
- 6,892
1
vote
1
answer
154
views
ExecutorService with single thread in Java Multithreading
What is the point of creating ExecutorService with single thread in java multithreading means why not to create single separate thread instead of ExecutorService with a single thread?
Which is ...
- 33
2
votes
0
answers
859
views
Spark task is running forever with WARN HangingTaskDetector message
I am trying to do a forecast of sales using Prophet in my Databricks cluster through a Grouped Map Pandas UDF. The problem is that each time I run it, either two or one executors get stuck running ...
0
votes
1
answer
1k
views
Apache-Spark: Retry on different executor
I'm in an organization where Hadoop/Spark is available, but I can't alter its configuration and can't access it either. So, as a client code, I was wondering if I can ask Spark to retry failing tasks ...
- 193
2
votes
0
answers
675
views
How to check currently running thread under given Task Executor with Spring @Async
I am not creating bean explicitly. I am using @EnableAsync and spring properties.
spring.task.execution.pool.core-size=10
spring.task.execution.pool.max-size=20
spring.task.execution.pool.queue-...
- 161
1
vote
2
answers
277
views
CachedThreadPool vs FixedThreadPool for huge number of task
I would like to know which one should i use in the particular scenario:
there are several tasks, usually 400k tasks to process.
most of the tasks took less than 4 sec to process but some of them (300 ...
-2
votes
2
answers
591
views
Callable thread for singleThreadExecutor is working as synchronously
I have tested the below callable sample code
ExecutorService executorService = Executors.newSingleThreadExecutor();
Future<String> futureResMap = executorService.submit(new Callable<String>...
0
votes
1
answer
207
views
Why executor input for spark application running on amazon emr is showing more than the actual file size its processing?
I am running an amazon emr cluster with 20 spark applications having cluster configurations as 1 master node and 2 worker node as c5.24xlarge instance.
Giving 3 executors and one driver to each ...
- 1
0
votes
1
answer
450
views
apache zeppelin and spark on kubernetes no communication between executors possible over shuffle
I've implemented apache zeppelin 0.10.1 on kubernetes. It uses spark version 3.2.1 .
My problem is that the executors cant communicate with each other while shuffling, but still can exchange data with ...
- 77
0
votes
1
answer
378
views
Java thread hangs after CompletionService.take()
I've got simple test code:
public static void main(String[] args) {
CompletionService<Integer> cs = new ExecutorCompletionService<>(Executors.newCachedThreadPool());
cs.submit(new ...
- 8,027
0
votes
1
answer
778
views
Spark SparkFiles.get() returns driver path instead of worker path
I am piping the partitions of an RDD through an external executable. I use sparkContext.addFiles() so that the executable will be available to the workers.
I when I attempt to run the code I am ...
- 1,039
1
vote
1
answer
3k
views
How to use an executor from a boost::asio object to dispatch stuff into the same execution thread?
Ok, I don't have enough code yet for a fully working program, but I'm already running into issues with "executors".
EDIT: this is Boost 1.74 -- Debian doesn't give me anything more current. ...
- 12.5k
1
vote
2
answers
2k
views
ThreadPoolExecutor communication with separate asyncio loop
I have a task that is IO bound running in a loop. This task does a lot of work and is often times hogging the loop (Is that the right word for it?). My plan is to run it in a separate process or ...
0
votes
1
answer
271
views
Spark --num-execotors options not clear
I do not understand this option. It seems like its the maximum number of executors.
If there is not enough memory on the nodes in the cluster this number is doing nothing and there are fewer executors ...
- 913
3
votes
2
answers
998
views
How can I increase spark.driver.memoryOverhead in Google dataproc?
I am getting two types of errors in running a job on Google dataproc and it is causing executors to be lost one by one until the last executor is lost and the job fails. I have set my master node to ...
- 71