0

I am working on my Spark Streaming project with the goal of creating a simple app to notify users when a data stream meets a condition (e.g., sending a notification when a stock price > x).

df = spark.readStream...\
          .filter(col > x)

However, I am unsure how to address the issue of allowing users to change the condition (such as changing the value of x) while the data is still streaming without generating redundant messages (messages for the old price or condition).

Could you please help me with some solutions using Spark Structured Streaming, preferably in Python, to solve this problem?

Thank you so much!

Improve this question

1 Answer 1

Reset to default
0

You have to restart the StreamingQuery to reflect changes in x. StructuredStreaming won't process "old" messages again if you have configured the checkpointLocation, see Checkpointing.

Improve this answer
1
  • Hi Boyangeor, could you provide a straightforward example? It would help me better understand what I need to do
    – Baubau Tran
    Commented Nov 23, 2023 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.