205 questions
0
votes
1
answer
58
views
Prefect 3 - Error running task with a custom Docker container
I have a project with a simple python file that is containerized with a Dockerfile.
The dockerfile is:
FROM python:3.9-slim
WORKDIR /app
COPY . .
RUN pip install --no-cache-dir -r requirements.txt
...
- 181
0
votes
0
answers
17
views
Can't find local python package in parallel directory when using Prefect + Docker
I have setup a Prefect deployment on a Docker image. The problem is that my flow package depends on a local package which is in a parallel directory:
main_folder
|_ my_flow
|_ src
|_ my_flow
...
- 315
0
votes
1
answer
94
views
How to create a run flow using prefect api?
I'm trying to run a prefect flow from a deployment I've already created. The name of my deployment is my-deployment-test. Context: I basically need to create a function that receives the deployment ...
0
votes
1
answer
43
views
prefect warning message regarding task performance for longer running tasks
i have been getting the below warning message with tasks with larger run time
most of my tasks take more than 10 seconds
it says
Try wrapping large task parameters with `prefect.utilities.annotations....
- 406
0
votes
1
answer
32
views
Can i integrate prefectHQ workflow orchestrator with saltstack
I'm new to this tools, i'm using this tools for migration tool in networking. Can I integrate PrefectHQ with SaltStack?
I installed PrefecHq on VS code and understanding the basics of how flow, tasks ...
0
votes
0
answers
26
views
PostgreSQL Table Size Increased After Aborted VACUUM FULL—Unable to Reclaim Space [duplicate]
I am using PostgreSQL deployed in a Kubernetes cluster, and I connect to it using pgAdmin via port forwarding. I recently ran the maintenance toolkit in pgAdmin on my Prefect database. During this ...
0
votes
0
answers
85
views
Prefect 2.0.x MinIO Storage Block using in Fastapi environment cannot even save newly created Block or Load existing Block
I facing a problem when trying to use Prefect Storage Block in Fastapi. The error is below when I trying to call load or newly created save Block :
File "/usr/lib/python3.8/json/__init__.py&...
0
votes
0
answers
32
views
Efficiently processing large molecular datasets with Dask Disctributed, DataFrames and Prefect,
I'm working with a large dataset of molecular structures (approximately 240,000 records) stored in a PostgreSQL database. I need to perform computations on each molecule using RDKit. I'm using Dask ...
- 461
1
vote
0
answers
85
views
Prefect with Google Cloud Run: Creating a New Execution ID for Each Flow in a Single Job
I am currently using Prefect cloud to orchestrate different tasks, and I am deploying these tasks as jobs on Google Cloud Run using the CloudRun Push workpool.
Here's the situation I am facing: every ...
- 73
2
votes
1
answer
612
views
How can I run Prefect flows using a Docker Worker pool and a local custom Docker image?
I'm new to Prefect and have a limited experience with Docker and I'm trying to deploy some existing Prefect flows so that they run inside a Docker container running an image I built myself, but ...
- 568
1
vote
0
answers
258
views
Prefect is not running on Docker Container
I'm new to docker and prefect and trying to run a basic prefect flows with docker container from my local windows.
Here is my Docker file
# We're using the latest version of Prefect with Python 3.10
...
- 11
0
votes
0
answers
23
views
Prefect Doesn't Terminate Completed Flows in Windows
I came across the following situation:
When prefect finishes a task, it continues to run in the windows server task manager.
E.g.:
The change name flow ended with full status, but it is still running ...
0
votes
0
answers
131
views
Prefect worker using docker infrastructure not working properly after deploy
I'm trying to run a prefect worker on a work pool based on a docker infrastructure. I don't know why, it always gives me problems when the worker submits the flow and the container is run up. Maybe it ...
0
votes
0
answers
48
views
Prefect Database high Lock time
I am running prefect 2.7.1 on Kubernetes on AWS EKS.
I am using AWS RDS m7g.4xlarge as the database for prefect
I am able to see spike in the wait time in the performance monitoring of the RDS going ...
- 599
0
votes
0
answers
101
views
How do I get the Prefect "flow id" (uuid) from within a "task" in a programmatic way using Python?
How can I get the "flow id" (not the "flow run id") from Prefect using Python from within a "task?" The task is being executed by a flow. I'm using Prefect version 2.19.1 ...
- 23
1
vote
1
answer
359
views
Accessing the scheduled time from within a Prefect flow run
I plan to run Prefect 2 flows on a cron schedule - but is there any way to access, at runtime, the timestamp at which this particular run was scheduled?
For example, if the flow is scheduled to run ...
- 42.6k
0
votes
1
answer
214
views
Unable to run the Prefect server on machine's IP address when using NSSM
I have configured Prefect server on our Windows VM so that it runs on the machine's IP address. Here's how I did it.
env\Scripts\activate.bat
(env) prefect config set PREFECT_SERVER_API_HOST='SERVER-...
- 105
0
votes
0
answers
78
views
Prefect server :prefect.exceptions.ParameterBindError: Error binding parameters for function
i am attempting to use prefect to manage my flow,but adding @flow will result in an error.Can methods modified by @ flow not be related to classes?
enter image description here
Can methods modified ...
0
votes
1
answer
225
views
Dockerfile runs runs locally but fails on Google Cloud Run
I have a Dockerfile, shown below:
FROM --platform=linux/amd64 prefecthq/prefect:2-python3.12
RUN apt-get update && apt-get -y install libpq-dev gcc
RUN pip install --upgrade pip
RUN pip ...
- 1,560
1
vote
1
answer
763
views
Run docker image as prefect task or flow
My question is effectively a duplicate of this question however the only answer is related to Prefect 1. I would like to know if anyone is aware of how to do such a thing in Prefect 2.
I am able to ...
- 80
0
votes
1
answer
156
views
Prefect: Unknown work pool type 'docker'. Please choose from block, process
I am attempting to follow the tutorial in the Prefect docs and I have gotten to the section where I need to create a docker work pool using the following command:
prefect work-pool create --type ...
- 80
0
votes
1
answer
171
views
Prefect and aiosqlite dependency problem - sqlalchemy.exc.OperationalError: (sqlite3.OperationalError)
I'm running a script with prefect using some tasks and flows. This script has run without problems before, but i'm experiencing problems with it without having changed it.
script
I just call ...
0
votes
0
answers
356
views
ERROR | prefect.flow_runs.runner - Process for flow run exited with status code: -7
I am getting this Prefect error while I am running my script. The error is the following:
0%| | 0/2070 [00:00<?, ?it/s]09:36:01.917 | ERROR | prefect.flow_runs.runner - Process for flow ...
0
votes
0
answers
32
views
Training sklearn estimators on large datasets throws aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer
I am trying to run prefect flow on Kubernetes. I have bigger dataset and want to train multiple estimators in parallel. See the below code for more in details.
from sklearn.datasets import ...
- 1
0
votes
0
answers
260
views
Prefect: How to Make a Worker Activate a Flow Specific venv Before Flow Run?
Preface: this GitHub issue is semi-related
I’m wondering what the correct method would be to associate a flow with a venv so that all flow runs are ran in the venv. I’m using a prefect worker and work ...
- 364
0
votes
1
answer
65
views
Secret Field not be imported when using prefect
ImportError: cannot import name 'SecretField' from 'pydantic'
Hi guys, i having some trouble when using prefect
I was installed prefect, activate venv and i had this :D
ImportError: cannot import name ...
- 1
0
votes
1
answer
158
views
Why doesn't a prefect task fail, if a contained dask.distributed task fails?
I'm running a workflow using Prefect using a DaskTaskRunner, which creates and holds a dask.distibuted.LocalCluster instance.
Inside a prefect task I use a dask_ml.RandomSearchCV and fit it, which by ...
- 7
0
votes
2
answers
603
views
ImportError: cannot import name 'SecretField' from 'pydantic'
I'm using prefect on a gitlab CI and it was working fine until now when it had this error:
from pydantic import BaseModel, Field, SecretField
ImportError: cannot import name 'SecretField' from '...
- 310
1
vote
1
answer
605
views
prefect orion start error: No such command 'orion'
I am learning workflow orchestration from data engineering zoomcamp and have successfully ingested the data to postgresql with prefect flow but when I try to run command prefect orion start to see the ...
- 1,241
0
votes
1
answer
121
views
Prefect trigger docker image
So I have a docker image on Azure container instance which I can start manually on azure portal. Now I want to run this image using prefect. Is there any way to create a deployment where it just runs ...
- 417
5
votes
1
answer
4k
views
Prefect : ImportError: cannot import name 'SecretField' from 'pydantic'
I'm currently using prefect to orchestrate some simple task in python. It's working fine until I get this error :
Traceback (most recent call last):
File "test.py", line 2, in <module&...
- 310
0
votes
1
answer
821
views
How to run locally scheduled flow in Prefect V2
I used to work with local scheduled flows in prefect v1 like :
from prefect import Flow, Task
from prefect.schedules import CronSchedule
@Task
def hello_world():
print("hello world")
...
-1
votes
1
answer
58
views
Is Prefect's VertexRun a GCP or AWS solution?
In Prefect's documentation, under the definition of vertex run the example definition was for Google's "e2-standard-4".
However, it says there that the spec should be workerPoolSpec linking ...
- 974
3
votes
0
answers
205
views
Connect to a Keycloak securised Prefect api url with Prefect CLI
I've built my own gke cluster to run jobs on it
I've deployed prefect-server, let's say on https://prefect.my-site.com/
but since it has no authentification system, i've deployed keycloak on top of it ...
- 327
0
votes
1
answer
115
views
Prefect - Unable to Persist Dask Dataframe to Dask Client
I'm fairly new Prefect and struggling with using Dask persistence within a Prefect Flow. I'm not sure if this an issue with code or a limitation within Prefect.
I made the following trivial example of ...
- 51
1
vote
0
answers
122
views
using HAProxy to expose service UI - behind haproxy on Kubernetes
My problem:
Exposing a UI for service behind haproxy fails
Details:
I'm using a server (Prefect) that exposes an API and UI in different urls.
Using haproxy for routing and authentication with the API ...
- 11
0
votes
1
answer
530
views
Prefect: User "system:serviceaccount:hm-prefect:prefect-worker" cannot create resource "jobs" in API group "batch" in the namespace "default"
As Prefect work pools and workers are now generally available after 2.11.0. I am trying to switch from Prefect Agent to Prefect Worker.
I deployed Prefect Server by
helm upgrade \
prefect-server \
...
- 49.5k
1
vote
1
answer
380
views
How to get a Prefect flow to run inside a conda environment
I'm new to Prefect and I'm trying to get my flow to run inside a conda environment. I'm trying to follow the yaml manifest deployment outlined here:
https://discourse.prefect.io/t/how-to-run-my-...
2
votes
0
answers
572
views
Kubernetes prefect UI cannot connect to api
So I am having an issue with prefect-server UI connecting to my API within my kubernetes environment. I am using a nginx as my ingress.
I have my ingress configuration for the prefect-server service ...
- 103
1
vote
1
answer
321
views
Why is "pip install prefect" downloading different versions of Prefect on different computers
When running a prefect python (3.11) script on a Windows OS, I am getting the error:
lib\site-packages\prefect\utilities\filesystem.py", line 5, in <module>
import resource
...
- 13
0
votes
1
answer
1k
views
Wait for successful execution of previous flow on Prefect
I'm trying to use Prefect 2 to configure the execution of a few tasks on Windows.
I often have problems with some tasks, that I might need to fix manually and re-run it. After this, I want other tasks ...
- 1,116
0
votes
1
answer
689
views
Starting Prefect server on Windows
After installing prefect version 2.10.15 and having the Scripts folder for python in my PATH variable I ran the command prefect server start to find the error:
___ ___ ___ ___ ___ ___ _____
| _ \ _ \ ...
- 1,116
0
votes
1
answer
249
views
prefect-shell: Task run encountered an exception: RuntimeError: PID xxx failed with return code 6
When I run Rclone to list the difference between a local folder and a S3 folder in shell directly, it succeed.
➜ rclone copy --dry-run --include="*.tdms" --use-json-log /Users/hongbo-miao/...
- 49.5k
0
votes
1
answer
461
views
sending on_failure email in prefect.io
I am trying to send myself an email whenever a task fails but I am having some issues. Here is my code for context:
failure_notification_task.py
from prefect_email import email_send_message, ...
- 55
0
votes
1
answer
178
views
Deploy work queue in a paused state Prefect 2.0
I'm working in Prefect 2.0 and want to deploy a work pool with some work queues inside. These work queues each contain a specific task. This is the YAML file I'm using for the deployment:
###
### A ...
- 31
0
votes
2
answers
252
views
How to delete Prefect blocks in Python?
I am using Prefect 2.
I am able to create Prefect blocks, for example, AWS Credentials and Kubernetes Job blocks by
from prefect_aws import AwsCredentials
from prefect.infrastructure import ...
- 49.5k
1
vote
1
answer
1k
views
How to run Prefect over https with nginx?
I have a server running CentOS 7. On this server Prefect is installed and runs over port 4200. I access Prefect over http://localhost:4200/xyz. I would like to run Prefect over https. So the goal is ...
- 11
1
vote
1
answer
101
views
Prefect role deployment
Currently I'm working with Prefect and learning about it. I saw in the documentation that it's possible to assign and create different roles within Prefect Cloud for the necesarry access control. Now ...
- 31
1
vote
1
answer
679
views
I'm unable to access my Prefect 2 server remotely
I'm trying to set-up a Prefect 2 docker container but I can't access the running server instance from outside.
I'm 100% sure this is a Prefect problem because I tested everything with netcat and curl ...
- 49
0
votes
1
answer
300
views
Prefect changing the value a variable when writing to google storage
Hi I am beginning to think that prefect is changing the value of a variable, here is the situation:
when writing to a google cloud storage, when I use the upload_from_path on my code you will see that ...
- 667