Kubernetes prefect UI cannot connect to api

Question

So I am having an issue with prefect-server UI connecting to my API within my kubernetes environment. I am using a nginx as my ingress.

I have my ingress configuration for the prefect-server service as so:

https://myhost.com/prefect/ -> http://localhost:3000/api

I am able to access the UI, but it saying it cannot connect to the api:

deployment.yml for prefect-server:

...
      - image: <my registry>
        imagePullPolicy: Always
        name: prefect-server
        command: ["prefect", "server", "start", "--host", "0.0.0.0", "--log-level", "WARNING", "--port", "3000"]
        workingDir: /opt/prefect
        env:
          - name: PREFECT_UI_API_URL 
            value:  "http://localhost:3000/api"

deployment.yml for agent:

...
      - image: <my registry>
        imagePullPolicy: Always
        name: prefect-agent
        command: ["prefect", "agent", "start", "--work-queue", "dev"]
        workingDir: /opt/prefect
        env:
          - name: PREFECT_API_URL 
            value:  "http://prefect-server.prefect.svc.cluster.local:3000/api" #WORKS

Here is the output of kubectl describe service -n prefect

Name:              prefect-server
Namespace:         prefect
Labels:            app.kubernetes.io/managed-by=Helm
                   run=prefect-server
Annotations:       meta.helm.sh/release-name: prefect
                   meta.helm.sh/release-namespace: prefect
Selector:          run=prefect-server
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                11.11.11.11 <redacted>
IPs:               11.11.11.11 <redacted>
Port:              <unset>  80/TCP
TargetPort:        3000/TCP
Endpoints:         11.22.22.22:3000 <redacted>
Session Affinity:  None
Events:            <none>

The agent is able to connect to the service container with no issues. For the prefect-server PREFECT_UI_API_URL, I have tried:

"http://prefect-server.prefect.svc.cluster.local:3000/api"
"https://myhost.com/prefect/api"
"http://127.0.0.1:3000/api"

UPDATE: Thank you VonC for your help, I think I am heading in the right direction. I made the changes to the PREFECT_UI_API_URL and ingress. Also had to change the agent url to http://prefect-server.prefect.svc.cluster.local:3000/prefect/api. However, the "cannot to to Server API.." error goes away but the page is still 404 when I visit htts://myhost.com/prefect/flow-runs endpoint.

Output for kubectl describe ingress -n prefect

Name:             prefect
Namespace:        prefect
Address:          X.X.X.50
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  /mysecret terminates 
Rules:
  Host                            Path  Backends
  ----                            ----  --------
  myhost.com  
                                  /prefect       prefect-server:80 (X.X.X.131:3000)
                                  /prefect/api   prefect-server:3000 (X.X.X.131:3000)
Annotations:                      field.cattle.io/publicEndpoints:
                                    [{"addresses":["X.X.X.50"],"port":80,"protocol":"HTTP","serviceName":"prefect:prefect-server","ingressName":"prefect:prefect","hostname"...
                                  ingress.kubernetes.io/secure-backends: true
                                  ingress.kubernetes.io/ssl-redirect: true
                                  kubernetes.io/ingress.class: f5
                                  meta.helm.sh/release-name: prefect
                                  meta.helm.sh/release-namespace: prefect
                                  virtual-server.f5.com/balance: round-robin
                                  
                                  virtual-server.f5.com/http-port: 443
                                  virtual-server.f5.com/ip: X.X.X.50
                                  virtual-server.f5.com/partition: kubernetes

My ingress.yml:

{{- if .Values.ingress.enabled }}
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: prefect
  annotations:
{{- if eq .Values.ingress.class "f5" }}
    {{- if .Values.ingress.tls }}
    ingress.kubernetes.io/secure-backends: "true"
    {{- end }}
    {{- $f5healthpath := printf "%s%s" .Values.ingress.hostname .Values.ingress.path }}
    {{- $f5path := trimPrefix "/" .Values.ingress.path }}
    {{- $f5partition := .Values.ingress.f5partition }}
    {{- $f5httpport := .Values.ingress.f5httpport}}
    {{- $f5ip := .Values.ingress.f5ip }}
    {{- $f5ssl := .Values.ingress.f5ssl }}
    {{- $f5class:= .Values.ingress.class}}
    {{- range $key, $value := .Values.ingress.annotations }}
    {{ $key }}: {{ $value | replace "F5HEALTHPATH" $f5healthpath | replace "F5PATITION" $f5partition | replace "F5HTTPPORT" $f5httpport | replace "F5IP" $f5ip | replace "F5SSL" $f5ssl | replace "F5CLASS" $f5class | replace "F5PATH" $f5path | quote }}
    {{- end }}
{{- else }}
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
{{- end }}
spec:
  tls:
  - secretName: /mysecret
  rules:
  - host: "myhost.com"
    http:
      paths:
      - path: "/prefect"
        pathType: Prefix
        backend:
          serviceName: prefect-server
          servicePort: 80
      - path: "/prefect/api"
        pathType: Prefix
        backend:
          serviceName: prefect-server
          servicePort: 3000       
{{- end }}

I believe the issue may be within my ingress, I would like all the paths to the prefect-server have the prefix path /prefect/