Add support for loading and inferencing on aws-neuron and aws-neuronx #18199

takipipo · 2024-12-12T09:46:11Z

Given that the current Ultralytics does not support AWS-Neuron, AWS-NeuronX, which is one of the unsupported options by Ultralytics, AWS-NeuronX is a cost-effective way to productionize the YOLO models.

Here is the Proposal method for exporting the Ultralytics model to AWS-Neuron.

from ultralytics import NeuronYOLO

model = NeuronYOLO("yolov8n.pt")
model.export(format="neuron")

neuron_model = NeuronYOLO("yolov8n.neuron")
neuron_model.predict("https://ultralytics.com/images/bus.jpg")

Here is the Proposal method for exporting the Ultralytics model to AWS-NeuronX

from ultralytics import NeuronYOLO

model = NeuronYOLO("yolov8n.pt")
model.export(format="neuronx")

neuronx_model = NeuronYOLO("yolov8n.neuronx")
neuronx_model.predict("https://ultralytics.com/images/bus.jpg")

cc: @nirattisai-t @luangtatipsy

Support aws neuronx

fix: NeuronYOLO base class

fix: support python < 3.10

Feature/support inf1

Sync fork

sentry-io · 2024-12-12T09:46:25Z

🔍 Existing Issues For Review

Your pull request is modifying functions with the following pre-existing issues:

📄 File: ultralytics/cfg/init.py

Function	Unhandled Issue
`check_cfg`	TypeError: 'epochs=epoch' is of invalid type str. 'epochs' must be an int (i.e. 'epochs=8') ... `Event Count:` 4
`check_cfg`	TypeError: 'epochs=50 imgsz=640' is of invalid type str. 'epochs' must be an int (i.e. 'epochs=8') ... `Event Count:` 2

_{Did you find this useful? React with a 👍 or 👎}

github-actions · 2024-12-12T09:46:25Z

Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

I have read the CLA Document and I sign the CLA

2 out of 3 committers have signed the CLA.
✅ (takipipo)[https://github.com/takipipo]
✅ (UltralyticsAssistant)[https://github.com/UltralyticsAssistant]
❌ @ubuntu
Ubuntu seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You can retrigger this bot by commenting recheck in this Pull Request.}_{Posted by the CLA Assistant Lite bot.}

takipipo · 2024-12-12T09:47:07Z

I have read the CLA Document and I sign the CLA

codecov · 2024-12-12T09:51:32Z

Codecov Report

Attention: Patch coverage is 7.51534% with 603 lines in your changes missing coverage. Please review.

Project coverage is 71.52%. Comparing base (1cfe60e) to head (2c81115).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
ultralytics/nn/neuron_autobackend.py	5.96%	394 Missing ⚠️
ultralytics/engine/neuron_exporter.py	0.00%	186 Missing ⚠️
ultralytics/engine/neuron_predictor.py	0.00%	11 Missing ⚠️
ultralytics/engine/neuron_model.py	50.00%	6 Missing ⚠️
ultralytics/models/yolo/detect/neuron_predict.py	55.55%	4 Missing ⚠️
ultralytics/models/yolo/neuron_model.py	75.00%	2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (1cfe60e) and HEAD (2c81115). Click for more details.

HEAD has 8 uploads less than BASE

Flag BASE (1cfe60e) HEAD (2c81115)

Tests 8 4

Benchmarks 6 3

GPU 2 1

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #18199      +/-   ##
==========================================
- Coverage   78.52%   71.52%   -7.01%     
==========================================
  Files         128      134       +6     
  Lines       17138    17785     +647     
==========================================
- Hits        13458    12721     -737     
- Misses       3680     5064    +1384

Flag	Coverage Δ
Benchmarks	`33.97% <7.51%> (-1.07%)`	⬇️
GPU	`37.32% <7.51%> (-3.79%)`	⬇️
Tests	`65.54% <7.51%> (-6.92%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ambitious-octopus

Hey @takipipo , the AWS-Neuron inference integration you're working on looks very promising. Thank you for your effort and contributions!
That said, the PR requires some adjustments and cleanup to align with our codebase.

Duplicate Code: You've introduced a new exporter.py and autobackend.py, which results in a lot of duplicate code. Instead of creating new files, please integrate your changes into the existing exporter.py and autobackend.py to maintain consistency and reduce redundancy.
NeuronYOLO Model: There’s no need to create a separate NeuronYOLO model for this integration. Both inference and export functionality should be handled within autobackend.py and exporter.py as part of the existing framework.

Unfortunately, I’m unable to test your PR in its current state. Once you've made these changes I’ll take another look.

Thanks again for your contributions!

ambitious-octopus · 2024-12-13T17:06:31Z

ultralytics/cfg/__init__.py

Why did you simplify the docstrings here?

ambitious-octopus · 2024-12-13T17:06:58Z

ultralytics/engine/exporter.py

@@ -614,7 +614,7 @@ def export_mnn(self, prefix=colorstr("MNN:")):

    @try_export
    def export_ncnn(self, prefix=colorstr("NCNN:")):
-        """YOLO NCNN export using PNNX https://github.com/pnnx/pnnx."""
+        """YOLOv8 NCNN export using PNNX https://github.com/pnnx/pnnx."""


Suggested change

"""YOLOv8 NCNN export using PNNX https://github.com/pnnx/pnnx."""

"""YOLO NCNN export using PNNX https://github.com/pnnx/pnnx."""

takipipo and others added 27 commits June 26, 2024 15:11

Add export and load to aws neuronx

9041c1d

feat: create new file to support neuronx

471e6b7

feat: add neuron autobackend

2e11f9b

fix: utilize Neuron AutoBackend

9134643

fix: remove neuronx from default exporter and auto backend

da61d33

Merge pull request #1 from takipipo/support-aws-neuronx

04dd2b5

Support aws neuronx

Merge branch 'ultralytics:main' into main

0427be0

fix: NeuronYOLO base class

6d94204

Merge pull request #3 from wisesight/neuronx

19c137e

fix: NeuronYOLO base class

Merge branch 'ultralytics:main' into main

0c23ed4

fix: support python < 3.10

8f5e5f5

Merge pull request #4 from wisesight/neuronx

fb0708e

fix: support python < 3.10

feat: add neuron loader and exporter

93405bd

fix: add missing torch_neuron import

c5c7d23

fix: missing statement for neuron

66f5e32

Auto-format by https://ultralytics.com/actions

c4f18df

refactor: remove unused statements

adeeb4e

Auto-format by https://ultralytics.com/actions

3af7dfa

fix: add import neuron neuronx when load model to memory

4c3e99a

Auto-format by https://ultralytics.com/actions

91f13a9

fix: add import neuron neuronx when load model to memory

509bf16

Auto-format by https://ultralytics.com/actions

ca32004

Merge pull request #6 from wisesight/feature/support-inf1

ab37921

Feature/support inf1

sync with ultralytics main at 1cfe60e

1630fec

refactor: public explorer in __init__ file

3c9ac12

Auto-format by https://ultralytics.com/actions

7ec1a8f

Merge pull request #8 from wisesight/sync-fork

2c81115

Sync fork

glenn-jocher self-requested a review December 12, 2024 18:21

ambitious-octopus requested changes Dec 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for loading and inferencing on aws-neuron and aws-neuronx #18199

Add support for loading and inferencing on aws-neuron and aws-neuronx #18199

takipipo commented Dec 12, 2024 •

edited

Loading

sentry-io bot commented Dec 12, 2024

github-actions bot commented Dec 12, 2024 •

edited

Loading

takipipo commented Dec 12, 2024

codecov bot commented Dec 12, 2024 •

edited

Loading

ambitious-octopus left a comment

ambitious-octopus Dec 13, 2024

ambitious-octopus Dec 13, 2024

	"""YOLOv8 NCNN export using PNNX https://github.com/pnnx/pnnx."""
	"""YOLO NCNN export using PNNX https://github.com/pnnx/pnnx."""

Add support for loading and inferencing on aws-neuron and aws-neuronx #18199

Are you sure you want to change the base?

Add support for loading and inferencing on aws-neuron and aws-neuronx #18199

Conversation

takipipo commented Dec 12, 2024 • edited Loading

sentry-io bot commented Dec 12, 2024

🔍 Existing Issues For Review

github-actions bot commented Dec 12, 2024 • edited Loading

takipipo commented Dec 12, 2024

codecov bot commented Dec 12, 2024 • edited Loading

Codecov Report

ambitious-octopus left a comment

Choose a reason for hiding this comment

ambitious-octopus Dec 13, 2024

Choose a reason for hiding this comment

ambitious-octopus Dec 13, 2024

Choose a reason for hiding this comment

takipipo commented Dec 12, 2024 •

edited

Loading

github-actions bot commented Dec 12, 2024 •

edited

Loading

codecov bot commented Dec 12, 2024 •

edited

Loading