Pynvme: An Open, Fast and Extensible Nvme SSD Test Tool: Crane Chu, Engineer, Founder Geng Yun Technology Pte LTD

pynvme: an open, fast and
extensible NVMe SSD test tool
Crane Chu, Engineer, Founder

GENG YUN Technology Pte Ltd
2020 Storage Developer Conference. © GENG YUN Technology Pte Ltd. All Rights Reserved. 1
Requirement
Changes in SSD
• SSD has been changing for the decade:
• media:
• SLC, MLC, TLC, QLC
• 2D => 3D
• PCM, 3D-Xpoint...
• host:
• PATA, SATA, PCIe/NVMe
• open-channel
• up coming: ZNS, KV, ... ?
• DRAM
• form factor
• Agile: good for the constant change and uncertainty
Agile Testing
• Developing and Testing are done interactively and iteratively.
• QA verifies the product of Dev/Test for customers.
• Testing and QA are different. Testing tools and QA tools are also different.
• Most available tools in the market are QA tools.
test QA
Developing
for developer for customer
before checkin before release
automatic manual Testing
white-box black-box
short long
changing stable QA Release
Experience with dnvme
• extended dnvme with a python wrapper in user space, thus
developers can write test scripts in Python.
• Some essential problems:

• performance:
• IOPS, latency, consistency
• test efficiency
• stress tests
• maintainness: kernel module
• memory: user can only allocate virtual memory
• Then, SPDK comes ...

• high performance
• user space driver
• DMA memory
• python + nvme: pynvme https://github.com/pynvme/pynvme

Design
SPDK
Architecture
library (ongoing): ZNS TCG psd
API: controller namespace qpair pcie buffer
SPDK: nvme driver cmdlog checksum ioworker
Open Ecosystem
Hardware
High Performance
4K read IOPS
400
350
300
250
200
150
100
50
0
1 2 4
fio test 1 fio test 2 fio test 3

pynvme test 1 pynvme test 2 pynvme test 3
Low Latency
latency
1600
1400
1200
1000
800
600
400
200
0
1 10 50 90 99 99.9 99.99
fio test 1 fio test 2 fio test 3 pynvme test 1 pynvme test 2 pynvme test 3
Design
FIO pynvme
NVMe driver
in userspace
NVMe SSD
psd: Python Space Driver
psd: IOSQ IOCQ SQE CQE PRPList
API: controller namespace qpair pcie buffer
SPDK: nvme driver cmdlog checksum ioworker
NVMe SSD
Features
• access PCI configuration space
• access NVMe registers in BAR space
• send any NVMe admin/IO commands
• support callback functions for NVMe commands
• support MSI/MSIx interrupts
• transparent checksum verification on every LBA
• generates IO workload of high performance and low latency
• support multiple namespaces
• support multiple tests on different controllers
• integrate with the test framework pytest
• integrate with VSCode to display cmdlog in GUI
• support NVMe over TCP targets
• doc: https://pynvme.readthedocs.io/
Examples
Example: hello world
• pytest fixtures
• callback functions for commands
• sync point: waitdone()
• run test:
> make setup
> make test TESTS=scripts/test_examples.py::test_hello_world
pciaddr=0000:3d:00.0
pciaddr=172.168.5.44
Example: psd
• create IOCQ/IOSQ respectively with
specified sqid and cqid
• fill commands in SQ entries
• manually trigger doorbell
Example: ioworker
• fio-like IO generator
• Python API
• better performance
• sending IO in separated processes
• define IO pattern in parameters
• support multiple ioworkers
• support workload from simple to
complex
Example: dirty power cycle
• ioworker sending IO in a separated
process
• cut power in main process when
ioworker is working
Example: customized nvme init
• pynvme has a default nvme
initializaiton process
• users can define different nvme
initialization process in test scripts
Example: latency of JEDEC workload
• ioworker generates IO according to
JEDEC enterprise workload
• scripts get latency at different
percentile (99%, 99.99%, 99.9999%)
Example: check on error code
• pynvme through a warning when
command completes with an error
• scripts can use pytest to capture and
check the warning
Example: sanitize an aer
• check if sanitize is
supported
• start sanitize operation
• send a write command,
and check if it is
aborted due to the
sanitize operation in
progress
• monitor the sanitize
progress till it is
completed
• check the final sanitize
status in logpage
Example: multiple processes
• process 1: ioworker with flush, read,
trim commands
• process 2: ioworker with flush, write,
trim commands
• main process: print temperature
value in SMART data for every
second
Live Demo in VSCode
pynvme builds your own tests.
Ecosystem Scripts Extendability
Hardware Performance Service

Please take a moment
to rate this session.
Your feedback matters to us.
Thank you!!!

Pynvme: An Open, Fast and Extensible Nvme SSD Test Tool: Crane Chu, Engineer, Founder Geng Yun Technology Pte LTD

Uploaded by

Copyright:

Available Formats

Pynvme: An Open, Fast and Extensible Nvme SSD Test Tool: Crane Chu, Engineer, Founder Geng Yun Technology Pte LTD

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Pynvme: An Open, Fast and Extensible Nvme SSD Test Tool: Crane Chu, Engineer, Founder Geng Yun Technology Pte LTD

Uploaded by

Copyright:

Available Formats

pynvme: an open, fast and

extensible NVMe SSD test tool

Crane Chu, Engineer, Founder

• Agile: good for the constant change and uncertainty

before checkin before release

automatic manual Testing

changing stable QA Release

• Some essential problems:

• Then, SPDK comes ...

• python + nvme: pynvme https://github.com/pynvme/pynvme

library (ongoing): ZNS TCG psd

API: controller namespace qpair pcie buffer

SPDK: nvme driver cmdlog checksum ioworker

fio test 1 fio test 2 fio test 3

psd: IOSQ IOCQ SQE CQE PRPList

API: controller namespace qpair pcie buffer

SPDK: nvme driver cmdlog checksum ioworker

Ecosystem Scripts Extendability

Hardware Performance Service

Your feedback matters to us.

You might also like