ThunderboltDevice Driver Programming Guide by Apple
ThunderboltDevice Driver Programming Guide by Apple
ThunderboltDevice Driver Programming Guide by Apple
Driver Programming
Guide
Contents
Enabling MSIs 16
Single Bottleneck Routine (MMIO Read/Write) 18
The Thunderbolt interface is a new I/O technology that supports high-resolution displays and high-performance
data devices through a single expansion interface.
At a Glance
The Thunderbolt interface is made up of an Intel controller chip designed to tunnel DisplayPort (DP) version
1.1 and PCI Express (PCIe) information. Its dual-link cables are designed to carry 10 gigabits per second (Gbps)
of data bidirectionally on each link, for a total of 40 Gbps per cable. Thunderbolt ports may be hot-plugged,
daisy chained with up to six devices in depth, and can be connected host-to-host.
Thunderbolt ports use a Mini DisplayPort (mDP) connector and provide 10 W of power to downstream devices.
Thunderbolt cables may be connected by either end (both ends are exactly the same). Thunderbolt technology
is a host-centered and host-managed system (like USB), but also allows for host-to-host connections.
A Thunderbolt port or cable provides two 10 Gbps bidirectional links, but these two links cannot be bonded
into a single channel. Host software must assign specific paths (for example, DP, PCIe, native) to each link to
balance the load. This functionality is included in EFI and in OS X. For example, if there are two Thunderbolt
enabled displays attached, it can choose to route the DisplayPort (DP) over each link and use the remaining
bandwidth for PCI. If there is only one Thunderbolt display, software can route the DisplayPort traffic over the
optimal link. The Thunderbolt interface has no requirement for fixed routing.
It is possible to boot directly from Thunderbolt devices. Apple provides a Unified Target Disk Mode (UTDM),
which supports FireWire and Thunderbolt connections. Users may boot from UTDM over FireWire or Thunderbolt
connections as well.
Prerequisites
Drivers must be built as Universal and be able to support systems with greater than 2 GB of RAM. Refer to the
section entitled Kernel Extensions and Drivers in the 64-Bit Transition Guide for more details. In particular,
all drivers must use IODMACommand for scatter-gather list support.
See Also
Apple provides the following reference material. For more information please refer to these documents.
Reference Material
This chapter describes the basic components and functions of the Thunderbolt technology and how these are
used with DisplayPort (DP) and PCI Express (PCIe) devices.
The Intel controller chip can be used in both host mode and endpoint mode. In host mode, the controller has
a Gen2 x4 uplink to the system PCI Express Root Complex and one or more DisplayPort (DP) inputs (depending
on the graphics capabilities of the system). Additionally, there is a PCI switch and a collection of DMA engines,
referred to as the Native Host Interface (NHI). The PCI switch enables PCI uplink for downstream devices and
the NHI is used for software protocols and device discovery. The controller chip includes four Thunderbolt
ports as outputs.
Note: Each open circle in the diagram indicates an adapter, that translates the native transport (NHI,
PCI, DP) for transmission over Thunderbolt ports and cables.
In endpoint mode, the controller chip provides a Gen2 x4 downlink (supports x1, x2, x4 at Gen2 speeds) or
Gen1 4x1 downlinks to support multiple devices. Figure 1-2 provides an example of the controller chip as an
endpoint with both a x4 Gen2 PCI and a DisplayPort (DP) output option.
Figure 1-2
Note: The PCI switch gets its PCI input from the upstream Thunderbolt connection through the
Thunderbolt switch. Similarly, the DisplayPort (DP) out must be connected through the Thunderbolt
topology to a DP in adapter.
In Figure 1-3, a PCI path has been established by routing the PCI upstream connection from the PCI switch in
the Mac host through the Thunderbolt switch and across the cable to the Thunderbolt switch in the PCI chassis.
From the Thunderbolt switch inside the PCI chassis, the path is routed up to the PCI switch in the controller
chip. The PCI fanout from the switch provides 4x1 connectivity.
PCI paths can utilize fanout or can connect directly to PCI adapters in the Mac host. The host software controls
how the paths are established. Wherever possible, OS X and EFI attempt to use the same algorithms for path
selection to provide a similar user experience. Devices can also provide hints to software, indicating how much
and the type of bandwidth the device consumes in order for the software to make a more optimal selection.
An example of fanout is shown in Figure 1-4.
Figure 1-4
The disk labeled Disk #1 has a PCI switch inside the controller chip used for fanout. In Figure 1-5, the host
software can connect Disk #2 directly to the host, instead of using fanout.
Figure 1-5
Only four PCI adapters are available through the Mac host. The host determines the optimal routes for all of
the attached devices.
This chapter provides guidelines for working with the Thunderbolt technology. Keep these guidelines in mind
when creating compatible devices.
10
If your driver uses timers to detect device failures, you may need to lengthen the timeout value.
If your driver needs to perform an operation the moment an interrupt occurs, you may have to find another
way to schedule the operation, such as telling the device to delay the action until the next interrupt or
using a software-defined phase-locked loop.
If your device expects the driver to send some data and then wait for the device to acknowledge it before
sending more data, then the device needs to be more tolerant of delays than it otherwise might need to
be, or else the driver needs to handle failure gracefully.
If your driver tells an isochronous endpoint to stop sending data, it may continue to send data longer than
usual.
11
Listing 2-1
<key>IOPCITunnelCompatible</key>
<true/>
You must include the key in the IOKitPersonalities section of the Info.plist file or the IOPCIFamily will
not load the drivers for Thunderbolt connected PCI devices. This opt-in key protects consumers against older
drivers that have not yet been updated to support Thunderbolt technology. This method also makes it easy
for third party developers, who have properly updated their driver, to signify they have done so. (See example
driver Listing 2-2 (page 12).)
Note: A PCI device driver must not ship with this key set, unless it fully supports the requirements
set forth in this document. This opt-in key is specific to a personality for the driver, but the personality
can contain additional information in the form of key/value pairs in order to provide additional
context to the device driver should the IOPCITunnelCompatible key not be enough.
An example driver that includes this key in one of its personalities is AppleAHCIPort.kext, which is shown
in the PCI Driver Personality (Thunderbolt Key Set).
Listing 2-2
<key>GenericAHCI</key>
<dict>
<key>CFBundleIdentifier</key>
<string>com.apple.driver.AppleAHCIPort</string>
<key>Chipset Name</key>
<string>AHCI Standard Controller</string>
<key>IOClass</key>
<string>AppleAHCI</string>
<key>IOPCIClassMatch</key>
<string>0x01060100&0xffffff00</string>
<key>IOProbeScore</key>
<integer>800</integer>
<key>IOProviderClass</key>
<string>IOPCIDevice</string>
<key>Vendor Name</key>
<string>Unknown</string>
12
key>IOPCITunnelCompatible/key>
<true/>
</dict>
Additionally, PCI device drivers that present storage to the operating system must ensure the Physical
Interconnect Location key is set to indicate that the device is externally connected. (See physical interconnect
location key example Listing 2-4 (page 13) .)
Listing 2-4
setProperty ( kIOPropertyPhysicalInterconnectLocationKey,
kIOPropertyExternalKey );
Note: This key must be set before any devices are instantiated (that is, early in the drivers start()
routine or when it loads based on its IOKitPersonality in the Info.plist file).
13
This chapter provides guidelines to help you support Thunderbolt devices using Message Signaled Interrupts
(MSI) and to understand how to respond to hardware devices that do not support MSI. There is also a section
on hot plug operations with PCI Devices that describes how to change PCI drivers so that they are able to deal
with unplanned disconnections.
Most modern PCI devices support flexibility when dealing with interrupt routing. With the advent of PCI-X (PCI
eXtended) and PCIe (PCI Express), Message Signaled Interrupts were introduced as an in-band mechanism for
asserting interrupts.
OS X services all interrupts on CPU 0 and uses the secondary interrupt context threads to defer work to task-level
interrupts rather than primary-interrupt levels. This method ensures that multiple drivers can run in parallel
on the available CPUs, that the OS can schedule real-time threads with accuracy, and that the system remains
responsive to user interaction. The best practice methods, described in I/O Kit Fundamentals , encourage the
use of the IOWorkLoop abstraction model for device drivers in order to defer work to the IOWorkLoop thread.
There may be a small number of devices that require work to be done in the primary interrupt context, however,
drivers should spend the least amount of time possible in the primary-interrupt context.
14
Note: Since Message Signaled Interrupts are never shared, most device drivers can use the MSI
vector with an IOInterruptEventSource instance and avoid spending time in the primary interrupt
context.
The filter is called, but the device has not signaled an interrupt.
This may be due to another device sharing the same interrupt pin. The filter routine should return false.
2.
3.
The driver can handle the interrupt at this level and return the device to a non interrupting state. If
the interrupt action needs to be run, the filter routine should call signalInterrupt().
Note: The filter may be called again either before or during the execution of the interrupt
action. The filter routine should return false.
b.
The driver cannot handle the interrupt at this level, but it can prevent the device from signaling
another interrupt (for example, by manipulating an internal mask).
The filter routine should prevent the device from signaling another interrupt and then call
signalInterrupt() to cause the interrupt action to be run. The interrupt action should handle the
interrupt and return the device to a state where it can again signal an interrupt. The filter routine
should return false.
15
Note: The MMIO read cycles and PCI Configuration cycles of any type (read or write) are non-posted
transactions from the CPU, which turn into PCIe transactions and may take many thousands of clock
cycles to complete . These transactions should be minimized whenever possible.
Apple provides developers with tools that can track primary interrupt times and help developers to minimize
the time needed for primary interrupts.
Enabling MSI
To enable MSI, a device driver should do the following, assuming that the provider is an IOPCIDevice instance:
Note: The device driver should use source as the last argument when creating an
IOInterruptEventSource or IOFilterInterruptEventSource.
Listing 3-1
Enabling MSIs
int
index
= 0;
int
source = 0;
= kIOReturnSuccess;
int interruptType
= 0;
16
As a basic guideline, developers should modify their drivers to handle a return value of 0xFFFFFFFF. If any
thread, callback, interrupt filter, or code path in a driver receives 0xFFFFFFFF indicating the device has been
unplugged, then all threads, callbacks, interrupt filters, interrupt handlers, and other code paths in that driver
must cease MMIO reads and writes immediately and prepare for termination.
Note: Use an instance variable to enforce modifying the drivers to return a value of 0xFFFFFFFF
and prevent the driver from issuing additional MMIO or PCI Configuration cycles, which may cause
additional transaction timeouts.
If 0xFFFFFFFF is a legal value for a particular register offset, one additional read of a different register, which
is known to never return 0xFFFFFFFF is the preferred mechanism for determining if the device is still connected.
Finally, if I/O Kit has already performed termination and called the drivers willTerminate() method, no
further accesses should be performed.
17
Once it has been determined that a device is no longer connected, do not try to clean up or reset the hardware
as attempts to communicate with the hardware may lead to further delays.
Apple recommends auditing usage of MMIO writes when no further access should be performed. MMIO writes
are posted transactions and it is possible for device drivers to queue up multiple writes in a row.
Note: It is best to prevent large numbers (i.e. > 2) of writes without an intervening read access even
though writes are preferred over reads in PCI.
A typical way for a developer to solve this problem is to provide a single bottleneck routine for all MMIO reads
and have that routine check the status of the device before beginning the actual transaction. An example of
such a routine using little endian fields for its memory mapped apertures follows:
Listing 3-2
class AppleSamplePCI
{
...
bool
fDeviceRemoved;
volatile unit8_t *
fBaseAddressRegister;
...
unit32_t
ReadRegister ( uint32_t
offset );
virtual bool
);
};
unit32_t
AppleSamplePCI::ReadRegister ( unint32_t offset )
{
unint32_t
if
result = 0xFFFFFFFF;
( !fDeviceRemoved )
{
result = OSReadLittleInt32 ( fBaseAddressRegister, offset );
if ( result == 0xFFFFFFFF )
fDeviceRemoved = true;
18
}
return result;
}
bool
AppleSamplePCI::willTerminate ( IOService * provider, IOOptionBits options )
{
fDeviceRemoved = true;
return super::willTerminate ( provider, options );
}
Similar routines can be written for PCI Configuration cycle transactions, which may receive a return value of
0xFFFFFFFF and MMIO reads of smaller sizes.
Note: Be sure to initialize the variables fBaseAddressRegister and fDeviceRemoved before
use.
19
<true/>
Important: If your driver does not explicitly declare support for pausing, your driver will never receive
pause requests. As a result, devices may fail to appear when the user plugs in additional devices, particularly
on hardware with multiple Thunderbolt ports. For this reason, you are strongly encouraged to support this
functionality as soon as possible.
20
0, 0, 0 },
To avoid disrupting service, write your code in a way that makes entering and exiting the pause state as fast
as possible. In particular, you do not need to run all of the code that you would use for a
kIOPCIDeviceOffState/kIOPCIDeviceOnState transition, because the device remains powered on
through the state transition, making a full reinitialization unnecessary.
However, when OS X tells your driver to pause, your driver should still do many of the things that it would do
when the computer goes into safe sleepthat is, it should tell the device to stop issuing additional transactions
and then wait until all outstanding transactions have finished before telling OS X that the devices power state
has changed.
While the driver is paused:
The driver should not access the device using memory-mapped I/O or configuration space transactions
The device should not generate any interrupts, whether MSI or pin-based interrupts
21
Note: This quiesced (inactive) behavior is similar to what your driver should do in all of the other
inactive power states (that is, all power states other than kIOPCIDeviceOnState), so the driver
state should be very similar to the off and doze states. However, because the system is not actually
being powered down or going to sleep, the device does not receive any PCI broadcast messages,
such as PME_turn_off.
When a driver is resumed after a pause, the driver should act as though the computer just woke from safe
sleep, but without performing any unnecessary hardware initialization (because the device remained powered).
In particular, it must determine whether the device has changed addresses, and if it has, it must use the new
physical addresses for all future communication with the device. The following values may have changed:
The devices MSI capability register block values for address and value, but not the number of MSIs allocated
The PCI power management configuration block registers of the devicethat is, the device will not be
put into a device sleep state (runtime D3)
22
Most drivers have few or no dependencies on the items above being changed. If they do, while entering the
state kIOPCIDeviceOnState (from any other state), these dependencies should be updated to the current
configuration of the device.
23
This chapter contains debugging tips that may be useful when tracking down problems in Thunderbolt drivers.
If you are writing DMA programs, although physical segments returned by IOMemoryDescriptor or
IODMACommand are contiguous ranges of I/O addresses, the I/O addresses for the segments may be
different than the CPU physical addresses, and the ranges are no longer necessarily contiguous in physical
memory from perspective of the CPU.
As a result, segments will usually be the same size as the full IOMemoryDescriptor size (for a virtually
contiguous IOMemoryDescriptor object with a single source buffer).
Note: This assumes you are using the default behavior for IODMACommand (mappingOptions
= kMapped) or getPhysicalSegment (without the kIOMapperNone option set). Specifying
nondefault options produces different behavior.
By default, requests for physically contiguous memory return memory that is contiguous only from the
perspective of I/O devices, and may not be physically contiguous in RAM.
24
If the memory must be contiguous to the CPU as well, you must pass the
kIOMemoryHostPhysicallyContiguous option to IOBufferMemoryDescriptor when you request
the memory. As before, such requests can fail if no physically contiguous blocks are available.
Transactions now fail if memory descriptors or DMA commands are not properly prepared for reading or
writing.
The prepare method adds a mapping entry into the I/O address translation table. You must call prepare
(and specify the correct direction) before you tell the device to read data from or write data to that region
of memory.
You must also balance each prepare call with a matching call to the complete method. If you do not,
your driver will eventually fill up the mapping table, and you will get a kernel panic the next time a driver
tries to add a mapping table entry.
Important: The I/O direction matters. If you prepare a descriptor or DMA command for reading and
your device then tries to write to that memory, the transaction will fail.
The prepare and complete methods have additional overhead, though this impact is partially
counterbalanced by shorter scatter-gather lists.
In the example above, 13:0:0 is the bus, device, and function of the device that generated the fault. You can
determine if your device produced the fault by looking at the pci-debug field in the output of the ioreg or
in the IORegistryExplorer application and comparing the values.
For a list of reason codes, see the Intel Virtualization Technology for Directed I/O Specification.
The R indicates that a read operation triggered the fault. (A W indicates a write.) The final value is the address
that the device was trying to read or write (expressed as an I/O-space address).
When debugging faults, it can be useful to configure your kernel to panic whenever a fault occurs. To do this,
add the following flag in your kernel boot args:
25
pci=0x100
Disabling VT-d
When debugging PCIe device drivers, it is often useful to temporarily disable VT-d so that I/O addresses are
the same as the corresponding physical addresses. To disable VT-d, add the following to your kernel boot args:
dart=0x0
If the problem you are debugging goes away in this mode, it usually indicates one of the following mistakes:
Some part of your code is incorrectly passing a physical address in RAM to your device for DMA purposes
instead of an I/O address.
Your code failed to call prepare or called prepare incorrectly on an IOMemoryDescriptor object
before using it to perform DMA.
26
Listing 4-1
void
AppleSamplePCI::taggedRetain ( const void * tag ) const
{
void *
bt[16] = { 0 };
Using standard symbolication tools allows you to determine which functions or methods caused the reference(s)
to be taken. Furthermore, you can override taggedRelease() and match the retains and releases to find
calls to taggedRetain(), which have no corresponding call to taggedRelease().
Note: The const keyword at the end of the method signatures is required to properly override
these methods.
27
This table describes the changes to Thunderbolt Device Driver Programming Guide .
Date
Notes
2013-10-22
2011-12-21
28
Apple Inc.
Copyright 2013 Apple Inc.
All rights reserved.
No part of this publication may be reproduced,
stored in a retrieval system, or transmitted, in any
form or by any means, mechanical, electronic,
photocopying, recording, or otherwise, without
prior written permission of Apple Inc., with the
following exceptions: Any person is hereby
authorized to store documentation on a single
computer for personal use only and to print
copies of documentation for personal use
provided that the documentation contains
Apples copyright notice.
No licenses, express or implied, are granted with
respect to any of the technology described in this
document. Apple retains all intellectual property
rights associated with the technology described
in this document. This document is intended to
assist application developers to develop
applications only for Apple-labeled computers.
Apple Inc.
1 Infinite Loop
Cupertino, CA 95014
408-996-1010
Apple, the Apple logo, FireWire, Leopard, Mac,
Mac Pro, MacBook, OS X, and Snow Leopard are
trademarks of Apple Inc., registered in the U.S.
and other countries.
Intel and Intel Core are registered trademarks of
Intel Corporation or its subsidiaries in the United
States and other countries.
Even though Apple has reviewed this document,
APPLE MAKES NO WARRANTY OR REPRESENTATION,
EITHER EXPRESS OR IMPLIED, WITH RESPECT TO THIS
DOCUMENT, ITS QUALITY, ACCURACY,
MERCHANTABILITY, OR FITNESS FOR A PARTICULAR
PURPOSE. AS A RESULT, THIS DOCUMENT IS PROVIDED
AS IS, AND YOU, THE READER, ARE ASSUMING THE
ENTIRE RISK AS TO ITS QUALITY AND ACCURACY.
IN NO EVENT WILL APPLE BE LIABLE FOR DIRECT,
INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL
DAMAGES RESULTING FROM ANY DEFECT OR
INACCURACY IN THIS DOCUMENT, even if advised of
the possibility of such damages.
THE WARRANTY AND REMEDIES SET FORTH ABOVE
ARE EXCLUSIVE AND IN LIEU OF ALL OTHERS, ORAL
OR WRITTEN, EXPRESS OR IMPLIED. No Apple dealer,
agent, or employee is authorized to make any
modification, extension, or addition to this warranty.
Some states do not allow the exclusion or limitation
of implied warranties or liability for incidental or
consequential damages, so the above limitation or
exclusion may not apply to you. This warranty gives
you specific legal rights, and you may also have other
rights which vary from state to state.