Documentation / admin-guide / thunderbolt.rst

Based on kernel version 6.8. Page generated on 2024-03-11 21:26 EST.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336
.. SPDX-License-Identifier: GPL-2.0

 USB4 and Thunderbolt
USB4 is the public specification based on Thunderbolt 3 protocol with
some differences at the register level among other things. Connection
manager is an entity running on the host router (host controller)
responsible for enumerating routers and establishing tunnels. A
connection manager can be implemented either in firmware or software.
Typically PCs come with a firmware connection manager for Thunderbolt 3
and early USB4 capable systems. Apple systems on the other hand use
software connection manager and the later USB4 compliant devices follow
the suit.

The Linux Thunderbolt driver supports both and can detect at runtime which
connection manager implementation is to be used. To be on the safe side the
software connection manager in Linux also advertises security level
``user`` which means PCIe tunneling is disabled by default. The
documentation below applies to both implementations with the exception that
the software connection manager only supports ``user`` security level and
is expected to be accompanied with an IOMMU based DMA protection.

Security levels and how to use them
The interface presented here is not meant for end users. Instead there
should be a userspace tool that handles all the low-level details, keeps
a database of the authorized devices and prompts users for new connections.

More details about the sysfs interface for Thunderbolt devices can be
found in ``Documentation/ABI/testing/sysfs-bus-thunderbolt``.

Those users who just want to connect any device without any sort of
manual work can add following line to

  ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1"

This will authorize all devices automatically when they appear. However,
keep in mind that this bypasses the security levels and makes the system
vulnerable to DMA attacks.

Starting with Intel Falcon Ridge Thunderbolt controller there are 4
security levels available. Intel Titan Ridge added one more security level
(usbonly). The reason for these is the fact that the connected devices can
be DMA masters and thus read contents of the host memory without CPU and OS
knowing about it. There are ways to prevent this by setting up an IOMMU but
it is not always available for various reasons.

Some USB4 systems have a BIOS setting to disable PCIe tunneling. This is
treated as another security level (nopcie).

The security levels are as follows:

    All devices are automatically connected by the firmware. No user
    approval is needed. In BIOS settings this is typically called
    *Legacy mode*.

    User is asked whether the device is allowed to be connected.
    Based on the device identification information available through
    ``/sys/bus/thunderbolt/devices``, the user then can make the decision.
    In BIOS settings this is typically called *Unique ID*.

    User is asked whether the device is allowed to be connected. In
    addition to UUID the device (if it supports secure connect) is sent
    a challenge that should match the expected one based on a random key
    written to the ``key`` sysfs attribute. In BIOS settings this is
    typically called *One time saved key*.

    The firmware automatically creates tunnels for Display Port and
    USB. No PCIe tunneling is done. In BIOS settings this is
    typically called *Display Port Only*.

    The firmware automatically creates tunnels for the USB controller and
    Display Port in a dock. All PCIe links downstream of the dock are

    PCIe tunneling is disabled/forbidden from the BIOS. Available in some
    USB4 systems.

The current security level can be read from
``/sys/bus/thunderbolt/devices/domainX/security`` where ``domainX`` is
the Thunderbolt domain the host controller manages. There is typically
one domain per Thunderbolt host controller.

If the security level reads as ``user`` or ``secure`` the connected
device must be authorized by the user before PCIe tunnels are created
(e.g the PCIe device appears).

Each Thunderbolt device plugged in will appear in sysfs under
``/sys/bus/thunderbolt/devices``. The device directory carries
information that can be used to identify the particular device,
including its name and UUID.

Authorizing devices when security level is ``user`` or ``secure``
When a device is plugged in it will appear in sysfs as follows::

  /sys/bus/thunderbolt/devices/0-1/authorized	- 0
  /sys/bus/thunderbolt/devices/0-1/device	- 0x8004
  /sys/bus/thunderbolt/devices/0-1/device_name	- Thunderbolt to FireWire Adapter
  /sys/bus/thunderbolt/devices/0-1/vendor	- 0x1
  /sys/bus/thunderbolt/devices/0-1/vendor_name	- Apple, Inc.
  /sys/bus/thunderbolt/devices/0-1/unique_id	- e0376f00-0300-0100-ffff-ffffffffffff

The ``authorized`` attribute reads 0 which means no PCIe tunnels are
created yet. The user can authorize the device by simply entering::

  # echo 1 > /sys/bus/thunderbolt/devices/0-1/authorized

This will create the PCIe tunnels and the device is now connected.

If the device supports secure connect, and the domain security level is
set to ``secure``, it has an additional attribute ``key`` which can hold
a random 32-byte value used for authorization and challenging the device in
future connects::

  /sys/bus/thunderbolt/devices/0-3/authorized	- 0
  /sys/bus/thunderbolt/devices/0-3/device	- 0x305
  /sys/bus/thunderbolt/devices/0-3/device_name	- AKiTiO Thunder3 PCIe Box
  /sys/bus/thunderbolt/devices/0-3/key		-
  /sys/bus/thunderbolt/devices/0-3/vendor	- 0x41
  /sys/bus/thunderbolt/devices/0-3/vendor_name	- inXtron
  /sys/bus/thunderbolt/devices/0-3/unique_id	- dc010000-0000-8508-a22d-32ca6421cb16

Notice the key is empty by default.

If the user does not want to use secure connect they can just ``echo 1``
to the ``authorized`` attribute and the PCIe tunnels will be created in
the same way as in the ``user`` security level.

If the user wants to use secure connect, the first time the device is
plugged a key needs to be created and sent to the device::

  # key=$(openssl rand -hex 32)
  # echo $key > /sys/bus/thunderbolt/devices/0-3/key
  # echo 1 > /sys/bus/thunderbolt/devices/0-3/authorized

Now the device is connected (PCIe tunnels are created) and in addition
the key is stored on the device NVM.

Next time the device is plugged in the user can verify (challenge) the
device using the same key::

  # echo $key > /sys/bus/thunderbolt/devices/0-3/key
  # echo 2 > /sys/bus/thunderbolt/devices/0-3/authorized

If the challenge the device returns back matches the one we expect based
on the key, the device is connected and the PCIe tunnels are created.
However, if the challenge fails no tunnels are created and error is
returned to the user.

If the user still wants to connect the device they can either approve
the device without a key or write a new key and write 1 to the
``authorized`` file to get the new key stored on the device NVM.

De-authorizing devices
It is possible to de-authorize devices by writing ``0`` to their
``authorized`` attribute. This requires support from the connection
manager implementation and can be checked by reading domain
``deauthorization`` attribute. If it reads ``1`` then the feature is

When a device is de-authorized the PCIe tunnel from the parent device
PCIe downstream (or root) port to the device PCIe upstream port is torn
down. This is essentially the same thing as PCIe hot-remove and the PCIe
toplogy in question will not be accessible anymore until the device is
authorized again. If there is storage such as NVMe or similar involved,
there is a risk for data loss if the filesystem on that storage is not
properly shut down. You have been warned!

DMA protection utilizing IOMMU
Recent systems from 2018 and forward with Thunderbolt ports may natively
support IOMMU. This means that Thunderbolt security is handled by an IOMMU
so connected devices cannot access memory regions outside of what is
allocated for them by drivers. When Linux is running on such system it
automatically enables IOMMU if not enabled by the user already. These
systems can be identified by reading ``1`` from
``/sys/bus/thunderbolt/devices/domainX/iommu_dma_protection`` attribute.

The driver does not do anything special in this case but because DMA
protection is handled by the IOMMU, security levels (if set) are
redundant. For this reason some systems ship with security level set to
``none``. Other systems have security level set to ``user`` in order to
support downgrade to older OS, so users who want to automatically
authorize devices when IOMMU DMA protection is enabled can use the
following ``udev`` rule::

  ACTION=="add", SUBSYSTEM=="thunderbolt", ATTRS{iommu_dma_protection}=="1", ATTR{authorized}=="0", ATTR{authorized}="1"

Upgrading NVM on Thunderbolt device, host or retimer
Since most of the functionality is handled in firmware running on a
host controller or a device, it is important that the firmware can be
upgraded to the latest where possible bugs in it have been fixed.
Typically OEMs provide this firmware from their support site.

There is also a central site which has links where to download firmware
for some machines:

  `Thunderbolt Updates <>`_

Before you upgrade firmware on a device, host or retimer, please make
sure it is a suitable upgrade. Failing to do that may render the device
in a state where it cannot be used properly anymore without special

Host NVM upgrade on Apple Macs is not supported.

Once the NVM image has been downloaded, you need to plug in a
Thunderbolt device so that the host controller appears. It does not
matter which device is connected (unless you are upgrading NVM on a
device - then you need to connect that particular device).

Note an OEM-specific method to power the controller up ("force power") may
be available for your system in which case there is no need to plug in a
Thunderbolt device.

After that we can write the firmware to the non-active parts of the NVM
of the host or device. As an example here is how Intel NUC6i7KYK (Skull
Canyon) Thunderbolt controller NVM is upgraded::

  # dd if=KYK_TBT_FW_0018.bin of=/sys/bus/thunderbolt/devices/0-0/nvm_non_active0/nvmem

Once the operation completes we can trigger NVM authentication and
upgrade process as follows::

  # echo 1 > /sys/bus/thunderbolt/devices/0-0/nvm_authenticate

If no errors are returned, the host controller shortly disappears. Once
it comes back the driver notices it and initiates a full power cycle.
After a while the host controller appears again and this time it should
be fully functional.

We can verify that the new NVM firmware is active by running the following

  # cat /sys/bus/thunderbolt/devices/0-0/nvm_authenticate
  # cat /sys/bus/thunderbolt/devices/0-0/nvm_version

If ``nvm_authenticate`` contains anything other than 0x0 it is the error
code from the last authentication cycle, which means the authentication
of the NVM image failed.

Note names of the NVMem devices ``nvm_activeN`` and ``nvm_non_activeN``
depend on the order they are registered in the NVMem subsystem. N in
the name is the identifier added by the NVMem subsystem.

Upgrading on-board retimer NVM when there is no cable connected
If the platform supports, it may be possible to upgrade the retimer NVM
firmware even when there is nothing connected to the USB4
ports. When this is the case the ``usb4_portX`` devices have two special
attributes: ``offline`` and ``rescan``. The way to upgrade the firmware
is to first put the USB4 port into offline mode::

  # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline

This step makes sure the port does not respond to any hotplug events,
and also ensures the retimers are powered on. The next step is to scan
for the retimers::

  # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan

This enumerates and adds the on-board retimers. Now retimer NVM can be
upgraded in the same way than with cable connected (see previous
section). However, the retimer is not disconnected as we are offline
mode) so after writing ``1`` to ``nvm_authenticate`` one should wait for
5 or more seconds before running rescan again::

  # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan

This point if everything went fine, the port can be put back to
functional state again::

  # echo 0 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline

Upgrading NVM when host controller is in safe mode
If the existing NVM is not properly authenticated (or is missing) the
host controller goes into safe mode which means that the only available
functionality is flashing a new NVM image. When in this mode, reading
``nvm_version`` fails with ``ENODATA`` and the device identification
information is missing.

To recover from this mode, one needs to flash a valid NVM image to the
host controller in the same way it is done in the previous chapter.

Networking over Thunderbolt cable
Thunderbolt technology allows software communication between two hosts
connected by a Thunderbolt cable.

It is possible to tunnel any kind of traffic over a Thunderbolt link but
currently we only support Apple ThunderboltIP protocol.

If the other host is running Windows or macOS, the only thing you need to
do is to connect a Thunderbolt cable between the two hosts; the
``thunderbolt-net`` driver is loaded automatically. If the other host is
also Linux you should load ``thunderbolt-net`` manually on one host (it
does not matter which one)::

  # modprobe thunderbolt-net

This triggers module load on the other host automatically. If the driver
is built-in to the kernel image, there is no need to do anything.

The driver will create one virtual ethernet interface per Thunderbolt
port which are named like ``thunderbolt0`` and so on. From this point
you can either use standard userspace tools like ``ifconfig`` to
configure the interface or let your GUI handle it automatically.

Forcing power
Many OEMs include a method that can be used to force the power of a
Thunderbolt controller to an "On" state even if nothing is connected.
If supported by your machine this will be exposed by the WMI bus with
a sysfs attribute called "force_power".

For example the intel-wmi-thunderbolt driver exposes this attribute in:

  To force the power to on, write 1 to this attribute file.
  To disable force power, write 0 to this attribute file.

Note: it's currently not possible to query the force power state of a platform.