Based on kernel version 4.3. Page generated on 2015-11-02 12:50 EST.
1 Linux kernel media framework 2 ============================ 3 4 This document describes the Linux kernel media framework, its data structures, 5 functions and their usage. 6 7 8 Introduction 9 ------------ 10 11 The media controller API is documented in DocBook format in 12 Documentation/DocBook/media/v4l/media-controller.xml. This document will focus 13 on the kernel-side implementation of the media framework. 14 15 16 Abstract media device model 17 --------------------------- 18 19 Discovering a device internal topology, and configuring it at runtime, is one 20 of the goals of the media framework. To achieve this, hardware devices are 21 modelled as an oriented graph of building blocks called entities connected 22 through pads. 23 24 An entity is a basic media hardware building block. It can correspond to 25 a large variety of logical blocks such as physical hardware devices 26 (CMOS sensor for instance), logical hardware devices (a building block 27 in a System-on-Chip image processing pipeline), DMA channels or physical 28 connectors. 29 30 A pad is a connection endpoint through which an entity can interact with 31 other entities. Data (not restricted to video) produced by an entity 32 flows from the entity's output to one or more entity inputs. Pads should 33 not be confused with physical pins at chip boundaries. 34 35 A link is a point-to-point oriented connection between two pads, either 36 on the same entity or on different entities. Data flows from a source 37 pad to a sink pad. 38 39 40 Media device 41 ------------ 42 43 A media device is represented by a struct media_device instance, defined in 44 include/media/media-device.h. Allocation of the structure is handled by the 45 media device driver, usually by embedding the media_device instance in a 46 larger driver-specific structure. 47 48 Drivers register media device instances by calling 49 50 media_device_register(struct media_device *mdev); 51 52 The caller is responsible for initializing the media_device structure before 53 registration. The following fields must be set: 54 55 - dev must point to the parent device (usually a pci_dev, usb_interface or 56 platform_device instance). 57 58 - model must be filled with the device model name as a NUL-terminated UTF-8 59 string. The device/model revision must not be stored in this field. 60 61 The following fields are optional: 62 63 - serial is a unique serial number stored as a NUL-terminated ASCII string. 64 The field is big enough to store a GUID in text form. If the hardware 65 doesn't provide a unique serial number this field must be left empty. 66 67 - bus_info represents the location of the device in the system as a 68 NUL-terminated ASCII string. For PCI/PCIe devices bus_info must be set to 69 "PCI:" (or "PCIe:") followed by the value of pci_name(). For USB devices, 70 the usb_make_path() function must be used. This field is used by 71 applications to distinguish between otherwise identical devices that don't 72 provide a serial number. 73 74 - hw_revision is the hardware device revision in a driver-specific format. 75 When possible the revision should be formatted with the KERNEL_VERSION 76 macro. 77 78 - driver_version is formatted with the KERNEL_VERSION macro. The version 79 minor must be incremented when new features are added to the userspace API 80 without breaking binary compatibility. The version major must be 81 incremented when binary compatibility is broken. 82 83 Upon successful registration a character device named media[0-9]+ is created. 84 The device major and minor numbers are dynamic. The model name is exported as 85 a sysfs attribute. 86 87 Drivers unregister media device instances by calling 88 89 media_device_unregister(struct media_device *mdev); 90 91 Unregistering a media device that hasn't been registered is *NOT* safe. 92 93 94 Entities, pads and links 95 ------------------------ 96 97 - Entities 98 99 Entities are represented by a struct media_entity instance, defined in 100 include/media/media-entity.h. The structure is usually embedded into a 101 higher-level structure, such as a v4l2_subdev or video_device instance, 102 although drivers can allocate entities directly. 103 104 Drivers initialize entities by calling 105 106 media_entity_init(struct media_entity *entity, u16 num_pads, 107 struct media_pad *pads, u16 extra_links); 108 109 The media_entity name, type, flags, revision and group_id fields can be 110 initialized before or after calling media_entity_init. Entities embedded in 111 higher-level standard structures can have some of those fields set by the 112 higher-level framework. 113 114 As the number of pads is known in advance, the pads array is not allocated 115 dynamically but is managed by the entity driver. Most drivers will embed the 116 pads array in a driver-specific structure, avoiding dynamic allocation. 117 118 Drivers must set the direction of every pad in the pads array before calling 119 media_entity_init. The function will initialize the other pads fields. 120 121 Unlike the number of pads, the total number of links isn't always known in 122 advance by the entity driver. As an initial estimate, media_entity_init 123 pre-allocates a number of links equal to the number of pads plus an optional 124 number of extra links. The links array will be reallocated if it grows beyond 125 the initial estimate. 126 127 Drivers register entities with a media device by calling 128 129 media_device_register_entity(struct media_device *mdev, 130 struct media_entity *entity); 131 132 Entities are identified by a unique positive integer ID. Drivers can provide an 133 ID by filling the media_entity id field prior to registration, or request the 134 media controller framework to assign an ID automatically. Drivers that provide 135 IDs manually must ensure that all IDs are unique. IDs are not guaranteed to be 136 contiguous even when they are all assigned automatically by the framework. 137 138 Drivers unregister entities by calling 139 140 media_device_unregister_entity(struct media_entity *entity); 141 142 Unregistering an entity will not change the IDs of the other entities, and the 143 ID will never be reused for a newly registered entity. 144 145 When a media device is unregistered, all its entities are unregistered 146 automatically. No manual entities unregistration is then required. 147 148 Drivers free resources associated with an entity by calling 149 150 media_entity_cleanup(struct media_entity *entity); 151 152 This function must be called during the cleanup phase after unregistering the 153 entity. Note that the media_entity instance itself must be freed explicitly by 154 the driver if required. 155 156 Entities have flags that describe the entity capabilities and state. 157 158 MEDIA_ENT_FL_DEFAULT indicates the default entity for a given type. 159 This can be used to report the default audio and video devices or the 160 default camera sensor. 161 162 Logical entity groups can be defined by setting the group ID of all member 163 entities to the same non-zero value. An entity group serves no purpose in the 164 kernel, but is reported to userspace during entities enumeration. The group_id 165 field belongs to the media device driver and must not by touched by entity 166 drivers. 167 168 Media device drivers should define groups if several entities are logically 169 bound together. Example usages include reporting 170 171 - ALSA, VBI and video nodes that carry the same media stream 172 - lens and flash controllers associated with a sensor 173 174 - Pads 175 176 Pads are represented by a struct media_pad instance, defined in 177 include/media/media-entity.h. Each entity stores its pads in a pads array 178 managed by the entity driver. Drivers usually embed the array in a 179 driver-specific structure. 180 181 Pads are identified by their entity and their 0-based index in the pads array. 182 Both information are stored in the media_pad structure, making the media_pad 183 pointer the canonical way to store and pass link references. 184 185 Pads have flags that describe the pad capabilities and state. 186 187 MEDIA_PAD_FL_SINK indicates that the pad supports sinking data. 188 MEDIA_PAD_FL_SOURCE indicates that the pad supports sourcing data. 189 190 One and only one of MEDIA_PAD_FL_SINK and MEDIA_PAD_FL_SOURCE must be set for 191 each pad. 192 193 - Links 194 195 Links are represented by a struct media_link instance, defined in 196 include/media/media-entity.h. Each entity stores all links originating at or 197 targeting any of its pads in a links array. A given link is thus stored 198 twice, once in the source entity and once in the target entity. The array is 199 pre-allocated and grows dynamically as needed. 200 201 Drivers create links by calling 202 203 media_entity_create_link(struct media_entity *source, u16 source_pad, 204 struct media_entity *sink, u16 sink_pad, 205 u32 flags); 206 207 An entry in the link array of each entity is allocated and stores pointers 208 to source and sink pads. 209 210 Links have flags that describe the link capabilities and state. 211 212 MEDIA_LNK_FL_ENABLED indicates that the link is enabled and can be used 213 to transfer media data. When two or more links target a sink pad, only 214 one of them can be enabled at a time. 215 MEDIA_LNK_FL_IMMUTABLE indicates that the link enabled state can't be 216 modified at runtime. If MEDIA_LNK_FL_IMMUTABLE is set, then 217 MEDIA_LNK_FL_ENABLED must also be set since an immutable link is always 218 enabled. 219 220 221 Graph traversal 222 --------------- 223 224 The media framework provides APIs to iterate over entities in a graph. 225 226 To iterate over all entities belonging to a media device, drivers can use the 227 media_device_for_each_entity macro, defined in include/media/media-device.h. 228 229 struct media_entity *entity; 230 231 media_device_for_each_entity(entity, mdev) { 232 /* entity will point to each entity in turn */ 233 ... 234 } 235 236 Drivers might also need to iterate over all entities in a graph that can be 237 reached only through enabled links starting at a given entity. The media 238 framework provides a depth-first graph traversal API for that purpose. 239 240 Note that graphs with cycles (whether directed or undirected) are *NOT* 241 supported by the graph traversal API. To prevent infinite loops, the graph 242 traversal code limits the maximum depth to MEDIA_ENTITY_ENUM_MAX_DEPTH, 243 currently defined as 16. 244 245 Drivers initiate a graph traversal by calling 246 247 media_entity_graph_walk_start(struct media_entity_graph *graph, 248 struct media_entity *entity); 249 250 The graph structure, provided by the caller, is initialized to start graph 251 traversal at the given entity. 252 253 Drivers can then retrieve the next entity by calling 254 255 media_entity_graph_walk_next(struct media_entity_graph *graph); 256 257 When the graph traversal is complete the function will return NULL. 258 259 Graph traversal can be interrupted at any moment. No cleanup function call is 260 required and the graph structure can be freed normally. 261 262 Helper functions can be used to find a link between two given pads, or a pad 263 connected to another pad through an enabled link 264 265 media_entity_find_link(struct media_pad *source, 266 struct media_pad *sink); 267 268 media_entity_remote_pad(struct media_pad *pad); 269 270 Refer to the kerneldoc documentation for more information. 271 272 273 Use count and power handling 274 ---------------------------- 275 276 Due to the wide differences between drivers regarding power management needs, 277 the media controller does not implement power management. However, the 278 media_entity structure includes a use_count field that media drivers can use to 279 track the number of users of every entity for power management needs. 280 281 The use_count field is owned by media drivers and must not be touched by entity 282 drivers. Access to the field must be protected by the media device graph_mutex 283 lock. 284 285 286 Links setup 287 ----------- 288 289 Link properties can be modified at runtime by calling 290 291 media_entity_setup_link(struct media_link *link, u32 flags); 292 293 The flags argument contains the requested new link flags. 294 295 The only configurable property is the ENABLED link flag to enable/disable a 296 link. Links marked with the IMMUTABLE link flag can not be enabled or disabled. 297 298 When a link is enabled or disabled, the media framework calls the 299 link_setup operation for the two entities at the source and sink of the link, 300 in that order. If the second link_setup call fails, another link_setup call is 301 made on the first entity to restore the original link flags. 302 303 Media device drivers can be notified of link setup operations by setting the 304 media_device::link_notify pointer to a callback function. If provided, the 305 notification callback will be called before enabling and after disabling 306 links. 307 308 Entity drivers must implement the link_setup operation if any of their links 309 is non-immutable. The operation must either configure the hardware or store 310 the configuration information to be applied later. 311 312 Link configuration must not have any side effect on other links. If an enabled 313 link at a sink pad prevents another link at the same pad from being enabled, 314 the link_setup operation must return -EBUSY and can't implicitly disable the 315 first enabled link. 316 317 318 Pipelines and media streams 319 --------------------------- 320 321 When starting streaming, drivers must notify all entities in the pipeline to 322 prevent link states from being modified during streaming by calling 323 324 media_entity_pipeline_start(struct media_entity *entity, 325 struct media_pipeline *pipe); 326 327 The function will mark all entities connected to the given entity through 328 enabled links, either directly or indirectly, as streaming. 329 330 The media_pipeline instance pointed to by the pipe argument will be stored in 331 every entity in the pipeline. Drivers should embed the media_pipeline structure 332 in higher-level pipeline structures and can then access the pipeline through 333 the media_entity pipe field. 334 335 Calls to media_entity_pipeline_start() can be nested. The pipeline pointer must 336 be identical for all nested calls to the function. 337 338 media_entity_pipeline_start() may return an error. In that case, it will 339 clean up any of the changes it did by itself. 340 341 When stopping the stream, drivers must notify the entities with 342 343 media_entity_pipeline_stop(struct media_entity *entity); 344 345 If multiple calls to media_entity_pipeline_start() have been made the same 346 number of media_entity_pipeline_stop() calls are required to stop streaming. The 347 media_entity pipe field is reset to NULL on the last nested stop call. 348 349 Link configuration will fail with -EBUSY by default if either end of the link is 350 a streaming entity. Links that can be modified while streaming must be marked 351 with the MEDIA_LNK_FL_DYNAMIC flag. 352 353 If other operations need to be disallowed on streaming entities (such as 354 changing entities configuration parameters) drivers can explicitly check the 355 media_entity stream_count field to find out if an entity is streaming. This 356 operation must be done with the media_device graph_mutex held. 357 358 359 Link validation 360 --------------- 361 362 Link validation is performed by media_entity_pipeline_start() for any 363 entity which has sink pads in the pipeline. The 364 media_entity::link_validate() callback is used for that purpose. In 365 link_validate() callback, entity driver should check that the properties of 366 the source pad of the connected entity and its own sink pad match. It is up 367 to the type of the entity (and in the end, the properties of the hardware) 368 what matching actually means. 369 370 Subsystems should facilitate link validation by providing subsystem specific 371 helper functions to provide easy access for commonly needed information, and 372 in the end provide a way to use driver-specific callbacks.