Based on kernel version 4.16.1. Page generated on 2018-04-09 11:53 EST.
1 Written by: Neil Brown 2 Please see MAINTAINERS file for where to send questions. 3 4 Overlay Filesystem 5 ================== 6 7 This document describes a prototype for a new approach to providing 8 overlay-filesystem functionality in Linux (sometimes referred to as 9 union-filesystems). An overlay-filesystem tries to present a 10 filesystem which is the result over overlaying one filesystem on top 11 of the other. 12 13 The result will inevitably fail to look exactly like a normal 14 filesystem for various technical reasons. The expectation is that 15 many use cases will be able to ignore these differences. 16 17 This approach is 'hybrid' because the objects that appear in the 18 filesystem do not all appear to belong to that filesystem. In many 19 cases an object accessed in the union will be indistinguishable 20 from accessing the corresponding object from the original filesystem. 21 This is most obvious from the 'st_dev' field returned by stat(2). 22 23 While directories will report an st_dev from the overlay-filesystem, 24 non-directory objects may report an st_dev from the lower filesystem or 25 upper filesystem that is providing the object. Similarly st_ino will 26 only be unique when combined with st_dev, and both of these can change 27 over the lifetime of a non-directory object. Many applications and 28 tools ignore these values and will not be affected. 29 30 In the special case of all overlay layers on the same underlying 31 filesystem, all objects will report an st_dev from the overlay 32 filesystem and st_ino from the underlying filesystem. This will 33 make the overlay mount more compliant with filesystem scanners and 34 overlay objects will be distinguishable from the corresponding 35 objects in the original filesystem. 36 37 Upper and Lower 38 --------------- 39 40 An overlay filesystem combines two filesystems - an 'upper' filesystem 41 and a 'lower' filesystem. When a name exists in both filesystems, the 42 object in the 'upper' filesystem is visible while the object in the 43 'lower' filesystem is either hidden or, in the case of directories, 44 merged with the 'upper' object. 45 46 It would be more correct to refer to an upper and lower 'directory 47 tree' rather than 'filesystem' as it is quite possible for both 48 directory trees to be in the same filesystem and there is no 49 requirement that the root of a filesystem be given for either upper or 50 lower. 51 52 The lower filesystem can be any filesystem supported by Linux and does 53 not need to be writable. The lower filesystem can even be another 54 overlayfs. The upper filesystem will normally be writable and if it 55 is it must support the creation of trusted.* extended attributes, and 56 must provide valid d_type in readdir responses, so NFS is not suitable. 57 58 A read-only overlay of two read-only filesystems may use any 59 filesystem type. 60 61 Directories 62 ----------- 63 64 Overlaying mainly involves directories. If a given name appears in both 65 upper and lower filesystems and refers to a non-directory in either, 66 then the lower object is hidden - the name refers only to the upper 67 object. 68 69 Where both upper and lower objects are directories, a merged directory 70 is formed. 71 72 At mount time, the two directories given as mount options "lowerdir" and 73 "upperdir" are combined into a merged directory: 74 75 mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,\ 76 workdir=/work /merged 77 78 The "workdir" needs to be an empty directory on the same filesystem 79 as upperdir. 80 81 Then whenever a lookup is requested in such a merged directory, the 82 lookup is performed in each actual directory and the combined result 83 is cached in the dentry belonging to the overlay filesystem. If both 84 actual lookups find directories, both are stored and a merged 85 directory is created, otherwise only one is stored: the upper if it 86 exists, else the lower. 87 88 Only the lists of names from directories are merged. Other content 89 such as metadata and extended attributes are reported for the upper 90 directory only. These attributes of the lower directory are hidden. 91 92 whiteouts and opaque directories 93 -------------------------------- 94 95 In order to support rm and rmdir without changing the lower 96 filesystem, an overlay filesystem needs to record in the upper filesystem 97 that files have been removed. This is done using whiteouts and opaque 98 directories (non-directories are always opaque). 99 100 A whiteout is created as a character device with 0/0 device number. 101 When a whiteout is found in the upper level of a merged directory, any 102 matching name in the lower level is ignored, and the whiteout itself 103 is also hidden. 104 105 A directory is made opaque by setting the xattr "trusted.overlay.opaque" 106 to "y". Where the upper filesystem contains an opaque directory, any 107 directory in the lower filesystem with the same name is ignored. 108 109 readdir 110 ------- 111 112 When a 'readdir' request is made on a merged directory, the upper and 113 lower directories are each read and the name lists merged in the 114 obvious way (upper is read first, then lower - entries that already 115 exist are not re-added). This merged name list is cached in the 116 'struct file' and so remains as long as the file is kept open. If the 117 directory is opened and read by two processes at the same time, they 118 will each have separate caches. A seekdir to the start of the 119 directory (offset 0) followed by a readdir will cause the cache to be 120 discarded and rebuilt. 121 122 This means that changes to the merged directory do not appear while a 123 directory is being read. This is unlikely to be noticed by many 124 programs. 125 126 seek offsets are assigned sequentially when the directories are read. 127 Thus if 128 129 - read part of a directory 130 - remember an offset, and close the directory 131 - re-open the directory some time later 132 - seek to the remembered offset 133 134 there may be little correlation between the old and new locations in 135 the list of filenames, particularly if anything has changed in the 136 directory. 137 138 Readdir on directories that are not merged is simply handled by the 139 underlying directory (upper or lower). 140 141 renaming directories 142 -------------------- 143 144 When renaming a directory that is on the lower layer or merged (i.e. the 145 directory was not created on the upper layer to start with) overlayfs can 146 handle it in two different ways: 147 148 1. return EXDEV error: this error is returned by rename(2) when trying to 149 move a file or directory across filesystem boundaries. Hence 150 applications are usually prepared to hande this error (mv(1) for example 151 recursively copies the directory tree). This is the default behavior. 152 153 2. If the "redirect_dir" feature is enabled, then the directory will be 154 copied up (but not the contents). Then the "trusted.overlay.redirect" 155 extended attribute is set to the path of the original location from the 156 root of the overlay. Finally the directory is moved to the new 157 location. 158 159 There are several ways to tune the "redirect_dir" feature. 160 161 Kernel config options: 162 163 - OVERLAY_FS_REDIRECT_DIR: 164 If this is enabled, then redirect_dir is turned on by default. 165 - OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW: 166 If this is enabled, then redirects are always followed by default. Enabling 167 this results in a less secure configuration. Enable this option only when 168 worried about backward compatibility with kernels that have the redirect_dir 169 feature and follow redirects even if turned off. 170 171 Module options (can also be changed through /sys/module/overlay/parameters/*): 172 173 - "redirect_dir=BOOL": 174 See OVERLAY_FS_REDIRECT_DIR kernel config option above. 175 - "redirect_always_follow=BOOL": 176 See OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW kernel config option above. 177 - "redirect_max=NUM": 178 The maximum number of bytes in an absolute redirect (default is 256). 179 180 Mount options: 181 182 - "redirect_dir=on": 183 Redirects are enabled. 184 - "redirect_dir=follow": 185 Redirects are not created, but followed. 186 - "redirect_dir=off": 187 Redirects are not created and only followed if "redirect_always_follow" 188 feature is enabled in the kernel/module config. 189 - "redirect_dir=nofollow": 190 Redirects are not created and not followed (equivalent to "redirect_dir=off" 191 if "redirect_always_follow" feature is not enabled). 192 193 When the NFS export feature is enabled, every copied up directory is 194 indexed by the file handle of the lower inode and a file handle of the 195 upper directory is stored in a "trusted.overlay.upper" extended attribute 196 on the index entry. On lookup of a merged directory, if the upper 197 directory does not match the file handle stores in the index, that is an 198 indication that multiple upper directories may be redirected to the same 199 lower directory. In that case, lookup returns an error and warns about 200 a possible inconsistency. 201 202 Because lower layer redirects cannot be verified with the index, enabling 203 NFS export support on an overlay filesystem with no upper layer requires 204 turning off redirect follow (e.g. "redirect_dir=nofollow"). 205 206 207 Non-directories 208 --------------- 209 210 Objects that are not directories (files, symlinks, device-special 211 files etc.) are presented either from the upper or lower filesystem as 212 appropriate. When a file in the lower filesystem is accessed in a way 213 the requires write-access, such as opening for write access, changing 214 some metadata etc., the file is first copied from the lower filesystem 215 to the upper filesystem (copy_up). Note that creating a hard-link 216 also requires copy_up, though of course creation of a symlink does 217 not. 218 219 The copy_up may turn out to be unnecessary, for example if the file is 220 opened for read-write but the data is not modified. 221 222 The copy_up process first makes sure that the containing directory 223 exists in the upper filesystem - creating it and any parents as 224 necessary. It then creates the object with the same metadata (owner, 225 mode, mtime, symlink-target etc.) and then if the object is a file, the 226 data is copied from the lower to the upper filesystem. Finally any 227 extended attributes are copied up. 228 229 Once the copy_up is complete, the overlay filesystem simply 230 provides direct access to the newly created file in the upper 231 filesystem - future operations on the file are barely noticed by the 232 overlay filesystem (though an operation on the name of the file such as 233 rename or unlink will of course be noticed and handled). 234 235 236 Multiple lower layers 237 --------------------- 238 239 Multiple lower layers can now be given using the the colon (":") as a 240 separator character between the directory names. For example: 241 242 mount -t overlay overlay -olowerdir=/lower1:/lower2:/lower3 /merged 243 244 As the example shows, "upperdir=" and "workdir=" may be omitted. In 245 that case the overlay will be read-only. 246 247 The specified lower directories will be stacked beginning from the 248 rightmost one and going left. In the above example lower1 will be the 249 top, lower2 the middle and lower3 the bottom layer. 250 251 252 Sharing and copying layers 253 -------------------------- 254 255 Lower layers may be shared among several overlay mounts and that is indeed 256 a very common practice. An overlay mount may use the same lower layer 257 path as another overlay mount and it may use a lower layer path that is 258 beneath or above the path of another overlay lower layer path. 259 260 Using an upper layer path and/or a workdir path that are already used by 261 another overlay mount is not allowed and may fail with EBUSY. Using 262 partially overlapping paths is not allowed but will not fail with EBUSY. 263 If files are accessed from two overlayfs mounts which share or overlap the 264 upper layer and/or workdir path the behavior of the overlay is undefined, 265 though it will not result in a crash or deadlock. 266 267 Mounting an overlay using an upper layer path, where the upper layer path 268 was previously used by another mounted overlay in combination with a 269 different lower layer path, is allowed, unless the "inodes index" feature 270 is enabled. 271 272 With the "inodes index" feature, on the first time mount, an NFS file 273 handle of the lower layer root directory, along with the UUID of the lower 274 filesystem, are encoded and stored in the "trusted.overlay.origin" extended 275 attribute on the upper layer root directory. On subsequent mount attempts, 276 the lower root directory file handle and lower filesystem UUID are compared 277 to the stored origin in upper root directory. On failure to verify the 278 lower root origin, mount will fail with ESTALE. An overlayfs mount with 279 "inodes index" enabled will fail with EOPNOTSUPP if the lower filesystem 280 does not support NFS export, lower filesystem does not have a valid UUID or 281 if the upper filesystem does not support extended attributes. 282 283 It is quite a common practice to copy overlay layers to a different 284 directory tree on the same or different underlying filesystem, and even 285 to a different machine. With the "inodes index" feature, trying to mount 286 the copied layers will fail the verification of the lower root file handle. 287 288 289 Non-standard behavior 290 --------------------- 291 292 The copy_up operation essentially creates a new, identical file and 293 moves it over to the old name. The new file may be on a different 294 filesystem, so both st_dev and st_ino of the file may change. 295 296 Any open files referring to this inode will access the old data. 297 298 Unless "inode index" feature is enabled, if a file with multiple hard 299 links is copied up, then this will "break" the link. Changes will not be 300 propagated to other names referring to the same inode. 301 302 Unless "redirect_dir" feature is enabled, rename(2) on a lower or merged 303 directory will fail with EXDEV. 304 305 Changes to underlying filesystems 306 --------------------------------- 307 308 Offline changes, when the overlay is not mounted, are allowed to either 309 the upper or the lower trees. 310 311 Changes to the underlying filesystems while part of a mounted overlay 312 filesystem are not allowed. If the underlying filesystem is changed, 313 the behavior of the overlay is undefined, though it will not result in 314 a crash or deadlock. 315 316 When the overlay NFS export feature is enabled, overlay filesystems 317 behavior on offline changes of the underlying lower layer is different 318 than the behavior when NFS export is disabled. 319 320 On every copy_up, an NFS file handle of the lower inode, along with the 321 UUID of the lower filesystem, are encoded and stored in an extended 322 attribute "trusted.overlay.origin" on the upper inode. 323 324 When the NFS export feature is enabled, a lookup of a merged directory, 325 that found a lower directory at the lookup path or at the path pointed 326 to by the "trusted.overlay.redirect" extended attribute, will verify 327 that the found lower directory file handle and lower filesystem UUID 328 match the origin file handle that was stored at copy_up time. If a 329 found lower directory does not match the stored origin, that directory 330 will not be merged with the upper directory. 331 332 333 334 NFS export 335 ---------- 336 337 When the underlying filesystems supports NFS export and the "nfs_export" 338 feature is enabled, an overlay filesystem may be exported to NFS. 339 340 With the "nfs_export" feature, on copy_up of any lower object, an index 341 entry is created under the index directory. The index entry name is the 342 hexadecimal representation of the copy up origin file handle. For a 343 non-directory object, the index entry is a hard link to the upper inode. 344 For a directory object, the index entry has an extended attribute 345 "trusted.overlay.upper" with an encoded file handle of the upper 346 directory inode. 347 348 When encoding a file handle from an overlay filesystem object, the 349 following rules apply: 350 351 1. For a non-upper object, encode a lower file handle from lower inode 352 2. For an indexed object, encode a lower file handle from copy_up origin 353 3. For a pure-upper object and for an existing non-indexed upper object, 354 encode an upper file handle from upper inode 355 356 The encoded overlay file handle includes: 357 - Header including path type information (e.g. lower/upper) 358 - UUID of the underlying filesystem 359 - Underlying filesystem encoding of underlying inode 360 361 This encoding format is identical to the encoding format file handles that 362 are stored in extended attribute "trusted.overlay.origin". 363 364 When decoding an overlay file handle, the following steps are followed: 365 366 1. Find underlying layer by UUID and path type information. 367 2. Decode the underlying filesystem file handle to underlying dentry. 368 3. For a lower file handle, lookup the handle in index directory by name. 369 4. If a whiteout is found in index, return ESTALE. This represents an 370 overlay object that was deleted after its file handle was encoded. 371 5. For a non-directory, instantiate a disconnected overlay dentry from the 372 decoded underlying dentry, the path type and index inode, if found. 373 6. For a directory, use the connected underlying decoded dentry, path type 374 and index, to lookup a connected overlay dentry. 375 376 Decoding a non-directory file handle may return a disconnected dentry. 377 copy_up of that disconnected dentry will create an upper index entry with 378 no upper alias. 379 380 When overlay filesystem has multiple lower layers, a middle layer 381 directory may have a "redirect" to lower directory. Because middle layer 382 "redirects" are not indexed, a lower file handle that was encoded from the 383 "redirect" origin directory, cannot be used to find the middle or upper 384 layer directory. Similarly, a lower file handle that was encoded from a 385 descendant of the "redirect" origin directory, cannot be used to 386 reconstruct a connected overlay path. To mitigate the cases of 387 directories that cannot be decoded from a lower file handle, these 388 directories are copied up on encode and encoded as an upper file handle. 389 On an overlay filesystem with no upper layer this mitigation cannot be 390 used NFS export in this setup requires turning off redirect follow (e.g. 391 "redirect_dir=nofollow"). 392 393 The overlay filesystem does not support non-directory connectable file 394 handles, so exporting with the 'subtree_check' exportfs configuration will 395 cause failures to lookup files over NFS. 396 397 When the NFS export feature is enabled, all directory index entries are 398 verified on mount time to check that upper file handles are not stale. 399 This verification may cause significant overhead in some cases. 400 401 402 Testsuite 403 --------- 404 405 There's testsuite developed by David Howells at: 406 407 git://git.infradead.org/users/dhowells/unionmount-testsuite.git 408 409 Run as root: 410 411 # cd unionmount-testsuite 412 # ./run --ov