Based on kernel version 4.16.1. Page generated on 2018-04-09 11:53 EST.
1 Freezing of tasks 2 (C) 2007 Rafael J. Wysocki <rjw@sisk.pl>, GPL 3 4 I. What is the freezing of tasks? 5 6 The freezing of tasks is a mechanism by which user space processes and some 7 kernel threads are controlled during hibernation or system-wide suspend (on some 8 architectures). 9 10 II. How does it work? 11 12 There are three per-task flags used for that, PF_NOFREEZE, PF_FROZEN 13 and PF_FREEZER_SKIP (the last one is auxiliary). The tasks that have 14 PF_NOFREEZE unset (all user space processes and some kernel threads) are 15 regarded as 'freezable' and treated in a special way before the system enters a 16 suspend state as well as before a hibernation image is created (in what follows 17 we only consider hibernation, but the description also applies to suspend). 18 19 Namely, as the first step of the hibernation procedure the function 20 freeze_processes() (defined in kernel/power/process.c) is called. A system-wide 21 variable system_freezing_cnt (as opposed to a per-task flag) is used to indicate 22 whether the system is to undergo a freezing operation. And freeze_processes() 23 sets this variable. After this, it executes try_to_freeze_tasks() that sends a 24 fake signal to all user space processes, and wakes up all the kernel threads. 25 All freezable tasks must react to that by calling try_to_freeze(), which 26 results in a call to __refrigerator() (defined in kernel/freezer.c), which sets 27 the task's PF_FROZEN flag, changes its state to TASK_UNINTERRUPTIBLE and makes 28 it loop until PF_FROZEN is cleared for it. Then, we say that the task is 29 'frozen' and therefore the set of functions handling this mechanism is referred 30 to as 'the freezer' (these functions are defined in kernel/power/process.c, 31 kernel/freezer.c & include/linux/freezer.h). User space processes are generally 32 frozen before kernel threads. 33 34 __refrigerator() must not be called directly. Instead, use the 35 try_to_freeze() function (defined in include/linux/freezer.h), that checks 36 if the task is to be frozen and makes the task enter __refrigerator(). 37 38 For user space processes try_to_freeze() is called automatically from the 39 signal-handling code, but the freezable kernel threads need to call it 40 explicitly in suitable places or use the wait_event_freezable() or 41 wait_event_freezable_timeout() macros (defined in include/linux/freezer.h) 42 that combine interruptible sleep with checking if the task is to be frozen and 43 calling try_to_freeze(). The main loop of a freezable kernel thread may look 44 like the following one: 45 46 set_freezable(); 47 do { 48 hub_events(); 49 wait_event_freezable(khubd_wait, 50 !list_empty(&hub_event_list) || 51 kthread_should_stop()); 52 } while (!kthread_should_stop() || !list_empty(&hub_event_list)); 53 54 (from drivers/usb/core/hub.c::hub_thread()). 55 56 If a freezable kernel thread fails to call try_to_freeze() after the freezer has 57 initiated a freezing operation, the freezing of tasks will fail and the entire 58 hibernation operation will be cancelled. For this reason, freezable kernel 59 threads must call try_to_freeze() somewhere or use one of the 60 wait_event_freezable() and wait_event_freezable_timeout() macros. 61 62 After the system memory state has been restored from a hibernation image and 63 devices have been reinitialized, the function thaw_processes() is called in 64 order to clear the PF_FROZEN flag for each frozen task. Then, the tasks that 65 have been frozen leave __refrigerator() and continue running. 66 67 68 Rationale behind the functions dealing with freezing and thawing of tasks: 69 ------------------------------------------------------------------------- 70 71 freeze_processes(): 72 - freezes only userspace tasks 73 74 freeze_kernel_threads(): 75 - freezes all tasks (including kernel threads) because we can't freeze 76 kernel threads without freezing userspace tasks 77 78 thaw_kernel_threads(): 79 - thaws only kernel threads; this is particularly useful if we need to do 80 anything special in between thawing of kernel threads and thawing of 81 userspace tasks, or if we want to postpone the thawing of userspace tasks 82 83 thaw_processes(): 84 - thaws all tasks (including kernel threads) because we can't thaw userspace 85 tasks without thawing kernel threads 86 87 88 III. Which kernel threads are freezable? 89 90 Kernel threads are not freezable by default. However, a kernel thread may clear 91 PF_NOFREEZE for itself by calling set_freezable() (the resetting of PF_NOFREEZE 92 directly is not allowed). From this point it is regarded as freezable 93 and must call try_to_freeze() in a suitable place. 94 95 IV. Why do we do that? 96 97 Generally speaking, there is a couple of reasons to use the freezing of tasks: 98 99 1. The principal reason is to prevent filesystems from being damaged after 100 hibernation. At the moment we have no simple means of checkpointing 101 filesystems, so if there are any modifications made to filesystem data and/or 102 metadata on disks, we cannot bring them back to the state from before the 103 modifications. At the same time each hibernation image contains some 104 filesystem-related information that must be consistent with the state of the 105 on-disk data and metadata after the system memory state has been restored from 106 the image (otherwise the filesystems will be damaged in a nasty way, usually 107 making them almost impossible to repair). We therefore freeze tasks that might 108 cause the on-disk filesystems' data and metadata to be modified after the 109 hibernation image has been created and before the system is finally powered off. 110 The majority of these are user space processes, but if any of the kernel threads 111 may cause something like this to happen, they have to be freezable. 112 113 2. Next, to create the hibernation image we need to free a sufficient amount of 114 memory (approximately 50% of available RAM) and we need to do that before 115 devices are deactivated, because we generally need them for swapping out. Then, 116 after the memory for the image has been freed, we don't want tasks to allocate 117 additional memory and we prevent them from doing that by freezing them earlier. 118 [Of course, this also means that device drivers should not allocate substantial 119 amounts of memory from their .suspend() callbacks before hibernation, but this 120 is a separate issue.] 121 122 3. The third reason is to prevent user space processes and some kernel threads 123 from interfering with the suspending and resuming of devices. A user space 124 process running on a second CPU while we are suspending devices may, for 125 example, be troublesome and without the freezing of tasks we would need some 126 safeguards against race conditions that might occur in such a case. 127 128 Although Linus Torvalds doesn't like the freezing of tasks, he said this in one 129 of the discussions on LKML (http://lkml.org/lkml/2007/4/27/608): 130 131 "RJW:> Why we freeze tasks at all or why we freeze kernel threads? 132 133 Linus: In many ways, 'at all'. 134 135 I _do_ realize the IO request queue issues, and that we cannot actually do 136 s2ram with some devices in the middle of a DMA. So we want to be able to 137 avoid *that*, there's no question about that. And I suspect that stopping 138 user threads and then waiting for a sync is practically one of the easier 139 ways to do so. 140 141 So in practice, the 'at all' may become a 'why freeze kernel threads?' and 142 freezing user threads I don't find really objectionable." 143 144 Still, there are kernel threads that may want to be freezable. For example, if 145 a kernel thread that belongs to a device driver accesses the device directly, it 146 in principle needs to know when the device is suspended, so that it doesn't try 147 to access it at that time. However, if the kernel thread is freezable, it will 148 be frozen before the driver's .suspend() callback is executed and it will be 149 thawed after the driver's .resume() callback has run, so it won't be accessing 150 the device while it's suspended. 151 152 4. Another reason for freezing tasks is to prevent user space processes from 153 realizing that hibernation (or suspend) operation takes place. Ideally, user 154 space processes should not notice that such a system-wide operation has occurred 155 and should continue running without any problems after the restore (or resume 156 from suspend). Unfortunately, in the most general case this is quite difficult 157 to achieve without the freezing of tasks. Consider, for example, a process 158 that depends on all CPUs being online while it's running. Since we need to 159 disable nonboot CPUs during the hibernation, if this process is not frozen, it 160 may notice that the number of CPUs has changed and may start to work incorrectly 161 because of that. 162 163 V. Are there any problems related to the freezing of tasks? 164 165 Yes, there are. 166 167 First of all, the freezing of kernel threads may be tricky if they depend one 168 on another. For example, if kernel thread A waits for a completion (in the 169 TASK_UNINTERRUPTIBLE state) that needs to be done by freezable kernel thread B 170 and B is frozen in the meantime, then A will be blocked until B is thawed, which 171 may be undesirable. That's why kernel threads are not freezable by default. 172 173 Second, there are the following two problems related to the freezing of user 174 space processes: 175 1. Putting processes into an uninterruptible sleep distorts the load average. 176 2. Now that we have FUSE, plus the framework for doing device drivers in 177 userspace, it gets even more complicated because some userspace processes are 178 now doing the sorts of things that kernel threads do 179 (https://lists.linux-foundation.org/pipermail/linux-pm/2007-May/012309.html). 180 181 The problem 1. seems to be fixable, although it hasn't been fixed so far. The 182 other one is more serious, but it seems that we can work around it by using 183 hibernation (and suspend) notifiers (in that case, though, we won't be able to 184 avoid the realization by the user space processes that the hibernation is taking 185 place). 186 187 There are also problems that the freezing of tasks tends to expose, although 188 they are not directly related to it. For example, if request_firmware() is 189 called from a device driver's .resume() routine, it will timeout and eventually 190 fail, because the user land process that should respond to the request is frozen 191 at this point. So, seemingly, the failure is due to the freezing of tasks. 192 Suppose, however, that the firmware file is located on a filesystem accessible 193 only through another device that hasn't been resumed yet. In that case, 194 request_firmware() will fail regardless of whether or not the freezing of tasks 195 is used. Consequently, the problem is not really related to the freezing of 196 tasks, since it generally exists anyway. 197 198 A driver must have all firmwares it may need in RAM before suspend() is called. 199 If keeping them is not practical, for example due to their size, they must be 200 requested early enough using the suspend notifier API described in 201 Documentation/driver-api/pm/notifiers.rst. 202 203 VI. Are there any precautions to be taken to prevent freezing failures? 204 205 Yes, there are. 206 207 First of all, grabbing the 'pm_mutex' lock to mutually exclude a piece of code 208 from system-wide sleep such as suspend/hibernation is not encouraged. 209 If possible, that piece of code must instead hook onto the suspend/hibernation 210 notifiers to achieve mutual exclusion. Look at the CPU-Hotplug code 211 (kernel/cpu.c) for an example. 212 213 However, if that is not feasible, and grabbing 'pm_mutex' is deemed necessary, 214 it is strongly discouraged to directly call mutex_[un]lock(&pm_mutex) since 215 that could lead to freezing failures, because if the suspend/hibernate code 216 successfully acquired the 'pm_mutex' lock, and hence that other entity failed 217 to acquire the lock, then that task would get blocked in TASK_UNINTERRUPTIBLE 218 state. As a consequence, the freezer would not be able to freeze that task, 219 leading to freezing failure. 220 221 However, the [un]lock_system_sleep() APIs are safe to use in this scenario, 222 since they ask the freezer to skip freezing this task, since it is anyway 223 "frozen enough" as it is blocked on 'pm_mutex', which will be released 224 only after the entire suspend/hibernation sequence is complete. 225 So, to summarize, use [un]lock_system_sleep() instead of directly using 226 mutex_[un]lock(&pm_mutex). That would prevent freezing failures. 227 228 V. Miscellaneous 229 /sys/power/pm_freeze_timeout controls how long it will cost at most to freeze 230 all user space processes or all freezable kernel threads, in unit of millisecond. 231 The default value is 20000, with range of unsigned integer.