Based on kernel version 4.16.1. Page generated on 2018-04-09 11:53 EST.
1 Interaction of Suspend code (S3) with the CPU hotplug infrastructure 2 3 (C) 2011 - 2014 Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com> 4 5 6 I. How does the regular CPU hotplug code differ from how the Suspend-to-RAM 7 infrastructure uses it internally? And where do they share common code? 8 9 Well, a picture is worth a thousand words... So ASCII art follows :-) 10 11 [This depicts the current design in the kernel, and focusses only on the 12 interactions involving the freezer and CPU hotplug and also tries to explain 13 the locking involved. It outlines the notifications involved as well. 14 But please note that here, only the call paths are illustrated, with the aim 15 of describing where they take different paths and where they share code. 16 What happens when regular CPU hotplug and Suspend-to-RAM race with each other 17 is not depicted here.] 18 19 On a high level, the suspend-resume cycle goes like this: 20 21 |Freeze| -> |Disable nonboot| -> |Do suspend| -> |Enable nonboot| -> |Thaw | 22 |tasks | | cpus | | | | cpus | |tasks| 23 24 25 More details follow: 26 27 Suspend call path 28 ----------------- 29 30 Write 'mem' to 31 /sys/power/state 32 sysfs file 33 | 34 v 35 Acquire pm_mutex lock 36 | 37 v 38 Send PM_SUSPEND_PREPARE 39 notifications 40 | 41 v 42 Freeze tasks 43 | 44 | 45 v 46 disable_nonboot_cpus() 47 /* start */ 48 | 49 v 50 Acquire cpu_add_remove_lock 51 | 52 v 53 Iterate over CURRENTLY 54 online CPUs 55 | 56 | 57 | ---------- 58 v | L 59 ======> _cpu_down() | 60 | [This takes cpuhotplug.lock | 61 Common | before taking down the CPU | 62 code | and releases it when done] | O 63 | While it is at it, notifications | 64 | are sent when notable events occur, | 65 ======> by running all registered callbacks. | 66 | | O 67 | | 68 | | 69 v | 70 Note down these cpus in | P 71 frozen_cpus mask ---------- 72 | 73 v 74 Disable regular cpu hotplug 75 by increasing cpu_hotplug_disabled 76 | 77 v 78 Release cpu_add_remove_lock 79 | 80 v 81 /* disable_nonboot_cpus() complete */ 82 | 83 v 84 Do suspend 85 86 87 88 Resuming back is likewise, with the counterparts being (in the order of 89 execution during resume): 90 * enable_nonboot_cpus() which involves: 91 | Acquire cpu_add_remove_lock 92 | Decrease cpu_hotplug_disabled, thereby enabling regular cpu hotplug 93 | Call _cpu_up() [for all those cpus in the frozen_cpus mask, in a loop] 94 | Release cpu_add_remove_lock 95 v 96 97 * thaw tasks 98 * send PM_POST_SUSPEND notifications 99 * Release pm_mutex lock. 100 101 102 It is to be noted here that the pm_mutex lock is acquired at the very 103 beginning, when we are just starting out to suspend, and then released only 104 after the entire cycle is complete (i.e., suspend + resume). 105 106 107 108 Regular CPU hotplug call path 109 ----------------------------- 110 111 Write 0 (or 1) to 112 /sys/devices/system/cpu/cpu*/online 113 sysfs file 114 | 115 | 116 v 117 cpu_down() 118 | 119 v 120 Acquire cpu_add_remove_lock 121 | 122 v 123 If cpu_hotplug_disabled > 0 124 return gracefully 125 | 126 | 127 v 128 ======> _cpu_down() 129 | [This takes cpuhotplug.lock 130 Common | before taking down the CPU 131 code | and releases it when done] 132 | While it is at it, notifications 133 | are sent when notable events occur, 134 ======> by running all registered callbacks. 135 | 136 | 137 v 138 Release cpu_add_remove_lock 139 [That's it!, for 140 regular CPU hotplug] 141 142 143 144 So, as can be seen from the two diagrams (the parts marked as "Common code"), 145 regular CPU hotplug and the suspend code path converge at the _cpu_down() and 146 _cpu_up() functions. They differ in the arguments passed to these functions, 147 in that during regular CPU hotplug, 0 is passed for the 'tasks_frozen' 148 argument. But during suspend, since the tasks are already frozen by the time 149 the non-boot CPUs are offlined or onlined, the _cpu_*() functions are called 150 with the 'tasks_frozen' argument set to 1. 151 [See below for some known issues regarding this.] 152 153 154 Important files and functions/entry points: 155 ------------------------------------------ 156 157 kernel/power/process.c : freeze_processes(), thaw_processes() 158 kernel/power/suspend.c : suspend_prepare(), suspend_enter(), suspend_finish() 159 kernel/cpu.c: cpu_[up|down](), _cpu_[up|down](), [disable|enable]_nonboot_cpus() 160 161 162 163 II. What are the issues involved in CPU hotplug? 164 ------------------------------------------- 165 166 There are some interesting situations involving CPU hotplug and microcode 167 update on the CPUs, as discussed below: 168 169 [Please bear in mind that the kernel requests the microcode images from 170 userspace, using the request_firmware() function defined in 171 drivers/base/firmware_class.c] 172 173 174 a. When all the CPUs are identical: 175 176 This is the most common situation and it is quite straightforward: we want 177 to apply the same microcode revision to each of the CPUs. 178 To give an example of x86, the collect_cpu_info() function defined in 179 arch/x86/kernel/microcode_core.c helps in discovering the type of the CPU 180 and thereby in applying the correct microcode revision to it. 181 But note that the kernel does not maintain a common microcode image for the 182 all CPUs, in order to handle case 'b' described below. 183 184 185 b. When some of the CPUs are different than the rest: 186 187 In this case since we probably need to apply different microcode revisions 188 to different CPUs, the kernel maintains a copy of the correct microcode 189 image for each CPU (after appropriate CPU type/model discovery using 190 functions such as collect_cpu_info()). 191 192 193 c. When a CPU is physically hot-unplugged and a new (and possibly different 194 type of) CPU is hot-plugged into the system: 195 196 In the current design of the kernel, whenever a CPU is taken offline during 197 a regular CPU hotplug operation, upon receiving the CPU_DEAD notification 198 (which is sent by the CPU hotplug code), the microcode update driver's 199 callback for that event reacts by freeing the kernel's copy of the 200 microcode image for that CPU. 201 202 Hence, when a new CPU is brought online, since the kernel finds that it 203 doesn't have the microcode image, it does the CPU type/model discovery 204 afresh and then requests the userspace for the appropriate microcode image 205 for that CPU, which is subsequently applied. 206 207 For example, in x86, the mc_cpu_callback() function (which is the microcode 208 update driver's callback registered for CPU hotplug events) calls 209 microcode_update_cpu() which would call microcode_init_cpu() in this case, 210 instead of microcode_resume_cpu() when it finds that the kernel doesn't 211 have a valid microcode image. This ensures that the CPU type/model 212 discovery is performed and the right microcode is applied to the CPU after 213 getting it from userspace. 214 215 216 d. Handling microcode update during suspend/hibernate: 217 218 Strictly speaking, during a CPU hotplug operation which does not involve 219 physically removing or inserting CPUs, the CPUs are not actually powered 220 off during a CPU offline. They are just put to the lowest C-states possible. 221 Hence, in such a case, it is not really necessary to re-apply microcode 222 when the CPUs are brought back online, since they wouldn't have lost the 223 image during the CPU offline operation. 224 225 This is the usual scenario encountered during a resume after a suspend. 226 However, in the case of hibernation, since all the CPUs are completely 227 powered off, during restore it becomes necessary to apply the microcode 228 images to all the CPUs. 229 230 [Note that we don't expect someone to physically pull out nodes and insert 231 nodes with a different type of CPUs in-between a suspend-resume or a 232 hibernate/restore cycle.] 233 234 In the current design of the kernel however, during a CPU offline operation 235 as part of the suspend/hibernate cycle (cpuhp_tasks_frozen is set), 236 the existing copy of microcode image in the kernel is not freed up. 237 And during the CPU online operations (during resume/restore), since the 238 kernel finds that it already has copies of the microcode images for all the 239 CPUs, it just applies them to the CPUs, avoiding any re-discovery of CPU 240 type/model and the need for validating whether the microcode revisions are 241 right for the CPUs or not (due to the above assumption that physical CPU 242 hotplug will not be done in-between suspend/resume or hibernate/restore 243 cycles). 244 245 246 III. Are there any known problems when regular CPU hotplug and suspend race 247 with each other? 248 249 Yes, they are listed below: 250 251 1. When invoking regular CPU hotplug, the 'tasks_frozen' argument passed to 252 the _cpu_down() and _cpu_up() functions is *always* 0. 253 This might not reflect the true current state of the system, since the 254 tasks could have been frozen by an out-of-band event such as a suspend 255 operation in progress. Hence, the cpuhp_tasks_frozen variable will not 256 reflect the frozen state and the CPU hotplug callbacks which evaluate 257 that variable might execute the wrong code path. 258 259 2. If a regular CPU hotplug stress test happens to race with the freezer due 260 to a suspend operation in progress at the same time, then we could hit the 261 situation described below: 262 263 * A regular cpu online operation continues its journey from userspace 264 into the kernel, since the freezing has not yet begun. 265 * Then freezer gets to work and freezes userspace. 266 * If cpu online has not yet completed the microcode update stuff by now, 267 it will now start waiting on the frozen userspace in the 268 TASK_UNINTERRUPTIBLE state, in order to get the microcode image. 269 * Now the freezer continues and tries to freeze the remaining tasks. But 270 due to this wait mentioned above, the freezer won't be able to freeze 271 the cpu online hotplug task and hence freezing of tasks fails. 272 273 As a result of this task freezing failure, the suspend operation gets 274 aborted.