Based on kernel version 4.16.1. Page generated on 2018-04-09 11:53 EST.
1 Started Nov 1999 by Kanoj Sarcar <kanoj@sgi.com> 2 3 What is NUMA? 4 5 This question can be answered from a couple of perspectives: the 6 hardware view and the Linux software view. 7 8 From the hardware perspective, a NUMA system is a computer platform that 9 comprises multiple components or assemblies each of which may contain 0 10 or more CPUs, local memory, and/or IO buses. For brevity and to 11 disambiguate the hardware view of these physical components/assemblies 12 from the software abstraction thereof, we'll call the components/assemblies 13 'cells' in this document. 14 15 Each of the 'cells' may be viewed as an SMP [symmetric multi-processor] subset 16 of the system--although some components necessary for a stand-alone SMP system 17 may not be populated on any given cell. The cells of the NUMA system are 18 connected together with some sort of system interconnect--e.g., a crossbar or 19 point-to-point link are common types of NUMA system interconnects. Both of 20 these types of interconnects can be aggregated to create NUMA platforms with 21 cells at multiple distances from other cells. 22 23 For Linux, the NUMA platforms of interest are primarily what is known as Cache 24 Coherent NUMA or ccNUMA systems. With ccNUMA systems, all memory is visible 25 to and accessible from any CPU attached to any cell and cache coherency 26 is handled in hardware by the processor caches and/or the system interconnect. 27 28 Memory access time and effective memory bandwidth varies depending on how far 29 away the cell containing the CPU or IO bus making the memory access is from the 30 cell containing the target memory. For example, access to memory by CPUs 31 attached to the same cell will experience faster access times and higher 32 bandwidths than accesses to memory on other, remote cells. NUMA platforms 33 can have cells at multiple remote distances from any given cell. 34 35 Platform vendors don't build NUMA systems just to make software developers' 36 lives interesting. Rather, this architecture is a means to provide scalable 37 memory bandwidth. However, to achieve scalable memory bandwidth, system and 38 application software must arrange for a large majority of the memory references 39 [cache misses] to be to "local" memory--memory on the same cell, if any--or 40 to the closest cell with memory. 41 42 This leads to the Linux software view of a NUMA system: 43 44 Linux divides the system's hardware resources into multiple software 45 abstractions called "nodes". Linux maps the nodes onto the physical cells 46 of the hardware platform, abstracting away some of the details for some 47 architectures. As with physical cells, software nodes may contain 0 or more 48 CPUs, memory and/or IO buses. And, again, memory accesses to memory on 49 "closer" nodes--nodes that map to closer cells--will generally experience 50 faster access times and higher effective bandwidth than accesses to more 51 remote cells. 52 53 For some architectures, such as x86, Linux will "hide" any node representing a 54 physical cell that has no memory attached, and reassign any CPUs attached to 55 that cell to a node representing a cell that does have memory. Thus, on 56 these architectures, one cannot assume that all CPUs that Linux associates with 57 a given node will see the same local memory access times and bandwidth. 58 59 In addition, for some architectures, again x86 is an example, Linux supports 60 the emulation of additional nodes. For NUMA emulation, linux will carve up 61 the existing nodes--or the system memory for non-NUMA platforms--into multiple 62 nodes. Each emulated node will manage a fraction of the underlying cells' 63 physical memory. NUMA emluation is useful for testing NUMA kernel and 64 application features on non-NUMA platforms, and as a sort of memory resource 65 management mechanism when used together with cpusets. 66 [see Documentation/cgroup-v1/cpusets.txt] 67 68 For each node with memory, Linux constructs an independent memory management 69 subsystem, complete with its own free page lists, in-use page lists, usage 70 statistics and locks to mediate access. In addition, Linux constructs for 71 each memory zone [one or more of DMA, DMA32, NORMAL, HIGH_MEMORY, MOVABLE], 72 an ordered "zonelist". A zonelist specifies the zones/nodes to visit when a 73 selected zone/node cannot satisfy the allocation request. This situation, 74 when a zone has no available memory to satisfy a request, is called 75 "overflow" or "fallback". 76 77 Because some nodes contain multiple zones containing different types of 78 memory, Linux must decide whether to order the zonelists such that allocations 79 fall back to the same zone type on a different node, or to a different zone 80 type on the same node. This is an important consideration because some zones, 81 such as DMA or DMA32, represent relatively scarce resources. Linux chooses 82 a default Node ordered zonelist. This means it tries to fallback to other zones 83 from the same node before using remote nodes which are ordered by NUMA distance. 84 85 By default, Linux will attempt to satisfy memory allocation requests from the 86 node to which the CPU that executes the request is assigned. Specifically, 87 Linux will attempt to allocate from the first node in the appropriate zonelist 88 for the node where the request originates. This is called "local allocation." 89 If the "local" node cannot satisfy the request, the kernel will examine other 90 nodes' zones in the selected zonelist looking for the first zone in the list 91 that can satisfy the request. 92 93 Local allocation will tend to keep subsequent access to the allocated memory 94 "local" to the underlying physical resources and off the system interconnect-- 95 as long as the task on whose behalf the kernel allocated some memory does not 96 later migrate away from that memory. The Linux scheduler is aware of the 97 NUMA topology of the platform--embodied in the "scheduling domains" data 98 structures [see Documentation/scheduler/sched-domains.txt]--and the scheduler 99 attempts to minimize task migration to distant scheduling domains. However, 100 the scheduler does not take a task's NUMA footprint into account directly. 101 Thus, under sufficient imbalance, tasks can migrate between nodes, remote 102 from their initial node and kernel data structures. 103 104 System administrators and application designers can restrict a task's migration 105 to improve NUMA locality using various CPU affinity command line interfaces, 106 such as taskset(1) and numactl(1), and program interfaces such as 107 sched_setaffinity(2). Further, one can modify the kernel's default local 108 allocation behavior using Linux NUMA memory policy. 109 [see Documentation/vm/numa_memory_policy.txt.] 110 111 System administrators can restrict the CPUs and nodes' memories that a non- 112 privileged user can specify in the scheduling or NUMA commands and functions 113 using control groups and CPUsets. [see Documentation/cgroup-v1/cpusets.txt] 114 115 On architectures that do not hide memoryless nodes, Linux will include only 116 zones [nodes] with memory in the zonelists. This means that for a memoryless 117 node the "local memory node"--the node of the first zone in CPU's node's 118 zonelist--will not be the node itself. Rather, it will be the node that the 119 kernel selected as the nearest node with memory when it built the zonelists. 120 So, default, local allocations will succeed with the kernel supplying the 121 closest available memory. This is a consequence of the same mechanism that 122 allows such allocations to fallback to other nearby nodes when a node that 123 does contain memory overflows. 124 125 Some kernel allocations do not want or cannot tolerate this allocation fallback 126 behavior. Rather they want to be sure they get memory from the specified node 127 or get notified that the node has no free memory. This is usually the case when 128 a subsystem allocates per CPU memory resources, for example. 129 130 A typical model for making such an allocation is to obtain the node id of the 131 node to which the "current CPU" is attached using one of the kernel's 132 numa_node_id() or CPU_to_node() functions and then request memory from only 133 the node id returned. When such an allocation fails, the requesting subsystem 134 may revert to its own fallback path. The slab kernel memory allocator is an 135 example of this. Or, the subsystem may choose to disable or not to enable 136 itself on allocation failure. The kernel profiling subsystem is an example of 137 this. 138 139 If the architecture supports--does not hide--memoryless nodes, then CPUs 140 attached to memoryless nodes would always incur the fallback path overhead 141 or some subsystems would fail to initialize if they attempted to allocated 142 memory exclusively from a node without memory. To support such 143 architectures transparently, kernel subsystems can use the numa_mem_id() 144 or cpu_to_mem() function to locate the "local memory node" for the calling or 145 specified CPU. Again, this is the same node from which default, local page 146 allocations will be attempted.