Persistent memory for transient data

Arguably, the most notable characteristic of persistent memory is that it is persistent: it retains its contents over power cycles. One other important aspect of these persistent-memory arrays that, we are told, will soon be everywhere, is their sheer size and low cost; persistent memory is a relatively inexpensive way to attach large amounts of memory to a system. Large, cheap memory arrays seem likely to be attractive to users who may not care about persistence and who can live with slower access speeds. Supporting such users is the objective of a pair of patch sets that have been circulating in recent months.

Current kernels treat persistent memory as a device. The memory managed by that device can host a filesystem or be mapped directly into a process's address space (or both), but it will only be used by processes that ask for it explicitly. This patch set from Dave Hansen can change that behavior, though. It creates a new device driver that takes any persistent memory assigned to it and hotplugs it into the system as a range of ordinary RAM; after that, it will be given over to processes to satisfy normal memory-allocation requests. A portion (or all) of the system's persistent memory can be given over to this use, as the system administrator wishes.

Persistent memory used in this mode looks like ordinary memory, but the two are still not exactly the same. In particular, while persistent memory is fast, it is still not as fast as normal RAM. So users may well want to ensure that some applications use regular memory (DRAM) while others are relegated to persistent memory that is masquerading as the regular variety. When persistent memory is added to the system in this way, it shows up under one or more special NUMA nodes, so the usual memory-policy mechanisms can be used to control which processes use it. As Hansen suggested, a cloud provider could use this mechanism to offer two classes of virtual machines, with the cheaper ones confined mostly to the slower, persistent memory.

Hansen's patches are mostly uncontroversial; that is not entirely true of the other patch set, though. Intel has developed a hardware feature known as "memory mode", which is another way to use persistent memory as if it were DRAM. It differs, though, in that memory mode also takes over the system's real DRAM and uses it as a direct-mapped cache for persistent memory. An application that exhibits good cache behavior will be able to use persistent memory at something close to DRAM speeds; things will slow down, though, in the presence of a lot of cache contention.

The fact that the cache is direct-mapped can make contention problems worse. Unlike an associative cache, a direct-mapped cache only has one slot available for any given memory address; if the data of interest is not in that particular slot, it cannot be in the cache at all. Making effective use of such an arrangement requires a memory-allocation pattern that will spread accesses across the entire cache. Otherwise, an applications memory may end up mapped to a relatively small number of cache slots and it will end up contending with itself — and running slowly.

The Linux memory-management system has no awareness of this kind of caching, though, and thus makes no provisions for using the cache effectively. The result is inconsistent performance at best, and heavy cache contention at worst; cache utilization tends to degrade over time, leading to situations where some high-performance users end up periodically rebooting their systems to restore performance. Linux might achieve world domination even with such behavior, but parts of that world would be likely to be looking for a new overlord.

The proposed solution, in the form of this patch set from Dan Williams, is simple enough: randomize the order in which memory appears on the free lists so that allocations will be more widely scattered. The initial randomization is done at system boot, when memory (in relatively large blocks) is shuffled. Over time, though, the system is likely to undo that randomization; mechanisms like memory compaction are designed to clean up fragmentation messes, for example. To avoid the creation of too much order, the patch set randomizes the placement of new large blocks in the free lists as they are created, hopefully keeping access patterns scattered over the lifetime of the system.

Williams cited some benchmarks that show performance improvements from this randomization when a direct-mapped cache is in use. Perhaps most importantly, the long-term performance levels out and remains predictable over the life of the system rather than degrading over time. Even so, this patch set has proved to be a hard sell with the memory-management developers, who fear its effects on performance in general. The shuffling only happens if the system is detected to be running in memory mode (or if it has been explicitly enabled with a command-line parameter), so it should not have any effect on most systems. Michal Hocko eventually came around to a grudging acceptance of the patches. Mel Gorman, instead, has withheld his approval, though he has also chosen not to try to block it from being merged.

One other developer who does support the patch is Kees Cook, who sees some potential security benefits from the randomization. The security benefits have, in general, been even harder to sell than the performance benefits, especially since nobody has provided an example of an attack that would be blocked by the free-list shuffling. Kernel developers can be unfavorably inclined toward security patches even when clear security benefits have been demonstrated; protestations that a change might, maybe, make things better, possibly, someday, tend not to get too far.

At this point, the work is seemingly complete and has gone to Andrew Morton, who will have to make a decision on whether to accept it. He has not tipped his hand so far, so the direction he will go is not clear. In the end, though, this is a relatively focused patch set that should help some use cases while having no effect on the rest. It would not be surprising if it found its way in sometime well before we all get our persistent-memory laptops to use in our autonomous flying cars.

Index entries for this article
Kernel	Memory management/Nonvolatile memory

---

to post comments