Demartek Comments on IDF2013 and NVMe

19 September 2013

By Dennis Martin, Demartek President NVM Express

Intel Developer Forum 2013

I recently attended the Intel Developer Forum (IDF) 2013 in San Francisco in order to follow-up on the progress of NVM Express (NVMe). While there, I attended an NVMe technical session and spent a large amount of time in the NVMe Pavilion discussing NVMe with most of the vendors there. NVMe has made good progress since last year (see last year’s comments), with the first NVMe SSD announced and a robust ecosystem developing to support this technology.

NVMe — What Is It?

NVMe is a scalable host controller interface designed for enterprise and client systems that use PCI Express SSDs. NVMe is being developed by an open industry consortium of 90+ members and is directed by a 13 company Promoter Group including Cisco, Dell, EMC, Intel, LSI, Micron, NetApp, Oracle, PMC-Sierra, Samsung, SanDisk, Seagate and sTec (now a part of Western Digital). The original NVMe 1.0 specification was published in March 2011. The 1.0c specification was published in February 2012. Version 1.1 of the NVMe specification was published in October of 2012. The NVMe workgroup is continuing to develop the specification and is expecting the next revision to be completed approximately mid-year 2014.

PCI Express

As we have been explaining for some time, acceptance of SSDs is growing in enterprise datacenters. Although there are several good interfaces available for SSD technology, PCI Express (PCIe) in particular provides very good bandwidth and very low latency characteristics, with the ability to support the next generation of non-volatile memory.

The current generation of PCIe (3.0) provides 1 GB/sec of bandwidth per lane, and many PCIe devices in this category use four or eight lanes of PCIe, which means that 4 GB/sec or 8 GB/sec is achievable with these storage devices.

The current generation of servers using PCIe 3.0 can support up to 40 lanes of PCIe per processor, meaning that many of today’s two-socket servers support 80 lanes of PCIe. Many of these servers have several PCIe slots that can support more than one of these PCIe SSDs, along with other types of adapters.

Hardware and Software Latency

The designers of NVMe are anticipating some interesting changes with respect to performance of the entire hardware and software infrastructure. With today’s technology, an I/O request spends the majority of its time working through the hardware infrastructure, including the NAND flash media, the flash controller, the adapter and related hardware. A small portion of the time is spent working through the host software I/O stack.

Future non-volatile memory technologies, some of which are beginning to emerge, have the potential of performing much faster than today’s NAND flash technology. When this occurs, using current software stacks and protocols, an I/O request will spend the majority of its time working through the software stack, rather than through the hardware infrastructure.

NVMe is designed to streamline the I/O stack in order to reduce the software stack latency. Intel compared the Linux NVMe software stack to the Linux SCSI/SAS software stack. They found that NVMe reduced the software overhead by more than 50% in their tests, from 6.0 µs (microseconds) to 2.8 µs. The NVMe stack also reduced the number of instruction cycles from 19,500 to 9,100.

Because NVMe is designed for non-volatile memory technologies, it is designed with support for many more queues than other protocols that were designed for slower storage technologies. NVMe supports 64K commands per queue and 64K queues so that it can fully support very fast hardware.

NVMe is designed to work with non-volatile memory technology that includes today’s NAND flash technology and future non-volatile memory technologies. NVMe also currently assumes that the storage is connected to the PCIe bus. However, I have seen at least two other technologies, NVDIMM and TeraDIMM (also known as ULLtraDIMM), that put flash memory on the memory bus and can present itself as storage. These could potentially take advantage of the NVMe stack in the future.

Enterprise vs. Client Devices

Just as with hard disk drive technology, there are two basic classes of SSD technology: Enterprise and Client, each having a different purpose and having different operating characteristics. We explain the technical differences between Enterprise and Client devices in our SSD Deployment Guide. NVMe provides optional features that are designed to support both enterprise SSDs and client SSDs.

NVMe 1.1 Specification Features

The NVMe 1.1 specification includes features that support end-to-end data protection, multi-path support including reservations, power optimizations and enhancements to the NVMe command set. The multi-path support, used primarily in enterprise devices, supports dual-port devices and supports devices behind a PCIe switch. The reservations feature supports multi-host environments and is compatible with implementations that use SCSI reservations.

The NVMe 1.1 specification includes power optimization features that are directed towards client SSDs. The Autonomous Power State Transition can direct the NVMe controller to automatically transition to a lower power state after a certain idle period.

Future revisions to the NVMe specification are expected to support namespace management, live firmware updates and other features.

NVMe Drivers

Driver development has progressed nicely over the last year, and there continues to be a great deal of activity in this area. NVMe device drivers include:

Windows: “in-box” for Windows 8.1 and Windows Server 2012 R2 (StorNVMe)
Linux: native O.S. driver since kernel 3.3 (January 2012)
UNIX: FreeBSD driver upstream and ready for release
Solaris: planned for version S12
VMware: vmklinux driver expected by December 2013
UEFI: open source driver available on SourceForge

NVMe Plugfests

The first NVMe plugfest was held in May 2013 with 11 companies participating, and another plugfest is being planned for 2014. Companies with products passing the Compliance Tests and Interoperability procedures can choose to list their products on the NVMe Integrators List at www.iol.unh.edu/services/testing/NVMe/integratorslist.php.

SNIA NVM Programming TWG

The Storage Networking Industry Association (SNIA) has a Non Volatile Memory (NVM) Programming Technical Work Group (TWG) that is working on a common set of software interfaces that provide access to non-volatile memory. The TWG is not limiting its work to NVMe, but exposes some of the features of NVMe to applications through attributes. This TWG has a working draft of the NVM Programming Model available on the SNIA website at snia.org/forums/sssi/nvmp.

Samsung

Samsung announced their first NVMe SSD earlier this year, claiming significantly faster sequential read speed and random read speed than their other enterprise SSDs.

Microsoft

Microsoft plainly stated that they are committed to NVMe, and presented some impressive performance gains using NVMe compared to SAS and SATA interface devices. They also discussed their roadmap for features planned for both enterprise and client NVMe devices.

Microsoft showed data that produced a more than 4x improvement in IOPS using an NVMe device with their driver compared to SAS and SATA devices. They also showed reduction in end-to-end latency by more than 8x in the same tests.

EMC

I spent some time talking with EMC in their booth about some interesting things that they are doing with NVMe. In addition to the obvious use case of flash-based SSDs, it turns out that NVMe can support other applications such as non-volatile RAM. EMC took existing NVMe components including the FreeBSD driver, utility and a standard reference controller to create a faster NVRAM software and controller package that can become a component of several of their storage systems.

NVMe Protocol and Electrical Tools

I also spent time with several other companies at their booths discussing NVMe. Two companies, Agilent Technologies and Teledyne LeCroy were demonstrating NVMe protocol analyzers, while one company, Tektronix, was demonstrating NVMe electrical validation tools, all of which are generally available today.

NVMe Technology Demonstrations

I also saw technology demonstrations, or prototypes, of NVMe SSDs and NVMe infrastructure from Dell, Intel, LSI, SanDisk, sTec and Western Digital. These included NVMe controllers, PCIe card form factor SSDs and drive form factor SSDs. All of the prototype SSDs I saw were enterprise NVMe SSDs, but LSI was showing their NVMe controller that supports both enterprise and client SSDs. I was told that some of these products are getting close to shipping, some possibly by the end of this calendar year or in the first half of 2014. Discussions with the other companies who were present all lead me to believe that we will see many NVMe products announced throughout 2014. From what I was able to observe and learn, the first batch of NVMe SSDs will be primarily enterprise products, with client SSDs following a few months later.

The UEFI 2.4 specification contains updates for NVMe, and full BIOS support of NVMe devices is expected to appear in products beginning in 2014. In addition, a native VMware vSphere driver is expected in 2014.

The companies with booths in the IDF2013 NVMe Pavilion included Agilent Technologies, Dell, EMC, Intel, LSI, Micron, PMC-Sierra, SanDisk, SNIA, sTec, Tektronix, Teledyne LeCroy, Viking Technology and Western Digital.

Demartek NVMe-related Publications

We plan to include an NVMe section in the next edition of our SSD Deployment Guide, and we expect to provide quite a bit more information on NVMe products in 2014. There is additional NVMe technical information on the Demartek Storage Interface Comparison page.

NVMExpress.org

The NVMe website (www.nvmexpress.org) has news, specifications, drivers, whitepapers and much more.