We are pleased to announce that Principled Technologies has acquired Demartek.
Demartek Comments on IDF2014 and NVMe, Thunderbolt 2, USB 3.1 and PCIe 4.0
24 September 2014
By Dennis Martin, Demartek President
Intel Developer Forum 2014
Earlier this month I attended the Intel Developer Forum (IDF) 2014 in San Francisco. I was especially interested in four technologies: NVM Express (NVMe), Thunderbolt 2, USB 3.1 and PCIe 4.0. I also observed a few other interesting tidbits.
NVM Express (NVMe)
I wrote about NVMe at IDF2013 previously, so I won’t repeat most of what I said in
last year’s comments other
than to say that NVMe is a scalable host controller interface designed for enterprise and client systems that use
PCI Express (PCIe) non-volatile memory devices such as SSDs. The goal is to reduce the host software stack in order
to keep pace with the reduced latencies of NAND flash SSDs and the technologies that will follow NAND flash in the future.
We had the opportunity to run some performance tests in our lab with two different brands of shipping NVMe SSDs, and I shared these results in my Flash Memory Summit 2014 presentation entitled Real-World Performance of Flash-Based Storage Systems that you can view or download. These NVMe SSDs use the SFF-8639 backplane interface in the server, and more information on SFF-8639 is available on our popular Storage Interface Comparison reference page. For these tests, we ran database workloads with Microsoft SQL Server using the native in-box NVMe drivers for Windows Server 2012 R2. The latencies for the NVMe SSDs were quite low, and bandwidth exceeded 2 GB (gigabytes) per second for a single NVMe SSD with a real-world database workload.
According to some estimates, PCIe is projected to be the leading enterprise SSD interface by the year 2018, surpassing SAS and SATA device interfaces as we know them today. PCIe bandwidth is significantly higher than SAS or SATA, and NVMe has lower latency than SAS or SATA interfaces. This calendar year has seen delivery of a few enterprise NVMe SSDs. I was told to expect more enterprise NVMe SSDs to be announced and begin shipping later this year and into next year.
Client NVMe SSDs are expected to appear starting in 2015, in a variety of form factors. These form factors for client NVMe SSDs include M.2 for laptops and similar devices, and the standard 2.5-inch drive form factor. The M.2 form factor NVMe devices support up to four lanes of PCIe and work with the Intel X99 chipsets on desktop-class motherboards. A new BGA (ball grid array) form factor has been proposed by Intel, SanDisk, Toshiba and HP in the PCI-SIG for thin laptops and possibly smaller devices. One of the key attributes of client SSDs is low power consumption. NVMe client SSDs are expected to draw less power than SATA SSDs while delivering higher performance.
Native NVMe drivers are available for Windows Server 2012 R2 and Linux kernels 3.12 and higher. NVMe drivers
are also available for FreeBSD, Solaris and others. UEFI open source drivers are available on SourceForge,
as well as an open source vmklinux driver for VMware. A native NVMe driver is planned for VMware vSphere 6.0.
Several organizations had booths in the NVMe Community at IDF2014, including EMC, Intel Corporation, JDSU, Keysight Technologies (formerly Agilent), Micron, Microsoft, Mobiveil, PMC, Samsung, Seagate (LSI), SuperMicro, Teledyne LeCroy and SNIA. They showed various NVMe products running, as well as NVMe test equipment.
Version 1.2 of the NVMe specification is expected to be complete by the end of this calendar year. It is expected to include features such as active/idle power, temperature thresholds, namespace management, live firmware updates and more. NVMe currently supports S.M.A.R.T. data for SSDs.
Amber Huffman from Intel gave a great NVMe presentation to a packed room of more than 200 people, I estimate, at IDF2014. Every seat was taken, and there were people standing along the sides and in the back of the room. The slides from this session (SSDS003), titled “NVM Express: Going Mainstream and What’s Next” are available at this link on the IDF2014 website.
NVM Express over Fabrics
About a week prior to IDF2014, NVM Express, Inc., the organization that developed the NVM Express specification, announced that it had begun work on a new specification known as NVM Express over Fabrics. The goal is to extend the benefits of NVMe onto fabrics such as Ethernet, Fibre Channel, InfiniBand and Intel Omni Scale, and accomplish this using a single industry-wide approach.
These fabrics can support large numbers of NVMe SSDs, and these SSDs can be at some distance from the host
servers that need them. The goal is to make “remote NVMe” have no more than approximately 10 µs
(microseconds) of added latency compared to attaching NVMe SSDs to the local PCIe bus.
I saw a demonstration of NVM Express over Fabrics at one of the booths in the NVMe Community. This particular implementation used two desktop-class computers, each with a single NVMe SSD inside. The two systems were connected via 40Gb Ethernet running iWARP RDMA.
In this demo, the remote NVMe latency was 8 µs higher than the local latency. The IOPS for both systems were
similar, with the remote NVMe SSD running at 450,000 IOPS and the local NVMe SSD running slightly faster,
as seen in the photo. Remote NVMe does not require RDMA, but works very well with an RDMA-enabled network connection.
I believe that NVMe in general is going to have a significant impact on the SSD marketplace. I also expect that NVMe over Fabrics will generate lots of activity among the Ethernet, Fibre Channel and InfiniBand product companies. I made a few other comments regarding NVMe in my Flash Memory Summit 2014 Commentary and in my Next Generation Storage Networking presentation at the Storage Developer Conference 2014.
NVM Express and the Demartek SSD Deployment Guide
We are currently updating the Demartek SSD Deployment Guide and will have a section devoted to NVMe. The Demartek Deployment Guides are a series of technical deployment guides designed to provide practical guidance for planning and deploying various computing technologies, such as SSDs. These guides are intended for managers and technical professionals within IT departments of all sizes and for technical sales engineers.
The Demartek SSD Deployment Guide covers several categories of SSD technology, such as NAND flash characteristics,
SSD form factors, individual SSD devices and storage systems with SSD technology. It covers caching, tiering,
hybrid arrays, performance and a variety of related topics. It also shows how to deploy the technology in
physical and virtual server environments, discusses best practices and includes separate sections for vendors
of these technology products and solutions.
NVMExpress.org
The NVMe website (www.nvmexpress.org) has news, specifications, drivers, whitepapers and much more. The schedule for upcoming events, including NVMe plugfests, is available on this site.
Thunderbolt™ 2 (20 Gbps)
I spent some time in the Thunderbolt Community area, discussing some of the advances in Thunderbolt 2.
Although I usually work with enterprise datacenter technologies, I believe that Thunderbolt 2 may find
its way into the datacenter over the next year or two.
Thunderbolt is an I/O interface that carries DisplayPort (HD video and HD audio) and PCI Express data bi-directionally over a single cable. The original Thunderbolt ran at 10 Gbps, and Thunderbolt 2 doubles that rate to 20 Gbps. Thunderbolt 2 products began shipping last year. Using a Thunderbolt connection, up to six devices can be daisy-chained together. Thunderbolt can also be used to transmit up to 10 watts of power to a device.
The target audience for Thunderbolt technology is media creators and editors who use premium laptops,
desktops, workstations and peripherals that connect to them. The peripherals typically include
high-resolution video cameras and high-speed portable storage devices. Although the original
Thunderbolt technology was only available on Apple Mac computers, the new Thunderbolt 2 technology
is available on both Apple Mac and PC platforms. HP workstations now include Thunderbolt 2, and there
is an entire ecosystem of Thunderbolt 2 products, including from 4K video cameras, portable SSDs,
motherboards, add-in cards and even Thunderbolt 2 hubs.
One of the interesting applications for Thunderbolt 2 is connecting a PC and a Mac together via a 20 Gbps Thunderbolt 2 cable and running 10 Gbps Ethernet, dragging and dropping files between the two systems seamlessly and very quickly. I also saw that Promise Technology provides two Thunderbolt 2 bridge devices. One connects to 10GbE switches and the other connects to Fibre Channel SANs. These would certainly allow a Thunderbolt 2 computer or device to participate in the datacenter.
USB 3.1 (10 Gbps) and USB Power Delivery
The new USB 3.1, with shipping products expected by year-end, runs at 10 Gbps, doubling the speed
of USB 3.0, and is compatible with previous versions of USB. USB 3.1 can also deliver up to 100
watts of power, and this portion of the specification is known as USB Power Delivery. USB 3.1
can deliver audio, video, data and power concurrently and bi-directionally.
At the same time as USB 3.1 appears, the new USB type-C connector will become available. This is
an entirely new design for USB, with a reversible plug orientation and cable direction. This means
no more USB cables that have one type of connector for the device, and a different type of connector
for the computer or charging station. These new type-C cables are electronically marked, so that
cable information is available to the devices. I have a close-up photo of this cable in my
Next Generation Storage Networking presentation
from the Storage Developer Conference 2014.
In the photo, I’m holding a prototype USB type-C cable and standing in front of a demonstration of a computer connected to an SSD via USB 3.1. This SSD is reporting more than 800 MB/sec of bandwidth, which exceeds what is possible with previous versions of USB and today’s 6 Gb SATA interface.
I saw an interesting demonstration of USB 3.1 that I described previously in my International CES 2014 Commentary, involving power and video delivery between a laptop computer and a video display.
Second Screen
I wandered through the Technology Showcase area and saw several interesting technologies on display.
One that I found especially interesting is called Second Screen. One prototype implementation that
I saw uses a second screen on a back of a laptop, using the same electronic ink technology as some
e-readers, that uses very low power and is highly visible. In this example, the presenter could be
running presentation software or a web browser in the normal manner, and can share a portion of the
screen to the second screen. This has some very interesting possible uses for real-time notifications,
and I believe it is much more interesting than simply putting a sticker on a back of a laptop computer.
In this example, we sent the Demartek website home page to the second screen, and the screen updates as the images change every few seconds. Once loaded onto the second screen, the computer can be powered down, and the images on the second screen continue to be refreshed in real-time. This type of second screen could certainly be used for very low power display panels on datacenter equipment just as easily as the prototype laptop computer shown in the photo.
Water Cooling
Many years ago, I worked on water-cooled mainframe computers. I saw a new example of this in the IBM
booth at IDF2014.
On the far left side of the photo, we see a traditional air-cooled two-processor server that fits into half of a 1U server form factor. In the center and right side, we see a pair of two-processor servers, each half of a 1U width, sharing a water cooling system.
Because water is a much more efficient coolant than air, this pair of servers runs cooler, and because the only fans are in the power supplies, not the main part of the server, it makes very little noise. Notice that the water cools all the processors and all the memory sticks.
PCI-Express (PCIe) 4.0
I spent a little time with the PCI-SIG, which is the industry organization chartered with the development and management of the PCI bus specification. Specifically, I was interested in the PCIe 4.0 specification, which is currently under development.
In November 2011, the PCI-SIG announced the approval of 16 giga-transfers per second (GT/s) as the bit rate for the upcoming PCIe 4.0. This is twice the rate of the current PCIe 3.0 specification available in computers today. There is quite a bit of work to do to complete this specification. The PCI-SIG has targeted version 0.5 of the PCIe 4.0 specification to be available to members in the second half of this year (2H 2014), and version 0.9 to be available by 1H 2016.
The PCIe bus is complex and must work properly with processors, chipsets, I/O devices and other components. Based on these target dates, I would expect that products with PCIe 4.0 would be available no earlier than 2017.
Other IDF Commentaries