NVMe-oF: Simple, invisible fabric for distributed storage networks

The Non-Volatile Memory Express (NVMe) specification has been the dominant interface for storage devices over the past several years. NVMe-oF is an extension to NVMe with similar advantages. In this paper, we bring out the importance of NVMe-oF for modern-day applications and discuss the advantages and disadvantages of the various fabrics that are compatible with NVMe-oF. The use cases for the different compatible fabrics and the role of NVMe-oF QVIP in the exhaustive verification of RDMA-based, long distance storage are highlighted.

Applications where NVMe-oF is a good fit

In today’s fast paced world, we need seamless access to huge chunks of data and new-world
technologies, such as artificial intelligence (AI), machine learning (ML), cloud computing, and
real-time data analytics. AI researchers are deriving applications such as cyber security analysis and intelligent virtual assistants (IVA) where the computer needs to process an intense amount of data. Therefore, researchers need an improvement to the existing infrastructure, one that has a high-performance matrix and low latency, enabling access to storage over a network at a faster pace. This solution is provided by real-time data analytics, which requires high throughput and low latency networks.

AI models require a huge amount of data, comprising millions of examples to train the AI
models so that they can make correct decisions. For the longest time, we kept focusing on improving the design of memory to increase the speed of the memory read/write operations, but we missed the role of the underlying transport layer in improving the performance. With its blazing speed, NVMe-oF paves a way for enterprises to utilize the full potential of NVMe SSDs. Figure 1 shows a complete overview of NVMe-oF with remote data memory access (RDMA) as the underlying fabric. NVMe-oF and RDMA are the best fit for each other because the fewer layers in RDMA reduces the latency factor. NVMe-oF can also help in reducing overloads of storage from the CPU and free up a greater number of cycles of the CPU. These cycles can be used for further computation purposes, thereby increasing performance.

However, every technology comes with its downsides. In the case of NVMe-oF, at a broader level, these downsides are cost and complexity, which impede faster adoption of NVMe-oF in enterprises. Currently, IT infrastructures use an Ethernet network and the TCP/IP protocol for communication over that network. Replacing the existing spinning disks at the data center doesn’t solve the problem, because the interface protocols that are used limit the performance of SSDs, as these protocols do not support the level of parallelism that the NVMe technology supports.