Server SAN Functionality Exceeds Traditional Storage Arrays

Premise

The functionality of data services deployed from within the server (named Server SAN functionality by Wikibon) is now the equivalent or better than best-of-breed storage arrays.

Server SAN Functionality

In 2014, Wikibon introduced the term Server SAN, and projected that the Server SAN storage model would emerge as the primary storage model, eclipsing the traditional storage array model during the 2020s. The core drivers behind our prediction are cost and performance. The term Server SAN reflects the movement of data services which traditionally had been supplied in a proprietary storage controller to the server itself. The data services can be part of an OS and/or a virtual machine hypervisor. The most common platforms are Linux, Microsoft Windows and VMware’s vSphere (e.g., VSAN). Wikibon has defined and forecast the market in 2015 and 2016.

The greatest inhibitors to the migration from traditional storage arrays to server-based storage services have been:

1. The availability of the equivalent functionality for storage services in Server SAN, and
2. The cost of migration of existing enterprise storage arrays.

Since our original prediction, hyperscale cloud providers have been demonstrating the advantages of Server SAN and have overcoming these barriers. In addition 2016 has seen the strong introduction of new protocols such as NVMe and NVMf together with high speed interconnects from Melanox and other providers. These functions are enabling very low latency direct attachment of servers to storage, and are achieving storage performance unobtainable on traditional SANs.

The major storage functionality that is provided by traditional storage arrays includes:

• Management of Cache and Slow Hard Disk Drives
  • The use of flash and flash caches has meant that superior performance can be delivered with higher levels of flash at lower cost.
• Snapshot Management
  •  QEMU freeze/thaw functionality provides both crash-consistent and application-consistent capabilities in the Linux stack, supported by Red Hat.
  • Requirements for snapshot management are discussed in the White Spaces section below.
• Storage-level Replication
  • The advent of flash has reduced the value of traditional synchronous replication from longer distances to be replaced by local replication. Long distances are no longer supported by the normal sub-second IO response times of flash, and the technology will not be able to support sub 50 microsecond response times that will rapidly become the norm.
  • The use of very high-speed interconnects between servers with fast system interconnect technologies such as RoCE over flexible 40Gb or 100Gb links together with protocols such as NVMf eliminates the requirement for synchronous replication.
  • Asynchronous replication is still a requirement, but has been brought to new levels of efficiency with new application-led methods of minimizing data transfer such as WANdisco’s Paxos technology and SympliVity’s (in the process of being acquired by HPE) advanced file system.
• Encryption
  • Encryption in storage arrays has mainly been at the drive level, which leaves data exposed over the many steps from application to the drive and back. Encryption is supported in a number of modes by the Linux stack.
  • The modern and much safer process is to use end-to-end encryption with the application and database as the focal point for both encryption and decryption. Special instructions, GPUs, and FPGAs in servers can assist in reducing encryption overhead.
• Quality of Service
  • Quality of service management to ensure minimum levels of service is available in all storage arrays and in the Linux stack.
  • The one area that some storage arrays still have the advantage (e.g., NetApp’s Solidfire QoS) is in the area of ensuring maximum levels of service and the ability to burst. Both these options are important for optimizing the revenue model of service providers, and will also become important for enterprise Server SAN. Giving away performance early and then dialing it back is not an effective strategy for customer satisfaction and revenue optimization.

• Data Reduction
  • The four areas of data reduction are zero suppression, thin provisioning, compression and de-duplication. Data sharing is a fifth and important one for the future.
  • Thin provisioning and zero suppression are table stakes, and have been available in Red Hat and other Server SAN solutions for a long time
  • The most significant shortfall in the Linux stack has been the lack of an integrated data reduction capability, including zero suppression, thin provisioning, de-duplication and compression. This void has been filled by the recent support of Permabit’s VDO data reduction stack (which includes all the data reduction components) by Red Hat. As a Red Hat Technology partner, Permabit has early access to Red Hat software, which allows early VDO testing. Permabit certifies VDO data reduction for use with Red Hat Ceph Storage and Red Hat Gluster Storage, and makes it available directly or through their resellers.

The complete availability of quality storage services available at the file system level that cover the functionality provided by traditional storage arrays is a major milestone in the migration to a Server SAN model. Vendor integration and support comes from a variety of Server SAN vendors, including total system vendors such as Nutanix, ScaleIO from Dell EMC, SimpliVity (now being acquired by HPE), Pivot3 and others. There are also software-only vendors such as VMware VSAN and Red Hat. See Wikibon research report for a discussion of the vendors and ecosystem support technologies (e.g., Permabit, Mellanox and others).

This progress does not suggest that there is no room for innovation and deployment in Server SAN. These are discussed in the next section.

White Spaces for Server SAN

The key white space areas for Server SAN are mainly in the storage and system management areas.  They include:

• Virtualized Data Sharing
  • The automation of data flows from a production systems from a snapshot (for example) to development copies of system and data for developers in line with QA.
  • Reduction in the cycle-time for data-dispersion within an enterprise from weeks, to hours, to minutes.
• Advanced Cataloging
  • Creating a record of who/when/where/how has created/modified/deleted data for compliance and early detection of anomalous behavior.
  • Because of the speed of storage and storage services such as snapshots, the cataloging function must be an extension of the data services rather than a separate solution entity.
  • Companies such as Catalogic have been assembling a cross-storage solutions that are potentially far more useful than proprietary programs that support just the storage of a single vendor.
• Converged Data Virtualization, Encryption and Data Reduction Solutions
  • The implementation of security and end-to-end encryption at the application level is potentially a killer for down-stream data reduction services.
  • Tight integration of data reduction and replication services is required to improve application speed and reduce the cost of security.
  • These services all exist within the current Linux stack
• Rapid Recovery Solutions
  • The number of software and human mistakes continues to grow, making recovery and recovery time vitally important. Server SAN offers a way of escaping from the batch-driven storage-centric solutions (e.g., Purpose-built Backup Appliances, or PBBAs) to application- and system- led recovery systems that include the stack, the application, database binaries, and the data to minimize data loss and recovery time.
  • Many regulatory requirements (especially in the trading and banking areas) are moving from batch to real-time.  Enterprises need to rapidly recover backend systems of record and need the ability to recover real-time application data streams that feed real-time compliance applications that are executing key business functions (e.g., trades).  Without this ability, enterprises will be at risk of making poor choices in their compliance-related decision-making.
• Compliance Solutions
  • Many regulatory requirements (especially in fraud detection, trading, and other banking areas) have exponentially increased in importance, and are moving from batch to real-time.
  • As such, catalogs, rapid recovery, the ability to monitor recovery readiness, the ability to apply analytics and AI to detect, intercept and correct accidental and malicious incidents.
• Immutability
  • One of the greatest threats to businesses comes from external data modification – a single bit change can destroy an encrypted database.
  • Therefore immutability must be included in the data protection and recovery processes and be available as a protection against ransomware.  This is an essential function that will almost certainly be well tested!
• Orchestration, Automation, Management and Maintenance of Data Services
  • One of the challenges of the new data services is managing how, when and in which order data services are deployed within the server. Previously, software from the SAN vendors provided expensive solutions in this area. Solution frameworks for this are emerging from OpenStack (distributed by Red Hat and others) and other initiative from both from for-profit ISVs and Open Source projects.

IT executives should be planning to address these white spaces, and find automation and orchestration components to ensure they are deployed effectively.

Conclusions

Solutions based on data services in the server (i.e., Server SAN or Hyper-converged solutions) are ready for prime time, especially since they can now take storage cost advantage of the recent data reduction functionality now provided Permabit as a partner of Red Hat.  Applications that can exploit the low latencies of Server SAN will often be the first deployed, in addition to application data deployed in Tier 2 and Tier 3 storage arrays. Of course, IT executives should weigh the status of depreciation of their traditional storage arrays before moving to Server SAN.

Action Item

Wikibon recommends that infrastructure architects plan to migrate data services from the SAN to the Server, and enable them for middleware, applications and ISV packages. Senior IT executives to initiate a plan for migrations to Server SAN or hyper converged solutions, if one is not already in place. Applications that are very performance sensitive may be the first to migrate, to take advantage of the very low latencies available with Server SAN. Particular attention should be paid to the ability to orchestrate, automate manage and maintain these environments.