As part of an initiative to expand their presence in cloud/datacenter, video and telecommunication verticals, digital solutions provider EchoStreams was looking for a new system to add to their FlacheSAN line. These new servers would provide high performance SSD storage for content delivery networks (CDNs), and video production and post-production applications — all of which typically require both high transfer rates and high numbers of input/output operations per second (IOps). With video file sizes reaching over a terabyte per hour of video, they required some serious bandwidth. On top of all that, customers wanted newer solutions that used NVMe SSDs to achieve higher performance over the existing SATA-based SSDs, while reducing the size and power requirements of the storage systems.

EchoStreams’ customers were requesting faster systems for their CDNs, video production and post-production storage, and big data systems. The OEM/ODM solutions provider determined to meet customer needs with a new version of their FlacheSAN hardware incorporating the latest high-performance SSDs. These new systems would have to exhibit the highest available transfer rates as well as above-average IOps.

Video production and post-production in particular demand very high transfer rates throughput. With higher resolution video (4K or 8K) and more color depth (10bit or more), HDR and higher frame rates (60fps or even 120fps), movie studios need super fast storage to transfer files for editors in post-production. Requirements for CDNs are more mixed — some will need high throughput, some will need high IOps and some will need both.

Previous solutions had relied on lots of drives to deliver aggregate performance with high enough transfer rates and IOps to satisfy application demands, but systems with large numbers of SATA drives also produce a lot of heat and require larger power supplies.

There are several things that limit the number of drives that can fit in a single enclosure. Some are obvious, like the volume of each drive or the amount of heat that each drive generates, which must be dissipated. Other issues, like the total number of connections the interface can support, are less obvious. Echostreams had to find a way to increase the speed and power of their systems without increasing the number of drives.

Systems engineers began development of two systems. The first used 20 U.2 NVMe PCIe SSDs. The second included 36 M.2 NVMe PCIe SSDS. One of the first hurdles was creating a system that could connect the extensive number of NVMe devices. For that, they turned to the new AMD EPYC processor that can support 128 PCIe lanes, and to two Samsung SSDs — the Samsung 983 DCT and the 970 PRO.

EchoStreams engineered a system that allows two NVMe drives to share one 4x PCIe connection, allowing 20 U.2 drives with Intel motherboards, and up to 36 M.2 drives with an AMD EPYC system. The lower power usage of NVMe drives enables systems that run relatively cool and use less power than the SATA drives that were previously in use.

While they design and manufacture all other boards such as backplane, risers and the chassis and system packaging, the EchoStreams NVMe systems were designed using a Tyan/Mitac motherboard. With SATA, it was not uncommon to stuff 60 or more drives onto a backplane. With NVMe, the speed of the drives requires a high-speed backplane, with many motherboards. With Xeon CPUs, PCIe is currently limited to 40 lanes, while the new AMD EPYC system supports 128.

The previous FlacheSAN1L-D4 system, featuring 20 SATA SSDs and dual Xeon E5-2600 v4 (Broadwell), achieved approximately 10GB/s throughput with about 1 million IOps (4K block size QD 32). The new system under development, the FlacheSAN1N20U-UN with 20 Samsung 983 DCT NVMe drives and a single AMD EPYC, hit 34GB/s — a more than 300 percent improvement — and about 4 million IOps — a 400 percent improvement. The first system prototype is currently shipping to limited customers.

The old SATA-based systems could only sustain 550MB/s, while the new systems support 3GB/s read with the Samsung 983 DCTs. The old systems could hit writes 500MB/s, while the new ones hit 1.9 GB/s with the 983 DCTs. IOps went from 100,000 random reads to 500,000 random reads (4K block size) per drive.

The previous FlacheSAN1L-D4 with 20 SATA SSDs and dual Xeon E5-2600 v4 (Broadwell) achieved about 10GB/s sustained throughput with about 1 million IOps (4K block size QD 32). The new FlacheSAN1N20U-UN (featuring 20 Samsung 983 DCT NVMe drives and a single AMD EPYC 7401 24-core CPU with 132GB RAM) was able to hit 34GB/s — an improvement of more than 300 percent — and about 4 million IOps, an improvement of about 400 percent.

The IOps per dollar and performance per watt are also both greatly improved. Power consumption is about the same for the whole system, but generates much higher performance, rising from 500Mbps to 3000MB/s with the same power consumption.

CDNs typically use many relatively small drives (rather than a few large drives), since that results in less data loss per drive in the event of a failure, and more drives provide better results. Media systems supporting 4K to 8K production and HDR need the most capacity: one movie in production can use one terabyte per hour of video.

The old system was a 2U chassis with 40 SATA SSDs and had hit a maximum throughput of 18GB/s; the new system is 40GB/s with half the power consumption. With three 100Gb NICs, 16 to 18 Samsung 983 DCT SSDs can be used to saturate the NICs. More drives can reduce the latency and increase IOps, but higher throughput would require faster NICs.

The new systems are still being developed and evaluated, and will continue to improve. However, even at this preliminary stage, it’s clear that the new NVMe SSDs offer higher performance at a lower operating cost, with the ability to add more SSDs to a single chassis without increasing power consumption or cooling requirements.