It‘s scary how smart and fast the Internet has become. Massive data centers can serve up – nearly instantaneously – the answers to virtually any question I have and provide a never ending stream of posts, pictures and videos of my friends. I easily find out the shirt I was looking at the other day is now on sale and my favorite band is coming to town with a “click here to buy tickets now” reminder.
Behind the scenes of this instant gratification are approaches to processing data. Among these new approaches is the use flash storage technologies (also called non-volatile memory). This approach allows sub-microsecond access to terabytes of data, enabling data scientists to view and process much more data across distributed systems at the same time – a “have your cake and eat it too” moment.
New storage technologies and new approaches
As we consider how to best leverage flash storage technology, one thing becomes obvious: the advent of flash technology changes the established balance of processing and storage. Flash storage can offer data at rates that are over a thousand times faster than hard drives. While CPUs overall have gotten faster, they haven’t gotten a thousand times faster over the last 10 years. So, to take advantage of the faster storage, we need to rethink how we process data.
One approach that NXP and other industry leaders are exploring is the placement of multicore CPUs tightly coupled to, and distributed with, the flash storage itself. This development has led to the creation of an Intelligent Flash Storage platform that can be used as a testbed for a number of new emerging storage technologies. These include various forms of non-volatile memory, the new NVM Express protocol running over PCI Express or Ethernet Fabrics and the use of ARM®-based processors directly attached to the flash storage. These technologies introduce powerful new paradigms that provide new ways of solving big data processing problems.
In terms of form factor, the platform is a PCI Express form factor board that integrates the following: eight high performance 64-bit ARM CPU cores, 40 Gbps of Ethernet network connectivity and protocol acceleration, Gen 3 PCI Express connectivity to a host processor over the standard PCI Express slot interface and an FPGA-based flash subsystem that supports malleable connectivity to a variety of flash memory modules.
The intersection of IO and processing is especially fruitful for research
We are still in the early years of exploring how to process big data and there is much more innovation and experimentation to come. The ability to contribute to this exciting area is something to be proud of.
By bringing together teams from many of the top universities, tier-1 research labs and some of the largest data centers in the world, collectively, we are creating not only a platform, but we want to push the limits of intelligent storage systems.
The ability to explore new heterogeneous memory architectures will guide the industry in how storage tiering will evolve. Being able to recreate what was a previously complex system in a small solid state drive will increase performance over an order of magnitude.
The multi-disciplined industry team is meeting in Boston this week to spend a few days collaborating, presenting ideas and challenges, as well as sharing findings from use of the platform. The goal over the next few months is to use this industry team for real world research on big data and distributed storage problems.
How do you think storage tiering will evolve? Where do you see the future of intelligent storage systems?