Business Drivers for High Performance Real-time Analytics
With the increasing availability of cheap compute capacity, data processing tools and learning frameworks, it is becoming increasingly achievable to incorporate “intelligence” into applications. However, it is crucial to add this intelligence in without disrupting performance or responsiveness of applications. Customer facing applications in particular have to respond to users in under 100ms, to meet user experience expectations. With Internet latencies often at up to 50ms round trip, application processing, data access and responses have to be generated in 50ms. To achieve this at scale, the database used must be capable of delivering sub-millisecond response times under conditions of any load. Add in the need to achieve intelligence, and this means that your analytic operations on live data need to be accomplished at the same speeds and often as the same operation. In other words, in-database analytics become critical to meeting performance and functionality requirements.
Figure 1. End-to-end application response time requirements
Analytical Capabilities of Redis
Sets, Hashes, Lists, Strings, Bitmap and Hyperloglog. The data structures provide not just mechanisms to store variably
structured data, they come with built-in operations that perform complex in-database analytics on the data, right in memory, where it is stored. This approach eliminates network and computing overhead while also radically simplifying application development complexity.
As an example, take the Sorted Set structure with Redis.
Sorted Sets order their members by a score. Retrieving members by score ranges is trivial with Sorted Set operations and make them a natural fit for time-series data, real-time bid management, purchases by order amounts, most viewed articles, top scores etc. Sorted Sets are built in Redis with mechanisms that provide high performance sorting, operations like ranking, range operations, counts in a range and also set operations like generating intersections, unions, are executed with the maximum possible efficiency and simplicity. Using Sorted Sets for analyses like time-series data analysis is usually an order of magnitude or two faster using Redis than with any other regular key/value store or with disk based databases.
Figure 2. Sorted Set representation
These analytical operations exist side by side with data processing structures like Lists, Hashes, Sets, functionality like Publish/Subscribe and commands like key expiration, increment/decrement by values, pushing and popping from lists and many more, allowing developers to use different data structures like LEGO building blocks within their application.
Figure 3. Redis data structures are like building blocks
However, don’t stop there: keep learning more and download now the complete and free ebook “Redis for Real-Time Analytics” and good reading!