Business Drivers for High Performance Real-time Analytics

23 Jan 2020

Next generation applications, at the cutting edge of user interaction, often tell the user, “We noticed you liked X. We think you will like Y.” Or when the user is presented with choices on his next step or next purchase, they get a message “Many of our users seem happiest if they choose C with A. We thought you should know.” Or sometimes, “You just booked Q – some reviews of R are here, you may want to check out R afterwards.”
These types of savvy applications are presenting users with offers in CONTEXT of their other actions/decisions, when they are still warm, and ready to enhance their experience. Such offers are often much more powerful than those made through a cold email or a cold phone call several days later. This additional intelligence in applications also makes for a delightful user experience, a user who just booked tickets to a show at “The Second City in Chicago”, might want to make a dinner reservation prior to the show and would be delighted to see discount offers or deals at businesses close by. Lastly, this type of intelligence makes a ton of business sense, because competing businesses could easily grab loyal customers, were they to offer a better experience.
A measure of “smartness” of applications could also include detecting odd or unusual behavior that marks different types of fraud from gaming fraud, to identity fraud or abuse of promotions. This scenario is even more intense. To be effective, analytic processing of current behavior and detection of such fraud needs to complete before the user concludes their fraudulent activity. These types of applications could be using data from several different sources and comparing with current user behavior, to detect and prevent misuse.
With the increasing availability of cheap compute capacity, data processing tools and learning frameworks, it is becoming increasingly achievable to incorporate “intelligence” into applications. However, it is crucial to add this intelligence in without disrupting performance or responsiveness of applications. Customer facing applications in particular have to respond to users in under 100ms, to meet user experience expectations. With Internet latencies often at up to 50ms round trip, application processing, data access and responses have to be generated in 50ms. To achieve this at scale, the database used must be capable of delivering sub-millisecond response times under conditions of any load. Add in the need to achieve intelligence, and this means that your analytic operations on live data need to be accomplished at the same speeds and often as the same operation. In other words, in-database analytics become critical to meeting performance and functionality requirements.
With the increasing availability of cheap compute capacity, data processing tools and learning frameworks, it is becoming increasingly achievable to incorporate “intelligence” into applications. However, it is crucial to add this intelligence in without disrupting performance or responsiveness of applications. Customer facing applications in particular have to respond to users in under 100ms, to meet user experience expectations. With Internet latencies often at up to 50ms round trip, application processing, data access and responses have to be generated in 50ms. To achieve this at scale, the database used must be capable of delivering sub-millisecond response times under conditions of any load. Add in the need to achieve intelligence, and this means that your analytic operations on live data need to be accomplished at the same speeds and often as the same operation. In other words, in-database analytics become critical to meeting performance and functionality requirements.

Figure 1. End-to-end application response time requirements

Analytical Capabilities of Redis

Redis is an in-memory database platform most well-known for its high performance, extreme versatility with data structures and modular extensibility to any data processing, analysis or storage use case.

When it comes to performance, Redis has been benchmarked to handle >1M operations/second at sub-millisecond latencies, with a single modest AWS instance. When it comes to end to end application throughput and latencies, Redis handily outperforms all other NoSQL databases in the market with the fewest resources.
But when it comes to analytics, the true performance boost is provided by its data structures which include Sets, Sorted
Sets, Hashes, Lists, Strings, Bitmap and Hyperloglog. The data structures provide not just mechanisms to store variably
structured data, they come with built-in operations that perform complex in-database analytics on the data, right in memory, where it is stored. This approach eliminates network and computing overhead while also radically simplifying application development complexity.
As an example, take the Sorted Set structure with Redis.

Sorted Sets order their members by a score. Retrieving members by score ranges is trivial with Sorted Set operations and make them a natural fit for time-series data, real-time bid management, purchases by order amounts, most viewed articles, top scores etc. Sorted Sets are built in Redis with mechanisms that provide high performance sorting, operations like ranking, range operations, counts in a range and also set operations like generating intersections, unions, are executed with the maximum possible efficiency and simplicity. Using Sorted Sets for analyses like time-series data analysis is usually an order of magnitude or two faster using Redis than with any other regular key/value store or with disk based databases.

Figure 2. Sorted Set representation

These analytical operations exist side by side with data processing structures like Lists, Hashes, Sets, functionality like Publish/Subscribe and commands like key expiration, increment/decrement by values, pushing and popping from lists and many more, allowing developers to use different data structures like LEGO building blocks within their application.
Essas operações analíticas existem lado a lado com estruturas de processamento de dados como Listas, Hashes, Conjuntos, funcionalidades como Publicar / Assinar e comandos como expiração de chave, incremento / decréscimo de valores, envio e exibição de listas e muito mais, permitindo que os desenvolvedores usem dados diferentes estruturas como blocos de construção LEGO dentro de sua aplicação.
Figure 3. Redis data structures are like building blocks

Keep reading!

Download the complete content and keep expanding your knowledge. It’s free!

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Siga-nos!
AdviseU IT logo
Siga-nos!
Copyright © 2019 AdviseU. Todos os direitos reservados. | Um blog dedicado a quem ama aprender.
Share This