申请试用
HOT
登录
注册
 
Redis + Structured Streaming—A Perfect Combination to Scale-Out Your Continuous

Redis + Structured Streaming—A Perfect Combination to Scale-Out Your Continuous

Spark开源社区
/
发布于
/
8764
人观看
“Continuous applications” supported by Apache Spark’s Structured Streaming API enable real-time decision making in the areas such as IoT, AI, fraud mitigation, personalized experience, etc. All continuous applications have one thing in common: they collect data from various sources (devices in IoT, for example), process them in real-time (example: ETL), and deliver them to machine learning serving layer for decision making. Continuous applications face many challenges as they grow to production. Often, due to the rapid increase in the number devices or end-users or other data sources, the size of their data set grows exponentially. This results in a backlog of data to be processed. The data will no longer be processed in near-real-time. Redis, the open-source, in-memory database offers many options to handle this situation in a cost-effective manner. First and foremost, you could insert Redis into an existing continuous application without disrupting its architecture, and with minimal code changes. Redis, being in-memory, allows over a million writes per second with sub-millisecond latency. The Redis Stream data structure enables you to collect both binary and text data in the time series format. The consumer groups of Redis Stream help you match the data processing rate of your continuous application with the rate of data arrival from various sources. In this session, I will perform a live demonstration of how to integrate a continuous application using Apache Spark’s Structured Streaming API with open source Redis. I will also walk through the code, and run a live IoT continuous application.
0 点赞
0 收藏
3下载
确认
3秒后跳转登录页面
去登陆