Connecting the Dots: Integrating Apache Spark into Production Pipelines

Have you ever struggled to smoothly integrate Spark into a production workflow? Spark is an excellent tool for processing data on the terabyte scale, but building a system to move from raw data through featurization, modeling, and prediction serving involves interacting with numerous other components. Over the past year and half my team at ShopRunner has built a production Spark workflow for data science from scratch. In this talk you’ll learn about the tools we use, the challenges we encountered, an open-source library we wrote to work through them, and how you can avoid the detours we took along the way. Data science work often begins in an interactive notebook environment, exploring data and testing out different modeling approaches. However, moving towards a production environment means building reproducible workflows, packaging libraries, setting up scheduling and monitoring of jobs, and figuring out ways to serve results to clients in real time. After testing out a variety of tools, we at ShopRunner have settled on a stack including Databricks, Snowflake, Datadog, Jenkins, and S3, ECS, and RDS from the suite of AWS services. These tools each offer unique benefits for their area of focus, but crafting a cohesive pipeline from this range of tools presented a challenge. Come learn how to integrate a Spark workflow into a pipeline that analyzes many terabytes of data, builds machine learning models at scale, and serves predictions to a variety of customer-facing tools. Whether you’re just getting started using Spark in productions systems or you already have Spark running in production and want to smooth the process, this talk will leave you better equipped to find and connect the tools that suit your needs.
展开查看详情

1.WIFI SSID:SparkAISummit | Password: UnifiedAnalytics

2.Connecting the Dots Integrating Apache Spark into Production Pipelines Hanna Torrence Data Scientist #UnifiedAnalytics #SparkAISummit

3. Amazon Prime for everyone else: Our 6 million members get free two-day shipping, returns, and deals across a growing network of 140+ retailers. #UnifiedAnalytics #SparkAISummit 3

4.Data Science Projects • Trending products • Product recommendations • Retailer propensity models • Churn modeling • Taxonomy classification • Attribute tagging #UnifiedAnalytics #SparkAISummit 4

5.Data Science Product business data modelling production need exploration #UnifiedAnalytics #SparkAISummit 5

6.Data Science Product business data modelling production need exploration #UnifiedAnalytics #SparkAISummit 6

7.Data Science Product business data modelling production need exploration #UnifiedAnalytics #SparkAISummit 7

8.Important Business Need or #UnifiedAnalytics #SparkAISummit 8

9.Exploratory Phase • Wrangling relevant data • Playing with different models • Continuing conversations to clarify the business problem #UnifiedAnalytics #SparkAISummit 7

10. Exploration #UnifiedAnalytics #SparkAISummit 10

11. Production • Maintainable Code • Scheduled Jobs • APIs #UnifiedAnalytics #SparkAISummit 11

12.Maintainable Code • scripts cleaned up + turned into functions/classes • code review to improve code and share knowledge • unit tests + continuous integration for safer, easier changes #UnifiedAnalytics #SparkAISummit 12

13.Maintainable Code #UnifiedAnalytics #SparkAISummit 13

14.Maintainable Code #UnifiedAnalytics #SparkAISummit 14

15.Maintainable Code #UnifiedAnalytics #SparkAISummit 15

16.Scheduled Jobs • Databricks job scheduler manages clusters • Jenkins manages library updates • We wrote apparate to manage communication between the two apparate #UnifiedAnalytics #SparkAISummit 16

17.Scheduled Jobs Create a Job Update library manual update in UI Build a new egg Upload to Databricks manual inspection manual update in UI Find all jobs using that library Update each job #UnifiedAnalytics #SparkAISummit 17

18.Scheduled Jobs #UnifiedAnalytics #SparkAISummit 18

19.Scheduled Jobs Create a Job Update library apparate Build a new egg Upload to Databricks apparate apparate Find all jobs using that library Update each job #UnifiedAnalytics #SparkAISummit 19

20.#UnifiedAnalytics #SparkAISummit 20

21.Scheduled Jobs #UnifiedAnalytics #SparkAISummit 21

22. Scheduled Jobs GitHub repo: https://github.com/ShopRunner/apparate Databricks blog post: Apparate: Managing Libraries in Databricks with CI/CD #UnifiedAnalytics #SparkAISummit 22

23.APIs • Approach to serving results varies by use case • Flask API in a Docker container deployed on a Kubernetes cluster via Spinnaker • Deploy APIs using ShopRunner’s standard production pipeline #UnifiedAnalytics #SparkAISummit 23

24.APIs #UnifiedAnalytics #SparkAISummit 24

25.APIs api: crookshanks/cat_or_dog post: image: “cat_1.jpg” post: image: “dog_1.jpg” post: image: “dog_2.jpg” vector: […] vector: […] vector: […] #UnifiedAnalytics #SparkAISummit 25

26.APIs image: “cat_1.jpg” image: “dog_1.jpg” image: “dog_2.jpg” prediction: “cat” prediction: “dog” prediction: “cat” #UnifiedAnalytics #SparkAISummit 26

27. So we have … • Moved exploratory code into a python package • Branch/Pull Request • code reviewed workflow with Jenkins CI • tested • version-controlled • single source of truth • Scheduled a batch jobs • Updates with apparate • Deployed an API • Deploys via Spinnaker CD #UnifiedAnalytics #SparkAISummit 27

28.Data Science Product business data modelling production need exploration #UnifiedAnalytics #SparkAISummit 28

29.Lessons Learned • Take advantage of existing infrastructure + best-in-class tools • Be aware of friction points in the process • Build out solutions to ease frustrating connections #UnifiedAnalytics #SparkAISummit 29

user picture
Spark开源社区
+关注
由Apache Spark PMC & Committers发起。致力于发布与传播Apache Spark + AI技术，生态，最佳实践，前沿信息。
https://spark.apache.org/
已将链接复制至剪贴板
相关文档