Finding Bad Acorns
展开查看详情
1. ANDREW GAO FINDING BAD & ACORNS JEFF SHARPE FLINK FORWARD 2018
2.ANDREW GAO JEFF SHARPE
3.Our journey to build a Fraud Decisioning Platform and use Flink to build out the use cases Developing a Fraud Fraud Defense at the Defense Platform Teller Using Flink
4.DEVELOPING A FRAUD DEFENSE PLATFORM
5. OUR USERS Fraud Customer Operator Data Data Scientist Analyst Product Engineer Owner
6.
7. OUR USERS Fraud Customer Operator Data Data Scientist Analyst Product Engineer Owner
8. ARCHITECTURE DATA ACTIONS MAGIC!
9.RUNNING ON
10.
11.RUNNING ON
12.Developing on Kubernetes has been challenging but very rewarding PROS CONS • Community support for • Maintaining your own Docker/Kube Kubernetes solution • Resilient • Containing blast radius • Easy to tear down and bring • Edge cases when combining # back of technology solutions • Maximizing resource efficiency
13.
14.FRAUD DEFENSE AT THE TELLER
15.
16. A FLINK MONOLITH • Problem: Develop a stream processing workflow for two legacy batch data sources • First Attempt: Do everything in Flink and take advantage of Flink Connected Streams
17.Using Flink operators to build our application workflow 2 3 4 1
18.AWS Lambda is a good fit for our use case and works well with our underlying technologies PROS CONS • Cheap • Not truly stateless • Start-up time • Not a lot of Code/Config • Scalability / Availability • Deployments are a breeze
19.Using Flink operators to build our application workflow 2 3 4 1
20.CUSTOM WINDOWS FOR OPTIMIZATION AND PORTABILITY 90 Day Storage Window 30 Day Virtual View 90 Day Filtered View
21. CUSTOM WINDOWS FOR OPTIMIZATION AND PORTABILITY Most-Recent-Beyond-24-Hours Window 24 Hour Offset Dynamic Window
22.Using Flink operators to build our application workflow 2 3 4 1
23. USING JYTHON TO BRIDGE THE GAP TO DATA SCIENTISTS Windows Data Flink Data Featur Featur Featur Featur Jython Adapter e e e e Featur Featur Featur Featur e e e e .py .py .py .py .py .py .py .py
24. GITFLOW AND JYTHON IMPROVE TRACEABILITY Featur Maven Develop Denied e JAR Import Build v1.0.42 Failed Junit Junit Tests Flink Tests Merge Job JAR Pull Build Request Commit
25.Using Flink operators to build our application workflow 2 3 4 1
26.FEATURES EXIST TO FEED MODELS Model Model Score Feature Feature H20 Tensor Flow Seldon (whatever)
27.
28. BREAKING UP THE MONOLITH • Problem: Back Pressure leading to Delayed Transactions • Solution: Break up the monolith Flink App into small Queryable State Apps
29.CHIPMUNKS