Finding Bad Acorns

在金融技术领域,一个最主要的工作职责是在实时的流式数据中,通过机器学习模型,找到金融诈骗者。这篇演讲将会解释Capital One工程师用Kafka/Flink/AWS Lambda等技术来完成实时的金融欺诈行为判定。
展开查看详情

1. ANDREW GAO FINDING BAD & ACORNS JEFF SHARPE FLINK FORWARD 2018

2.ANDREW GAO JEFF SHARPE

3.Our journey to build a Fraud Decisioning Platform and use Flink to build out the use cases Developing a Fraud Fraud Defense at the Defense Platform Teller Using Flink

4.DEVELOPING A FRAUD DEFENSE PLATFORM

5. OUR USERS Fraud Customer Operator Data Data Scientist Analyst Product Engineer Owner

6.

7. OUR USERS Fraud Customer Operator Data Data Scientist Analyst Product Engineer Owner

8. ARCHITECTURE DATA ACTIONS MAGIC!

9.RUNNING ON

10.

11.RUNNING ON

12.Developing on Kubernetes has been challenging but very rewarding PROS CONS • Community support for • Maintaining your own Docker/Kube Kubernetes solution • Resilient • Containing blast radius • Easy to tear down and bring • Edge cases when combining # back of technology solutions • Maximizing resource efficiency

13.

14.FRAUD DEFENSE AT THE TELLER

15.

16. A FLINK MONOLITH • Problem: Develop a stream processing workflow for two legacy batch data sources • First Attempt: Do everything in Flink and take advantage of Flink Connected Streams

17.Using Flink operators to build our application workflow 2 3 4 1

18.AWS Lambda is a good fit for our use case and works well with our underlying technologies PROS CONS • Cheap • Not truly stateless • Start-up time • Not a lot of Code/Config • Scalability / Availability • Deployments are a breeze

19.Using Flink operators to build our application workflow 2 3 4 1

20.CUSTOM WINDOWS FOR OPTIMIZATION AND PORTABILITY 90 Day Storage Window 30 Day Virtual View 90 Day Filtered View

21. CUSTOM WINDOWS FOR OPTIMIZATION AND PORTABILITY Most-Recent-Beyond-24-Hours Window 24 Hour Offset Dynamic Window

22.Using Flink operators to build our application workflow 2 3 4 1

23. USING JYTHON TO BRIDGE THE GAP TO DATA SCIENTISTS Windows Data Flink Data Featur Featur Featur Featur Jython Adapter e e e e Featur Featur Featur Featur e e e e .py .py .py .py .py .py .py .py

24. GITFLOW AND JYTHON IMPROVE TRACEABILITY Featur Maven Develop Denied e JAR Import Build v1.0.42 Failed Junit Junit Tests Flink Tests Merge Job JAR Pull Build Request Commit

25.Using Flink operators to build our application workflow 2 3 4 1

26.FEATURES EXIST TO FEED MODELS Model Model Score Feature Feature H20 Tensor Flow Seldon (whatever)

27.

28. BREAKING UP THE MONOLITH • Problem: Back Pressure leading to Delayed Transactions • Solution: Break up the monolith Flink App into small Queryable State Apps

29.CHIPMUNKS