Seamless End-to-End Production Machine Learning with Seldon and MLFlow

1 .End to end Machine Learning in kubernetes with Seldon and MLFlow Adaptive. Intelligent. Agile. Intuitive. Alive. Inspiring. 

2 .About me agm@seldon.io Adrian Gonzalez-Martin @kaseyo23 Machine Learning Engineer github.com/adriangonz 

3 .About Seldon We are hiring! seldon.io/careers/ 

4 .Outline Example use case. Training models with MLFlow. Deploying models with Seldon. Demo! 

5 .Example use case 

6 .Wine e-commerce https://wine.seldon.io/... 6.5 2.3 ? 

7 .Wine quality dataset Regression problem where goal is to predict wine quality. Fixed Volatile Citric ... Sulphates Alcohol Quality Acidity Acidity Acid 7 0.27 0.36 0.45 8.8 6 6.3 0.3 0.34 0.49 9.5 7 8.1 0.28 0.4 0.44 10.1 1 7.2 0.23 0.32 0.4 9.9 2 7.2 0.23 0.32 0.4 9.9 5 ... 

8 .Wine quality dataset Train two versions of ElasticNet. Model A A/B Test Deploy both and do A/B test. Router Model B 

9 .Wine quality dataset ElasticNet Linear regression with L1 and L2 regularisers. Two hyperparameters: 

10 .Training models with MLFlow 

11 .Training models at scale is hard Keep track of experiments (e.g. versioning). Optimise hyperparameters at scale. Share models with the rest of the team. Plain notebooks may not be the best solution!! secret-project ├── eda.ipynb ├── requirements.txt ├── requirements-tf2.txt └── experiments ├── attention-embedd.ipynb ├── attn.ipynb ├── hmm.ipynb ├── LSTM-2.ipynb └── lstm.ipynb 

12 .MLFlow can help! MLtracking API to track results and hyperparameters. These can be stored remotely. MLproject Define environment, parameters and model’s interface. Hyperparameter optimisation can be run Databricks or Kubernetes. MLmodel Snapshot/version of the model. 

13 .MLFlow can help! MLtracking Track both hyperparameters: Track metrics: MLproject Pip dependencies and entry point template. MLmodel Snapshot of regression coefficients and hyperparameters: 

14 .MLproject file and training MLproject $ mlflow run ./training -P alpha=0.5 name: mlflow-talk $ mlflow run ./training -P alpha=1.0 conda_env: conda.yaml entry_points: main: parameters: alpha: float l1_ratio: {type: float, default: 0.1} command: "python train.py {alpha} {l1_ratio}" 

15 .MLmodel snapshot MLmodel artifact_path: model flavors: python_function: data: model.pkl env: conda.yaml loader_module: mlflow.sklearn python_version: 3.6.9 sklearn: pickled_model: model.pkl serialization_format: cloudpickle sklearn_version: 0.19.1 run_id: 5a6be5a1ef844783a50a6577745dbdc3 utc_time_created: '2019-10-02 14:21:15.783806' 

16 .MLflow UI + MLtrack 

17 .Deploying models with Seldon 

18 .Deploying models is hard Serving infrastructure. Seldon can help with these... Scaling up (and down!). Monitoring models. Machine Machine Updating models. Data Verification Resource Resource Management Management Data Collection Monitoring Monitoring Serving Serving ML Configuration Analysis Tools Analysis Tools Infrastructure Infrastructure Code Feature Extraction Process Process Management Management Tools Tools [1] Hidden Technical Debt in Machine Learning Systems, NIPS 2015 

19 .Inference graph Seldon/3rd party MLFlow model component Inference graph (k8s pod) Model A API Explainer A/B Test (REST, gRPC) Router Model B 

20 .Cloud Native Built on top of Kubernetes Cloud Native APIs. All major cloud providers are supported. On-prem providers such as OpenShift are also supported. On-prem https://docs.seldon.io/projects/seldon-core/en/latest/examples/notebooks.html 

21 . Seldon Architecture Kubernetes clusters running Seldon Core Data scientists, engineers and Deployment Controller Kubernetes Operator (kubectl, CI/CD, Seldon Deploy) managers API Business Ingress Inference Applications REST API or gRPC (Ambassador, Istio) Components MLFlow models Seldon docker Public docker Client docker (S3, GCS, etc.) registry registry registry 

22 .Inference graph apiVersion: machinelearning.seldon.io/v1alpha2 kind: SeldonDeployment metadata: name: wines-classifier spec: name: wines-classifier predictors: - graph: children: [] implementation: MLFLOW_SERVER modelUri: gs://seldon-models/mlflow/model-a name: wines-classifier name: model-a replicas: 1 traffic: 50 - graph: children: [] implementation: MLFLOW_SERVER modelUri: gs://seldon-models/mlflow/model-b name: wines-classifier name: model-b replicas: 1 traffic: 50 

23 .Feedback Kubernetes clusters running Seldon Core Prediction Inference graph Feedback (reward signal) 

24 .Feedback We can build a rough reward signal using the squared error. 

25 .Seldon Analytics 

26 .Demo! https://github.com/adriangonz/mlflow-talk 

27 .Thanks! agm@seldon.io We are hiring! @kaseyo23 seldon.io/careers/ github.com/adriangonz