Seamless End-to-End Production Machine Learning with Seldon and MLFlow

Deploying and managing machine learning models at scale introduces new complexities. Fortunately, there are tools that simplify this process. In this talk we walk you through an end-to-end hands on example showing how you can go from research to production without much complexity by leveraging the Seldon Core and MLflow frameworks. We will train a set of ML models, and we will showcase a simple way to deploy them to a kubernetes cluster through sophisticated deployment methods, including canary deployments, shadow deployments and we’ll touch upon richer ML graphs such as explainer deployments.


1.End to end Machine Learning in kubernetes with Seldon and MLFlow Adaptive. Intelligent. Agile. Intuitive. Alive. Inspiring.

2.About me Adrian Gonzalez-Martin @kaseyo23 Machine Learning Engineer

3.About Seldon We are hiring!

4.Outline Example use case. Training models with MLFlow. Deploying models with Seldon. Demo!

5.Example use case

6.Wine e-commerce 6.5 2.3 ?

7.Wine quality dataset Regression problem where goal is to predict wine quality. Fixed Volatile Citric ... Sulphates Alcohol Quality Acidity Acidity Acid 7 0.27 0.36 0.45 8.8 6 6.3 0.3 0.34 0.49 9.5 7 8.1 0.28 0.4 0.44 10.1 1 7.2 0.23 0.32 0.4 9.9 2 7.2 0.23 0.32 0.4 9.9 5 ...

8.Wine quality dataset Train two versions of ElasticNet. Model A A/B Test Deploy both and do A/B test. Router Model B

9.Wine quality dataset ElasticNet Linear regression with L1 and L2 regularisers. Two hyperparameters:

10.Training models with MLFlow

11.Training models at scale is hard Keep track of experiments (e.g. versioning). Optimise hyperparameters at scale. Share models with the rest of the team. Plain notebooks may not be the best solution!! secret-project ├── eda.ipynb ├── requirements.txt ├── requirements-tf2.txt └── experiments ├── attention-embedd.ipynb ├── attn.ipynb ├── hmm.ipynb ├── LSTM-2.ipynb └── lstm.ipynb

12.MLFlow can help! MLtracking API to track results and hyperparameters. These can be stored remotely. MLproject Define environment, parameters and model’s interface. Hyperparameter optimisation can be run Databricks or Kubernetes. MLmodel Snapshot/version of the model.

13.MLFlow can help! MLtracking Track both hyperparameters: Track metrics: MLproject Pip dependencies and entry point template. MLmodel Snapshot of regression coefficients and hyperparameters:

14.MLproject file and training MLproject $ mlflow run ./training -P alpha=0.5 name: mlflow-talk $ mlflow run ./training -P alpha=1.0 conda_env: conda.yaml entry_points: main: parameters: alpha: float l1_ratio: {type: float, default: 0.1} command: "python {alpha} {l1_ratio}"

15.MLmodel snapshot MLmodel artifact_path: model flavors: python_function: data: model.pkl env: conda.yaml loader_module: mlflow.sklearn python_version: 3.6.9 sklearn: pickled_model: model.pkl serialization_format: cloudpickle sklearn_version: 0.19.1 run_id: 5a6be5a1ef844783a50a6577745dbdc3 utc_time_created: '2019-10-02 14:21:15.783806'

16.MLflow UI + MLtrack

17.Deploying models with Seldon

18.Deploying models is hard Serving infrastructure. Seldon can help with these... Scaling up (and down!). Monitoring models. Machine Machine Updating models. Data Verification Resource Resource Management Management Data Collection Monitoring Monitoring Serving Serving ML Configuration Analysis Tools Analysis Tools Infrastructure Infrastructure Code Feature Extraction Process Process Management Management Tools Tools [1] Hidden Technical Debt in Machine Learning Systems, NIPS 2015

19.Inference graph Seldon/3rd party MLFlow model component Inference graph (k8s pod) Model A API Explainer A/B Test (REST, gRPC) Router Model B

20.Cloud Native Built on top of Kubernetes Cloud Native APIs. All major cloud providers are supported. On-prem providers such as OpenShift are also supported. On-prem

21. Seldon Architecture Kubernetes clusters running Seldon Core Data scientists, engineers and Deployment Controller Kubernetes Operator (kubectl, CI/CD, Seldon Deploy) managers API Business Ingress Inference Applications REST API or gRPC (Ambassador, Istio) Components MLFlow models Seldon docker Public docker Client docker (S3, GCS, etc.) registry registry registry

22.Inference graph apiVersion: kind: SeldonDeployment metadata: name: wines-classifier spec: name: wines-classifier predictors: - graph: children: [] implementation: MLFLOW_SERVER modelUri: gs://seldon-models/mlflow/model-a name: wines-classifier name: model-a replicas: 1 traffic: 50 - graph: children: [] implementation: MLFLOW_SERVER modelUri: gs://seldon-models/mlflow/model-b name: wines-classifier name: model-b replicas: 1 traffic: 50

23.Feedback Kubernetes clusters running Seldon Core Prediction Inference graph Feedback (reward signal)

24.Feedback We can build a rough reward signal using the squared error.

25.Seldon Analytics


27.Thanks! We are hiring! @kaseyo23