利用Intel Analytics Zoo在大数据平台上构建深度学习应用

1 . 区 cn 社 g. 能 cu 智 Use Intel Analytics Zoo to build deep ai 工 w. G人 learning applications on big data platforms ww CU AI Kai Huang 

2 . Outline 区 cn 社 g. 能 cu 智 • Motivation and introduction ai 工 w. G人 • Highlighted features and examples ww CU AI • Read-world applications • Conclusion 

3 . Trends & Motivations 区 • Data scale is driving deep learning process. cn 社 • Huge gap exists between deep learning and big data communities. g. 能 cu 智 ai 工 w. G人 ww CU AI The Chasm Deep learning experts Average users (big data users, data scientists, analysts, etc.) “Machine Learning Yearning”, Andrew Ng, 2016 

4 . Trends & Motivations • Real-world ML/DL systems are complex big data analytics pipelines. 区 cn 社 • Apache Hadoop and Spack drives big data solutions. g. 能 cu 智 ai 工 w. G人 ww CU AI Hidden Technical Debt in Machine Learning Systems”, Google, NIPS 2015 paper 

5 . BigDL Bringing Deep Learning To Big Data Platforms 区 cn 社 • Distributed deep learning framework for Apache Spark* g. 能 • Make deep learning more accessible to big data users cu 智 and data scientists: ai 工 • Write deep learning applications as standard Spark programs. DataFrame w. G人 • Run on existing Spark/Hadoop clusters (no changes needed). ML Pipeline ww CU • Feature parity with popular deep learning frameworks: SQL SparkR Streaming AI • E.g., Caffe, Torch, TensorFlow, etc. MLlib GraphX • High performance (on CPU): Spark Core • Powered by Intel MKL and multi-threaded programming. https://github.com/intel-analytics/BigDL • Efficient scale-out: https://bigdl-project.github.io/ • Leveraging Spark for distributed training & inference. 

6 . Analytics Zoo Unified Analytics + AI Platform for Big Data 区 cn 社 Distributed TensorFlow, Keras and BigDL on Spark g. 能 • Distributed TensorFlow and Keras on Spark. cu 智 High-Level Pipeline APIs • Native support for transfer learning, Spark DataFrame and ML Pipelines. ai 工 • Model serving API for inference pipelines. w. G人 Feature Engineering Feature transformations for image, 3D image, text, time series, speech, etc. ww CU Built-in Deep Learning Models AI Image classification, object detection, text classification, text matching, sequence-to-sequence, recommendations, anomaly detection, etc. Anomaly detection, sentiment analysis, fraud detection, image generation, Reference Use Cases chatbot, etc. Backends Spark, TensorFlow, Keras, BigDL, MKL-DNN, etc. https://github.com/intel-analytics/analytics-zoo/ https://analytics-zoo.github.io/ 

7 . Keras-Style API Use Keras-Style API to create an Analytics Zoo model and train, evaluate or tune it in a distributed fashion. 区 cn 社 from zoo.pipeline.api.keras.models import Sequential from zoo.pipeline.api.keras.layers import * g. 能 cu 智 model = Sequential() model.add(Reshape((1, 28, 28), input_shape=(28, 28, 1))) ai 工 model.add(Convolution2D(6, 5, 5, activation="tanh", name="conv1_5x5")) w. G人 model.add(MaxPooling2D()) model.add(Convolution2D(12, 5, 5, activation="tanh", name="conv2_5x5")) ww CU model.add(MaxPooling2D()) model.add(Flatten()) AI model.add(Dense(100, activation="tanh", name="fc1")) model.add(Dense(10, activation="softmax", name="fc2")) model.compile("sparse_categorical_crossentropy", "adadelta", ["accuracy"]) model.set_tensorboard(log_dir, app_name) model.set_checkpoint(path) model.fit(x, y=None, batch, epochs, validation_data) model.predict(x, batch) model.predict_classes(x, batch) model.evaluate(x, y=None, batch) 

8 . Auto-grad API 区 Autograd API provides automatic differentiation for math operations to easily define custom layers or losses. cn 社 g. 能 cu 智 import zoo.pipeline.api.autograd as A ai 工 log = A.log(in_node + 1.0) w. G人 dot = A.batch_dot(embed1, embed2, axes=[2, 2]) ww CU AI from zoo.pipeline.api.autograd import * def mean_absolute_error(y_true, y_pred): result = mean(abs(y_true - y_pred), axis=1) return result 

9 . Transfer Learning API Use transfer learning APIs to easily customize pre-trained models for feature extraction or fine-tuning: 区 cn 社 from zoo.pipeline.api.net import * g. 能 from zoo.pipeline.api.keras.layers import Dense, Input, Flatten from zoo.pipeline.api.keras.models import Model cu 智 ai 工 # Load a pre-trained inception model full_model = Net.load_caffe(def_path, model_path) w. G人 # Remove the last few layers ww CU model = full_model.new_graph(outputs=["pool5/drop_7x7_s1"]).to_keras()) AI # Freeze the first few layers model.freeze_up_to(["pool4/3x3_s2"]) # Append a few layers input = Input(shape=(3, 224, 224)) inception= model.to_keras()(input) flatten = Flatten()(inception) logits = Dense(2)(flatten) new_model = Model(input, logits) 

10 . Working with Image 1. Read images into local or distributed ImageSet 区 cn 社 from zoo.common.nncontext import init_nncontext g. 能 from zoo.feature.image import * cu 智 sc = init_nncontext() ai 工 local_image_set = ImageSet.read(image_path) distributed_image_set = ImageSet.read(image_path, sc, 2) w. G人 2. Image augmentations using built-in ImageProcessing operations ww CU transformer = ChainedPreprocessing([ImageBytesToMat(), AI ImageColorJitter(), ImageExpand(max_expand_ratio=2.0), ImageResize(300, 300, -1), ImageHFlip()]) transformed_local_image_set = transformer(local_image_set) transformed_distributed_image_set = transformer(distributed_image_set) Image Augmentations Using Built-in Image Transformations (w/ OpenCV on Spark) 

11 . Working with Text 区 1. Read text into local or distributed TextSet cn 社 from zoo.common.nncontext import init_nncontext g. 能 from zoo.feature.text import * cu 智 sc = init_nncontext() ai 工 local_text_set = TextSet.read(text_path) w. G人 distributed_text_set = TextSet.read(text_path, sc, 2) ww CU 2. Build text transformation pipeline using built-in operations AI transformed_text_set = text_set.tokenize() \ .normalize() \ .word2idx() \ .shape_sequence(len) 

12 . nnframes Native DL support in Spark DataFrames and ML Pipelines 区 cn 社 1. Initialize NNContext and load images into DataFrames using NNImageReader g. 能 from zoo.common.nncontext import * cu 智 from zoo.pipeline.nnframes import * ai 工 sc = init_nncontext() imageDF = NNImageReader.readImages(image_path, sc) w. G人 2. Process loaded data using DataFrame transformations ww CU getName = udf(lambda row: ...) AI df = imageDF.withColumn("name", getName(col("image"))) 3. Processing image using built-in feature engineering operations from zoo.feature.image import * transformer = ChainedPreprocessing( [RowToImageFeature(), ImageChannelNormalize(123.0, 117.0, 104.0), ImageMatToTensor(), ImageFeatureToTensor()]) 

13 . nnframes Native DL support in Spark DataFrames and ML Pipelines 区 cn 社 4. Define model using Keras-style API g. 能 from zoo.pipeline.api.keras.layers import * cu 智 from zoo.pipeline.api.keras.models import * ai 工 model = Sequential() .add(Convolution2D(32, 3, 3, activation='relu', input_shape=(1, 28, 28))) \ w. G人 .add(MaxPooling2D(pool_size=(2, 2))) \ .add(Flatten()).add(Dense(10, activation='softmax'))) ww CU AI 5. Train model using Spark ML Pipelines estimater = NNEstimater(model, CrossEntropyCriterion(), transformer) \ .setLearningRate(0.003).setBatchSize(40).setMaxEpoch(1) \ .setFeaturesCol("image").setCachingSample(False) nnModel = estimater.fit(df) 

14 . Distributed TensorFlow on Spark in Analytics Zoo 区 1. Data wrangling and analysis using PySpark cn 社 g. 能 from zoo.common.nncontext import init_nncontext cu 智 from zoo.pipeline.api.net import TFDataset ai 工 sc = init_nncontext() w. G人 # Each record in the train_rdd consists of a list of NumPy ndrrays train_rdd = sc.parallelize(file_list) ww CU .map(lambda x: read_image_and_label(x)) AI .map(lambda image_label: decode_to_ndarrays(image_label)) # TFDataset represents a distributed set of elements, # in which each element contains one or more TensorFlow Tensor objects. dataset = TFDataset.from_rdd(train_rdd, names=["features", "labels"], shapes=[[28, 28, 1], [1]], types=[tf.float32, tf.int32], batch_size=…) 

15 . Distributed TensorFlow on Spark in Analytics Zoo 区 2. Deep learning model development using TensorFlow cn 社 g. 能 import tensorflow as tf cu 智 slim = tf.contrib.slim ai 工 w. G人 images, labels = dataset.tensors labels = tf.squeeze(labels) with slim.arg_scope(lenet.lenet_arg_scope()): ww CU logits, end_points = lenet.lenet(images, num_classes=10, is_training=True) AI loss = tf.reduce_mean(tf.losses.sparse_softmax_cross_entropy(logits=logits, labels=labels)) 

16 . Distributed TensorFlow on Spark in Analytics Zoo 区 3. Distributed training on Spark and BigDL cn 社 g. 能 from bigdl.optim.optimizer import MaxIteration, Adam, MaxEpoch, TrainSummary cu 智 from zoo.pipeline.api.net import TFOptimizer ai 工 optimizer = TFOptimizer.from_loss(loss, Adam(1e-3)) w. G人 optimizer.set_train_summary(TrainSummary("/tmp/az_lenet", "lenet")) optimizer.optimize(end_trigger=MaxEpoch(5)) ww CU 4. For Keras users AI optimizer = TFOptimizer.from_keras(keras_model, dataset) optimizer.optimize(end_trigger=MaxEpoch(5)) predictor = TFPredictor.from_keras(model, dataset) predictions_rdd = predictor.predict() 

17 . Models Interoperability Support 区 Load existing TensorFlow, Keras, Caffe, Torch, ONNX models: cn 社 • Useful for inference and model fine-tuning. g. 能 • Allows for transition from single-node for distributed application deployment. cu 智 • Allows for model sharing between data scientists and production engineers. ai 工 w. G人 from zoo.pipeline.api.net import Net ww CU AI Net.load_tf(path, inputs=None, outputs=None, byte_order=“little_endian”, bin_file=None) Net.load_keras(hdf5_path, json_path=None, by_name=False) Net.load_caffe(def_path, model_path) Net.load_torch(path) 

18 . POJO-Style Model Serving API import com.intel.analytics.zoo.pipeline.inference.AbstractInferenceModel; 区 cn 社 public class TextClassification extends AbstractInferenceModel { public RankerInferenceModel(int concurrentNum) { g. 能 super(concurrentNum); cu 智 } ... ai 工 } w. G人 public class ServingExample { ww CU public static void main(String[] args) throws IOException { TextClassification model = new TextClassification(); AI model.load(modelPath, weightPath); texts = … List<JTensor> inputs = preprocess(texts); for (JTensor input : inputs) { List<Float> result = model.predict(input.getData(), input.getShape()); ... } } 

19 . Built-in Deep Learning Models 区 • Object detection (SSD, Faster-RCNN, etc.) cn 社 g. 能 • Image classification (VGG, Inception, ResNet, MobileNet, etc.) cu 智 ai 工 • Text classification (Using CNN, LSTM, etc.) w. G人 • Text matching (For either ranking or classification.) ww CU AI • Sequence-to-sequence (Encoder-decoder structure using RNN.) • Recommendation (Neural Collaborative Filtering, Wide and Deep Learning, etc.) • Anomaly detection (Unsupervised time series anomaly detection using LSTM.) 

20 . Object Detection API 区 1. Load pretrained model in Detection Model Zoo cn 社 from zoo.common.nncontext import * g. 能 from zoo.models.image.objectdetection import * cu 智 sc = init_nncontext() model = ObjectDetector.load_model(model_path) ai 工 2. Off-the-shell inference using the loaded model w. G人 image_set = ImageSet.read(img_path, sc) ww CU output = model.predict_image_set(image_set) AI 3. Visualize the results using utility methods config = model.get_config() visualizer = Visualizer(config.label_map(), encoding="jpg") visualized = visualizer(output).get_image(to_chw=False).collect() Off-the-shell Inference Using Analytics Zoo Object Detection API https://github.com/intel-analytics/analytics-zoo/tree/master/pyzoo/zoo/examples/objectdetection 

21 . Reference Use Cases • Anomaly Detection 区 • Using LSTM network to detect anomalies in time series data. cn 社 g. 能 • Fraud Detection cu 智 • Using feed-forward neural network to detect frauds in credit card transaction data. ai 工 • Recommendation w. G人 • Use Analytics Zoo Recommendation API on data with explicit feedback. ww CU • Sentiment Analysis AI • Using neural network models (e.g. CNN, GRU, Bi-LSTM) to handle user feedbacks. • Variational Autoencoder • Using VAE to generate faces and digital numbers. • Web Services • Using Analytics Zoo model serving APIs for model inference in web servers. https://github.com/intel-analytics/analytics-zoo/tree/master/apps 

22 .Object Detection and Image Feature Extraction at JD.com 区 cn 社 g. 能 cu 智 ai 工 w. G人 ww CU AI 

23 . Similar Image Search 区 cn 社 Query g. 能 cu 智 Search Result ai 工 w. G人 ww CU Query Search Result AI --- --- Source: “Bringing deep learning into big data analytics using BigDL”, Xianyan Jia and Zhenhua Wang, Strata Data Conference Singapore 2017 

24 . Challenges of Productionizing Large-Scale Deep Learning Solutions 区 cn 社 g. 能 • Very complex and error-prone in managing large-scale distributed systems cu 智 • E.g., resource management and allocation, data partitioning, task balance, fault tolerance, ai 工 model deployment, etc. w. G人 • Low end-to-end performance in GPU solutions ww CU • E.g., reading images out from HBase takes about half of the total time AI • Very inefficient to develop the end-to-end processing pipeline • E.g., image pre-processing on HBase can be very complex 

25 .Production Deployment with Analytics Zoo for Spark and BigDL 区 cn 社 g. 能 cu 智 ai 工 w. G人 ww CU AI • Reuse existing Hadoop/Spark clusters for deep learning with no changes (image search, IP protection, etc.) • Efficiently scale out on Spark with superior performance (3.83x speed-up vs. GPU severs) as benchmarked by JD http://mp.weixin.qq.com/s/xUCkzbHK4K06-v5qUsaNQQ https://software.intel.com/en-us/articles/building-large-scale-image-feature-extraction-with-bigdl-at-jdcom 

26 . NLP Based Customer Service Chatbot for Microsoft Azure 区 cn 社 g. 能 cu 智 ai 工 w. G人 ww CU AI https://software.intel.com/en-us/articles/use-analytics-zoo-to-inject-ai-into-customer-service- platforms-on-microsoft-azure-part-1 

27 . Real-World Applications 区 Industrial Inspection Platform in Midea* and KUKA* cn 社 https://software.intel.com/en-us/articles/industrial-inspection-platform-in-midea-and-kuka-using- g. 能 distributed-tensorflow-on-analytics cu 智 ai 工 Image Similarity Based House Recommendation for MLSlistings* w. G人 https://software.intel.com/en-us/articles/using-bigdl-to-build-image-similarity-based-house- recommendations ww CU Building users-items propensity models for Mastercard* AI Service AI https://software.intel.com/en-us/articles/deep-learning-with-analytic-zoo-optimizes-mastercard- recommender-ai-service Job Recommendations in Jobs2Career* https://software.intel.com/en-us/articles/talroo-uses-analytics-zoo-and-aws-to-leverage-deep- learning-for-job-recommendations 

28 . 区 cn 社 g. 能 cu 智 ai 工 w. G人 ww CU AI Unified Analytics + AI Platform Distributed TensorFlow, Keras and BigDL on Apache Spark https://github.com/intel-analytics/analytics-zoo 

29 . AI ww CU w. G人 ai 工 cu 智 g. 能 cn 社 区