Applications of Deep Learning in Telematics

Smart phones are equipped with many sensors which provide detailed and continuous information of the device’s location and movement. The use of such signals for vehicle movement inference presents many challenges due to signal noise, unknown phone orientation, varying device sensor quality and so on. Signal processing and feature engineering are generally difficult and require deep domain knowledge and manual pattern recognition. We discuss how deep learning can be leveraged in this context for automatic signal processing and feature engineering. We present several applications of deep learning in vehicle telematics as well as the deep learning architecture designed for learning sensor embeddings for vehicle movement events. One challenge we face is that model training requires huge volumes of sensor data, which must be processed efficiently. We present a solution using Spark for model development and batch deployment.

1.Applications of Deep Learning in Telematics Wayne Zhang, Uber #UnifiedAnalytics #SparkAISummit

2.Stand for Safety We want Uber to be the safest transportation platform on the planet. Safety should be our number-one priority. We have to, as a company, stand for safety.” Dara Khosrowshahi (2018)

3.Example Driving Safety Products Driving Hours Limit Speed Limit Alert Ride Check

4. Telematics - Wide availability - Lower quality Source: Smartphone-based Vehicle Telematics - A Ten-Year Anniversary - Cheap - Measure phone motion - Short upgrade cycle 4

5.Sensor Data (Driver Device) ● GPS ○ Absolute location, velocity and time ○ Low frequency (~0.5Hz) ● IMU ○ Relative motion of phone ○ Accelerometer: 3D linear acceleration ○ Gyroscope: 3D angular velocity ○ High frequency (~25Hz)

6.Motivation ● High-frequency signals ○ Intricate and diverse patterns ○ Dynamic over time Driving Handling Driving Walking On Train Phone


8.Sequence Classification Classify whole sequence to certain events: ● Crash ● Driving events (brake, turn, speeding) ● Phone handling ● Rider complaint ● ...

9.Sequence-Sequence Prediction Input sequence Output sequence (Phone sensor data) Vehicle sensor Align to other sensor phone => vehicle Turn event (binary encoding) Pinpoint telematics events (turn, activity)

10.Design Choice ● High-frequency data result in huge # time steps ○ Pre-filtering: identify specific time window of interest ○ Window segmentation: divide input sequence into small windows Pre-filtering Window Segmentation window0 window2 windowT window1

11.Window Prediction Embedding LSTM [optional] Raw data

12.Feature Extraction Raw data window 1 window 2 window t window T Sample Summary 1-D CNN - Time domain stats (min, max, mean, sd) - Frequency domain feature (FFT) New Feature Vector LSTM LSTM

13.Data Augmentation ● Sensor readings depend on phone orientation ● Create augmented data by artificially rotating phone ○ New sensor readings ○ Label stays the same

14. Model Dev Pipeline Data Label/Feature Model Score Non-DL - Sensor (Driver) - Telematics - SparkML Transformer - Score and classify - Map - Trip - XgBoost - Trip - xM trips in training - Other - xM trips in validation - Saved model pipeline Data Label/Feature Model Score - Sensor (Driver) - Event definition - Multi-layer LSTM - Sensor embedding DL - Feature - xM trips in training - Saved protocol buffer

15.Horovod ● Open source library developed at Uber ● Distributed training for TensorFlow, Keras & PyTorch ● Uses bandwidth-optimal communication protocols & makes use of advanced networking ● Seamlessly installs via pip install horovod

16.Petastorm ● Open source library developed at Uber ATG Apache Parquet as a dataframe with ● Enables deep learning directly from Parquet tensors ● Supports Tensorflow, PyTorch, and PySpark Hedgehog Fog Horse nd-arrays, scalars (e.g. images, Apache Parquet lidar point store clouds)

17.Performance DL model DL model

18.Proprietary and confidential © 2019 Uber Technologies, Inc. All rights reserved. No part of this document may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval systems, without permission in writing from Uber. This document is intended only for the use of the individual or entity to whom it is addressed and contains information that is privileged, confidential or otherwise exempt from disclosure under applicable law. All recipients of this document are notified that the information contained herein includes proprietary and confidential information of Uber, and recipient may not make use of, disseminate, or in any way disclose this document or any of the enclosed information to any person other than employees of addressee to the extent necessary for consultations with authorized personnel of Uber.

19.Thank You! Wayne Zhang