《基于自动化机器学习的图神经网络架构设计》赵欢

1 .Automated Graph Neural Network Research and applications in 4Paradigm 赵欢 资深研究员，第四范式 2022.1.15 

2 .国际领先的人工智能技术与服务提供商 2 

3 . 个人主页 个人介绍 • 2019.7-至今 第四范式 资深研究员 • 负责AutoGraph的研究与落地 • 发表学术论文22篇，12 篇为第一作者，申请国家发明专利 4 项 • 担任 TKDE, TOIS 和 IJCAI, AAAI, ICML, NeurIPS，ICLR等国际顶级人工智能数据挖掘期刊与 会议的审稿人 • 2017.10-2019.7 淘宝搜索推荐事部 实习/工作 • 研究并落地基于 Graph embedding的召回算法 • 负责图计算平台研发和手淘推荐算法 • 2019.1 香港科技大学获得计算机系博士学位 • 基于图学习的推荐算法研究 

4 .Outline • Background • Graph Neural Network (GNN) • Neural Architecture Search (NAS) • Graph Neural architecture search • Reinforcement learning based search • Differentiable architecture search • Real-world applications in 4Paradigm • Tabular data prediction • Double-strand DNA breaks (DSBs) prediction • Conclusion 

5 .Graph-based applications Search Engine Recommendation Social Network Image Credit: Jure Stanford CS224 Image Credit: Shreyansh@Medium Image Credit: Wikipedia biomedicine Traffic prediction Chip Design Image Credit: Jure Stanford CS224 Image Credit: Google Blog Image Credit: Jure Stanford CS224 

6 .Graph is an universal language for describing and analyzing entities with relations/interactions. Prof. Jure Leskovec in Stanford CS224 6 

7 .Basic definitions of Graph • Node set • Edge set • Adjacency Matrix A • Neighbor set • Feature Matrix Image Credit: Ma et al. @KDD 2020 

8 .Graph Neural Networks (GNN) • GNNs have been new state-of-the-art (SOTA) for graph-based tasks. Image Credit: Tecent AI Lab Image Credit: Xavier Bresson 

9 .Illustrative architecture of GNN 9 Image Credit: Jure Stanford CS224 

10 .GNN 1 • Message passing framework 2 5 • Node embedding updated by neighbors Self contained • K-layer GNN access K-hop neighbors Neighborhood 3 4 • "Neighborhood aggregation" 2 AGG • Variants of GNN 1 3 • GCN: normalized sum aggregator AGG AGG • GraphSAGE: MEAN, MAX, SUM, LSTM • GAT: Attention aggregator 2 5 2 4 • GIN: Multi-Layer Perceptrons (MLP) AGG AGG AGG AGG 10 1 3 1 4 1 3 3 5 

11 .Graph Neural Network Revisited 1 2 5 • Message passing framework • Instantiate existing GNN models. [Zhao et al. 2021] 3 4 2 AGG 1 3 AGG AGG 2 5 2 4 AGG AGG AGG AGG 11 Zhao et al. Search to aggregate neighborhood for graph neural networks. ICDE 2021 1 3 1 4 1 3 3 5 

12 .Design Space for GNN [You et al. 2020] • The performance of GNN models vary • Combinations of 12 key design dimensions. • 315,000 instances 12 You et al. Design Space for Graph Neural Networks. NeurIPS 2020 

13 .Neural architecture search for graph neural network 13 

14 .Neural Architecture Search • Explore the possibility of automatically searching for unexplored architectures beyond human-designed ones. • A bi-level optimization problem • Derive a search space based on domain knowledge • Search objective is usually validation performance • Training objective usually comes from classical learning models • Search constraint is usually resource budgets min 𝑀 𝐹 𝑤 ∗ ; 𝜆 , 𝐷%&' Search Objective Search Space !∈𝒮 min 𝐿 𝐹 𝑤; 𝜆 , 𝐷)*& Training Objective ( s. t. 𝐺 𝜆 ≤𝐶 Search Constraints 14 Zoph et al. Neural Architecture Search with Reinforcement Learning. ICLR 2017 

15 .Two representative approaches NASNet [Zoph et al. 2017] • Stand alone (Reinforcement learning) • An RNN controller to sample a candidate architecture • Train till convergency • Update the RNN controller by the validation accuracy DARTS [Liu et al. 2017] • One-shot (Differentiable architecture search) • Train a supernet including all candidate architectures • Derive a chidnet from the supernet as the search architecture Zoph et al. Neural Architecture Search with Reinforcement Learning. ICLR 2017 15 Liu et al. Darts: Differentiable architecture search. ICLR 2019 

16 . SNAG [Zhao et al. 2020] NAS for GNN • Two representative approaches • Search for task-specific GNN architectures. • Node-level • SNAG [Zhao et al. 2020] SANE [Zhao et al. 2021] • Reinforcement learning based method • SANE [Zhao et al. 2021] • Differentiable architecture search Zhao et al. simplifying architecture search for graph neural network. CIKM-CSSA 2020 16 Zhao et al. Search to aggregate neighborhood for graph neural networks. ICDE 2021 

17 .SNAG • Overview framework 17 Zhao et al. Simplifying Architecture Search for Graph Neural Network. CIKM-CSSA 2020 

18 .Search Space • Search space • Node aggregators + Layer aggregators 18 Zhao et al. Simplifying Architecture Search for Graph Neural Network. CIKM-CSSA 2020 

19 .Search method • Search method: ENAS/GraphNAS • Reward • Policy gradient • MC approximation 19 Zhao et al. Simplifying Architecture Search for Graph Neural Network. CIKM-CSSA 2020 

20 .Experiments • Settings • Four benchmark datasets • Baselines • GNNs • NAS methods: Random, Bayesian, GraphNAS/GraphNAS-WS 20 Zhao et al. Simplifying Architecture Search for Graph Neural Network. CIKM-CSSA 2020 

21 .Experiments • Results • transductive 21 Zhao et al. Simplifying Architecture Search for Graph Neural Network. CIKM-CSSA 2020 

22 .Experiments • Results • inductive 22 Zhao et al. Simplifying Architecture Search for Graph Neural Network. CIKM-CSSA 2020 

23 .Experiments • Results • Speedup in searching • Influences of layer aggregators 23 Zhao et al. Simplifying Architecture Search for Graph Neural Network. CIKM-CSSA 2020 

24 .Quick Summary • A first graph neural network architecture search by RL • Effective but expensive due to "trial-and-error" manner • Stand alone • More efficient search algorithm is needed • One-shot 24 Gao et al. GraphNAS: Graph Neural Architecture Search with Reinforcement Learning. IJCAI 2020 

25 .SANE [Zhao et al. 2021] • Search to Aggregate Neighborhood (SANE) for graph neural networks • Differentiable architecture search • A research work from our group. • A more compact search space only search for node and layer aggregation functions • The expressive ability of GNNs mainly rely on the aggregation function[Hu et al. 2019] • The intermediate layers can further improve the expressive ability [Xu et al. 2018] • SOTA performance Zhao et al. Search to aggregate neighborhood for graph neural networks. ICDE 2021 Hu et al. How Powerful are Graph Neural Networks? ICLR 2019 25 Xu et al. Representation Learning on Graphs with Jumping Knowledge Networks. ICML 2018 

26 .Search Space • Message passing framework • Node aggregator • Layer aggregator • Comparing to existing GNNs 26 Zhao et al. Search to aggregate neighborhood for graph neural networks. ICDE 2021 

27 .Search Space • More explanations on node aggregator 27 Zhao et al. Search to aggregate neighborhood for graph neural networks. ICDE 2021 

28 .Differentiable search algorithm • Supernet (DAG) • Continuous relaxation • Mixed OPs • Computation process 28 Zhao et al. Search to aggregate neighborhood for graph neural networks. ICDE 2021 

29 .Differentiable search algorithm • Search{𝛼! , 𝛼" , 𝛼# } • Derive architecture • Choose the OP with the largest weight. 29 Zhao et al. Search to aggregate neighborhood for graph neural networks. ICDE 2021