《基于自动化机器学习的图神经网络架构设计》赵欢

发布于

828

人观看

赵欢
香港科技大学博士 / 第四范式科学技术部资深研究员
负责自动化图神经网络（AutoGraph）相关方向的研究与落地工作，研究方向涵盖推荐系统，图表征学习，图神经网络，在人工智能顶会上发表学术论文20篇，申请国家发明专利4项。曾就职于淘宝搜索推荐事业部，并于2017年双十一期间推动团队在淘宝落地业界首个大规模基于graph embedding的召回算法，相关工作也发表在国际顶级数据挖掘会议KDD 2018上，现在已经成为各大互联网公司召回算法的参考基线。

本次活动，赵老师将为大家分享《AutoML in Graph Neural Networks：基于自动化机器学习的图神经网络架构设计》，近些年，图神经网络（GNN）成为工业界和学术界的重要方向。但由于应用场景的差异性，不同架构的GNN效果变化较大，因此如何设计任务自适应的图神经网络架构，是一个非常重要的研究课题，围绕AutoGraph，本次主要分享以下内容：
GNN的典型应用场景和时下最主流的基于邻居聚合的GNN
围绕节点分类和图分类的任务，基于NAS自动设计GNN架构的应用分享
AutoGraph在第四范式的应用，重点在表数据预测和生物医学场景的应用

展开查看详情

1 .Automated Graph Neural Network Research and applications in 4Paradigm 赵欢资深研究员，第四范式 2022.1.15

2 .国际领先的人工智能技术与服务提供商 2

3 . 个人主页个人介绍 • 2019.7-至今第四范式资深研究员 • 负责AutoGraph的研究与落地 • 发表学术论文22篇，12 篇为第一作者，申请国家发明专利 4 项 • 担任 TKDE, TOIS 和 IJCAI, AAAI, ICML, NeurIPS，ICLR等国际顶级人工智能数据挖掘期刊与会议的审稿人 • 2017.10-2019.7 淘宝搜索推荐事部实习/工作 • 研究并落地基于 Graph embedding的召回算法 • 负责图计算平台研发和手淘推荐算法 • 2019.1 香港科技大学获得计算机系博士学位 • 基于图学习的推荐算法研究

4 .Outline • Background • Graph Neural Network (GNN) • Neural Architecture Search (NAS) • Graph Neural architecture search • Reinforcement learning based search • Differentiable architecture search • Real-world applications in 4Paradigm • Tabular data prediction • Double-strand DNA breaks (DSBs) prediction • Conclusion

5 .Graph-based applications Search Engine Recommendation Social Network Image Credit: Jure Stanford CS224 Image Credit: Shreyansh@Medium Image Credit: Wikipedia biomedicine Traffic prediction Chip Design Image Credit: Jure Stanford CS224 Image Credit: Google Blog Image Credit: Jure Stanford CS224

6 .Graph is an universal language for describing and analyzing entities with relations/interactions. Prof. Jure Leskovec in Stanford CS224 6

7 .Basic definitions of Graph • Node set • Edge set • Adjacency Matrix A • Neighbor set • Feature Matrix Image Credit: Ma et al. @KDD 2020

8 .Graph Neural Networks (GNN) • GNNs have been new state-of-the-art (SOTA) for graph-based tasks. Image Credit: Tecent AI Lab Image Credit: Xavier Bresson

9 .Illustrative architecture of GNN 9 Image Credit: Jure Stanford CS224

10 .GNN 1 • Message passing framework 2 5 • Node embedding updated by neighbors Self contained • K-layer GNN access K-hop neighbors Neighborhood 3 4 • "Neighborhood aggregation" 2 AGG • Variants of GNN 1 3 • GCN: normalized sum aggregator AGG AGG • GraphSAGE: MEAN, MAX, SUM, LSTM • GAT: Attention aggregator 2 5 2 4 • GIN: Multi-Layer Perceptrons (MLP) AGG AGG AGG AGG 10 1 3 1 4 1 3 3 5

11 .Graph Neural Network Revisited 1 2 5 • Message passing framework • Instantiate existing GNN models. [Zhao et al. 2021] 3 4 2 AGG 1 3 AGG AGG 2 5 2 4 AGG AGG AGG AGG 11 Zhao et al. Search to aggregate neighborhood for graph neural networks. ICDE 2021 1 3 1 4 1 3 3 5

12 .Design Space for GNN [You et al. 2020] • The performance of GNN models vary • Combinations of 12 key design dimensions. • 315,000 instances 12 You et al. Design Space for Graph Neural Networks. NeurIPS 2020

13 .Neural architecture search for graph neural network 13

14 .Neural Architecture Search • Explore the possibility of automatically searching for unexplored architectures beyond human-designed ones. • A bi-level optimization problem • Derive a search space based on domain knowledge • Search objective is usually validation performance • Training objective usually comes from classical learning models • Search constraint is usually resource budgets min 𝑀 𝐹 𝑤 ∗ ; 𝜆 , 𝐷%&' Search Objective Search Space !∈𝒮 min 𝐿 𝐹 𝑤; 𝜆 , 𝐷)*& Training Objective ( s. t. 𝐺 𝜆 ≤𝐶 Search Constraints 14 Zoph et al. Neural Architecture Search with Reinforcement Learning. ICLR 2017

15 .Two representative approaches NASNet [Zoph et al. 2017] • Stand alone (Reinforcement learning) • An RNN controller to sample a candidate architecture • Train till convergency • Update the RNN controller by the validation accuracy DARTS [Liu et al. 2017] • One-shot (Differentiable architecture search) • Train a supernet including all candidate architectures • Derive a chidnet from the supernet as the search architecture Zoph et al. Neural Architecture Search with Reinforcement Learning. ICLR 2017 15 Liu et al. Darts: Differentiable architecture search. ICLR 2019

16 . SNAG [Zhao et al. 2020] NAS for GNN • Two representative approaches • Search for task-specific GNN architectures. • Node-level • SNAG [Zhao et al. 2020] SANE [Zhao et al. 2021] • Reinforcement learning based method • SANE [Zhao et al. 2021] • Differentiable architecture search Zhao et al. simplifying architecture search for graph neural network. CIKM-CSSA 2020 16 Zhao et al. Search to aggregate neighborhood for graph neural networks. ICDE 2021

17 .SNAG • Overview framework 17 Zhao et al. Simplifying Architecture Search for Graph Neural Network. CIKM-CSSA 2020

18 .Search Space • Search space • Node aggregators + Layer aggregators 18 Zhao et al. Simplifying Architecture Search for Graph Neural Network. CIKM-CSSA 2020

19 .Search method • Search method: ENAS/GraphNAS • Reward • Policy gradient • MC approximation 19 Zhao et al. Simplifying Architecture Search for Graph Neural Network. CIKM-CSSA 2020

20 .Experiments • Settings • Four benchmark datasets • Baselines • GNNs • NAS methods: Random, Bayesian, GraphNAS/GraphNAS-WS 20 Zhao et al. Simplifying Architecture Search for Graph Neural Network. CIKM-CSSA 2020

21 .Experiments • Results • transductive 21 Zhao et al. Simplifying Architecture Search for Graph Neural Network. CIKM-CSSA 2020

22 .Experiments • Results • inductive 22 Zhao et al. Simplifying Architecture Search for Graph Neural Network. CIKM-CSSA 2020

23 .Experiments • Results • Speedup in searching • Influences of layer aggregators 23 Zhao et al. Simplifying Architecture Search for Graph Neural Network. CIKM-CSSA 2020

24 .Quick Summary • A first graph neural network architecture search by RL • Effective but expensive due to "trial-and-error" manner • Stand alone • More efficient search algorithm is needed • One-shot 24 Gao et al. GraphNAS: Graph Neural Architecture Search with Reinforcement Learning. IJCAI 2020

25 .SANE [Zhao et al. 2021] • Search to Aggregate Neighborhood (SANE) for graph neural networks • Differentiable architecture search • A research work from our group. • A more compact search space only search for node and layer aggregation functions • The expressive ability of GNNs mainly rely on the aggregation function[Hu et al. 2019] • The intermediate layers can further improve the expressive ability [Xu et al. 2018] • SOTA performance Zhao et al. Search to aggregate neighborhood for graph neural networks. ICDE 2021 Hu et al. How Powerful are Graph Neural Networks? ICLR 2019 25 Xu et al. Representation Learning on Graphs with Jumping Knowledge Networks. ICML 2018

26 .Search Space • Message passing framework • Node aggregator • Layer aggregator • Comparing to existing GNNs 26 Zhao et al. Search to aggregate neighborhood for graph neural networks. ICDE 2021

27 .Search Space • More explanations on node aggregator 27 Zhao et al. Search to aggregate neighborhood for graph neural networks. ICDE 2021

28 .Differentiable search algorithm • Supernet (DAG) • Continuous relaxation • Mixed OPs • Computation process 28 Zhao et al. Search to aggregate neighborhood for graph neural networks. ICDE 2021

29 .Differentiable search algorithm • Search{𝛼! , 𝛼" , 𝛼# } • Derive architecture • Choose the OP with the largest weight. 29 Zhao et al. Search to aggregate neighborhood for graph neural networks. ICDE 2021

0点赞

0收藏

3下载