2.谢巍盛-翼支付在Neural Architecture Search领域的探索与实践

1 .

2 .翼支付在Neural Architecture Search的探索 与实践 谢巍盛 

3 .PART 1 AI Development History 

4 .

5 .Source: Medium 

6 .

7 .“The artificial intelligence (AI) market size is projected to surpass around US$ 1,597.1 billion by 2030 and is expanding growth at a CAGR of 38.1% from 2022 to 2030” -Precedence Research 

8 .PART 2 Neural Architecture Search (NAS) 

9 .

10 .ref:[2] 

11 .ref:[2] 

12 .ref:[1] 

13 .ref:[2] 

14 . 100000 10000 Model Parameters (Billion) (log Scale) 1000 100 10 1 BERT/GPT GPT-2 GPT-3 Google Brain (2018) (2019) (2020) (2021) GPT-3: 175 Billion, 355 GPU Year, $1.2M ref:[4] 

15 .N eural A rchitecture S earch (N A S ) S ize of S earch S pace: L A * O A * H A S ize of S earch S pace: L B * O B * H B ; w ith L B << L A L: N o. layers; O : N o. types of operations m acro-arch (chained or m ulti-branches) fixed or for search; layer-based or cell-based H : N o. hyper-param of operations(conditioned) ref:[5] 

16 .N eural A rchitecture S earch (N A S ) ref:[3] 

17 .N eural A rchitecture S earch (N A S ) ref:[5] 

18 .

19 .• Global Search • Search from Scratch • Discrete Search Process • Fully Trained ref:[7] 

20 .D A R T S ( IC LR ’ 19 ) • Cell-based Search (Micro) • Continuous Search Process • Approximate Gradient ref:[6] 

21 .

22 .ref:[3] 

23 .P erfo rm ance C o llap se D arts ( IC LR ’ 19 ) D arts+ ( IC LR ’ 19 ) P C -D arts ( C V P R ’ 19 ) P -D arts ( C V P R ’ 19 ) Fair D arts ( E C C V ’ 20 ) R obust D arts ( IC LR ’ 20 ) S D arts ( IC LR ’ 20 ) iD arts ( C V P R ’ 21 ) D arts- ( IC LR ’ 21 ) D arts-pt ( IC LR ’ 21 ) … 

24 .AAAI’2022 (Under Review) 

25 .PART 3 System Integration 

26 .source: H idden TechnicalD ebt in M achine Learning S ystem s, N IP S 2015 

27 .

28 .

29 .