展开查看详情
1 .TuneIn: How to get your jobs tuned
while sleeping
Manoj Kumar Arpan Agrawal Pralabh Kumar
Senior Software Engineer Software Engineer Senior Software Engineer
LinkedIn LinkedIn LinkedIn
2 . OUR VISION
Create economic opportunity for every
member of the global workforce
2
3 . OUR MISSION
Connect the world’s professionals to make them
more productive and successful
3
4 .Agenda
• Why TuneIn?
• How does TuneIn work?
• Architecture and framework features
• Road ahead
4
5 .Grid Scale at LinkedIn
2008 2018
1 cluster 10+ clusters
20 nodes 1000s of nodes
5 users 1000s of active users
MapReduce Pig, Hive, Spark, etc.
Few workflows 10000s workflows
5
6 .Typical Conversations
Hey, this Spark I will tune it to
job is running improve the
slowly. run time.
Manager Developer
6
7 .Typical Conversations
We have found
some jobs which I will ask my team
are consuming to tune those jobs
high resources on to reduce the
the cluster. resource usage.
Hadoop Admin Manager
7
8 .Typical Conversations
Is there a way
we can get this I will try to tune
daily report 30 it to reduce the
minutes early? run time.
Client Developer
8
9 .Why Tuning?
• Optimal parameter configuration:
– leads to better cluster utilization and thus savings
– reduces the execution time
• Default configuration is not always optimal
9
10 .Manual Tuning
PHASE 3 PHASE 1
Come up with next Execute
parameter set Manual Job
Tuning
Process
PHASE 2
Observe the Execution
Metrics
10
11 .Dr. Elephant: Heuristic based tuning
• Suggests tuning recommendations
PHASE 3 PHASE 1
based on pre-defined heuristics Come up with next Execute
parameter set Heuristics
Job
• No need to worry about the Based
Manual
hundreds of counters and Tuning
parameters
• Relies on user’s initiative to use the
recommendations
PHASE 2
• Expects some user expertise Look at the Dr. Elephant
recommendations
11
14 .Why Auto Tuning?
• 10000s of jobs to tune
• Increases developer productivity
• Tunes without any extra effort
• No expertise is expected
• Option of which objective function to tune for
– resource usage
– execution time etc.
14
15 .Let’s auto tune!
Photo: memecreator.org
15
16 .TuneIn
• Framework to automatically tune
recurring Hadoop and Spark jobs
• Iteratively tries to reach the
optimal configuration
• Results : 20-35% reduction in
Resource Usage
16
17 .Particle Swarm Optimization (PSO) [1]
• Mimics the behavior of
swarm of birds searching food
• Starts optimization by
introducing particles at random
positions in the search space
Source: Wikipedia
Particle Swarm Optimization by J. Kennedy et al., https://ieeexplore.ieee.org/document/488968/
17
18 .PSO (contd.)
• Points of attraction: personal and
global best known positions
• Particles converge to the region
with the minimum cost function
value
Source: Wikipedia
18
19 .Why PSO?
• Cost function is noisy
– PSO is gradient free and robust against noise [3]
• Spark and Hadoop are complex systems
– PSO is a metaheuristic black box optimization algorithm
• Fastest convergence
K. E. Parsopoulos et al., “Particle Swarm Optimizer in Noisy and Continuously Changing Environments,” in Artificial Intelligence and Soft Computing
19
20 .PSO Details [2]
• Swarm size of 3 gives the best result
– neither too small to cover the search space
– nor too big to do many first iteration random searches
• Good starting point is important to guide the swarm
Optimizing Hadoop parameter settings with gene expression programming guided PSO by Mukhtaj Khan et al.
20
21 .Cost function
• Resource usage per unit input
∑"#$%&'$()* +,-./0-12 314,25 ∗ +,-./0-12 78.041
9,./: ;-8<. =0>1
• Approximately input size invariant
21
22 .Search Space
• Parameters being tuned constitutes Param 3
the search space
• Depends on the cost function metric
Param 2
Param 1
22
23 .Search Space
Cost function: Resource Usage
Pig Spark
mapreduce.map.memory.mb spark.executor.memory
mapreduce.reduce.memory.mb spark.executor.cores
mapreduce.task.io.sort.mb spark.memory.fraction
mapreduce.task.io.sort.factor spark.yarn.executor.memoryOverhead
23
24 .Search Space Reduction
• Important to prevent failures
• Speeds up convergence
• Boundary parameter values
– e.g. !"#$%. '(')*+,$. ),$'! ∈ 1, 10
• Parameter interdependent constraints
– Captures the interdependence among the parameters
– e.g. 1#"$'2*)'. +#!%. 3,. !,$+. 14 < 0.60 ∗ 1#"$'2*)'. 1#". 1'1,$8. 14
24
25 .Search Space Reduction
• Different jobs can have different optimal boundary values
• Boundary values are tightened on the basis of previous
job execution counters
25
26 . Avoiding over optimization
• Undesirable to squeeze memory
so much that execution time
shoots up significantly
• Updated cost function:
∑"#$%&'$()* +,-./0-12 314,25 ∗ +,-./0-12 78.041
+ @1-/:.5
9,./: ;-8<. =0>1
Photo: Ian Burt
26
27 .Convergence
• No theoretical bound on the steps to converge
• Practically converges in 20 job executions
• TuneIn gets turned off for the job automatically on convergence
27
28 .Results
Job type Metric Reduction range
Spark Resource Usage 30 - 40 % per job
Pig Resource Usage 20 - 35 % per job
28
29 .Architecture
Dr. Elephant
1. Get Parameters
Rest API
2. Mapper memory: 2048
Sort Buffer: 200
3. Submit
Job
4. Fetch Metrics MapReduce/Spark
TuneIn Framework
Fetchers
29
