展开查看详情
1 . gohbase
Pure Go HBase Client
Andrey Elenskiy • Bug Breeder at Arista Networks
2 .What’s so special?
● A (sorta)-fully-functional driver for HBase written in Go
● Kinda based on AsyncHBase Java client
● Fast enough
● Small and simple codebase (for now)
● No Java (not a single AbstractFactoryObserverService)
3 .Top contributors (2,000 ++)
● Timoha (Andrey Elenskiy)
● tsuna (Benoit Sigoure)
● dgonyeo (Derek Gonyeo)
● CurleySamuel (Sam Curley)
4 .So much failure
● HBase's feature-set is huge → bunch of small projects → bugs
● Asynch, wannabe lock-free architecture + handling failures = :coding_horror:
● Benchmarking is tricky
● Found some HBase issues in the process
5 .Go is pretty cool I guess
6 .goroutines and channels FTW
func main() {
ch := make(chan string)
go func() {
time.Sleep(time.Second)
ch <- "...wait for it..."
}()
go func() {
time.Sleep(2 * time.Second)
ch <- "...dary"
}()
Legen...
fmt.Println("Legen...")
fmt.Println(<-ch) ...wait for it...
fmt.Println(<-ch)
...dary
}
Program exited.
7 .context.Context
func main() {
ch := make(chan string)
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
go func() {
time.Sleep(100 * time.Millisecond)
ch <- "SWAG"
}()
// 50/50 chance to fall into either
select { SWAG
case s := <-ch: // could be HERE
fmt.Println(s)
case <-ctx.Done():
Program exited.
fmt.Println("YOLO") // could or HERE
}
}
8 .Example
func main() {
client := gohbase.NewClient("localhost")
// set a timeout for get to be 100 ms
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
getRequest, err := hrpc.NewGetStr(ctx, "table", "row")
// this will fail if it takes longer than 100 ms
getResponse, err := client.Get(getRequest)
}
● context is usually used throughout a web app, so it fits to the API nicely
● chaining contexts is useful
func main() {
parent, cancel := context.WithCancel(context.Background())
child, _ := context.WithTimeout(parent, 100*365*24*time.Hour) // wait for 100 years
cancel()
<-child.Done()
fmt.Println("YO")
}
9 .Internal architecture in a nutshell
11 .Step 1: Get region in B+Tree
12 .Step 2: write() to RS
13 .Step 3: receiveRPCs()
14 .Step 4: rpc.ResultChan()<-res
15 .Case A: Normal
● Client’s goroutine writes rpc to RS connection
● One goroutine in RegionClient to read from RS connection
● Asynchronous internals. Synchronous API.
16 .Case B: Cache miss/failure
1. Go to B+tree cache for region of the RPC
2-100. ...Magic...
101. rpc.ResultChan()<-res
17 .Magic?
2. Mark region as unavailable in cache
3. Block all new RPCs for region by reading on its "availability" channel
func main() {
ch := make(chan struct{})
go func() { 2009-11-10 23:00:00 +0000 UTC sleeping
fmt.Println(time.Now(), "sleeping") 2009-11-10 23:00:01 +0000 UTC done
time.Sleep(time.Second)
close(ch)
}()
Program exited.
<-ch
fmt.Println(time.Now(), "done")
}
4. Start a goroutine to reestablish the region
5. Replace all overlapping regions in cache with new looked up region
6. Connect to Region Server
7. Probe the region to see if it’s being served
8. Close "availability" channel to unblock RPCs and let them find new region in cache
9. write() to RS
10-100. PROFIT!!!
18 .How do you benchmark this stuff?
Requirements:
● No disk IO
● No network
Tried:
● Standalone
● Pseudo-distributed (MiniHBaseCluster from HBaseTestingUtility)
● Distributed on the same node with Docker
● 16 node HBase cluster
19 .I want my cores
● Using 70% CPU per Region Server on client side
● Region Server is chilling and not using all CPUs per connection
● Where’s the bottleneck?
20 .Linux TCP loopback
● 100K QPS (10μs per operation)
● 2μs context switch on same hardware
● Where’s ~50% cpu?
21 .Linux TCP loopback
● 100K QPS (10μs per operation)
● 2μs context switch on same hardware
● Where’s ~50% cpu?
22 .Linux TCP loopback
● 100K QPS (10μs per operation)
● 2μs context switch on same hardware
● Where’s ~50% cpu?
● Read/Write syscall is slow?
● One connection, one core
● Gone too deep, I just wanted to
benchmark the client…
23 .fast.patch [HBASE-15594]
24 clients doing 1M random reads each to one HBase 1.3.1 regionserver
24 .fast.patch [HBASE-15594]
● With it, same %75 / %75 CPU utilization per connection
● Without it, RegionServer is 100% CPU per connection: probably wasting time
context switching
● More threads, more throughput
● More connections, even more throughput
25 .Benchmark Results
● 30M rows with 26 byte keys and 100 byte vlaues
● 200 regions
● 3 runs of each benchmark
● 16 regionservers with 32 cores 64gb ram
● One Arista switch ;)
26 .RandomRead
With lots of threads gohbase is 10% faster
3 times less memory allocated though
27 .Scan
With more threads, gohbase is comparable to HTable
30% less total memory allocated
28 .RandomWrite? (gohbase only)
Best: 250 threads, 270sec, 165,218mb total
29 .Benchmarking was “entertaining”
热门城市
