14. Haeinsa 이름의 유래
Haeinsa는 정말로 해인사에서 따온 이름입니다.
1. 해인사에는 팔만대장경이 보존되어 있습니다.
2. 팔만대장경은 81,258개의 목판본으로 이루어져 있습니다.
3. 오탈자 없이,52,382,960글자가 새겨져 있습니다.
4. 800년 가까이 아무 문제 없이 보존되었습니다.
5. 개발 시작 당시 VCNC의 사무실이 봉은사 옆에 있었습니다.
23. Lock Column
• Haeinsa에서 트랜잭션의 상태를 저장하기 위한 특수 칼럼
• Row 단위 칼럼이 하나씩 존재
• 읽기/쓰기 시마다 확인
Row key
bal
lock
Bob
4: $3
3: $10
State:PREWRITTEN
CommitTimestamp:6
PrewriteTimestamp:4
Secondaries:[Joe]
Joe
3: $2
State:STABLE
CommitTimestamp:3
24. Lock Column State Diagram
각 Row의 상태 변화는 다음과 같습니다.
Stable
Stable
Prewritten
Committed
트랜잭션 성공시
Prewritten
Aborted
트랜잭션 실패시
26. How it works
아래 예제를 통해서 Haeinsa의 동작을 알아봅시다.
BeginTransaction()
bobBalance = Read(Bob, balance)
Write(Bob, balance, bobBalance-$7)
joeBalance = Read(Joe, balance)
Write(Joe, balance, joeBalance+$7)
Commit()
27. How it works
bobBal = Read(Bob,bal)
𝐶
𝑅 𝐵𝑜𝑏
𝑅 𝐽𝑜𝑒
Get(Bob, lock)
Get(Bob, bal)
36. Commit Operation
𝑐𝑜𝑚𝑚𝑖𝑡(𝑅 𝑝𝑟𝑖𝑚𝑎𝑟𝑦 )
STABLE
PREWRITTEN
COMMITTED
𝑅 𝑃𝑟𝑖𝑚𝑎𝑟𝑦
𝑅 𝑆𝑒𝑐𝑜𝑛𝑑𝑎𝑟𝑦1
𝑅 𝑆𝑒𝑐𝑜𝑛𝑑𝑎𝑟𝑦2
Primary Row를 Commit 상태로 원자적으로 변경합니다.
이 이후에는 문제가 발생해도 이 트랜잭션은 성공한 것으로 처리합니다.
49. 실패시 동작
𝑚𝑎𝑘𝑒𝑆𝑡𝑎𝑏𝑙𝑒(𝑅 𝑝𝑟𝑖𝑚𝑎𝑟𝑦 )
STABLE
PREWRITTEN
COMMITTED
𝑅 𝑃𝑟𝑖𝑚𝑎𝑟𝑦
𝑅 𝑆𝑒𝑐𝑜𝑛𝑑𝑎𝑟𝑦1
ABORTED
𝑅 𝑆𝑒𝑐𝑜𝑛𝑑𝑎𝑟𝑦2
마지막으로 Primary를 Stable로 만듭니다.
이로써 트랜잭션은 복구가 완료되었습니다.
50. 어떻게 변경사항을 취소하는가?
Row key
Bal
lock
4: $3
3: $10
State:PREWRITTEN
CommitTimestamp:6
PrewriteTimestamp:4
Secondaries:[Joe]
Joe
4: $9
3: $2
State:PREWRITTEN
CommitTimestamp:6
PrewriteTimestamp:4
Primary:Bob
Alice
7:$10
State:STABLE
CommitTimestamp:7
Bob
데이터를 여러 Timestamp에 걸쳐
여러 버전을 저장해 둡니다.
51. 어떻게 변경사항을 취소하는가?
Row key
Bal
lock
4: $3
3: $10
State:PREWRITTEN
CommitTimestamp:6
PrewriteTimestamp:4
Secondaries:[Joe]
Joe
4: $9
3: $2
State:PREWRITTEN
CommitTimestamp:6
PrewriteTimestamp:4
Primary:Bob
Alice
7:$10
State:STABLE
CommitTimestamp:7
Bob
PrewriteTimestamp이 가리키는
버전을 지우면 변경사항을 취소할 수 있습니다.
52. 어떻게 변경사항을 취소하는가?
Row key
Bal
lock
Bob
4: $3
3: $10
State:PREWRITTEN
CommitTimestamp:6
PrewriteTimestamp:4
Secondaries:[Joe]
Joe
4: $9
3: $2
State:STABLE
CommitTimestamp:6
Alice
7:$10
State:STABLE
CommitTimestamp:7
PrewriteTimestamp이 가리키는
버전을 지우면 변경사항을 취소할 수 있습니다.
53. 어떻게 변경사항을 취소하는가?
Row key
Bal
lock
4: $3
3: $10
State:PREWRITTEN
CommitTimestamp:6
PrewriteTimestamp:4
Secondaries:[Joe]
Joe
3: $2
State:STABLE
CommitTimestamp:6
Alice
7:$10
State:STABLE
CommitTimestamp:7
Bob
PrewriteTimestamp이 가리키는
버전을 지우면 변경사항을 취소할 수 있습니다.
64. 그냥 적용해 봤습니다
• HBase의 기본 연산을 모두 제공합니다.
• Lock Column만 추가하면 기존 HBase클러스터에 쉽게 적용
이 가능합니다.
Between
테스트 서버
비트윈
클라이언트
Haeinsa
적용
HBase
(Cluster)
65. 그냥 적용했을때의 문제점
• 일반적인 경우의 충돌 비율: 0.004% ~ 0.010%
• 하지만 유저가 의도적으로 많은 요청을 보낼 때에는 충돌이 자
주 나는 현상이 발생하였습니다.
Between
테스트 서버
비트윈
클라이언트
HBase
(Cluster)
요청을 의도적으로
많이 보내는 경우,
충돌이 자주남
66. 특정 쓰레드로 요청을 스케쥴링
• 특정 유저의 연산을 특정 쓰레드가 처리하도록 구현
• 결과적으로 한 유저에 대해서는 Serial Scheduling이 되어 한
유저가 의도적으로 요청을 많이 보내도 Conflict이 나지 않음
비트윈
클라이언트
비트윈 서버내 쓰레드풀
67. Conflict Rate를 더욱 낮추기 위해서 재시도를 함
• 만약 Conflict이 나게 되면 요청을 처음 부터 다시 시작
• 다시 충돌이 나는 경우를 방지하기위해 적당히 Backoff를 함
• Conflict Rate가 크게 줄었음: 0.0003%~0.0010%
Conflict
발생시
비트윈
클라이언트
다시
재시도
HBase
(Cluster)
69. 정리
• Haeinsa는 HBase상에서 트랜잭션을 지원하면서 Lineary
Scalable한 클라이언트 라이브러리 입니다.
• 비트윈 서비스에서 2달간 적용되어 안정성을 확보하였습니다.
• 다른 트랜잭션 라이브러리에 비해 성능이 뛰어나고 HBase보
다 더 좋은 성능을 내기도 합니다.
75. Appendix A: How it works?
Client-side
BeginTransaction()
bobBal = Read(Bob, bal)
Write(Bob, bal, bobBal-$7)
joeBal = Read(Joe, bal)
Write(Joe, bal, joeBal+$7)
Commit()
HBase-side
Row key
Bal
Lock
Bob
3: $10
State:STABLE
CommitTimestamp:3
Joe
3: $2
State:STABLE
CommitTimestamp:3
<< Before Transaction >>
There is rows representing balance of Bob and Joe.
Let’s track how Haeinsa works by studying the
transaction that Bob giving the $7 to Joe.
76. Client-side
BeginTransaction()
bobBal = Read(Bob, bal)
Write(Bob, bal, bobBal-$7)
joeBal = Read(Joe, bal)
Write(Joe, bal, joeBal+$7)
Commit()
HBase-side
Row key
bal
lock
Bob
3: $10
State:STABLE
CommitTimestamp:3
Joe
3: $2
State:STABLE
CommitTimestamp:3
State of Transaction
Writes = []
Locks = {}
Nothing to do
77. Client-side
BeginTransaction()
bobBal = Read(Bob, bal)
Write(Bob, bal, bobBal-$7)
joeBal = Read(Joe, bal)
Write(Joe, bal, joeBal+$7)
Commit()
State of Transaction
HBase-side
Row key
bal
lock
Bob
3: $10
State:STABLE
CommitTimestamp:3
Joe
3: $2
State:STABLE
CommitTimestamp:3
Writes = []
Locks[Bob] = (STABLE, 3)
Read Bob’s Lock column first. And then read Bob’s
Balance column.
78. Client-side
BeginTransaction()
bobBal = Read(Bob, bal)
Write(Bob, bal, bobBal-$7)
joeBal = Read(Joe, bal)
Write(Joe, bal, joeBal+$7)
Commit()
State of Transaction
HBase-side
Row key
bal
lock
Bob
3: $10
State:STABLE
CommitTimestamp:3
Joe
3: $2
State:STABLE
CommitTimestamp:3
Writes =
[(Bob, bal, $3)]
Locks[Bob] = (STABLE, 3)
Bob’s new balance put into writes. Store on clientside memory. It will be write on Hbase on commit.
79. Client-side
BeginTransaction()
bobBal = Read(Bob, bal)
Write(Bob, bal, bobBal-$7)
joeBal = Read(Joe, bal)
Write(Joe, bal, joeBal+$7)
Commit()
State of Transaction
HBase-side
Row key
bal
lock
Bob
3: $10
State:STABLE
CommitTimestamp:3
Joe
3: $2
State:STABLE
CommitTimestamp:3
Writes =
[(Bob, bal, $3)]
Locks[Bob] = (STABLE, 3)
Locks[Joe] = (STABLE, 3)
Read Joe’s Lock column first. And then read Joe’s
Balance column.
80. Client-side
BeginTransaction()
bobBal = Read(Bob, bal)
Write(Bob, bal, bobBal-$7)
joeBal = Read(Joe, bal)
Write(Joe, bal, joeBal+$7)
Commit()
State of Transaction
HBase-side
Row key
bal
lock
Bob
3: $10
State:STABLE
CommitTimestamp:3
Joe
3: $2
State:STABLE
CommitTimestamp:3
Writes =
[(Bob, bal, $3),
(Joe, bal, $9)]
Locks[Bob] = (STABLE, 3)
Locks[Joe] = (STABLE, 3)
Joe’s new balance put into writes.
81. Client-side
BeginTransaction()
bobBal = Read(Bob, bal)
Write(Bob, bal, bobBal-$7)
joeBal = Read(Joe, bal)
Write(Joe, bal, joeBal+$7)
Commit()
State of Transaction
Writes =
[(Bob, bal, $3),
(Joe, bal, $9)]
Locks[Bob] =
(PREWRITTEN, 6, 4,
[Joe])
Locks[Joe] = (STABLE, 3)
HBase-side
Row key
bal
lock
Bob
4: $3
3: $10
State:PREWRITTEN
CommitTimestamp:6
PrewriteTimestamp:4
Secondaries:[Joe]
Joe
3: $2
State:STABLE
CommitTimestamp:3
Prewrite value on primary row. Primary row is
selected by particular algorithm by Haeinsa.
82. Client-side
BeginTransaction()
bobBal = Read(Bob, bal)
Write(Bob, bal, bobBal-$7)
joeBal = Read(Joe, bal)
Write(Joe, bal, joeBal+$7)
Commit()
HBase-side
Row key
Bob
bal
4: $3
3: $10
State:PREWRITTEN
CommitTimestamp:6
PrewriteTimestamp:4
Secondaries:[Joe]
4: $9
3: $2
State:PREWRITTEN
CommitTimestamp:6
PrewriteTimestamp:4
Primary:Bob
State of Transaction
Writes =
[(Joe, bal, $9)]
Locks[Bob] =
(PREWRITTEN, 6, 4, [Joe])
Locks[Joe] =
(PREWRITTEN, 6, 4, ,
Bob)
Joe
lock
Prewrite value on secondary row. Secondary row is
the row which is not primary row.
83. Client-side
BeginTransaction()
bobBal = Read(Bob, bal)
Write(Bob, bal, bobBal-$7)
joeBal = Read(Joe, bal)
Write(Joe, bal, joeBal+$7)
Commit()
State of Transaction
Writes = []
Locks[Bob] =
(COMMITTED, 6, , [Joe])
Locks[Joe] =
(PREWRITTEN, 6, 4, , Bob)
HBase-side
Row key
Bob
Joe
bal
lock
4: $3
3: $10
State:COMMITTED
CommitTimestamp:6
Secondaries:[Joe]
4: $9
3: $2
State:PREWRITTEN
CommitTimestamp:6
PrewriteTimestamp:4
Primary:Bob
If prewrite all succeed, change state of primary row
to COMMITED. The transaction can be treated as
succeed at this moment.
84. Client-side
BeginTransaction()
bobBal = Read(Bob, bal)
Write(Bob, bal, bobBal-$7)
joeBal = Read(Joe, bal)
Write(Joe, bal, joeBal+$7)
Commit()
State of Transaction
HBase-side
Row key
bal
lock
Bob
4: $3
3: $10
State:COMMITTED
CommitTimestamp:6
Secondaries:[Joe]
Joe
4: $9
3: $2
State:STABLE
CommitTimestamp:6
Writes = []
Locks[Bob] =
(COMMITTED, 6, , [Joe])
Locks[Joe] = (STABLE, 6)
Change state of secondary row to STABLE.
85. Client-side
BeginTransaction()
bobBal = Read(Bob, bal)
Write(Bob, bal, bobBal-$7)
joeBal = Read(Joe, bal)
Write(Joe, bal, joeBal+$7)
Commit()
State of Transaction
HBase-side
Row key
bal
lock
Bob
4: $3
3: $10
State:STABLE
CommitTimestamp:6
Joe
4: $9
3: $2
State:STABLE
CommitTimestamp:6
Writes = []
Locks[Bob] = (STABLE, 6)
Locks[Joe] = (STABLE, 6)
Change state of primary row to STABLE.
86. Client-side
BeginTransaction()
bobBal = Read(Bob, bal)
Write(Bob, bal, bobBal-$7)
joeBal = Read(Joe, bal)
Write(Joe, bal, joeBal+$7)
Commit()
State of Transaction
HBase-side
Row key
bal
lock
Bob
4: $3
3: $10
State:STABLE
CommitTimestamp:6
Joe
4: $9
3: $2
State:STABLE
CommitTimestamp:6
Writes = []
Locks={}
Transaction completed. All rows are in stable state.