- Wizard is a Go library that provides database partitioning and sharding capabilities. It supports vertical partitioning, horizontal sharding, and works with the XORM ORM. Users initialize a Wizard engine with database connections, create clusters, and get sessions from Wizard to execute queries across shards.

1. Copyright © 2009-2017 eureka, inc. All rights reserved.
golang.tokyo #7 Wizard
Takuma Morikawa / eureka, Inc.
2017-07-03

2. Who?
●
-
-
-

3. Agenda
● 1. About Us
● 2. What Lead Us to Create This?
● 3. How to Use It?

4. CONFIDENTIAL

5. About Us - Pairs
https://eure.jp/pairs-5million-thanks/

6. About Us - Couples

7. About Us - IAC/Match Group

8. 2017.01
2016.01
2016.09
2015.05
Our History
"Couples" gets over 4 million
downloads.
"Pairs" gets over 5 million user
and 32 million matchings.
Junya Ishibashi is elected CEO,
succeeding Yu Akasaka, who
retained chairman title.
IAC/InterActiveCorp acquires
eureka.
2014.06
2014.05
2013.10
2012.10
2011.09
2010.08
2008.11
"Pairs" gets over 1 million user
and 3.3 million matchings.
eureka launches communication
app for couple, "Couples".
eureka launches online-dating
service "Pairs" in Taiwan.
eureka launches online-dating
service "Pairs" in Japan.
eureka starts smartphone and
Facebook app services.
eureka starts blog marketing
services.
eureka is founded in Tokyo.

9. Today’s Talk...

10. Today’s talk...

11. Today’s talk...
* Gopher designed by Hinako Sakawa - Original Gopher designed by Renee French.
https://github.com/ohina

12. What’s Wizard?
● https://github.com/evalphobia/wizard
● Database Partitioning/Sharding Library
- Vertical partitioning
- Horizontal sharding
- Inspired by mixed_gauge (https://github.com/taiki45/mixed_gauge)
- Hash Slots
- Clusters
- Extra features
- Read only flag
- Transaction Management
- Query to all shards
- Wizard is NOT ORM
● Supported ORM
- xorm (https://github.com/go-xorm/xorm)

13. 2. What Lead Us to Create This?

14. One day, on very sunny day morning...
あるよく晴れた日の朝...

15. Mr. Me (alias)
私さん（仮名）
ʕº̫͡ºʔ
Note: All characters and stories are fictitious examples
※ 登場人物・話は全て架空の例です
Mr. Boss (alias)
ボス（仮名）
ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢

16. One day, on very sunny day morning… (2014, Winter)
ʕº̫͡ºʔ< "Awesome weather today, huh!"
（うーん、今日もいい天気）
ʕº̫͡ºʔ< "Nothing beats Golang on a day like this."
（こんな日はGoに限る）
ʕº̫͡ºʔ< "Alright, let’s write some codes."
（よーし、今日も仕事しちゃうぞ）

17. "Hi, Mr. Me" >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（私さん）
"I have something to tell you..." >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（ちょっと話があるんですが...）
One day, on very sunny day morning… (2014, Winter)

18. One day, on very sunny day morning… (2014, Winter)
ʕº̫͡ºʔ< "What's the matter?"
（え、なんでしょうか）
ʕº̫͡ºʔo0O (Something smells fishey...)
（これは危険なニオイがするぞ...）

19. One day, on very sunny day morning… (2014, Winter)
"About the next golang project," >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（Goフルスクラッチに際して）
"We needs some kind of database shardings,
right?" >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（DBシャーディング必要じゃないですか？）

20. One day, on very sunny day morning… (2014, Winter)
ʕº̫͡ºʔ< "Ah, well..."
（あー、そうですね。。。）
ʕº̫͡ºʔ< "But there isn’t such library in golang."
（でもちょっと探してみた限りシャーディング対応したライブラリなさそうなんですよ
ね。）
ʕº̫͡ºʔ< "I think it’s better to do it using some
middleware for MySQL"
（やっぱミドルウェア側でやるしかないんじゃないですかね）

21. One day, on very sunny day morning… (2014, Winter)
"Hmmmmm, whatever" >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（ふーん）
"You should fix the details and
add it to the schedule." >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（まあいいけどちゃんと詳細決めて見積もり入れておいてくださいよ）

22. One day, on very sunny day morning… (2014, Winter)
ʕº̫͡ºʔ< "Roger, sir!"
（ラジャー！）
"..." >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
(He replies cheerfully, but never keeps a schedule.)
O0oʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（返事だけはいつも調子いいからけどスケジュール全く守らないからな、
こいつ...）

23. 3 Months Later...
三ヶ月後...

24. 3 Months Later (Spring)
"Hi, Mr. Me" >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（私さん）
"What’s going on the sharding?" >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（シャーディングの件どうなってます？）

25. 3 Months Later (Spring)
ʕº̫͡ºʔ< "Ah, sharding! Well..."
（あーその件ですね。。。）
"Did you consider about it seriously?" >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（ちゃんと考えてました？）
ʕº̫͡ºʔo0O (he-he! Gotcha!)
（ふはは、かかったなバカめ！）

26. 3 Months Later (Spring)
ʕº̫͡ºʔ< "Let’s try to use Vitess by Youtube."
（YoutubeのVitess使いましょう）
"What?!?!" >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（は？！？！）
ʕº̫͡ºʔ< "Trust yourself."
（我々なら出来ますよ。自分を信じましょう。）

27. 3 Months Later (Spring)
"Are you sure?" >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（信じましょうって...）
"It’s just released, right?" >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（まだ出たばっかですよね？）
"Moreover, don’t you think that
you should maintain it yourself?" >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（それにあなた、自分で構築・運用する気ないですよね？）

28. 3 Months Later (Spring)
"If use it, you should do it from beginning to the
end." >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（Vitess採用するならあなた運用対応してくださいよ）
"I’ll catch you even if you escape." >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（決めるだけ決めて逃げたらどこまでも追いかけますからね）
"Right?" >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（それでもいいですか？）

29. 3 Months Later (Spring)
ʕº̫͡ºʔo0O (OMG...)
（そんなばかな...）
ʕº̫͡ºʔ< "Just kidding. I have another plan."
（いや、冗談です。実は本当の案があるんです。）
ʕº̫͡ºʔ< "We’ll write golang library for sharding."
（我々でGo言語のライブラリ作ります。）

30. 3 Months Later (Spring)
"Alright. Should’ve said it in the first place" >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（はじめからそう言ってください）
"We have 3 months to release," >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（3ヶ月後にはリリースですから）
"You don’t have much time to sleep. Just write code."
>ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（寝てる暇はありませんよ、さあ書きましょう）

31. 3 Months Later (Spring)
ʕº̫͡ºʔ< "..."
（...）
ʕº̫͡ºʔo0O (Fxxk!)
（ちきしょー！）

32. Another 3 Months Later...
さらに三ヶ月後...

33. Another 3 Months Later (Summer)
ʕº̫͡ºʔ< "Phew,"
（ふー）
ʕº̫͡ºʔ< "The release has been postponed and this
library is almost about complete."
（リリースも延期になったし、なんとかライブラリも仕上がりそうだ）

34. Another 3 Months Later (Summer)
"Hi, Mr. Me" >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（私さん）
"I have a good news" >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（嬉しいお知らせがあります）
"You must be surprised." >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（きっと驚きますよ？）

35. Another 3 Months Later (Summer)
ʕº̫͡ºʔ< "What’s the news?"
（どうしたんですか？）
"Listen," >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（実は...）
"AWS Aurora is just released" >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（AWS Auroraがリリースされました）

36. Another 3 Months Later (Summer)
ʕº̫͡ºʔ< "What?!"
（え？？？）
ʕº̫͡ºʔ (Checking release infomation of AWS Aurora)
（AWS Auroraのリリース情報をチェックする）

37. Another 3 Months Later (Summer)
https://aws.amazon.com/blogs/aws/now-available-amazon-aurora/

38. Another 3 Months Later (Summer)
https://aws.amazon.com/blogs/aws/now-available-amazon-aurora/
NO MORE SHARDING!!

39. Another 3 Months Later (Summer)
"So we don’t need shardings anymore." >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（これでもうシャーディングは不要ですね）
"What a peaceful world." >ʕ̡̢̡ʘ̅͟͜͡ ʘ̲̅ ʔ̢̡̢
（いい世の中になりました。）

40. Another 3 Months Later (Summer)
ʕº̫͡ºʔ< "..."
ʕº̫͡ºʔo0O (...)
ʕº̫͡ºʔo0O (I’ll go home and sleep, seriously.)
（今日はもう帰って寝よう...）

41. Another 3 Months Later (Summer)
Chapter 1.
“Tales of Sharding”
~ Fin ~

42. 3. How to use Wizard?

43. How to use Wizard?
https://github.com/evalphobia/wizard

44. How to use Wizard?
https://github.com/evalphobia/wizard
See README!!

45. How to use Wizard?
Chapter 2.
“How to use”
~ Fin ~

46. How to use Wizard? (Setup Engine)
● 1. Initialize New Engine
- xorm.NewEngine(“mysql”, “user:pass@tcp(0.0.0.0:3306)/dbname”)
● 2. Create New Cluster
- cluster := wizard.NewCluster(engine) // <= register master db
- cluster.RegisterSlave(engine2) // <= register slave db
● 3. Add cluster to new wizard struct
- w := wizard.NewWizard()
- w.SetDefault(cluster)
- ormEngine = wizardxorm.New(w)

47. How to use Wizard? (Get Session)
● 4. Get Session from wizard
- session, err := ormEngine.UseMasterSession(ctx, User{})
● first arg is identifier.
○ Use same identifier, you can get same session.
○ Suit for a series of transaction during the request.
○ e.g.) context.Context, *http.Request
○ Anything is OK!
● second arg is struct for table.
● 5. Execute SQL
- session.Sql("SELECT 1")

48. Wizard
What’s Wizard/Cluster/Engine?
Cluster (User)
DB
Master
M
DB
Slave1
S
DB
Slave2
S
(Engine)
(Engine)
(Engine)
Session
Cluster (Article)
DB
Master
M
DB
Slave1
S
DB
Slave2
S
(Engine)
(Engine)
(Engine)
Shard Cluster
(Favorite)
Shard #1 SSM
Shard #2 SSM
Shard #3 SSM
①
②
③
④

49. Code: Setup Engine
import (
_ "github.com/go-sql-driver/mysql"
"github.com/go-xorm/xorm"
"github.com/evalphobia/wizard"
wizardORM "github.com/evalphobia/wizard/orm/xorm"
)

50. Code: Setup Engine
var ormEngine wizardORM.ORM
// Initialize database. (Ignore errors for simpleness)
func initNormal() {
// xorm's engine(DB handler)
xormEngine, _ := xorm.NewEngine("mysql", "user:pass@tcp(0.0.0.0:3306)/dbname")
// Create DB Cluster (argument is master DB)
cluster := wizard.NewCluster(xormEngine)
// Create wizard (DB and session handler)
w := wizard.NewWizard()
w.SetDefault(cluster) // set default cluster
ormEngine = wizardxorm.New(w)
}

51. Code: Execute SQL #1 - select to user’s table
// Get User from database.
func sqlGetUser(ctx context.Context, userID int64) (User, error) {
session, _ := ormEngine.UseMasterSession(ctx, &User{})
session.And("user_id = ?", id)
session.ForUpdate() // Lock the row
user := &User{}
_, _ = session.Get(user) //
return user
}
// => SELECT * FROM user WHERE user_id = userID FOR UPDATE;

52. Code: Execute SQL #2 - update user’s table
// Update user’s point
func sqlUpdateUserPoint(ctx context.Context, userID int64, point int64) error {
session, _ := ormEngine.UseMasterSession(ctx, &User{})
session.Cols("point")
session.And("id = ?", userID)
return session.Update(&User{
Point: point,
})
}
// => UPDATE user SET point = point WHERE user_id = userID;

53. Code: Execute SQL #3 - update campaign table
// Update campaign status of the user
func sqlUpdateCampaignStatus(ctx context.Context, userID int64) error {
session, _ := ormEngine.UseMasterSession(ctx, &Campaign{})
session.Cols("status")
session.And("user_id = ?", userID)
return session.Update(&Campaign{
Status: "end",
})
}
// => UPDATE campaign SET status = "end" WHERE user_id = userID;

54. Code: Execute SQL - wrap up
// UPDATE user’s point and complete campaign.
func ExecSQL(ctx context.Context) {
ormEngine.SetAutoTransaction(ctx, true) // start transaction when SQL is executed
userID := 1
u, _ := sqlGetUser(ctx, userID) // => BEGIN and SELECT
_ = sqlUpdateUserPoint(ctx, u.UserID, u.Point + 1) // => UPDATE user
_ = sqlUpdateCampaignStatus(ctx, u.UserID) // => UPDATE campaign
orm.SetAutoTransaction(ctx, false)
err = ormEngine.CommitAll(ctx) // COMMIT
ormEngine.CloseAll(ctx) // Close all sessions
}

55. How to use Shard? (Setup Engine)
● 1. Initialize New Engine
- xorm.NewEngine(“mysql”, “user:pass@tcp(0.0.0.0:3306)/dbname”)
● 2. Create New Clusters
- baseCluster := wizard.NewCluster(engine)
- shardCluster01 := wizard.NewCluster(shardEngineA01)
- shardCluster01.RegisterSlave(shardEngineA02)
- shardCluster02 := wizard.NewCluster(shardEngineB01)
- shardCluster02.RegisterSlave(shardEngineB02)

56. How to use Shard? (Setup Shard Clusters)
● 3. Add clusters to new wizard struct
- w := wizard.NewWizard()
- w.SetDefault(baseCluster)
- // ↑add normal cluster
- shards := w.CreateShardCluster(&UserFavorite{}, 1021)
- shards.RegisterShard(0, 510, shardCluster01)
- shards.RegisterShard(511, 1020, shardCluster02)
- // ↑add shards cluster
- w.RegisterTables(shards, &UserLogin{}, ...)
- // ↑add sharded tables
- ormEngine = wizardxorm.New(w)

57. How to use Shard? (Get Session)
● 4. Get Session from wizard
- session, err := ormEngine.UseMasterSessionByKey(ctx, User{}, key)
● first arg is identifier.
● second arg is struct for table.
● third ard is shard key.
○ e.g. user_id
● 5. Execute SQL
- session.Sql("SELECT 1")
- // ↑ executed on one shard
- ormEngine.FindParallelByCondition([]*User, condition)
- // ↑ executed on all shards

58. Wrap up
● Wizard is DB sharding/session management library
● Only supports xorm
- Send PR!
● eureka use Wizard for production.
● Hard to test...
- Data
- DB instances
- clusters of MySQL/PostgreSQL/etc...

59. CONFIDENTIAL
Thank you
:)
Thank you :)
* Gopher designed by Hinako Sakawa - Original Gopher designed by Renee French.
https://github.com/ohina