http://badcamp.net/session/database-synchronization

1. Database
Synchronization
Shaun Haber
Warner Bros. Records

2. What is it?
• Merging content between a dev site and a
production site

3. Disclaimer
• No single answer
• No “Drupally” solution
• Not exclusive to Drupal
• Not magic

4. Who the hell am I?

7. Warner Music Group

8. Warner Bros. Records
• Subsidiary of Warner Music Group
• Family of labels (Reprise, Sire, etc.)
• Over 100 artists
• Top-selling albums
• It’s music biz after all!

9. So what?

10. WBR Tech
• Only label with an in-house Tech team
• “Start-up” mentality
• Fast-paced, hectic, and fun!
• We use Drupal... religiously

11. 93 Drupal Sites
1 new site every week

12. Launching like crazy!

14. Source: http://flickr.com/photos/krosinsky/2848288562/
Web sites in the wild!

15. Websites in the wild
• Always collecting new data!

16. Data Launch
Time

17. Not a bad thing,
obviously
• Want websites to grow
• More users + more data = PROFIT

18. But...
• How do we keep the site updated?
- New content
- New features
- Code fixes
- <insert your own update here>

19. Minor updates
Major updates
Source: http://flickr.com/photos/nimboo/132386298

20. Minor Updates
• CSS tweak
• template.php change
• Add a new Block
• Change settings on a View
• Install a new module

21. Major Updates
• Schema changes
• Information re-architecture
• Significant configuration changes
• User flow changes
• New theme integration

22. Strategy?
Maintain a separate Dev site!

23. Prod server
Dev server
New
Time

24. Prod server
Dev server
New QA
Time

25. Prod server
Prod
Dev server
New QA Dev
Time

26. Prod server
Prod Prod
Dev server
New QA Dev
Time

27. Prod server
Prod Prod
Dev server
New QA Dev Dev
Time

28. Prod server
?
Prod Prod
Dev server
New QA Dev Dev
Time

29. Syncing Databases Sucks
Code Easy
Files Easy
Database Hard

30. Prod server
Prod Prod
Dev server
New QA Dev Dev
Time

31. Prod server
Prod 2.0
Prod Prod
Dev server
New QA Dev Dev
Time

32. Order of Events
1. Develop a new site
2. Launch site
3. Take snapshot of prod site
4. Develop on snapshot
5. Magic? => Relaunch new version of site

33. But it’s not Magic!
1. Take dev site down
2. Shift sequenced IDs on Dev
3. Take prod site down
4. Merge content from Prod to Dev
5. QA “new” dev site
6. Copy dev site to prod site
7. Bring “new” prod site live

34. Source: http://flickr.com/photos/interplast/6339098/
It’s Database Surgery!

35. 2 Step Process
• Step 1 - Shift Sequenced IDs
• Step 2 - Merge content

36. 3
2
1

37. 3 3
2 2
1 1

38. 6
5 5
4 4
3 3
2 2
1 1

39. 11
10
6
5
4
3 3
2 2
1 1

40. 11
10
6 6
5 5
4 4
3 3
2 2
1 1

41. 11
10
6 6
5 5
4 4
3 3
2 2
1 1

42. 11
10
6 6
5 5
4 4
3 3
2a 2a
1 1

43. 11
10
6 6
5 5
4 4
3 3
2a 2a
1 1

44. Step 1 - Shifting IDs
• comments_cid
• files_fid
• node_revisions_vid
• node_nid
• users_uid

45. Need to know
• Highest common ID between Dev and
Prod
• Delta value to shift
• Reference of known tables and fields

46. Highest Common ID
• Top item on the “stack” at time of the
snapshot.
3
3 3
2 2
1 1

47. Delta value
• Amount to shift the conflicted items, with
extra padding
11
10
7
3

48. UPDATE table
SET id = id + $delta
WHERE id > $common

49. And that’s it for Step 1

50. Actually, it’s MUCH
more complicated...

51. What tables have nid?
comments.nid poll.nid
content_field_* nid.field_*_nid poll_choices.nid
content_type_* nid.field_*_nid poll_votes.nid
files.nid term_node.nid
forum.nid uc_cart_products.nid
forward_log.nid uc_order_products.nid
history.nid uc_product_features.nid
node.nid uc_products.nid
node_access.nid uc_roles_products.nid
node_comment_statistics.nid usernode.nid
node_counter.nid webform.nid
node_revisions.nid webform_component.nid
nodefamily.parent_nid, child_nid webform_submissions.nid
panels_node.nid webform_submitted_data.nid

52. Also...
• Special tables:
• location, sequences, url_alias, etc.
• node-nid.tpl.php
• Serialized PHP variables in DB
• PHP code in DB
• URLs in DB or elsewhere (e.g., /node/123)

53. Well shit!

54. Do the best we can!
• Reference of all known tables
• Reference of all known sequence fields
• Reference of all known “special cases”
• Automate as much as possible

55. Scripting Time!

56. Check for unknown tables
$rs = db_query(“SHOW TABLES”);
while ($row = db_fetch_row($rs)) {
if (!is_known_table($row[0]) {
log_unknown_table($row[0]);
}
}
if (found_unknown_tables()) {
print_unknown_tables();
exit;
}

57. Store all known tables in a
txt file
access buddylist_groups
accesslog buddylist_pending_requests
audio_widget_thumbnail cache*
audio_widget_track comments
authmap contact
blocks content_field_*
blocks_roles content_type_*
boxes devel_queries
buddylist devel_times
buddylist_buddy_group ...

59. Store all fields in separate
txt files
comments.nid node_comment_statistics.nid
content_field_* nid.field_*_nid node_counter.nid
content_type_* nid.field_*_nid node_revisions.nid
files.nid nodefamily.parent_nid, child_nid
forum.nid panels_node.nid
forward_log.nid poll.nid
history.nid poll_choices.nid
node.nid poll_votes.nid
node_access.nid ...

60. Now we can shift IDs!
• Iterate thru DB tables
• If table has known fields, shift IDs
(remember that SQL command?)
• Rinse and repeat for each sequenced ID

61. UPDATE table
SET id = id + $delta
WHERE id > $common

62. Special Cases

63. Sequences table
• Simply reset the value to new highest ID
• Do this after shifting IDs in the “primary”
table (node.nid, user.uid, etc.)

64. UPDATE sequences
SET `$seq` = $max

65. Location table
• Stores ID val in column `eid`
• Stores sequence type in column `type`
• type = node, user

66. UPDATE location
SET `eid` = `eid` + $delta
WHERE `eid` > $common
AND `type` = $type

67. Url_alias table
• ID values are embedded as strings
• Use pattern matching to parse the ID
• node: node/nid
• user: user/uid, blog/uid
• Add the delta, update new alias

68. Pseudo-code
SELECT * FROM url_alias WHERE src LIKE ‘node/%’
preg_match('/node/([0-9]*)/', $src, $matches)
$id = $matches[1]
$id = $id + $delta
UPDATE url_alias SET src = 'node/$id' WHERE pid = $pid

69. Manually
• Rename any node-nid.tpl.php files
• Search for ID vals in DB:
• Eval’ed PHP code
• Serialized PHP code
• URLs
• anything else?

70. Step 1 Recap
• Maintain indexes for tables and fields
• Automate using the indexes
• Review indexes before each shift
• Inspect for manual cases after each shift
• Document every new case you find!

71. At least most of this
can be automated!

72. Step 2 - Merging
Content

73. Merging Content
10
6 6
5 5
4 4
3 3

74. What to merge?
• Content
• Really, just the content
• No variables, settings, etc.

75. Need to know
• Highest Common ID (same from Step 1)
• Reference of tables

76. Process
• Iterate thru Prod tables:
• Skip
• INSERT IGNORE (I)
• REPLACE (R)
• DROP and INSERT (A)

78. Special Cases
• Url_alias table
• Sequences table
• Some nodes

79. Url_Alias table
• Don’t go by pid
• REPLACE INTO url_alias SET src = '$src',
dst = '$dst'

80. Sequences table
• Manually inspect sequence values!

81. Node timestamps
• Get timestamp of Highest Common nid
• Check for older nodes on Prod that have
been modified recently

82. Replace on Dev with
SELECT nid
FROM node
WHERE changed > $timestamp
AND nid > $common

83. That’s it... for now.

84. Future
• Share sequences table between Dev and
Prod
• Even/odd IDs (Drupal 6+)
• Macro recordings and playbacks

85. Questions?
• Shaun Haber
shaun.haber@wbr.com
http://srhaber.com
Twitter: @srhaber