Slides from talk on legacy data migration. Includes introduction of Trucker gem and covers common migration issues. This talk was given by Patrick Crowley and Rob Kaufman at RubyMidwest 2010 in Kansas City, MO.

1. Migrating Legacy Data

2. Patrick Crowley
Rob Kaufman

3. (We’re from SD Ruby)

4. Let’s talk about migrating

5. Suck => Less Suck

6. PHP => Rails

7. Perl => Rails

8. Java => Rails

9. Rails => Rails

10. Migrating = pain

11. • Develop new site
• Migrate data
• Deploy new site

12. Migrate data

13. • Dump database
• Drop some tables
• Rename attributes
• Tweak data by hand
• Import data into new app
• Pray things are okay

14. Sucks, right?

15. • Labor intensive
• Error prone
• One way trip

16. Trucker

17. Migrate legacy data
(with less suck)

18. • Migrate data from day one
• Improve data over time
• Launch with confidence

19. rake +
legacy classes +
migration helper

20. Let’s get started

21. sudo gem install trucker

22. config.gem “trucker”

23. script/generate truck

24. $ script/generate truck
create app/models/legacy
create app/models/legacy/legacy_base.rb
exists lib/tasks
create lib/tasks/legacy.rake
insert added legacy adapter to end of database.yml
insert added new load path to environment.rb

25. config/environment.rb
Rails::Initializer.run do |config|
config.load_paths += %W( #{RAILS_ROOT}/app/
models/legacy )
end

26. config/database.yml
legacy:
adapter: mysql
encoding: utf8
database: trucker_legacy
username: root
password:

27. app/models/legacy/legacy_base.rb
class LegacyBase < ActiveRecord::Base
self.abstract_class = true
establish_connection "legacy"
def migrate
new_record = self.class.to_s.gsub(/
Legacy/,'::').constantize.new(map)
new_record[:id] = self.id
new_record.save
end

28. app/models/legacy/legacy_post.rb
class LegacyPost < LegacyBase
set_table_name "blog_posts"
def map
{
:name => self.headline.squish,
:body => self.body.squish
}
end
end

29. app/models/legacy/legacy_post.rb
class LegacyPost < LegacyBase
set_table_name "blog_posts"
def map
{
:name => tweak(self.headline.squish),
:body => self.body.squish
}
end
def tweak(name)
name.capitalize.gsub(/teh/, "the")
end

30. lib/tasks/legacy.rake
namespace :db do
namespace :migrate do
desc 'Migrates posts'
task :posts => :environment do
Trucker.migrate :posts
end
end
end

31. Let’s do some migrating.

32. Don’t forget to import
your legacy database!

33. $ rake db:migrate:posts
Migrating all posts (1/10)
Migrating all posts (2/10)
Migrating all posts (3/10)
Migrating all posts (4/10)
Migrating all posts (5/10)
Migrating all posts (6/10)
Migrating all posts (7/10)
Migrating all posts (8/10)
Migrating all posts (9/10)
Migrating all posts (10/10)

34. $ rake db:migrate:posts limit=5
Migrating 5 posts (1/10)
Migrating 5 posts (2/10)
Migrating 5 posts (3/10)
Migrating 5 posts (4/10)
Migrating 5 posts (5/10)

35. $ rake db:migrate:posts limit=5 offset=5
Migrating 5 posts after 5 (6/10)
Migrating 5 posts after 5 (7/10)
Migrating 5 posts after 5 (8/10)
Migrating 5 posts after 5 (9/10)
Migrating 5 posts after 5 (10/10)

36. DEMO

37. Use helper method for
custom migrations

38. namespace :db do
namespace :migrate do
desc 'Migrate pain_in_the_ass model'
task :pain_in_the_ass => :environment do
Trucker.migrate :pain_in_the_ass, :helper => pain_in_the_ass_migration
end
end
end
def pain_in_the_ass_migration
# Custom code goes here
end

39. What about?!?

40. Trucker helps you move.

41. But you still need to
pack your stuff up.

42. Some other things
to think about

43. Encoding issues

44. • MySQL issues:
“ALTER TABLE mytable
CONVERT TO CHARACTER
SET utf8;
• iconv

45. Java to the rescue?!?

46. JDBC Adapters vs
ActiveRecord

47. • There are about 12 adapters in
the wild for pure ActiveRecord
• The ActiveRecord adapter for
sqlserver runs on Win/Unix,
but Unix setup is complicated
• Text adapter does CSV, Tab,
and other plain text formats

50. JDBC Adapters

51. ActiveRecord Tested:
 MySQL  Derby
 PostgreSQL  HSQLDB
 Oracle  H2
 Microsoft SQL  SQLite3
Server  Informix
 DB2
 FireBird

52. MSSQL – An Example

53. • Download MSSQL driver
(Google for it, they move it
around a lot)
• Copy sqljdbc4.jar into
RAILS_ROOT/lib
• Add require 'lib/sqljdbc4.jar' at
the top of your environment.rb
or application.rb

54. config/database.yml
legacy:
adapter: jdbc
username: USERNAME
password: PASSWORD
driver: com.microsoft.sqlserver.jdbc.SQLServerDriver
jdbc: sqlserver://63.134.199.59:1433

55. Take it live

56. • Run in prod when you can
• Run locally when you can't
(Heroku, VPNs, etc)

57. Resources

58. • http://github.com/mokolabs/trucker
• http://github.com/mokolabs/
trucker_sample_app
• http://spkr8.com/t/3749

59. Special thanks
to Dave Thomas

60. The End