Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

EDU 2.0 - Migrating to CloudSearch

999 visualizaciones

Publicado el

  • Sé el primero en comentar

EDU 2.0 - Migrating to CloudSearch

  1. 1. EDU 2.0Migrating to CloudSearch Graham Glass, Founder
  2. 2. Background● Cloud-hosted E-learning Platform (LMS)● for academia● for businesses● 1,000,000+ users● 15,000 new users a week● Customers include Disney, Large Universities, California School Districts, small Kindergartens, etc.
  3. 3. Search● Originally using Sphinx/ThinkingSphinx● All data came from RDS/MySQL● One Sphinx per app server● Full indexing once a day● Noticeable slowdown during indexing● Sphinx daemons would sometimes fail
  4. 4. CloudSearch● Decided to move to CloudSearch● Simple, scaleable● No Sphinx servers to manage● Reliable, fast, delta-indexing● Easy to index anything, including DynamoDB
  5. 5. Migration● 14 different types were being indexed● Decided to index just one item for testing● Use script to upload initial contents● Then index everything except for high volume items (messages, postings), which were migrated last of all.● Finally, index messages and postings
  6. 6. Configuration● Two search domains, one for each site● 20 index fields (only 2 text)● Truncate messages/postings to 1000 bytes
  7. 7. Rails Integration● Used aws_cloud_search gem● Added hooks into object model to add search update records to database● Separate workers update search every 15 minutes with records from database● Had issues with XML characters
  8. 8. Example Hooksdef after_create super update_searchenddef update_search if ((type = material_class).searchable? && (scope != None)) SearchUpdate.add(type, material_id) endend
  9. 9. Example DB update def self.add(type, ids) begin search_update = =>, :ids =>(ids.kind_of?(Array) ? ids.join(,) : ids), :operation => Update)! rescue Exception => exception puts "SearchUpdate.add exception: #{exception.message}" end end
  10. 10. End Result● Migrated all of search in about two weeks, spending 1-2 hours a day.● 7,000,000 documents, m2.xlarge● 500,000 documents, m1.small● Simplified architecture, positioned for scalability and DynamoDB● Only downside: $500/month for search