Slide deck from my talk at the SDN Conference in Amsterdam in Fall 2009

1. Nik Kalyani DotNetNuke Corporation Windows Azure: Building a Twitter Clone Session Code: NE.09

3. Technology Entrepreneur, 15 years

5. Twitter: @techbubble

7. Agenda Azure Architecture Storage Concepts Micro-blogging Application Requirements Architecture Processing

8. Azure Architecture Your Applications … ServiceBus Workflow Database Analytics Identity Contacts AccessControl … Reporting … Devices … Compute Storage Manage …

9. Azure Application Architecture n m Worker Role Web Role LB Cloud Storage (blob, table, queue)

10. Azure Durable Storage Durable, scalable, available store Simple interface REST ADO.NET Data Services Simple abstractions Blobs Tables Queues …

11. Azure Data Storage Concepts Container Blobs http://<account>.blob.core.windows.net/<container> Table Entities Account http://<account>.table.core.windows.net/<table> Queue Messages http://<account>.queue.core.windows.net/<queue>

12. Block Blob Container Account Blob Storage Concepts IMG001.JPG pictures IMG002.JPG erica Block 1 movies MOV1.AVI Block 2 Block 3

13. Blobs: Features and Functions Store large objects Associate metadata with blob Metadata is <name, value> pairs Set/Get with or separate from blob data bits Standard REST Interface Limitations Block Id - 64 Bytes Per Block PutBlob - 64 MB Total Blob MetaData - 8 KB Per Blob PutBlock / PutBlockList - 50 GB Each Blocks - 4 MB Each

14. Table Storage Concepts Entity Table Account Name =… Email = … users Name =… Email = … erica Photo ID =… Date =… photo index Photo ID =… Date =…

15. Tables: Features and Functions Massively scalable, highly available and durable structured storage Not RDMS -- no joins, no maintenance of foreign keys LINQ and REST interfaces via ADO.Net Data Services Partition key is exposed to the application Limitations Up to 255 Entity Properties (including PartitionKey, RowKey, and Timestamp) 1 MB Max for Combined Size of All Data in an Entity's Properties

16. Queue Storage Concepts Message Queue Account 128x128, http://… thumbnail jobs 256x256, http://… erica http://… photo processing jobs http://…

17. Queues: Features and Functions An account can create many queues.. Queue Name is scoped by the account A Queue contains messages.. No limit on number of messages stored in a queue A message is stored for at most a week in a queue http://<Account>.queue.core.windows.net/<QueueName> Messages.. Message size <= 8 KB To store larger data, store data in blob/entity storage, and the blob/entity name in the message

18. Micro-blogging ApplicationRequirements Relationships Any user can “follow” any other user A user can “unfollow” another user he/she is “following”

19. Micro-blogging ApplicationRequirements Micro-blogging Any user can post a message of up to 140 characters in length All message posts are public

20. Micro-blogging ApplicationRequirements Timeline A public timeline of all posted messages is available to anybody for up to 24 hours from the time the message was first posted A user timeline of all messages posted by a user is available to anybody forever A “following” timeline of all messages posted by all users that a user is following in the prior 24 hours is available to each user

21. Micro-blogging ApplicationArchitectural Considerations Micro-blogging The number of users will determine volume of messages However, common situation is that a small number of users will post majority of the messages

22. Micro-blogging ApplicationArchitectural Considerations Timelines The public timeline is transient and will have a high frequency of reads and writes A user timeline is permanent and will have a small number of writes and a large number of reads A “following” timeline is transient and will have a large number of irregularly timed writes and a small number of reads

23. Micro-blogging ApplicationProfile, Messages and Relationships Storage Table = User PartitionKey = Username RowKey=string.Format("{0:D19}", DateTime.MaxValue.Ticks – DateTime.UtcNow.Ticks); Use Table storage sharded by user Entities with the same PartitionKey are stored on the same storage node Obtaining recent messages, followers or following list require only a single query for each Adding or Deleting Follower or Following requires two writes (i.e. one for each user)

24. Micro-blogging ApplicationFollow User B Partition User A follows User B Following [User D] Message Queue Maintenance Queue User C Partition User A Partition Follower [User Y] Follower [User Z] Following [User A] Following [User X] Follower [User C]

25. Micro-blogging ApplicationFollow User B Partition Follower [User A] Following [User D] Message Queue Maintenance Queue User C Partition User A Partition Follower [User Y] Follower [User Z] Following [User A] Following [User X] Follower [User C] Following [User B]

26. Micro-blogging ApplicationFollow User B Partition User B follows User A Following [User D] Follower [User A] Message Queue Maintenance Queue User C Partition User A Partition Follower [User Y] Follower [User Z] Following [User A] Following [User X] Follower [User C] Following [User B]

27. Micro-blogging ApplicationFollow User B Partition Follower [User A] Following [User A] Following [User D] Message Queue Maintenance Queue User C Partition User A Partition Follower [User Y] Follower [User Z] Following [User A] Following [User X] Follower [User B] Follower [User C] Following [User B]

28. Micro-blogging ApplicationMessage Posting User B Partition User A posts a message Follower [User A] Following [User A] Following [User D] Message Queue Maintenance Queue User C Partition User A Partition Follower [User Y] Follower [User Z] Following [User A] Following [User X] Follower [User B] Follower [User C] Following [User B]

29. Micro-blogging ApplicationMessage Posting User B Partition Message added to Message Queue Follower [User A] Following [User A] Following [User D] Message Queue User A Message Maintenance Queue User C Partition User A Partition Follower [User Y] Follower [User Z] Following [User A] Following [User X] Follower [User B] Follower [User C] Following [User B]

30. Micro-blogging ApplicationMessage Posting User B Partition After Message Worker Role processes… Follower [User A] Following [User A] Following [User D] Message Queue Msg (A) Maintenance Queue User B User C User C Partition User A Partition Follower [User Y] Follower [User Z] Following [User A] Following [User X] Follower [User B] Follower [User C] Following [User B] Msg (A) Msg (A)

31. Micro-blogging ApplicationMessage Posting User B Partition After Maintenance Worker Role processes… Follower [User A] Following [User A] Following [User D] Message Queue Msg (A) Maintenance Queue User C Partition User A Partition Follower [User Y] Follower [User Z] Following [User A] Following [User X] Follower [User B] Follower [User C] Following [User B] Msg (A) Msg (A)

32. Micro-blogging ApplicationMessage Maintenance User B Partition User A posts a message Follower [User A] Following [User A] Following [User D] Message Queue Msg (A) Msg (D) Msg (D) 25 hours Maintenance Queue User C Partition User A Partition Follower [User Y] Follower [User Z] Following [User A] Following [User X] Follower [User B] Follower [User C] Following [User B] Msg (A) Msg (X) Msg (A)

33. Micro-blogging ApplicationMessage Maintenance User B Partition Message added to Message Queue Follower [User A] Following [User A] Following [User D] Message Queue User A Message Msg (A) Msg (D) Msg (D) 25 hours Maintenance Queue User C Partition User A Partition Follower [User Y] Follower [User Z] Following [User A] Following [User X] Follower [User B] Follower [User C] Following [User B] Msg (A) Msg (X) Msg (A)

34. Micro-blogging ApplicationMessage Maintenance User B Partition After Message Worker Role processes… Follower [User A] Following [User A] Following [User D] Message Queue Msg (A) Msg (D) Msg (D) 25 hours Msg (A) Maintenance Queue User B User C User C Partition User A Partition Follower [User Y] Follower [User Z] Following [User A] Following [User X] Follower [User B] Follower [User C] Following [User B] Msg (A) Msg (X) Msg (A) Msg (A) Msg (A)

35. Micro-blogging ApplicationMessage Maintenance User B Partition After Maintenance Worker Role processes… Follower [User A] Following [User A] Following [User D] Message Queue Msg (A) Msg (D) Msg (A) Maintenance Queue User C Partition User A Partition Follower [User Y] Follower [User Z] Following [User A] Following [User X] Follower [User B] Follower [User C] Following [User B] Msg (A) Msg (X) Msg (A) Msg (A) Msg (A)

36. Micro-blogging ApplicationTakeaways Cloud app storage significantly different from RDMS…requires shift in thinking Queues make apps efficient … processing is done asynchronously Duplication of data is often required and this is OK Sharding is an important consideration for the scalability of your cloud application Absence of join, sort make application design challenging but with planning this is manageable

37. Evaluation form Vul je evaluatieformulier in en maak kans op een van de prachtige prijzen!! Fill out your evaluation form and win one of the great prizes!! Session Code: NE.09