1. David Barshow
@barshow
barshow@dropbox.com

2. About
Early engineer at Orchestra, Inc.
Helped build and scale Mailbox
Currently operations tech lead for Mailbox

3. Talk
About Mailbox
Current architecture
Scaling: war stories
Learnings: solutions to scaling

7. Orchestra HQ Early 2011

9. Orchestra HQ Early 2012

10. What if we re-
invent the inbox
where tasks
already live?

17. Quick swipe gestures that make
managing email delightful

18. Defer messages until you’re
ready to respond

19. Auto-swipe messages away

20. Why have a server component
for a mail client, can’t it talk
directly to the server (IMAP)?

21. Syncing a inbox with 0 changes looking
for read / unread changes with IMAP

22. EHPE12 XLIST "" *
* XLIST (HasNoChildren) "/" "Ads"
* XLIST (HasNoChildren) "/" "Apple Mail To Do"
* XLIST (HasNoChildren) "/" "Archive"
* XLIST (HasNoChildren) "/" "Deleted Messages"
* XLIST (HasNoChildren) "/" "Done"
* XLIST (HasNoChildren) "/" "Drafts"
* XLIST (HasNoChildren) "/" "Facebook"
* XLIST (HasNoChildren Inbox) "/" "Inbox"
* XLIST (HasNoChildren) "/" "Junk E-mail"
* XLIST (HasNoChildren) "/" "Notes"
* XLIST (HasNoChildren) "/" "Sent"
* XLIST (HasNoChildren) "/" "Sent Messages"
* XLIST (HasNoChildren) "/" "Shipments"
* XLIST (HasNoChildren) "/" "Stores"
* XLIST (HasNoChildren) "/" "Trash"
* XLIST (Noselect HasChildren) "/" "[Gmail]"
* XLIST (HasNoChildren AllMail) "/" "[Gmail]/All Mail"
* XLIST (HasNoChildren Drafts) "/" "[Gmail]/Drafts"
* XLIST (HasNoChildren Important) "/" "[Gmail]/Important"
* XLIST (HasNoChildren) "/" "[Gmail]/Sent"
* XLIST (HasNoChildren Sent) "/" "[Gmail]/Sent Mail"
* XLIST (HasNoChildren Spam) "/" "[Gmail]/Spam"
* XLIST (HasNoChildren Starred) "/" "[Gmail]/Starred"
* XLIST (HasNoChildren) "/" "[Gmail]/Templates"
* XLIST (HasChildren Trash) "/" "[Gmail]/Trash"
* XLIST (HasNoChildren) "/" "[Gmail]/Trash/Apple Mail To Do"
* XLIST (HasNoChildren) "/" "[Gmail]/Trash/Deleted Messages"
* XLIST (HasChildren) "/" "[Mailbox]"
* XLIST (HasNoChildren) "/" "[Mailbox]/Boat"
* XLIST (HasNoChildren) "/" "[Mailbox]/Important"
* XLIST (HasNoChildren) "/" "[Mailbox]/In Boat"
* XLIST (HasNoChildren) "/" "[Mailbox]/Later"
* XLIST (HasNoChildren) "/" "[Mailbox]/Ops"
* XLIST (HasNoChildren) "/" "[Mailbox]/To Buy"
* XLIST (HasNoChildren) "/" "[Mailbox]/To Read"
* XLIST (HasNoChildren) "/" "[Mailbox]/To Watch"
EHPE12 OK Success
EHPE13 IDLE
+ idling
DONE
EHPE13 OK IDLE terminated (Success)
EHPE14 STATUS "[Gmail]/All Mail" (MESSAGES UNSEEN UIDNEXT)
EHPE15 STATUS "Inbox" (MESSAGES UNSEEN UIDNEXT)
EHPE16 STATUS "[Gmail]/Trash" (MESSAGES UNSEEN UIDNEXT)
EHPE17 STATUS "[Gmail]/Sent Mail" (MESSAGES UNSEEN UIDNEXT)
EHPE18 STATUS "[Mailbox]/Boat" (MESSAGES UNSEEN UIDNEXT)
EHPE19 STATUS "[Mailbox]/Important" (MESSAGES UNSEEN UIDNEXT)
EHPE20 STATUS "[Mailbox]/In Boat" (MESSAGES UNSEEN UIDNEXT)
EHPE21 STATUS "[Mailbox]/Later" (MESSAGES UNSEEN UIDNEXT)
EHPE22 STATUS "[Mailbox]/Ops" (MESSAGES UNSEEN UIDNEXT)
EHPE23 STATUS "[Mailbox]/To Buy" (MESSAGES UNSEEN UIDNEXT)
EHPE24 STATUS "[Mailbox]/To Read" (MESSAGES UNSEEN UIDNEXT)
EHPE25 STATUS "[Mailbox]/To Watch" (MESSAGES UNSEEN UIDNEXT)
* STATUS "[Gmail]/All Mail" (MESSAGES 71215 UIDNEXT 130633 UNSEEN 10038)
EHPE14 OK Success
* STATUS "Inbox" (MESSAGES 40 UIDNEXT 61644 UNSEEN 3)
EHPE15 OK Success
* STATUS "[Gmail]/Trash" (MESSAGES 5 UIDNEXT 9805 UNSEEN 0)
EHPE16 OK Success
* STATUS "[Gmail]/Sent Mail" (MESSAGES 3096 UIDNEXT 3155 UNSEEN 4)
EHPE17 OK Success
* STATUS "[Mailbox]/Boat" (MESSAGES 0 UIDNEXT 2 UNSEEN 0)
EHPE18 OK Success
* STATUS "[Mailbox]/Important" (MESSAGES 1 UIDNEXT 2 UNSEEN 0)
EHPE19 OK Success
* STATUS "[Mailbox]/In Boat" (MESSAGES 0 UIDNEXT 2 UNSEEN 0)
EHPE20 OK Success
* STATUS "[Mailbox]/Later" (MESSAGES 3 UIDNEXT 3490 UNSEEN 0)
EHPE21 OK Success
* STATUS "[Mailbox]/Ops" (MESSAGES 0 UIDNEXT 2 UNSEEN 0)
EHPE22 OK Success
* STATUS "[Mailbox]/To Buy" (MESSAGES 0 UIDNEXT 1 UNSEEN 0)
EHPE23 OK Success
* STATUS "[Mailbox]/To Read" (MESSAGES 0 UIDNEXT 3 UNSEEN 0)
EHPE24 OK Success
* STATUS "[Mailbox]/To Watch" (MESSAGES 0 UIDNEXT 1 UNSEEN 0)
EHPE25 OK Success
EHPE26 IDLE
+ idling
DONE
EHPE26 OK IDLE terminated (Success)
EHPE27 UID SEARCH (OR X-GM-MSGID 1469389077319771056 OR X-GM-MSGID 1469388522915173669 OR X-GM-MSGID 1469388213742478589 OR X-GM-MSGID 1469384978144379631 OR X-GM-MSGID 1469384963535310187 OR X-GM-MSGID 1469383698165135365 OR X-GM-MSGID 1469379320583718227 OR X-GM-MSGID 1469379173492779359 OR X-GM-MSGID
1469378854019826555 OR X-GM-MSGID 1469378866871990283 OR X-GM-MSGID 1469376496711137269 OR X-GM-MSGID 1469375115226023939 OR X-GM-MSGID 1469374816626104598 OR X-GM-MSGID 1469374587720191385 OR X-GM-MSGID 1469373484770299022 OR X-GM-MSGID 1469371278824000751 OR X-GM-MSGID 1469369463503544206 OR X-GM-MSGID
1469368296477183526 OR X-GM-MSGID 1469363178701887774 OR X-GM-MSGID 1469362693002971228 OR X-GM-MSGID 1469362256271961395 OR X-GM-MSGID 1469362137831918881 OR X-GM-MSGID 1469361012992020635 OR X-GM-MSGID 1469360363511391521 OR X-GM-MSGID 1469360133365467529 OR X-GM-MSGID 1469359023993889761 OR X-GM-MSGID
1469358604942033099 OR X-GM-MSGID 1469358564164497927 OR X-GM-MSGID 1469358496596344252 OR X-GM-MSGID 1469357133922826620 OR X-GM-MSGID 1469356721401555114 OR X-GM-MSGID 1469354017427494065 OR X-GM-MSGID 1469339159573607282 OR X-GM-MSGID 1469344546328026380 OR X-GM-MSGID 1469352998087252614 OR X-GM-MSGID
1469351200001914680 OR X-GM-MSGID 1469347639274682465 OR X-GM-MSGID 1469346995001325511 OR X-GM-MSGID 1468897649492533927 OR X-GM-MSGID 1468898065441874503 OR X-GM-MSGID 1468898195658626808 OR X-GM-MSGID 1469262431860803310 OR X-GM-MSGID 1469262537124458919 OR X-GM-MSGID 1469343951728738127 OR X-GM-MSGID
1469343945388847421 OR X-GM-MSGID 1460769988219897386 OR X-GM-MSGID 1460780952841734266 OR X-GM-MSGID 1469343836730370435 OR X-GM-MSGID 1469343837063795087 OR X-GM-MSGID 1469343752565365396 OR X-GM-MSGID 1461752797221770591 OR X-GM-MSGID 1461752891327548784 OR X-GM-MSGID 1469266412016200869 OR X-GM-MSGID
1469323353867013111 OR X-GM-MSGID 1469342659566467946 OR X-GM-MSGID 1469340835793306590 OR X-GM-MSGID 1469336371213479416 OR X-GM-MSGID 1469336453235925299 OR X-GM-MSGID 1469336477339104345 X-GM-MSGID 1469337311505966509)
* SEARCH 125214 125224 125810 125811 130294 130295 130296 130485 130486 130502 130553 130558 130559 130560 130562 130567 130574 130576 130577 130578 130579 130580 130581 130582 130584 130585 130587 130588 130589 130591 130592 130593 130594 130595 130596 130597 130598 130600 130601 130602 130603 130604
130606 130607 130610 130612 130613 130614 130615 130617 130618 130619 130621 130622 130623 130624 130625 130626 130627 130629
EHPE27 OK SEARCH completed (Success)
EHPE28 UID FETCH
125214,125224,125810,125811,130294,130295,130296,130485,130486,130502,130553,130558,130559,130560,130562,130567,130574,130576,130577,130578,130579,130580,130581,130582,130584,130585,130587,130588,130589,130591,130592,130593,130594,130595,130596,130597,130598,130600,130601,130602,130603,130604,130606,130607,
130610,130612,130613,130614,130615,130617,130618,130619,130621,130622,130623,130624,130625,130626,130627,130629 (FLAGS X-GM-LABELS UID X-GM-MSGID)
* 66552 FETCH (X-GM-MSGID 1460769988219897386 X-GM-LABELS () UID 125214 FLAGS (Seen))
* 66560 FETCH (X-GM-MSGID 1460780952841734266 X-GM-LABELS () UID 125224 FLAGS (Seen))
* 67017 FETCH (X-GM-MSGID 1461752797221770591 X-GM-LABELS () UID 125810 FLAGS (Seen))
* 67018 FETCH (X-GM-MSGID 1461752891327548784 X-GM-LABELS () UID 125811 FLAGS (Seen))
* 70908 FETCH (X-GM-MSGID 1468897649492533927 X-GM-LABELS () UID 130294 FLAGS (Seen))
* 70909 FETCH (X-GM-MSGID 1468898065441874503 X-GM-LABELS () UID 130295 FLAGS (Seen))
* 70910 FETCH (X-GM-MSGID 1468898195658626808 X-GM-LABELS () UID 130296 FLAGS (Seen))
* 71068 FETCH (X-GM-MSGID 1469262431860803310 X-GM-LABELS () UID 130485 FLAGS (Seen))
* 71069 FETCH (X-GM-MSGID 1469262537124458919 X-GM-LABELS () UID 130486 FLAGS (Seen))
* 71084 FETCH (X-GM-MSGID 1469266412016200869 X-GM-LABELS () UID 130502 FLAGS (Seen))
* 71132 FETCH (X-GM-MSGID 1469323353867013111 X-GM-LABELS () UID 130553 FLAGS (Seen))
* 71137 FETCH (X-GM-MSGID 1469336371213479416 X-GM-LABELS () UID 130558 FLAGS (Seen))
* 71138 FETCH (X-GM-MSGID 1469336453235925299 X-GM-LABELS () UID 130559 FLAGS (Seen))
* 71139 FETCH (X-GM-MSGID 1469336477339104345 X-GM-LABELS () UID 130560 FLAGS (Seen))
* 71140 FETCH (X-GM-MSGID 1469337311505966509 X-GM-LABELS () UID 130562 FLAGS (Seen))
* 71143 FETCH (X-GM-MSGID 1469339159573607282 X-GM-LABELS () UID 130567 FLAGS (Seen))
* 71148 FETCH (X-GM-MSGID 1469340835793306590 X-GM-LABELS ("Important") UID 130574 FLAGS (Seen))
* 71149 FETCH (X-GM-MSGID 1469342659566467946 X-GM-LABELS () UID 130576 FLAGS (Seen))
* 71150 FETCH (X-GM-MSGID 1469343752565365396 X-GM-LABELS () UID 130577 FLAGS (Seen))
* 71151 FETCH (X-GM-MSGID 1469343836730370435 X-GM-LABELS () UID 130578 FLAGS ())
* 71152 FETCH (X-GM-MSGID 1469343837063795087 X-GM-LABELS () UID 130579 FLAGS (Seen))
* 71153 FETCH (X-GM-MSGID 1469343945388847421 X-GM-LABELS () UID 130580 FLAGS (Seen))
* 71154 FETCH (X-GM-MSGID 1469343951728738127 X-GM-LABELS () UID 130581 FLAGS (Seen))
* 71155 FETCH (X-GM-MSGID 1469344546328026380 X-GM-LABELS () UID 130582 FLAGS (Seen))
* 71156 FETCH (X-GM-MSGID 1469346995001325511 X-GM-LABELS () UID 130584 FLAGS (Seen))
* 71157 FETCH (X-GM-MSGID 1469347639274682465 X-GM-LABELS () UID 130585 FLAGS (Seen))
* 71158 FETCH (X-GM-MSGID 1469351200001914680 X-GM-LABELS () UID 130587 FLAGS (Seen))
* 71159 FETCH (X-GM-MSGID 1469352998087252614 X-GM-LABELS () UID 130588 FLAGS (Seen))
* 71160 FETCH (X-GM-MSGID 1469354017427494065 X-GM-LABELS ("Important") UID 130589 FLAGS (Seen))
* 71161 FETCH (X-GM-MSGID 1469356721401555114 X-GM-LABELS ("Important") UID 130591 FLAGS (Seen))
* 71162 FETCH (X-GM-MSGID 1469357133922826620 X-GM-LABELS () UID 130592 FLAGS (Seen))
* 71163 FETCH (X-GM-MSGID 1469358496596344252 X-GM-LABELS ("Important") UID 130593 FLAGS (Seen))
* 71164 FETCH (X-GM-MSGID 1469358564164497927 X-GM-LABELS ("Important") UID 130594 FLAGS (Seen))
* 71165 FETCH (X-GM-MSGID 1469358604942033099 X-GM-LABELS ("Important") UID 130595 FLAGS (Seen))
* 71166 FETCH (X-GM-MSGID 1469359023993889761 X-GM-LABELS ("Important") UID 130596 FLAGS (Seen))
* 71167 FETCH (X-GM-MSGID 1469360133365467529 X-GM-LABELS ("Important") UID 130597 FLAGS (Seen))
* 71168 FETCH (X-GM-MSGID 1469360363511391521 X-GM-LABELS (Ads) UID 130598 FLAGS ())
* 71169 FETCH (X-GM-MSGID 1469361012992020635 X-GM-LABELS () UID 130600 FLAGS (Seen))
* 71170 FETCH (X-GM-MSGID 1469362137831918881 X-GM-LABELS ("Important") UID 130601 FLAGS (Seen))
* 71171 FETCH (X-GM-MSGID 1469362256271961395 X-GM-LABELS ("Important") UID 130602 FLAGS (Seen))
* 71172 FETCH (X-GM-MSGID 1469362693002971228 X-GM-LABELS ("Important") UID 130603 FLAGS (Seen))
* 71173 FETCH (X-GM-MSGID 1469363178701887774 X-GM-LABELS () UID 130604 FLAGS (Seen))
* 71174 FETCH (X-GM-MSGID 1469368296477183526 X-GM-LABELS () UID 130606 FLAGS (Seen))
* 71175 FETCH (X-GM-MSGID 1469369463503544206 X-GM-LABELS ("Important") UID 130607 FLAGS (Seen))
* 71177 FETCH (X-GM-MSGID 1469371278824000751 X-GM-LABELS ("Important") UID 130610 FLAGS (Seen))
* 71178 FETCH (X-GM-MSGID 1469373484770299022 X-GM-LABELS () UID 130612 FLAGS (Seen))
* 71179 FETCH (X-GM-MSGID 1469374587720191385 X-GM-LABELS ("Important") UID 130613 FLAGS (Seen))
* 71180 FETCH (X-GM-MSGID 1469374816626104598 X-GM-LABELS () UID 130614 FLAGS (Seen))
* 71181 FETCH (X-GM-MSGID 1469375115226023939 X-GM-LABELS () UID 130615 FLAGS (Seen))
* 71182 FETCH (X-GM-MSGID 1469376496711137269 X-GM-LABELS ("Important") UID 130617 FLAGS (Seen))
* 71183 FETCH (X-GM-MSGID 1469378854019826555 X-GM-LABELS ("Important") UID 130618 FLAGS (Seen))
* 71184 FETCH (X-GM-MSGID 1469378866871990283 X-GM-LABELS ("Important") UID 130619 FLAGS (Seen))
* 71186 FETCH (X-GM-MSGID 1469379173492779359 X-GM-LABELS ("Sent") UID 130621 FLAGS (Seen))
* 71187 FETCH (X-GM-MSGID 1469379320583718227 X-GM-LABELS () UID 130622 FLAGS (Seen))
* 71188 FETCH (X-GM-MSGID 1469383698165135365 X-GM-LABELS ("Important") UID 130623 FLAGS (Seen))
* 71189 FETCH (X-GM-MSGID 1469384963535310187 X-GM-LABELS ("Important") UID 130624 FLAGS (Seen))
* 71190 FETCH (X-GM-MSGID 1469384978144379631 X-GM-LABELS ("Important") UID 130625 FLAGS (Seen))
* 71191 FETCH (X-GM-MSGID 1469388213742478589 X-GM-LABELS () UID 130626 FLAGS (Seen))
* 71192 FETCH (X-GM-MSGID 1469388522915173669 X-GM-LABELS ("Important") UID 130627 FLAGS (Seen))
* 71193 FETCH (X-GM-MSGID 1469389077319771056 X-GM-LABELS () UID 130629 FLAGS (Seen))
EHPE28 OK Success
EHPE29 IDLE
+ idling
DONE
EHPE29 OK IDLE terminated (Success)
EHPE30 UID SEARCH (OR X-GM-MSGID 1469338849725941969 OR X-GM-MSGID 1469339657762199160 OR X-GM-MSGID 1469339608353734515 OR X-GM-MSGID 1469262350191527787 OR X-GM-MSGID 1469263235973306832 OR X-GM-MSGID 1469263521807612591 OR X-GM-MSGID 1469339203553539069 OR X-GM-MSGID 1469339204005384406 OR X-GM-MSGID
1469337706677163069 OR X-GM-MSGID 1469333798560376740 OR X-GM-MSGID 1469333306941213639 OR X-GM-MSGID 1469331759368840879 OR X-GM-MSGID 1469330163600515429 OR X-GM-MSGID 1469309326113238415 OR X-GM-MSGID 1469277171823234540 OR X-GM-MSGID 1469276590004048096 OR X-GM-MSGID 1469276461183894647 OR X-GM-MSGID
1469256406863677952 OR X-GM-MSGID 1469253343397587259 OR X-GM-MSGID 1469252067602142234 OR X-GM-MSGID 1469250171489558932 OR X-GM-MSGID 1469250159917973795 OR X-GM-MSGID 1469249091060277280 OR X-GM-MSGID 1469247588253846386 OR X-GM-MSGID 1469245585221124717 OR X-GM-MSGID 1469243083620163471 OR X-GM-MSGID
1469241390692167542 OR X-GM-MSGID 1469206474970171189 OR X-GM-MSGID 1469238735594350553 OR X-GM-MSGID 1469238527387112380 OR X-GM-MSGID 1469238305376710319 OR X-GM-MSGID 1469235792291555874 OR X-GM-MSGID 1469207274482603552 OR X-GM-MSGID 1469204075394088943 OR X-GM-MSGID 1469203631114028257 OR X-GM-MSGID
1469193340343004323 OR X-GM-MSGID 1469185601731282350 OR X-GM-MSGID 1468183290373869499 OR X-GM-MSGID 1468191822796905951 OR X-GM-MSGID 1468199207961802073 OR X-GM-MSGID 1469181508074615512 OR X-GM-MSGID 1469184300165403339 OR X-GM-MSGID 1469182783256241965 OR X-GM-MSGID 1441927701390957102 OR X-GM-MSGID
1441967844067269026 OR X-GM-MSGID 1469182299692417428 OR X-GM-MSGID 1469182300269402981 OR X-GM-MSGID 1450503757865770658 OR X-GM-MSGID 1469182225782057987 OR X-GM-MSGID 1469182226220105581 OR X-GM-MSGID 1469182131159530304 OR X-GM-MSGID 1469182131533786548 OR X-GM-MSGID 1431965482341585482 OR X-GM-MSGID
1459801415072667026 OR X-GM-MSGID 1459801789079791066 OR X-GM-MSGID 1469182125538239438 OR X-GM-MSGID 1469182125652888221 OR X-GM-MSGID 1469181833457331765 OR X-GM-MSGID 1460588018688506197 OR X-GM-MSGID 1469181753648051664 OR X-GM-MSGID 1469181753654918118 OR X-GM-MSGID 1462553780861977955 OR X-GM-MSGID
1469181705806022216 OR X-GM-MSGID 1469181705877145936 OR X-GM-MSGID 1469180993032339597 OR X-GM-MSGID 1469178103447113517 OR X-GM-MSGID 1469177473430852891 OR X-GM-MSGID 1469177447881255537 OR X-GM-MSGID 1469177113804924775 OR X-GM-MSGID 1469175204534544925 OR X-GM-MSGID 1469175060144621174 OR X-GM-MSGID
1469174995478559960 OR X-GM-MSGID 1469174822129808781 OR X-GM-MSGID 1469172862277216811 OR X-GM-MSGID 1469169678939938679 OR X-GM-MSGID 1469166635813710634 OR X-GM-MSGID 1469165832947257753 OR X-GM-MSGID 1469164554707302627 OR X-GM-MSGID 1469162748470736164 OR X-GM-MSGID 1469159452362839842 OR X-GM-MSGID
1469159417215626319 OR X-GM-MSGID 1469158460904334854 OR X-GM-MSGID 1469154649603975482 OR X-GM-MSGID 1469153526815850851 OR X-GM-MSGID 1469100136547144483 OR X-GM-MSGID 1467248592971472117 OR X-GM-MSGID 1467518822943225557 OR X-GM-MSGID 1467555385667315697 OR X-GM-MSGID 1467556396619328652 OR X-GM-MSGID
1467556612192450121 OR X-GM-MSGID 1467556612440168948 OR X-GM-MSGID 1467972421936751441 OR X-GM-MSGID 1467972550479309302 OR X-GM-MSGID 1468475198023064848 OR X-GM-MSGID 1468890674936050451 OR X-GM-MSGID 1468823605639116506 OR X-GM-MSGID 1467034568838158344 OR X-GM-MSGID 1466868326760276966 OR X-GM-MSGID
1463417324193870091 X-GM-MSGID 1469006683539827837)
* SEARCH 104403 111482 111517 118014 124605 124606 125110 126333 126834 129044 129148 129240 129454 129488 129490 129491 129492 129705 129706 129845 129852 129857 129990 130233 130287 130392 130413 130414 130415 130416 130417 130418 130420 130421 130422 130424 130426 130427 130428 130429 130430 130431
130432 130433 130434 130435 130436 130438 130439 130440 130441 130442 130443 130444 130445 130446 130447 130448 130449 130450 130451 130452 130453 130457 130458 130459 130460 130461 130466 130467 130468 130469 130470 130471 130473 130475 130476 130477 130478 130479 130481 130482 130484 130492 130495 130525
130526 130527 130548 130554 130555 130556 130557 130563 130564 130568 130569 130570 130571
EHPE30 OK SEARCH completed (Success)
EHPE31 UID FETCH
104403,111482,111517,118014,124605,124606,125110,126333,126834,129044,129148,129240,129454,129488,129490,129491,129492,129705,129706,129845,129852,129857,129990,130233,130287,130392,130413,130414,130415,130416,130417,130418,130420,130421,130422,130424,130426,130427,130428,130429,130430,130431,130432,130433,
130434,130435,130436,130438,130439,130440,130441,130442,130443,130444,130445,130446,130447,130448,130449,130450,130451,130452,130453,130457,130458,130459,130460,130461,130466,130467,130468,130469,130470,130471,130473,130475,130476,130477,130478,130479,130481,130482,130484,130492,130495,130525,130526,130527,
130548,130554,130555,130556,130557,130563,130564,130568,130569,130570,130571 (FLAGS X-GM-LABELS UID X-GM-MSGID)
* 50536 FETCH (X-GM-MSGID 1431965482341585482 X-GM-LABELS () UID 104403 FLAGS (Seen))
* 56623 FETCH (X-GM-MSGID 1441927701390957102 X-GM-LABELS () UID 111482 FLAGS (Seen))
* 56655 FETCH (X-GM-MSGID 1441967844067269026 X-GM-LABELS () UID 111517 FLAGS (Seen))
* 62127 FETCH (X-GM-MSGID 1450503757865770658 X-GM-LABELS () UID 118014 FLAGS ())
* 66061 FETCH (X-GM-MSGID 1459801415072667026 X-GM-LABELS () UID 124605 FLAGS (Seen))
* 66062 FETCH (X-GM-MSGID 1459801789079791066 X-GM-LABELS () UID 124606 FLAGS (Seen))
* 66473 FETCH (X-GM-MSGID 1460588018688506197 X-GM-LABELS () UID 125110 FLAGS (Seen))
* 67468 FETCH (X-GM-MSGID 1462553780861977955 X-GM-LABELS () UID 126333 FLAGS (Seen))
* 67905 FETCH (X-GM-MSGID 1463417324193870091 X-GM-LABELS ("Important" [Mailbox]/Important) UID 126834 FLAGS (Seen))
* 69821 FETCH (X-GM-MSGID 1466868326760276966 X-GM-LABELS ([Mailbox]/Later "Important") UID 129044 FLAGS (Seen))
* 69918 FETCH (X-GM-MSGID 1467034568838158344 X-GM-LABELS ([Mailbox]/Later "Important") UID 129148 FLAGS (Seen))
* 69996 FETCH (X-GM-MSGID 1467248592971472117 X-GM-LABELS ("Important") UID 129240 FLAGS (Seen))
* 70189 FETCH (X-GM-MSGID 1467518822943225557 X-GM-LABELS ("Important") UID 129454 FLAGS (Seen))
* 70220 FETCH (X-GM-MSGID 1467555385667315697 X-GM-LABELS ("Important") UID 129488 FLAGS (Seen))
* 70222 FETCH (X-GM-MSGID 1467556396619328652 X-GM-LABELS ("Important") UID 129490 FLAGS (Seen))
* 70223 FETCH (X-GM-MSGID 1467556612192450121 X-GM-LABELS ("Important") UID 129491 FLAGS (Seen))
* 70224 FETCH (X-GM-MSGID 1467556612440168948 X-GM-LABELS ("Important") UID 129492 FLAGS (Seen))
* 70407 FETCH (X-GM-MSGID 1467972421936751441 X-GM-LABELS ("Important") UID 129705 FLAGS (Seen))
* 70408 FETCH (X-GM-MSGID 1467972550479309302 X-GM-LABELS ("Important") UID 129706 FLAGS (Seen))
* 70518 FETCH (X-GM-MSGID 1468183290373869499 X-GM-LABELS () UID 129845 FLAGS (Seen))
* 70525 FETCH (X-GM-MSGID 1468191822796905951 X-GM-LABELS () UID 129852 FLAGS (Seen))
* 70529 FETCH (X-GM-MSGID 1468199207961802073 X-GM-LABELS () UID 129857 FLAGS (Seen))
* 70645 FETCH (X-GM-MSGID 1468475198023064848 X-GM-LABELS ("Important") UID 129990 FLAGS (Seen))
* 70860 FETCH (X-GM-MSGID 1468823605639116506 X-GM-LABELS ("Important") UID 130233 FLAGS (Seen))
* 70901 FETCH (X-GM-MSGID 1468890674936050451 X-GM-LABELS ("Important") UID 130287 FLAGS (Seen))
* 70987 FETCH (X-GM-MSGID 1469100136547144483 X-GM-LABELS ([Mailbox]/Later "Important") UID 130392 FLAGS (Seen))
* 71005 FETCH (X-GM-MSGID 1469153526815850851 X-GM-LABELS () UID 130413 FLAGS (Seen))
* 71006 FETCH (X-GM-MSGID 1469154649603975482 X-GM-LABELS () UID 130414 FLAGS (Seen))
* 71007 FETCH (X-GM-MSGID 1469158460904334854 X-GM-LABELS () UID 130415 FLAGS (Seen))
* 71008 FETCH (X-GM-MSGID 1469159417215626319 X-GM-LABELS ("Important") UID 130416 FLAGS (Seen))
* 71009 FETCH (X-GM-MSGID 1469159452362839842 X-GM-LABELS ("Important") UID 130417 FLAGS (Seen))
* 71010 FETCH (X-GM-MSGID 1469162748470736164 X-GM-LABELS () UID 130418 FLAGS (Seen))
* 71012 FETCH (X-GM-MSGID 1469164554707302627 X-GM-LABELS () UID 130420 FLAGS (Seen))
* 71013 FETCH (X-GM-MSGID 1469165832947257753 X-GM-LABELS () UID 130421 FLAGS (Seen))
* 71014 FETCH (X-GM-MSGID 1469166635813710634 X-GM-LABELS () UID 130422 FLAGS (Seen))
* 71015 FETCH (X-GM-MSGID 1469169678939938679 X-GM-LABELS () UID 130424 FLAGS (Seen))
* 71016 FETCH (X-GM-MSGID 1469172862277216811 X-GM-LABELS () UID 130426 FLAGS (Seen))
* 71017 FETCH (X-GM-MSGID 1469174822129808781 X-GM-LABELS ("Important") UID 130427 FLAGS (Seen))
* 71018 FETCH (X-GM-MSGID 1469174995478559960 X-GM-LABELS ("Important") UID 130428 FLAGS (Seen))
* 71019 FETCH (X-GM-MSGID 1469175060144621174 X-GM-LABELS (Stores) UID 130429 FLAGS ())
* 71020 FETCH (X-GM-MSGID 1469175204534544925 X-GM-LABELS () UID 130430 FLAGS (Seen))
* 71021 FETCH (X-GM-MSGID 1469177113804924775 X-GM-LABELS () UID 130431 FLAGS (Seen))
* 71022 FETCH (X-GM-MSGID 1469177447881255537 X-GM-LABELS ("Important") UID 130432 FLAGS (Seen))
* 71023 FETCH (X-GM-MSGID 1469177473430852891 X-GM-LABELS () UID 130433 FLAGS (Seen))
* 71024 FETCH (X-GM-MSGID 1469178103447113517 X-GM-LABELS () UID 130434 FLAGS (Seen))
* 71025 FETCH (X-GM-MSGID 1469180993032339597 X-GM-LABELS ("Important") UID 130435 FLAGS (Seen))
* 71026 FETCH (X-GM-MSGID 1469181508074615512 X-GM-LABELS () UID 130436 FLAGS (Seen))
* 71028 FETCH (X-GM-MSGID 1469181705806022216 X-GM-LABELS () UID 130438 FLAGS (Seen))
* 71029 FETCH (X-GM-MSGID 1469181705877145936 X-GM-LABELS () UID 130439 FLAGS ())
* 71030 FETCH (X-GM-MSGID 1469181753654918118 X-GM-LABELS () UID 130440 FLAGS (Seen))
* 71031 FETCH (X-GM-MSGID 1469181753648051664 X-GM-LABELS () UID 130441 FLAGS ())
* 71032 FETCH (X-GM-MSGID 1469181833457331765 X-GM-LABELS ("Important") UID 130442 FLAGS (Seen))
* 71033 FETCH (X-GM-MSGID 1469182125652888221 X-GM-LABELS () UID 130443 FLAGS ())
* 71034 FETCH (X-GM-MSGID 1469182125538239438 X-GM-LABELS () UID 130444 FLAGS (Seen))
* 71035 FETCH (X-GM-MSGID 1469182131159530304 X-GM-LABELS () UID 130445 FLAGS (Seen))
* 71036 FETCH (X-GM-MSGID 1469182131533786548 X-GM-LABELS () UID 130446 FLAGS ())
* 71037 FETCH (X-GM-MSGID 1469182225782057987 X-GM-LABELS () UID 130447 FLAGS (Seen))
* 71038 FETCH (X-GM-MSGID 1469182226220105581 X-GM-LABELS () UID 130448 FLAGS ())
* 71039 FETCH (X-GM-MSGID 1469182299692417428 X-GM-LABELS () UID 130449 FLAGS (Seen))
* 71040 FETCH (X-GM-MSGID 1469182300269402981 X-GM-LABELS () UID 130450 FLAGS ())
* 71041 FETCH (X-GM-MSGID 1469182783256241965 X-GM-LABELS () UID 130451 FLAGS (Seen))
* 71042 FETCH (X-GM-MSGID 1469184300165403339 X-GM-LABELS () UID 130452 FLAGS (Seen))
* 71043 FETCH (X-GM-MSGID 1469185601731282350 X-GM-LABELS ("Important") UID 130453 FLAGS (Seen))
* 71045 FETCH (X-GM-MSGID 1469193340343004323 X-GM-LABELS ("Important") UID 130457 FLAGS (Seen))
* 71046 FETCH (X-GM-MSGID 1469203631114028257 X-GM-LABELS ("Important") UID 130458 FLAGS (Seen))
* 71047 FETCH (X-GM-MSGID 1469204075394088943 X-GM-LABELS (Facebook) UID 130459 FLAGS ())
* 71048 FETCH (X-GM-MSGID 1469206474970171189 X-GM-LABELS ("Important") UID 130460 FLAGS (Seen))
* 71049 FETCH (X-GM-MSGID 1469207274482603552 X-GM-LABELS () UID 130461 FLAGS (Seen))
* 71051 FETCH (X-GM-MSGID 1469235792291555874 X-GM-LABELS () UID 130466 FLAGS (Seen))
* 71052 FETCH (X-GM-MSGID 1469238305376710319 X-GM-LABELS () UID 130467 FLAGS (Seen))
* 71053 FETCH (X-GM-MSGID 1469238527387112380 X-GM-LABELS (Ads) UID 130468 FLAGS ())
* 71054 FETCH (X-GM-MSGID 1469238735594350553 X-GM-LABELS ("Important") UID 130469 FLAGS (Seen))
* 71055 FETCH (X-GM-MSGID 1469241390692167542 X-GM-LABELS () UID 130470 FLAGS (Seen))
* 71056 FETCH (X-GM-MSGID 1469243083620163471 X-GM-LABELS () UID 130471 FLAGS (Seen))
* 71058 FETCH (X-GM-MSGID 1469245585221124717 X-GM-LABELS () UID 130473 FLAGS (Seen))
* 71060 FETCH (X-GM-MSGID 1469247588253846386 X-GM-LABELS () UID 130475 FLAGS (Seen))
* 71061 FETCH (X-GM-MSGID 1469249091060277280 X-GM-LABELS () UID 130476 FLAGS (Seen))
* 71062 FETCH (X-GM-MSGID 1469250159917973795 X-GM-LABELS () UID 130477 FLAGS (Seen))
* 71063 FETCH (X-GM-MSGID 1469250171489558932 X-GM-LABELS () UID 130478 FLAGS (Seen))
* 71064 FETCH (X-GM-MSGID 1469252067602142234 X-GM-LABELS () UID 130479 FLAGS (Seen))
* 71065 FETCH (X-GM-MSGID 1469253343397587259 X-GM-LABELS () UID 130481 FLAGS (Seen))
* 71066 FETCH (X-GM-MSGID 1469256406863677952 X-GM-LABELS () UID 130482 FLAGS (Seen))
* 71067 FETCH (X-GM-MSGID 1469262350191527787 X-GM-LABELS () UID 130484 FLAGS (Seen))
* 71075 FETCH (X-GM-MSGID 1469263235973306832 X-GM-LABELS () UID 130492 FLAGS (Seen))
* 71078 FETCH (X-GM-MSGID 1469263521807612591 X-GM-LABELS () UID 130495 FLAGS (Seen))
* 71106 FETCH (X-GM-MSGID 1469276461183894647 X-GM-LABELS ("Important") UID 130525 FLAGS (Seen))
* 71107 FETCH (X-GM-MSGID 1469276590004048096 X-GM-LABELS ("Important") UID 130526 FLAGS (Seen))
* 71108 FETCH (X-GM-MSGID 1469277171823234540 X-GM-LABELS ("Important") UID 130527 FLAGS (Seen))
* 71128 FETCH (X-GM-MSGID 1469309326113238415 X-GM-LABELS ("Important" "Sent") UID 130548 FLAGS (Seen))
* 71133 FETCH (X-GM-MSGID 1469330163600515429 X-GM-LABELS () UID 130554 FLAGS (Seen))
* 71134 FETCH (X-GM-MSGID 1469331759368840879 X-GM-LABELS (Facebook) UID 130555 FLAGS ())
* 71135 FETCH (X-GM-MSGID 1469333306941213639 X-GM-LABELS () UID 130556 FLAGS (Seen))
* 71136 FETCH (X-GM-MSGID 1469333798560376740 X-GM-LABELS ("Important") UID 130557 FLAGS (Seen))
* 71141 FETCH (X-GM-MSGID 1469337706677163069 X-GM-LABELS () UID 130563 FLAGS (Seen))
* 71142 FETCH (X-GM-MSGID 1469338849725941969 X-GM-LABELS () UID 130564 FLAGS (Seen))
* 71144 FETCH (X-GM-MSGID 1469339204005384406 X-GM-LABELS () UID 130568 FLAGS (Seen))
* 71145 FETCH (X-GM-MSGID 1469339203553539069 X-GM-LABELS () UID 130569 FLAGS (Seen))
* 71146 FETCH (X-GM-MSGID 1469339608353734515 X-GM-LABELS ("Important") UID 130570 FLAGS (Seen))
* 71147 FETCH (X-GM-MSGID 1469339657762199160 X-GM-LABELS () UID 130571 FLAGS (Seen))
EHPE31 OK Success
EHPE32 IDLE
+ idling
DONE
EHPE32 OK IDLE terminated (Success)
EHPE33 SELECT "[Gmail]/Trash"
* FLAGS (Answered Flagged Draft Deleted Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk)
* OK [PERMANENTFLAGS (Answered Flagged Draft Deleted Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk *)] Flags permitted.
* OK [UIDVALIDITY 602077766] UIDs valid.
* 5 EXISTS
* 0 RECENT
* OK [UIDNEXT 9805] Predicted next UID.
* OK [HIGHESTMODSEQ 6709154]
EHPE33 OK [READ-WRITE] [Gmail]/Trash selected. (Success)
EHPE34 UID SEARCH (X-GM-MSGID 1469006683539827837)
* SEARCH 9804
EHPE34 OK SEARCH completed (Success)
EHPE35 SELECT "[Gmail]/All Mail"
* FLAGS (Answered Flagged Draft Deleted Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk)
* OK [PERMANENTFLAGS (Answered Flagged Draft Deleted Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk *)] Flags permitted.
* OK [UIDVALIDITY 602077785] UIDs valid.
* 71196 EXISTS
* 0 RECENT
* OK [UIDNEXT 130633] Predicted next UID.
* OK [HIGHESTMODSEQ 6709154]
EHPE35 OK [READ-WRITE] [Gmail]/All Mail selected. (Success)
EHPE36 SELECT "[Gmail]/Trash"
* FLAGS (Answered Flagged Draft Deleted Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk)
* OK [PERMANENTFLAGS (Answered Flagged Draft Deleted Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk *)] Flags permitted.
* OK [UIDVALIDITY 602077766] UIDs valid.
* 5 EXISTS
* 0 RECENT
* OK [UIDNEXT 9805] Predicted next UID.
* OK [HIGHESTMODSEQ 6709154]
EHPE36 OK [READ-WRITE] [Gmail]/Trash selected. (Success)
EHPE37 UID FETCH 9804 (FLAGS X-GM-LABELS UID X-GM-MSGID)
* 5 FETCH (X-GM-MSGID 1469006683539827837 X-GM-LABELS ("Important") UID 9804 FLAGS (Seen))
EHPE37 OK Success
EHPE38 SELECT "[Gmail]/All Mail"
* FLAGS (Answered Flagged Draft Deleted Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk)
* OK [PERMANENTFLAGS (Answered Flagged Draft Deleted Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk *)] Flags permitted.
* OK [UIDVALIDITY 602077785] UIDs valid.
* 71196 EXISTS
* 0 RECENT
* OK [UIDNEXT 130633] Predicted next UID.
* OK [HIGHESTMODSEQ 6709154]
EHPE38 OK [READ-WRITE] [Gmail]/All Mail selected. (Success)
EHPE39 UID SEARCH (OR X-GM-MSGID 1469006996691250206 OR X-GM-MSGID 1468566332287868339 OR X-GM-MSGID 1468052215868306821 X-GM-MSGID 1466845749668929332)
* SEARCH
EHPE39 OK SEARCH completed (Success)
EHPE40 IDLE
+ idling
DONE
EHPE40 OK IDLE terminated (Success)
EHPE41 SELECT "[Gmail]/Trash"
* FLAGS (Answered Flagged Draft Deleted Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk)
* OK [PERMANENTFLAGS (Answered Flagged Draft Deleted Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk *)] Flags permitted.
* OK [UIDVALIDITY 602077766] UIDs valid.
* 5 EXISTS
* 0 RECENT
* OK [UIDNEXT 9805] Predicted next UID.
* OK [HIGHESTMODSEQ 6709154]
EHPE41 OK [READ-WRITE] [Gmail]/Trash selected. (Success)
EHPE42 UID SEARCH (OR X-GM-MSGID 1469006996691250206 OR X-GM-MSGID 1468566332287868339 OR X-GM-MSGID 1468052215868306821 X-GM-MSGID 1466845749668929332)
* SEARCH 9800 9801 9802 9803
EHPE42 OK SEARCH completed (Success)
EHPE43 SELECT "[Gmail]/All Mail"
* FLAGS (Answered Flagged Draft Deleted Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk)
* OK [PERMANENTFLAGS (Answered Flagged Draft Deleted Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk *)] Flags permitted.
* OK [UIDVALIDITY 602077785] UIDs valid.
* 71196 EXISTS
* 0 RECENT
* OK [UIDNEXT 130633] Predicted next UID.
* OK [HIGHESTMODSEQ 6709154]
EHPE43 OK [READ-WRITE] [Gmail]/All Mail selected. (Success)
EHPE44 SELECT "[Gmail]/Trash"
* FLAGS (Answered Flagged Draft Deleted Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk)
* OK [PERMANENTFLAGS (Answered Flagged Draft Deleted Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk *)] Flags permitted.
* OK [UIDVALIDITY 602077766] UIDs valid.
* 5 EXISTS
* 0 RECENT
* OK [UIDNEXT 9805] Predicted next UID.
* OK [HIGHESTMODSEQ 6709154]
EHPE44 OK [READ-WRITE] [Gmail]/Trash selected. (Success)
EHPE45 UID FETCH 9800,9801,9802,9803 (FLAGS X-GM-LABELS UID X-GM-MSGID)
* 1 FETCH (X-GM-MSGID 1466845749668929332 X-GM-LABELS ("Important") UID 9800 FLAGS (Seen))
* 2 FETCH (X-GM-MSGID 1468052215868306821 X-GM-LABELS () UID 9801 FLAGS (Seen))
* 3 FETCH (X-GM-MSGID 1468566332287868339 X-GM-LABELS () UID 9802 FLAGS (Seen))
* 4 FETCH (X-GM-MSGID 1469006996691250206 X-GM-LABELS ("Important") UID 9803 FLAGS (Seen))
EHPE45 OK Success
EHPE46 SELECT "[Gmail]/All Mail"
* FLAGS (Answered Flagged Draft Deleted Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk)
* OK [PERMANENTFLAGS (Answered Flagged Draft Deleted Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk *)] Flags permitted.
* OK [UIDVALIDITY 602077785] UIDs valid.
* 71196 EXISTS
* 0 RECENT
* OK [UIDNEXT 130633] Predicted next UID.
* OK [HIGHESTMODSEQ 6709154]
EHPE46 OK [READ-WRITE] [Gmail]/All Mail selected. (Success)
EHPE47 IDLE
+ idling

23. STATUS
STATUS
STATUS
STATUS
UIDSEARCH
UIDSEARCH
UIDSEARCH
UIDSEARCH
UIDFETCH
UIDFETCH
UIDFETCH
UIDFETCH
CAPABILITY AUTHENTICATE
IMAP protocol getting in sync!
with no changes
XLIST STATUS UIDSEARCH UIDFETCH
CAPABILITY OK XLIST
STATUS
STATUS
STATUS
STATUS
UIDSEARCH
UIDSEARCH
UIDSEARCH
UIDSEARCH
UIDFETCH
UIDFETCH
UIDFETCH
UIDFETCH
~8,000 ms & 32,247 bytes
IMAP
XLIST
XLIST
XLIST

24. We decided to do all
IMAP syncing from the cloud.
IMAP

26. Mailbox protocol updating read/unread
flags with no changes

27. {"method":"USER_AUTH_DEVICE","params":{"user_id":"-3371520013043391617","device_id":"7D2B12B9-24E2-42DF-9937-5F9C5AD792C4","secret":"imasupersecretkeythatwontworkyouhackers"}}
{“id":25,"method":"ACCOUNT_SNAPSHOT","params":{"account_id":"614590AC-38F8-1CDD-91A4-4057B20B13FC"}}
{“type”:”NOTIFICATION","method":"USER_STATE","params":{"user_id":"-3371520013043391617","device_id":"7D2B12B9-24E2-42DF-9937-5F9C5AD792C4","accounts":["614590AC-38F8-1CDD-91A4-4057B20B13FC"],"preferences":{"auto_swipe_enabled":true,"badge_type":1,"notif_type":2,"reminder_push":true,"push":true},"push_valid":false}}
{“id”:25,"result":{"docs":{}, "account_id":"614590AC-38F8-1CDD-91A4-4057B20B13FC" ,”sync_state”:{"list":1401328862805,"emails":1401328862805,"items":1401328862805}},"type":"ACCOUNT_SNAPSHOT"}

28. {"method":"AUTH_DEVICE","params":
{"user_id":"-3371520013043391617","device_id":"7D2B12B9-24E2-42DF-993
7-5F9C5AD792C4","secret":"imasupersecretkeythatwontworkyouhackers"}}
!
{“id":25,"method":"ACCOUNT_SNAPSHOT","params":
{"account_id":"614590AC-38F8-1CDD-91A4-4057B20B13FC"}}
!
{“type”:”NOTIFICATION","method":"STATE","params":
{"user_id":"-3371520013043391617","device_id":"7D2B12B9-4E2-2DF-9937-
5F9C5AD792C4","accounts":["614590C-38F8-
CDD-91A4-4057B20B13FC"],"preferences":
{"auto_swipe_enabled":true,"badge_type":1,"notif_type":
2,"reminder_push":true,"push":true},"push_valid":false}}
!
{“id”:25,"result":{"docs":{},
"account_id":"614590AC-38F8-1CDD-91A4-4057B20B13FC" ,”sync_state”:
{"list":1401328862805,"emails":1401328862805,"items":
1401328862805}},"type":"ACCOUNT_SNAPSHOT"}

29. Device
Authentication
Snapshot
Request
Snapshot
Response
Mailbox protocol getting in sync with no changes
~100ms & 2,051 bytes
Auth Response

30. Mailbox Architecture!
what it looks like in practice

31. connector!
processes
streaming
layer
queues
worker!
process
provider!
queues
provider!
worker
gmail
application!
db
provider!
db
email cache!
db
mongos!
27017
mongos!
27017
mongos
27018

32. connector!
processes
streaming
layer
queues
worker!
process
provider!
queues
provider!
worker
gmail
application!
db
provider!
db
email cache!
db
mongos!
27017
mongos!
27017
mongos
27018

33. connector!
processes
streaming
layer
queues
worker!
process
provider!
queues
provider!
worker
gmail
application!
db
provider!
db
email cache!
db
mongos!
27017
mongos!
27017
mongos
27018

34. connector!
processes
streaming
layer
queues
worker!
process
provider!
queues
provider!
worker
gmail
application!
db
provider!
db
email cache!
db
mongos!
27017
mongos!
27017
mongos
27018

35. connector!
processes
streaming
layer
queues
worker!
process
provider!
queues
provider!
worker
gmail
application!
db
provider!
db
email cache!
db
mongos!
27017
mongos!
27017
mongos
27018

36. connector!
processes
streaming
layer
queues
worker!
process
provider!
queues
provider!
worker
gmail
application!
db
provider!
db
email cache!
db
mongos!
27017
mongos!
27017
mongos
27018

37. connector!
processes
streaming
layer
queues
worker!
process
provider!
queues
provider!
worker
gmail
application!
db
provider!
db
email cache!
db
mongos!
27017
mongos!
27017
mongos
27018

38. A new email message
from the provider

39. connector!
processes
streaming
layer
queues
worker!
process
provider!
queues
provider!
worker
gmail
application!
db
provider!
db
email cache!
db
mongos!
27017
mongos!
27017
mongos
27018

40. connector!
processes
streaming
layer
queues
worker!
process
provider!
queues
provider!
worker
gmail
application!
db
provider!
db
email cache!
db
mongos!
27017
mongos!
27017
mongos
27018

41. connector!
processes
streaming
layer
queues
worker!
process
provider!
queues
provider!
worker
gmail
application!
db
provider!
db
email cache!
db
mongos!
27017
mongos!
27017
mongos
27018

42. Scaling!
with a small team

43. New users already have email

44. We have to make
those emails
available
0M
50M
100M
emails / day
Feb 2013 March 2013

45. Scaling war stories
Logging full (disk) after index missing
Ephemeral ports
Background indexes, block secondaries
Running queries on primaries accidentally ( mapreduce )

46. Scaling war stories
Logging full (disk) after index missing
Ephemeral ports
Background indexes, block secondaries
Running queries on primaries accidentally ( mapreduce )

47. Log volume filling up and missing
index
Everything seemed like a queuing problem ( slow deliveries )
Had profiling level setup for slow queries but didn't look
Replica’s log volume full ( of slow query messages )
Didn't have disk usage monitoring
Missing index for all pushes, data grew out of working set

48. Scaling war stories
Logging full (disk) after index missing
Ephemeral ports
Background indexes, block secondaries
Running queries on primaries accidentally ( mapreduce )

49. Ephemeral Ports
Our workers would connected to mongos would timeout on all
queries
mongos would log “config servers not in sync! no config
servers reachable”
Rebooting mongos fixed the problem

50. #!/bin/bash
echo "`date -u` [mongos_watcher] Starting to monitor our mongos.log file for config server reachability" >>
mongos_watcher.log
!
# for infinity, we will check the logs for mongo to make sure all is good
tail -fn1 /var/log/mongodb/mongos.log | while read line ; do
echo "$line" | grep "ERROR: config servers not in sync! no config servers reachable"
if [ $? = 0 ]
then
echo "`date -u` [mongos_watcher] FOUND CONFIG SERVER ERROR: BOUNCING MONGOS" >>
mongos_watcher.log
sudo service mongos restart
fi
done

51. Ephemeral Ports
Our workers would connected to mongos would timeout on all
queries
mongos would log “config servers not in sync! no config
servers reachable”
Rebooting mongos fixed the problem
Firewall rules blocked tcp reconnects on ephemeral
ports 32768 - 65535

52. Scaling war stories
Logging full (disk) after index missing
Ephemeral ports
Background indexes, block secondaries
Running queries on primaries accidentally ( mapreduce )

53. Index builds
Background index builds, only in the background jobs on the
primaries ( until mongo 2.6 )
With secondary reads enabled, this would make things crawl to
a halt.
SecondaryPreferred doesn’t mean try for a little while on a
secondary, it will try on a secondary if it exists, and wait until it
can acquire a lock.

54. Scaling war stories
Logging full (disk) after index missing
Ephemeral ports
Background indexes, block secondaries
Running queries on primaries accidentally

55. Manual queries in the console
map reduces are good at taking a whole database down if not
properly sent to only secondaries
any non indexed query can do the same
people move fast and with lots of terminals open can make
mistakes, local vs production

57. Learnings!
what came from our scaling challenges

58. Systems
Readonly shell ( Accountability )
Multi-shard
LVM snapshots
Compaction
Hydra -> why and what, moving to separate clusters

59. Systems
Readonly shell ( Accountability )
Multi-shard
LVM snapshots
Compaction
Hydra -> why and what, moving to separate clusters

60. Readonly Shell
Mongo cli can be dangerous
dropDatabase() … just seeing this makes me nervous
“use mailbux", actually would create the database mailbux
Accountability for mongo actions -> syslog -> hipchat
Read-only shell script, unless you sudo.

62. Systems
Readonly shell ( Accountability )
Multi-shard
LVM snapshots
Compaction
Hydra -> why and what, moving to separate clusters

63. Multi-shard on AWS EC2
hi.4xl & i2.8xl not even touching the cpu, or disk limits
Write locking is relieved by adding more shards
Boxes have multiple disks and software raiding is slow
Allows masters to be spread across boxes
Setting up the cluster is harder

64. disk 1
disk 2
disk 3
disk 4
disk 5
disk 6
disk 7
disk 8
software raid!
mdadm!
!
/srv/db
mongod!
27017
Traditional multi disk mongod setup

65. Software raid on EC2
(i2.8xlarge dd if=/dev/zero of=ddfile bs=1024k)
8 SSD
RAID STRIPE
80GB 77.908s 1.02GB/s
8SSD
INDIVIDUAL
10GBx8 25.116s 3.18GB/s

66. disk 1
disk 2
disk 3
disk 4
disk 5
disk 6
disk 7
disk 8
Multi-shard multi-disk mongod
mongod 27017
mongod 27018
mongod 27019
mongod 27020
mongod 27021
mongod 27022
mongod 27023
mongod 27024

67. shard 1
shard 2
shard 3
shard 4
shard 5
shard 6
shard 7
shard 8
shard 1
shard 2
shard 3
shard 4
shard 5
shard 6
shard 7
shard 8
shard 1
shard 2
shard 3
shard 4
shard 5
shard 6
shard 7
shard 8
AZ1 AZ2 AZ3
PRIMARY SECONDARY

68. Scale with multi-shards
50k ops per second on a single replica set (w:2)
less load per box, because primaries can be spread
around

69. Systems
Readonly shell ( Accountability )
Multi-shard
LVM snapshots
Compaction
Hydra -> why and what, moving to separate clusters

70. LVM & Backup
LVM (logical volume management)
Allows snapshotting of active volumes by keeping a COW table
(Copy-On-Write)
Has a performance impact, especially when a snapshot exists
mongodump doesn’t work at scale, snapshotting does
Like local EBS snapshots

71. shard 1
shard 2
shard 3
shard 4
shard 5
shard 6
shard 7
shard 8
shard 1
shard 2
shard 3
shard 4
shard 5
shard 6
shard 7
shard 8
shard 1
shard 2
shard 3
shard 4
shard 5
shard 6
shard 7
shard 8
AZ1 AZ2 AZ3 - LVMS
PRIMARY SECONDARY LOW PRIORITY SECONDARY

72. Systems
Readonly shell ( Accountability )
Multi-shard
LVM snapshots
Compaction
Hydra -> why and what, moving to separate clusters

73. Compaction with multi-shards
With lots of deletes on a write heavy database, compaction is a
must
You can not compact a primary so you have to step down
You want masters to be spread across hosts so you must
compact the replica you want to be master first

74. shard 1
shard 2
shard 3
shard 4
shard 5
shard 6
shard 7
shard 8
shard 1
shard 2
shard 3
shard 4
shard 5
shard 6
shard 7
shard 8
shard 1
shard 2
shard 3
shard 4
shard 5
shard 6
shard 7
shard 8
AZ1 AZ2 AZ3 - LVMS
PRIMARY SECONDARY LOW PRIORITY SECONDARY

75. THIRD
shard 2
shard 3
shard 4
shard 5
shard 6
shard 7
shard 8
FIRST
shard 2
shard 3
shard 4
shard 5
shard 6
shard 7
shard 8
SECOND
shard 2
shard 3
shard 4
shard 5
shard 6
shard 7
shard 8
AZ1 AZ2 AZ3 - LVMS
PRIMARY SECONDARY LOW PRIORITY SECONDARY
SECONDARY PRIMARY LOW PRIORITY SECONDARY

76. Systems
Readonly shell ( Accountability )
Multi-shard
LVM snapshots
Compaction
Hydra -> why and what, moving to separate clusters

77. Hydra: collection migration tool
Built to relieve stress on our primary application db
Moved emails out of our primary application cluster
Giant improvements in write lock

79. Lessons Learned
Monitor monitor monitor
nosql doesn't mean no management, investigate all the things!
Scaling is still hard, testing and research are still required

82. David Barshow
@barshow
barshow@dropbox.com