PayPal under the hood
•
1 recomendación•939 vistas
This is a peek at PayPal’s inner workings. Take a broad look at the technologies and foundations of the PayPal system, and the history and evolution behind its design. Take a tour of PayPal’s service-oriented architecture, the techniques the engineering team uses to achieve such secure financial transaction processing, and how PayPal innovates at scale.
Recomendados
Más contenido relacionado
Similar a PayPal under the hood
Similar a PayPal under the hood (20)
Más de X.commerce
Más de X.commerce (20)
Último
Último (20)
PayPal under the hood
- 3. “Lose a penny and everybody goes %#!’ing crazy!”
- 4. Introduction to PayPal technology Challenges & solutions Design Concepts
- 7. Technology
- 9. web/API C++/Java/Python Linux services C++/Java Linux permanent transient “mayfly” store Oracle store (database) IBM/AIX, Sun Linux
- 12. web API payment auth wallet processing
- 13. foreign risk exchange fees models limits receiving prefs. refunds IPN item info
- 14. Challenges/Solution s • reliability • security • scale
- 16. app 1 work 2 work 3 COMMIT Redo log standby offsite failover
- 17. app A 1 work 2 work 3 insert message 5 read message app B 4 COMMIT 6 if message isn’t “done” 7 work 8 mark message as “done” 9 COMMIT
- 18. 5 sync call app A 1 work 2 work 3 insert message 10 read message app B 4 COMMIT 6 if message isn’t “done” 7 work 8 mark message as “done” 9 COMMIT
- 20. physical security machine access controls firewalls service-level access control encryption on the wire encryption at rest hardware encryption (HSM)
- 21. balance log balance: $100 Account Activity Oct 1 open: $0 Oct 12 Add Funds $15 Oct 12 +$150 from Bank 0 Oct 13 - $50 Oct 13 Balance- $50 funded Payment
- 23. 2002 2011 payments/sec tens hundreds SLOC ~1M ~10M applications tens hundreds developers tens thousands release LTS monthly daily complexity some lots
- 25. server child process
- 26. server server server machine machine machine read read replica replica
- 27. app 1 app 2 app 3 app 4 database
- 31. app 1 app 2 app 3 app 4 database
- 32. app 1 app 2 app 3 app 4 cs risk transaction profile s
- 34. app 1 app 2 app 3 app 4 US EU profile txn profile txn risk cs risk cs
- 36. Account Servicing Financial Product Mobile Risk Compliance Help Money Security Consumer History Money Market Shipping Customer Service Incentive Notification Skype™ Dropbox Infrastructure Payflow Trinity™ FinOps/Recon Marketing Platform User FinSys Marketplaces Presentation Web Financial Instrument Merchant
- 37. app_A: app A: CONNECTS_TO app_B DEPENDS_ON foo.a CONNECTS_TO app_C library foo.a: … INCLUDES foo.o DEPENDS_ON bar.a ONLY_WITH payment_app
- 38. Design Concepts
- 41. reliable orchestration customer 1 customer 2 eventually consistent
- 42. payment processing payment customer processing balance recovery idem- potent
- 44. Wrap-up
- 45. “Lose a penny and everybody goes %#!’ing crazy!” “It’s all about the [transactions], [baby]”
- 46. Life Beyond Distributed Transactions The PayPal Wars
- 47. Thanks! Q&A
Notas del editor
- former engineercaptures our philosophy
- what paypal tech doesI picked some problemsideas influencing devsampler platetechnical, dense12:30-1:15 Thurs 10/13 Room 2018(40m, 5m Q&A)
- startuprelease nights, stayed up late, ate eggrolls, and crossed fingers that the site wouldn’t crashdb pw
- daughter, (new reason to stay up all night)still eat eggrolls for release, but now it happens during the day (don’t sleep under desks)even if I wanted, customer data with 10 foot polewhat’s inside?
- early engineerswere smart, motivated - take over the worldno problem unsolveable, no technique off-limitsinnovation (GL – impl of captcha)huge copy/paste, “magic” communication
- voila! inside kind of looks like this.
- spend money on reliable data storehttpd & geronimo
- 1998 ecosystemfounding culture of “we can build it better” – different nowC++ ecosystem is weak compared to what you see in Java, Pythonwhat does all this stuff do?
- trend: API box covering more
- tech problems, our solutions
- 3 themes for problems
- why do we consider this “reliable” at this point?redo log for raw datastandby – fast failoveroffsite – recover from disasterwhat about pieces in payments that have to work together
- lots of different systems involved in fulfilling a payment, working together reliablyif you wonder about delay
- ensure that a payment reaches the end stateinfra technique used many domainsas a business that deals with money, how do we build trust that nothing fishy is going on?
- trustworthy!how do you prevent or and detect tampering?
- responsibleexamples in the industrypreso: Bill Corry info secwhat about people that are allowed to touch these things?
- Two ways to answer the same questionchain of comparisons ultimately takes you to border between PP and external financial systemor penny-slicing, like insuperman III?round-off slicing doesn’t really apply – PP is in the middle of fxtxns, and round-off would have to be a txn
- tricky word because it can apply to a lot of things
- Does your codebase have “room”?tech organization that experienced huge, continuous growth, all of these dims have had “scaling” challenges
- payment processing capacitymem managementzombie boxes in/out rotationpush (connections)less eff, but SIMPLE to debug/operateput on read-only instances of DB’shorizontal scaling, indefiniteisolate problems of state-mgt
- our strategy for scaling readsauthentication, customer historybut what about state?
- monster box, could take 128 CPUs, started with 48
- too much work sync between cpuswork not totally independent (indices, etc.)
- business functionpartition by domain, independent machinesone machine lot of CPUs couldn’t do it
- dependencies!points of failure!work gets to be too big for one?
- don’t need all partitions to serve requestlocalize customer data to an in-country datacenterwhat about work spanning users (later)
- hiring more peoplelogisticsgetting people into a roommake room, keep small scope
- domainsexercise at first forced us to define “what paypalis”given these buckets, next question was how to ensure ensure dependencies don’t just turn it into a black holesmall scope, SOAbut something else that’s proved effective is…
- files that specify dependenciestopology and securitymaven has thistools that tell us, constrain, what talks to whatenforces boundaries – keeps things apartbut how do you get composition without coupling?
- some principles that we’re working with, to ensure scales and is reliablesum it up in one phrase
- ACID propertiesdo work, spanning lots of systems, together, in a way that ensures a consistent outcomeincomplete financial activityhow to classify “work” or transactions?
- how to keep these transactions consistent at scale?first two are easy – replicas, work on single partitions
- entities are customersother happens eventuallydecouples how they get their job done, contentionconsistent state change across entitiescloud storage APIs – this constraint in txnswhat’s the difficulty with this model?
- RPC/http/soapreliable, trustworthy - “unknown” is the worst possible answer!all systems with RPC-style interaction, with remote stateas you partition state, more places where this can happencoding this is a mess – have infra help you
- constraints, freedoms of: data access, RPC, memory/process model. your public APIs.Constraints define what happen when you scalewe underestimated the weight/cost of this, and didn’t invest enough early in engineering solutionsimplications can’t be completely hidden from app; rely on infrayour core competency, differentiator
- motivating designconstantly looking for ways to push this into the infrastructures.t.devs don’t have to worry about scale, throughput, reliability, correctness – our differentiator
- good read because:-short, with lots of pictures; appear intelligent without having to read a library- covers fundamental issues in large-scale, reliable, distributed systemsweakened ACIDPP Wars Eric Jackson, beginnings, war stories