Performance Testing ISV Apps to Scale 11/9/2016

1. How To Performance
Test Your Apps to
Scale
​ Vydianath Iyer Jaswinder Rattanpal

2. Vydianath Iyer
Principal Technical Evangelist
Salesforce.com
viyer@salesforce.com
/in/vydianath
Jaswdiner Rattanpal
Associate Technical Evangelist
Salesforce.com
jrattanpal@salesforce.com
/in/jrattanpal
2

3. Safe harbor statement under the Private Securities Litigation Reform Act of 1995:
This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties
materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results
expressed or implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed
forward-looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items
and any statements regarding strategies or plans of management for future operations, statements of belief, any statements concerning
new, planned, or upgraded services or technology developments and customer contracts or use of our services.
The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new
functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our
operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of any
litigation, risks associated with completed and any possible mergers and acquisitions, the immature market in which we operate, our
relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our
service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger
enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com, inc. is included in
our annual report on Form 10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter.
These documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section
of our Web site.
Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available
and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features
that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements.
Safe Harbor
3

4. Agenda
1
2
3
4
5
Scale Testing - What, Why & How
How to get a Large Data Volume (LDV) Sandbox Org
Tools and Techniques
Demo
Partner Resources & Related Offerings
6 Q & A
4

5. What is Scale Testing ?
5

6. What is Scale Testing (Salesforce) ?
Testing for Large Data Volumes (LDV) Testing for Large Concurrent
User Volumes
6

7. Test for realistic customer sizes
​ Large Data Loads
​ Large User Loads (Concurrent)
API
Platform Stress Testing
Platform Capacity Testing
Network Latency Testing
Testing 3rd Party Integrations
Server Availability/Failover Test
Web
IN SCOPE OUT OF SCOPE
What’s the scope?
7

8. Why Test ?
8

9. Increasing application complexity
Increased customer expectations
Minimize downtimes and disruption
Competitive Differentiator
User productivity & Satisfaction
Better ROI
Happy Customers = More Business.
Why Test for Performance ?
9

10. Where do we start ?
10

11. https://goo.gl/2Rq8Yf
Fill up the Data & API Consumption spreadsheet
Rules of thumb
Do significant objects have more than a few million records ?
Do you anticipate a thousand concurrent users or more ?
Seek your TE’s advise:
Help determine the scope of required testing
Help with tools, best practices and processes
To begin with...
11

12. High Level Steps - Test in your Developer Org
1 Single User Test in a Developer Org Pre-Requisite
Why is step 1 a pre-requisite ?
● Is your application designed well ?
● Does it follow Salesforce coding best practices ?
● Triggers Bulkified, Selective Queries, View State minimized ?
Max 10 concurrent users in a Developer Org
Prepare
2
12

13. 3
4
5
Single User Test
Performance/Load Testing - 100s of concurrent users
Test with 10 Concurrent Users
Critical Steps
Know the rules
High Level Steps - Test in a Large Data Volume (LDV) Org
13

14. • Open up a case 2 weeks before start of testing
• Mimic normal realistic “thinking time”
• Concurrent user test loads should be realistic
• Salesforce does not support stress testing
• Test only against sandbox instances
TESTING LOAD EXAMPLE:
5K users * 5 processes/ hour = 25 K processes /hour
1 business process = 3 transactions
25K processes/hour = 75K transactions/hour
Equates to 20 transactions per second (TPS)
Average load< 40 TPS
Moderately heavy load 40-100 TPS
Extremely Heavy load> 100 TPS
Salesforce will not provide back-end logs or results
Step 4 - Rules & Requirements
14

15. ​ Number of Atomic database actions* performed by a certain entity per second
Page DB Save DB SaveTrigger WF update DB Save
JS Query
JS
Page
DML
Single Transaction
Transaction 1
Transaction 2
What is a Transaction (Force.com context) ?
​ * Atomic action = guaranteed isolation from concurrent processes
15

16. How to get a LDV Org ?
16

17. LDV Org Options
17
​ Less than 100 GB { <50 million records }
​ Max 2 Orgs per calendar year
​ Max 60 days
​ Open up a case
​ Provisioned in 14 biz days
​ FREE !!!
​ Greater than 100 GB { >50 million records }
​ Unlimited number
​ Renewed Yearly (Can be cancelled/reduced)
​ Open up a case, Contact PAM (if you have)
​ Provisioned in 14-21 biz days
​ $96/GB/YEAR
FREE LDV ORG PAID LDV ORG

18. Tools & Techniques
18

19. Grinder Gatling Tsung JMeter **
OS Any Any Linux/Unix Any
GUI Console Only Recorder Only No Full
Test Recorder TCP (including HTTP) HTTP HTTP, Postgres HTTP
Test Language Python, Clojure Scala XML XML
Extension Language
Python, Clojure Scala
Erlang
Java, Beanshell, Javascript, Jexl
Load Reports Console HTML HTML
CSV, XML, Embedded Tables,
Graphs, Plugins
Protocols
HTTP
SOAP
JDBC
POP3
SMTP
LDAP
JMS
HTTP
JDBC
JMS
HTTP
WebDAV
Postgres
MySQL
XMPP
WebSocket
AMQP
MQTT
LDAP
HTTP
FTP
JDBC
SOAP
LDAP
TCP
JMS
SMTP
POP3
IMAP
Host Monitoring No No No Yes with PerfMon plugin
Miscellaneous
Python knowledge required,
reports are basic
Limited support for protocols,
Scala knowledge required
Supported only on Linux Reports are basic
Performance Test Tools Matrix (free open source)
Tool
Feature
** All sample scripts in the partner community chatter group are based on JMeter 19

20. 20
A Few Performance Analysis tools
Lightning Inspector Plugin (Chrome)
• Google Chrome Dev Tools extension
• Navigate component tree, inspect components and the DOM elements
• Monitor and Modify server responses which help in debugging server interactions
• Inspect component attributes, and profile component performance.
• Understand the sequence of event firing and handling
Apex Timeline
• SOQL queries used in the business transaction along with timings
• DMLs and objects and the cumulative time taken to update/insert/upsert the object
• A visual representation of the overall flow of events
• A JSON representation of the debug log for easier readability.
• Use the tooling API and automatically display the logs in your org
Web Page Test
• Open source tool developed and supported by Google
• Used to test a website performance
• Perform a visual comparison of urls and also against industry pages

21. ​
Some Other Tools ..
Gatling – Load Testing
Test Monitoring
Test Reporting
Event Log Analysis
21

22. 22
A Few Free Bulk Data Load Tools
Bulk Data Load Tool
• Reference architecture for creating mock data for Salesforce orgs using Ruby on Rails.
• Generates mock records and associations for any S object
• Uses native Rails functionality to generate mock data
• Demo: https://partners.salesforce.com/0693A000005q6XR
Salesforce Foundation data creation
• Test data generation and deployment ETL jobs
Demo Data/Metadata creator
• Heroku tool that uses Bulk API and MDAPI to create Data and Metadata

23. JMeter Caveats - LEX
23

24. Demo
24

25. Analyzing
Results
25

26. 26

27. 27

28. ​
BlazeMeter

29. 29

30. 30

31. Want to know more …
ISV Partner Resources
Partner Community Page - http://p.force.com/scaletesting
JMeter Scripts (Lightning and Classic)
Free Bulk Data Load Tools
Demo Orgs and Videos
LDV Org request process
Best Practices for Performance Optimization - https://goo.gl/fUk7SE
Partner Community Chatter Group - https://goo.gl/HPUlxX
Talk to your TE/
PAM if you are an
existing Partner
Navigate to the Partner
Community Chatter Group
http://p.force.com/scaletesting
31

32. Performance
Hammer
32

33. https://goo.gl/8NLNPr
Performance Hammer in a Nutshell ..
• Point in Time Performance Testing between Salesforce Releases
• Prevent Performance Regression
Partner Release
Salesforce Release
• Same version of package deployed in 2 different release sandboxes
• Partner provides JMeter Scripts to run tests
• Salesforce provides results (and stores summary of results)
33
FREE for App
Innovation Partners

34. Parting thoughts…
34

35. Key to Success
◆ Performance Starts with Design
◆ Implement Best Practices Early
◆ Optimize Single User Transactions
◆ Loop back Customer Behavior patterns
◆ Collaborate with your Salesforce Team
Incorporate Performance Testing in your App Life Cycle
35

36. Q & A
36