Testing for Success: How to Infuse Consistent Testing Into Your Fundraising Programs

Testing for Success –
How to Infuse Consistent Testing into Your
Fundraising Programs
Donna Arriaga
Senior Director
Digital Nonprofit Strategy
PMX Agency
Copyright © 2018 PMX Agency, LLC. All rights reserved. This information is deemed proprietary and confidential. Unauthorized use or disclosure is prohibited.

Goals for the Session:
 Understand the benefits of using a testing methodology vs
one-off “experiments”.
 Learn to identify common testing pitfalls and how to avoid
them.
 Understand how to use a testing framework to drive
learnings for performance optimization and iterative testing
strategies.
 Expand your testing toolbox with resources you can take
home today and start using tomorrow.

But first…What’s your testing expertise?
I’m on a roll, but could
always use some new
tips/tools
I have a robust, well-
developed testing
program
I’m just getting
started
???
???
???

How many feel like they….?
We already have a
robust testing
methodology and
framework in place
We have a process, but
it could use some
refinement
Methodology...what?
???
???
???

What are
the benefits
of testing?

Top-Level Benefits of Testing
Continually Improve the
Supporter Experience
Ongoing Performance
Optimization
and Iterative Learning
Mitigate Risk/Quantify
Potential Gain

Resist the temptation to use testing if….
 There’s not enough traffic or your
segment sizes are too small
 You’re focused on one-off tests with
limited future-facing applications
 The primary aim is to settle team
disagreements
 You haven’t created a testing plan yet
 There’s low risk and high anticipated
benefit of taking action right away

What is a test?

New Tactics versus Controlled Experiment
 “Exploratory Test”: New tactic, comparing performance against
historicals.
 Controlled Experiment: A/B/n test or multi-variant test; compare test
variants against the control.
Multi-Variant Test
A/B Test

Testing Methodology

Testing should be a strategic effort guided by a pre-defined,
ongoing process and methodology
It doesn’t have to look like this…. But it cannot look like this!

Spaghetti Testing…Don’t Do it!
It’s messy. It lacks rigor. And it doesn’t add to an
iterative knowledge base.

If You are Spaghetti Testing, Turn it into Something Meaningful
Look for the bigger “What” and “Why”

Testing Process & Methodology

Remember when….

1
SETUP &
DISCOVERY
Identify goals,
business
problems and
KPIs.
2
QUANTITATIVE
ANALYSIS
Identify patterns
and areas of
opportunity.
3
QUALITATIVE
ANALYSIS
Understand the
users’ needs.
5
HYPOTHESIS
FORMATION
This is the
basis for the
test. Include
the targeted
variable,
predicted
outcome, and
rationale.
6
TEST
PLANNING
Prioritize and
set up
experiments.
7
ASSET
CREATION
Design and
copy
variations to
be tested.
8
TESTING &
REPORTING
Run split test
with live
traffic.
Analyze and
report
results.
4– IDEAS TO TEST
Keep a running list of test ideas.
REPORT NO-BRAINERS
Fix things that are clearly broken.
Testing Methodology at PMX // Conversion Rate Optimization
(CRO) Process

Breaking Down the Methodology

Testing Methodology Breakdown
#1: SETUP & DISCOVERY
 Identify goals, business problems, and KPIs?
What are we trying to achieve and solve for?

#1: Examples of Business Problems & KPIs
Examples of KPIs
• Open Rate
• CTR
• On Page Conversion
• Average Gift
• Cost per Acquisition or Cost per
Donation
• Lead Quality Score
• Donor Retention
Examples of Business Problems
• Reduction in YoY Retention Rate
• Revenue is down due to average gift (conversion is up)
• Revenue is down due to conversion (average gift is up)
• Revenue is down due to lower conversion and average gift
• Cost to raise a dollar (CTRAD) is higher than the goal threshold;
need to reduce CTRAD
• New donor acquisition is down YoY.
• Lead-to-donor conversion is down.
• File has shifted to more lower-dollar, low-retention donors;
need to acquire and retain more higher-dollar donors.

#2&3: RESEARCH & ANALYSIS
 Collect and analyze qualitative and
quantitative data.
Quantitative Data
 Web Analytics
 Donation Platform
 Email marketing, social, display,
SEM, etc. platforms (depending
upon test)
Qualitative data
 Surveys
 Heat maps
 Stakeholder interviews
 Focus groups

#4: IDEAS TO TEST
 Develop ideas to solve the problem.
 Testing ideas are based on questions you have of the research and analysis. Why
does the current pattern persist? What's needed to change the pattern/behavior?
 Remember to test into micro-conversions AND macro-conversions.
 Why is it useful to brainstorm more than one
possible solution?
 The questions you ask of the data and your ideas to solve the
problem provide a foundation of testing options that feed into
a larger strategy.

#5: CREATE HYPOTHESIS
 Your hypothesis helps you define exactly what you aim to learn from the
experiment. Without a hypothesis and a clear learning goal, you are left with
results that have minimal, if any forward-facing application.
 Without a hypothesis, you effectively reduce the top 3 benefits of testing, down
to only risk mitigation.
 Continually improve the supporter experience.
 Ongoing performance optimization and iterative learning.
 Mitigate risk/quantify potential gain.

#5: How to formulate a strong hypothesis
If [variable], then [result] due to [rationale].
Image Source: Optimizely

#5: Examples:
If [variable], then [result] due to [rationale].
1. Donation Form Submit Button Test (controlled experiment)
If we change the donation form submit button copy to “send my gift” instead of “All of my
information is correct. Process my donation please.”, we can increase conversion due to reduced
friction and reduced user anxiety (e.g. by removing doubt-triggering copy such as “my info is correct.”)
2. Donor Stewardship (exploratory test)
If we deliver high-quality, regularly scheduled donor stewardship messages, then we can increase
donor retention due to an improved donor experience.

#6: TEST PLANNING
 Prioritize Experiments – Determine which tests to run asap, which to
hold onto for later testing, and which ideas to just implement.
 Implement (don’t test) ‘no-brainers’ such as broken elements and low-risk ideas
with high anticipated benefit.
 Define test dependencies and conflicts.
 Define experiments by isolating variables and KPIs;
these should already be reflected in your hypothesis.
 Estimate effort and return using a framework like an
ICE score (Impact, Confidence, Ease).

#6: TEST PLANNING
 Using an ICE score to prioritize tests:
Assign a number value from 1-10 to each of the below categories:
 Impact: How much of an impact to I expect this test to have?
 Confidence: How confident am I that the test will lead to an improvement?
 Ease: How easy is this test to set up?
The resulting score gives you a relative level of prioritization.
“Think of the ICE Score as a minimum viable prioritization
framework. It’s not objectively perfect but it’s good enough to get the
job done.” – Growth Hackers
ICE Score

#6: TEST PLANNING
 Caution: Do NOT run too many test variants.
 The more variants you run, the higher your probability of declaring one of
those variants a winner when it actually is not. (Type I Error, cumulative alpha
error)
 Important: Determine estimates for the test duration and sample
size BEFORE implementing the test.
 Determining your sample size and test duration in ADVANCE helps ensure you
will collect enough data points to confidently make predictions/conclusions
based on the results.

#6: TEST PLANNING
 Tip: Organize your tests in a testing matrix. Identify the tests
slated for implementation, and keep a parking lot of testing ideas
up-to-date.
Include the following elements in the testing matrix:
 Test Name
 Objective
 Hypothesis
 Testable Elements
 Primary KPI
 Secondary KPI
 Considerations (such as test conflicts
or dependencies)
 Anticipated Timeline or Test Cell Size
 Assumptions

Testing Matrix Example

#7: ASSET CREATION
 Develop assets and QA to ensure they mirror the
test plan and targeted variables.
 Ensure team members have buy-in of the test plan. Clear,
mutual understanding of the testing purpose will help
ensure assets are aligned with the test.
 During the feedback and review process, be cautious of
edits that may invalidate the test (e.g. radical change of a
variable or variables).

#8: TESTING & REPORTING
 Avoid common mishaps while your test is live.
 Test Run Time (and why it's important)
 Determining your sample size and test duration in ADVANCE helps ensure you will
collect enough data points to confidently make predictions/conclusions based on the
results.
 Don’t Stop the Test Too Early (and why "it's a trap!")
 Stopping the test too early means the target time/sample
size requirement has not been fulfilled. This lowers the test
power and validity, and it increases your chances of picking
an incorrect winner.

 Avoid common mishaps while your test is live.
 Don’t Stop the Test Too Early (cont.)
 Results often fluctuate dramatically during the early stages of the test. Because
the numbers are small during the start, the large fluctuations could be due to
chance. Don’t make decisions based
on these numbers. Allow the test to
run for its full duration.
Image: CXL

 KPI Reporting
 Report on the KPIs (primary and secondary, if a secondary KPI is defined) for
the control and test group(s). These KPIs are identified in your hypothesis.
 Resist the temptation to make test result decisions based on metrics other
than those identified in the original hypothesis. Doing so runs the risk of
distorting learnings.

 Statistical Significance
 Simply put, statistical signifignace is a way of determining how sure you are that a difference
or a relationship exists. Significance is dependent upon sample size and effect size.
 When we run an experiment, we’re measuring a sample of a larger population. Our aim is to
gain insights about a whole population via the observed sample. But within each sample,
there will be a degree of natural variance.
 Statistical significance tells us how likely it would be to obtain the observed difference
between the test and control, assuming no difference exists.
 Signifignace does not tell us the likelihood the variant will out perform the control, or chance to beat original (CTBO).
 A p-value of 0.05 means there’s a 5% chance of obtaining the observed results when there is no actual difference.
 Confidence level is the % of time each sample’s success rate will fall within the reported confidence interval.

Testing Examples

Example: Iterative Testing
Donor Impact Report

For the months of March through July, Donor Impact Report went through a series five distinct
and iterative tests to optimize open rate, CTR and user experience.
Optimizations over the period of 6 months resulted in a 29% lift in open rate and a 24% lift in
click through rate.
Donor Impact Report (DIR) Evolution & Iterative Testing
DIR Issue Delivered Open Rate CTR
Mar 2.8 mil 8.76% 0.42%
Apr 4.0 mil 8.33% 0.40%
May 4.0 mil 8.05% 0.42%
Jun 4.1 mil 9.08% 0.41%
Jul 1.4 mil 9.67% 0.45%
Aug 1.3 mil 11.26% 0.52%

 RESULTS: The test variant ("Your March Donor Impact Report is here") outperformed
the control with a 15.7% lift in open rate at 99% confidence.
1. March DIR: Subject Line Testing
The first area of focus in the iterative DIR testing series was open rate optimization. The test
sought to provide insights into the following research questions:
 If we use a donor-centric subject line, can we positively impact open rates? (e.g. "Choices you
make impact heart health")
 Given the audience engages favorably with DIR content over the course of several months, can we
positively impact open rate by including a reference to “Donor Impact Report” in the subject line?
(e.g. "Your March Donor Impact Report is here")
DIR Evolution & Iterative Testing // Results
Implications: For subsequent DIRs, the following approach was utilized:
• MAR Test Champion  Used following subject line: “Your [Month] Donor Impact Report is here”

2. April DIR: Body Copy Testing
The next area of focus in the iterative DIR testing series was
clickthrough rate optimization. The test sought to provide insights into
the following research question:
 If we shorten body copy within each story segment of the DIR, can we
reduce friction and increase clickthrough rates?
 RESULTS: The test variant with shorter body copy outperformed the control
with a 7.1% lift in CTR at 99% confidence.
control example
test variant
example
• MAR Test Champion  Used following subject line: “Your [Month] Donor Impact Report is here”
• APR Test Champion  Used shorter body copy

3. May DIR: Headline Testing
The next iterative DIR test sought to build upon clickthrough rate
optimization with a focus on content module headlines. The test
sought to provide insights into the following research question:
 If we shorten headlines for each story segment of the DIR, can we
control example
test variant example
 RESULTS: The test variant with shorter story headlines outperformed
the control with a 4.4% lift in CTR at 99% confidence.
• MAR Test Champion  Subject line: “Your [Month] Donor Impact Report is here”
• APR Test Champion  Shorter body copy
• MAY Test Champion  Shorter headlines

4. June DIR: Design & Layout Testing
The next iterative DIR test continued building upon clickthrough rate optimization
with a focus on an updated design and improved UX. The test sought to provide
insights into the following research question:
 If we update the DIR UX with cleaner blocked image formats for desktop and stacked content
on mobile, can we reduce friction and increase clickthrough rates?
Based on the clickthrough rate analysis, we see that the new, mobile optimized test variant
was associated with a 2.3% drop in clickthrough rate at 94% confidence.
However, with nearly 60% of all DIR opens occurring on either mobile or tablet, these results were very
surprising. Before accepting these results as a valid challenge of our assumptions (that a cleaner, mobile
optimized layout would drive stronger CTRs), we first needed to fulfill the following due diligence:
• Verify there were no technical issues with the a/b audience split
• Rule out any potential mitigating factors which may have impacted the KPI
• If results are found valid, form hypothesis as to why the control outperformed the test variant.
!

4. June DIR: Design & Layout Testing (continued)
After further investigation, we observed that the new test variant layout resulted in a latent consequence which
impacted the pre-header text.
 The DIR control contained pre-header text which served to
introduce email content.
 While the DIR test variant also contained this introductory
content, the pre-header was front-loaded with “JUNE 2017”
which is effectively a repeat subject line content and offers
no additional value to the supporter.
These pre-header text differences had a notable impact on open rate; the test variant pre-header was
associated with a 4.3% reduction in open rate at 99% confidence.
!

4. June DIR: Design & Layout Testing (continued)
To control for the differing open rates, we examined click-to-open rate (CTOR), or clicks as a percentage of
total opens, to gain insights into the impact which the new layout may have had on clicks.
Results: Controlling for opens using the CTOR KPI, we see that the test variant (the new layout) had
a 2.05% lift in CTOR at 91% confidence.
• JUN Test Champion  New design layout; and used pre-header text format from the old layout

5. July DIR: Headline & Teaser Copy Test
With the trend toward reduced copy outperforming longer copy in the DIR, the
next test sought to further advance the shortened copy approach in order to
determine a “how short is too short” threshold.
This test sought to provide insights into the following research question:
 If we remove the article teaser copy and use only headlines, can we
control example
test variant example
• JUN Test Champion  New design layout; and used pre-header text format from the old layout
 RESULTS: The headline-only test variant underperformed against the
control with a 6.9% drop in CTR at 99% confidence.

Example: Controlled Experiment & Exploratory Test Hybrid
Dynamic Ask String

Dynamic Ask String
HYPOTHESIS: If we use a dynamic ask string based on the donor’s last gift amount, we
can optimize revenue while minimizing donation form abandonment, due to increased
relevance in suggestive donation amounts.
 PRIMARY KPI: Percent change in DAS donation amount vs previous last gift amount
 SECONDARY KPI: Donation form conversion rate
// Non-inferiority test to ensure the new DAS “does no harm” to conversion compared to the control ask array
Implementation Notes:
The dynamic ask string was based on the following standard calculation…
Giving History (Last Gift Amount)
❑ $LGA ❑ $LGA x 1.5 [PRESELECT] ❑ $LGA x 2 ❑ Other: $ _________
Additional Considerations:
Default Array for Non-Donors (and non-logged in visitors)
Unique Rule Set for High-Value Donors: Display the default ask array for LGA ≥ $1,000

DAS Test // Performance Summary
PRIMARY KPI: Percent change in DAS donation amount vs previous last gift amount
 Topline Results: DAS drove an average lift of 19% over previous last gift amount. DAS was most
effective for 0-5 month donors, and it had the greatest lift for previous last gifts that were $150
and under.
SECONDARY KPI: Donation form conversion rate
// non-inferiority test to ensure the new DAS “does no harm” to conversion compared to the control ask array
 Topline Results: The DAS form had a 2.93% lift in conversion. The DAS test variant also had a
65% probability to be the best (with 4.81% expected loss, assuming an incorrect declaration of
winner). These results satisfied the accepted thresholds for the secondary KPI objective of the
test – ensure the new DAS “did no harm” to conversion.

DONATION AMOUNT
Primary KPI Analysis

PRIMARY KPI: Donation Amount // by Months Since Last Gift, All Sources
Months btw DAS
and Previous Gift
Difference btw DAS
and Previous Gift
% Difference btw DAS
and Previous Gift
Total DAS Revenue Total DAS Donations
0-5 months $21.09 62% $26,257 261
6-11 months $8.29 21% $65,917 672
12-17 months $7.01 15% $631,368 4,490
18-23 months $14.00 43% $22,248 234
24-29 months $3.02 22% $81,863 671
Grand Total $7.56 19% $827,653 6,328
• The DAS resulted in an overall 19% lift in donation amount over the previous last gift. The largest lift in donation
amount occurred among 0-5 month donors whose DAS donation was 62% higher than their previous last gift. The
second highest lift was observed among 18-23 month donors who had a 43% lift over their previous last gift.
62%
21%
15%
43%
22%
0%
20%
40%
60%
80%
% Difference btw DAS and Previous Gift
DAS Performance by Months Between Gifts // ALL Sources
0-5 months 6-11 months 12-17 months 18-23 months 24-29 months
*Suppress previous gifts ≥ $1,000
The largest lift in donation
amount occurred among
0-5 month donors whose
DAS donation was 62%
higher than their previous
last gift.
On average, the DAS
resulted in a 19% lift in
donation amount over the
previous last gift.

Months btw DAS
and Previous Gift
Difference btw DAS
and Previous Gift
% Difference btw DAS
and Previous Gift
Total DAS Revenue Total DAS Donations
0-5 months $20.38 58% $11,281 119
6-11 months $4.83 14% $33,585 375
12-17 months $0.92 15% $193,367 1,682
18-23 months $25.33 35% $10,611 106
24-29 months $5.76 22% $28,277 258
Grand Total $3.92 18% $277,121 2,540
• For EMAIL sourced gifts only, the DAS resulted in an overall 18% lift in donation amount over the previous last
gift. The largest lift in donation amount occurred among 0-5 month donors whose DAS donation was 58% higher
than their previous last gift. The second highest lift was observed among 18-23 month donors who had a 35% lift
over their previous last gift.
58%
14% 15%
35%
22%
0%
20%
40%
60%
80%
DAS Performance by Months Between Gifts // EMAIL
0-5 months 6-11 months 12-17 months 18-23 months 24-29 months
The largest lift in donation
amount for email sourced
gifts occurred among 0-5
month donors whose DAS
donation was 58% higher
than their previous last gift.
On average, the DAS for
email sourced gifts
resulted in a 18% lift in
donation amount over the
previous last gift.
PRIMARY KPI: Donation Amount // by Months Since Last Gift, EMAIL Sourced Gifts

• Overall, the DAS was most effective in increasing gift amounts for donors whose previous last gift was $150 and
under. The strongest lift in donation amount occurred among donors whose previous last gift $50 and under. This
segment experienced a 37% lift in donation amount.
37%
20%
13%
0%
5%
10%
15%
20%
25%
30%
35%
40%
DAS Performance by Previous Gift Amount // ALL Sources
$10-$50 $50-$100 $100-$150
Previous
Gift
Amount
Difference btw
DAS and
Previous Gift
% Difference
btw DAS and
Previous Gift
Total DAS
Revenue
Total DAS
Donations
$10-$50 $8.71 37% $68,047 1,946
$50-$100 $10.90 20% $92,460 1,416
$100-$150 $13.02 13% $200,296 1,765
$150-$200 -$0.85 -1% $23,158 153
$200-$250 -$2.47 -1% $45,655 230
$250-$300 $3.30 1% $94,760 374
$300-$350 -$11.29 -4% $16,217 56
$350-$400 $52.08 15% $4,845 12
$400-$450 -$44.67 -11% $7,462 21
$450-$500 -$75.00 -17% $750 2
$500-$550 $28.85 6% $110,025 208
$600-$650 -$67.50 -11% $2,135 4
$700-$750 -$16.67 -2% $2,050 3
$750-$800 -$150.00 -20% $4,800 8
$800-$850 $0.00 0% $1,600 2
$900-$950 $100.00 11% 1000 1
Previous gift amounts of $150 and under
accounted for 81% of all DAS donations with
previous gift history.
PRIMARY KPI: Donation Amount // by Previous Gift Amount, All Sources

• For email sourced gifts, the DAS was most effective in increasing gift amounts for donors whose previous last gift
was $150 and under. The strongest lift in donation amount occurred among donors whose previous last gift $50
and under. This segment experienced a 32% lift in donation amount.
32%
18%
13%
0%
5%
10%
15%
20%
25%
30%
35%
DAS Performance by Previous Gift Amount // EMAIL
$10-$50 $50-$100 $100-$150
Previous
Gift
Amount
Difference btw
DAS and
Previous Gift
% Difference
btw DAS and
Previous Gift
Total DAS
Revenue
Total DAS
Donations
$10-$50 $7.43 32% $30,480 895
$50-$100 $9.45 18% $35,946 562
$100-$150 $13.20 13% $80,185 705
$150-$200 $3.30 2% $6,835 44
$200-$250 -$10.58 -6% $14,650 77
$250-$300 $7.20 3% $30,350 118
$300-$350 -$11.11 -4% $5,200 18
$350-$400 $120.00 34% $2,350 5
$400-$450 $20.00 5% $2,100 5
$450-$500
$500-$550 -$29.71 -6% $32,450 69
$600-$650
$700-$750 -$50.00 -7% $1,300 2
$750-$800 $0.00 0% $750 1
$800-$850
$900-$950 $100.00 11% $1,000 1
Previous gift amounts of $150 and under
accounted for 85% of all email sourced DAS
donations with previous gift history.
PRIMARY KPI: Donation Amount // by Previous Gift Amount, EMAIL Sourced Gifts

CONVERSION RATE
Secondary KPI Analysis

A/B Testing
Clicks
(donation URL)
Total
Donations
Amount
Conversion
(donations /
clicks)
% change
from control
Probability to
be Best
Expected
Loss
Control Donation Form 561 196 $19,717 34.9% CONTROL 35% 7.12%
DAS Test Variant Form 723 260 $22,480 36.0% 2.93% 65% 4.81%
An A/B test was run on FY18 Giving Tuesday Email #3 to determine the DAS impact on conversion. Test
results indicated a 2.93% lift in conversion for the DAS donation form. The DAS test variant also had a 65%
probability to be the best (with 4.81% expected loss, assuming an incorrect declaration of winner).
• These results satisfied the accepted thresholds for the secondary KPI objective of the test – ensure the new
DAS “did no harm” to conversion.
SECONDARY KPI // Donation Form Conversion

Summary

SUMMARY: Basic “3 Principles” of Performance Testing
1. Decide what you’re testing ahead of time, and why (hypothesis
formulation, primary and secondary KPIs).
2. Isolate the testable element between control and challenger(s).
3. Reach statistical significance – KPI results alone aren’t enough!

SUMMARY: The (Testing) Bisque Analogy
If you set out to make a velvety, creamy
bisque, you have a set list of ingredients and
steps to achieve that delicious outcome.
Adding additional or alternative steps and
ingredients means your end result may no
longer be that same velvety bisque.
At best, it tastes good, but may fail to be a
bisque. At worst, it’s not even consumable.
Like the bisque, testing needs a set process.
With testing, adhere to your goals,
hypotheses and testing process. Staying on
track can help ensure your tests deliver
learnings, optimizations, reliability, and
validity.

Take Home Tips & Tools
 Sample Size Calculator http://www.evanmiller.org/ab-testing/sample-size.html
 Test Duration Calculator https://vwo.com/ab-split-test-duration/
 Chi-Square A/B Signifignace Test (for conversion)
https://www.kissmetrics.com/growth-tools/ab-significance-test/
 T-Test Signifignace calculator (for average gift) http://in-
silico.net/tools/statistics/ttest
 5 Factors that Lead to Conversion Rate Increases (Meclabs conversion heuristic)
https://www.meclabs.com/about/heuristic
 NextAfter Nonprofit Testing Logs https://www.nextafter.com/research/explore/
 How to Run Marketing Experiments the Right Way
https://conversionxl.com/blog/marketing-experiments/
 10 Statistics Traps in A/B Testing: The Ultimate Guide for Optimizers
https://conversionxl.com/blog/testing-statistics-mistakes

Thank You!
www.pmxagencynonprofit.com

Testing for Success: How to Infuse Consistent Testing Into Your Fundraising Programs

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (12)

Similar a Testing for Success: How to Infuse Consistent Testing Into Your Fundraising Programs

Similar a Testing for Success: How to Infuse Consistent Testing Into Your Fundraising Programs (20)

Último

Último (20)

Testing for Success: How to Infuse Consistent Testing Into Your Fundraising Programs

Notas del editor