The document describes an A/B test conducted by a public radio station during a pledge drive to determine the most effective online advertising format for driving conversions and donations. Four formats were tested - a control banner ad and three intrusive lightbox, push-down, and bottom-up ads. The results found no statistically significant differences in conversion rates, average donation amounts, or bounce rates between the control and test formats. It was concluded that more intrusive ad formats are not more effective than standard banner ads for online pledge drives. The test raised questions about how messaging, creative, and testing outside of pledge week may impact results.
7. The big questions
What online ad format is most effective at driving
increased conversion rate and pledge amount?
Do the intrusive ad formats negatively affect user
engagement (e.g., bounce rate)?
7
8. The test
Partnered with Michigan Radio
A/B test splitting traffic among 3 test formats and a
control
Same messaging and creative used for all versions
Deployed for pledge week (October 2011)
23k total home page visits during the test
8
9. Test setup
We split the traffic (using Core Publisher, our Drupal-
based CMS)
• Randomized visitors and assigned a visitor-level cookie for the
duration of the test
We used DFP (DoubleClick for Publishers) to serve the ads
Sent the test version in the DFP call
In Google Analytics, we used a visit/session-level custom
variable to capture the test version
• This custom variable was used to calculate:
• Conversion rate
• Pledge amount
• Bounce rate
9
15. The results
Control Lightbox Push-down Bottoms up
Conversion: 6.39% 5.65% 5.50% 5.49%
Avg pledge: $126.52 $138.30 $124.90 $132.83
Revenue: $46,181 $44,394 $39,968 $40,779
Bounce: 37% 37% 37% 36%
None of the options resulted in a statistically significant change in performance
15
16. What does this mean?
Creating intrusive ads for pledge week is essentially a
waste of time, resources, and money
The standard 300x250 is at least as effective as doing
something “special”
Time is probably better spent on more effective ways of
driving donations (e.g., email, mailings, offers)
16
17. Every test leads to more questions
What if we tested different messaging or creative?
Would the results be the same if we didn’t test during
pledge week?
What about the rest of the pledge funnel? For
example, does the presence of a landing page before the
pledge form help or hurt?
17
Custom Variables can be used to track dimensions that don’t exist in GA out of the box.Examples:Page category, Test Version, Membership statusThree Levels of scope with different persistence:Pageview – lasts only for the page viewVisit/session – Lasts for the duration of the session, expires at end of visitVisitor – Last for 24 months in the visitor cookieCode with 4 parameters: slot (1-5), name, value, scope (2=visit)_gaq.push('_setCustomVar',1,’PledgeTest',’AdVersion1', 2);Only Visitor Level custom variables are built in to standard reportsTo view page or visit level you will need to build a custom report