Transcript of a BriefingsDirect podcast on how one company is able to track video viewing on the Internet in real time, despite massive amounts of data flowing in continuously.

1. Learn How Visible Measures Measures an Expanding
Universe of Video and Viewer Use Big Data
Transcript of a BriefingsDirect podcast on how one company is able to track video viewing on
the Internet in real time, despite massive amounts of data flowing in continuously.
Listen to the podcast. Find it on iTunes. Sponsor: HP
Dana Gardner: Hello, and welcome to the next edition of the HP Discover Performance
Podcast Series. I'm Dana Gardner, Principal Analyst at Interarbor Solutions,
your moderator for this ongoing discussion of IT innovation and how it’s
making an impact on people’s lives.
Once again, we’re focusing on how IT leaders are improving their business
performance for better access, use and analysis of their data and information.
This time we’re coming to you directly from the HP Vertica Big Data
Conference in Boston. [Disclosure: HP is a sponsor of BriefingsDirect podcasts.]
Our next innovation case study interview examines how video advertising solutions provider
Visible Measures delivers impactful metrics on video use and patterns. To learn more about how
Visible Measures measures, please join me now in welcoming our guest. We're here with Chris
Meisl, the Chief Technology Officer at Visible Measures Corporation based in Boston. Welcome.
Chris Meisl: Thanks for having me, Dana.
Gardner: Tell us a little bit about video metrics. It seems that this is pretty straightforward, isn't
it? You just measure the number of downloads and you know how many people are watching a
video -- or is there more to it?
Meisl: You'd think it would be that straight-forward. Video is probably the fastest growing
component of the Internet right now. Video consumption is accelerating
unbelievably. When you measure a video, not only you are looking at did
someone view the video but how far they are into the video? Did they rewind
it, stop it, or replay certain parts? What happened at the end? Did they share
it?
There are all kinds of events that can happen around a video. It's not like in the
display advertising business, where you have an impression and you have a
click. With video, you have all kinds of interactions that happen.
You can really measure engagement in terms of how much people have actually watched the
video and how they've interacted with a video while it's playing.

2. Gardner: This is an additional level of insight beyond what happened traditionally with
television, where you need a Nielsen box or some other crude, if I could use that term, way of
measuring. This is much more granular and precise.
Census based
Meisl: Exactly. The cable industry tried to do this on various occasions with various set-up
boxes that would "phone home" with various information. But for the most part, like
Nielsen, it's panel-based. On the Internet, you can be more census-based. You can
measure every single video, which we do. So we now know about over half a
billion videos and we've measured over three trillion video events.
Because you have this very deep census data of everything that's happened, you
can use standard and interesting statistical processes to figure out exactly what's
happening in that space, without having to extend a relatively small panel. You
know what everyone is doing.
Gardner: And of course, this extends not only to an editorial or entertainment level of video, but
also to the advertising videos that would be embedded or precede or follow from those. Right?
Meisl: Exactly. Advertising and video is interesting, because it's not just standard television style
advertising. In standard television advertising, there are 30-second spots, that are translated into
the Internet space as pre-roll, post-roll, mid-roll, or what have you. You're watching the content
that you really want to watch, and then you get interrupted by these ads. This is something that
we at Visible Measures didn't like very much.
We're promoting this idea of content marketing through video, and content marketing is a very
well-established area. We're trying to encourage brands to use those kinds of techniques using the
video medium.
That means that brands will tell more extensive stories in maybe three- to five-minute video
segments -- that might be episodic -- and we then deliver that across thousands of publishers,
measure the engagement, measure the brand-lift, and measure how well those kinds of video-
storytelling features really help the brand to build up the trust that they want with their customers
in order to get the premium pricing that that brand has over something much more generic.
Gardner: Of course, the key word there was "measures." In order to measure, you have to
capture, store, and analyze. Tell us a little bit about the challenges that you faced in doing that at
this scale with this level of requirement. It sounds as if even the real time elements of being able
to feed back that information to the ad servers is important too.
Meisl: Right. The first part that you have to do is have a really comprehensive understanding of
what's going on in the video space.

3. Visible Measure started with measuring all video that’s out there. Everywhere we can, we work
with publishers to instrument their video players so that we get signals while people are watching
videos on their site.
For the publishers that don't want to allow us to instrument their players, then we can use more
traditional Google spidering techniques to capture information on the view count, comment
count, and things like that. We do that on a regular basis, a few times a day or at least once a day,
and then we can build up metrics on how the video is growing on those sites.
Massive database
So we ended up building this massive database of video and we would provide information, or
rather insight, based on that data, to advertisers on how well their campaigns were performing.
Eventually, advertisers started to ask us to just deliver the campaign itself, instead of giving just
the insight that they would then have to try to convince various other ad platforms to use in order
to get a more effective campaign. So we started to shift a couple of years ago into actual
campaign delivery.
Now, we have to do more of a real-time analysis, because as you mentioned, you want to, in real
time, figure out the best ways to target the best sites to send that video to, and the best way to
tune that campaign in order to get the best performance for the brand.
Gardner: And so faced with these requirements, I assume you did some proofs of concept
(POCs). You looked around the marketplace for what’s available and you’ve come up with some
infrastructure that is so far meeting your needs.
Meisl: Yes. We started with Hadoop, because we had to build this massive database of video, and
we would then aggregate the information in Hadoop and pour that into MySQL.
We quickly got to the point where it would take us so long to load all that information into
MySQL that we were just running out of hours in the day. It took us 11 hours to load MySQL.
We couldn’t actually use the MySQL. It was a sharded MySQL cluster. We couldn’t actually use
it while it was being loaded. So you’d have to have two banks of it.
You only have a 12-hour window. Otherwise, you’ve blown your day. That's when we started
looking around for alternate solutions for storing this information and making it available to our
customers. We elected to use Vertica -- this was about four years ago -- because that same 11-
hour load took 2 hours in Vertica. And we're not going to run out of money buying hard drives,
because they compress it. They have impressive compression.
Now, as we move more into the campaign delivery for the brands that we represent, we have to
do our measurement in real time. We use Storm, which is a real-time stream processing platform
and that writes to Vertica as the events happen.

4. So we can ask questions of Vertica as they happen. That allows our ad service, for example, to
have much more intelligence about what's going on with campaigns that are in-flight. It allows us
to do much more sophisticated fraud detection. There are all kinds of possibilities that you can
only do if you have access to the data as soon as it was generated.
Gardner: Clearly if a load takes 11 hours, you're well into the definition of big data. But I'm
curious, for you what constitutes big data? Where does big data begin from medium or non-big
data?
Several dimensions
Meisl: There are several dimensions to big data. Obviously, there's the size of it. We process
what we receive, maybe half a billion events per day, and we might peak at near a million events
a minute. There is quite a bit of lunchtime video viewing in America, but typically in the
evening, there is a lot more.
The other aspect of big data is the nature of what's in that data, the unstructured nature, the
complexity of it, the unexpectedness of the data. You don't know exactly what you're going to get
ahead of time.
For information that’s coming from our instrumented players, we know what that’s going to be,
because we wrote the code to make that. But we receive feeds from all kinds of social networks.
We know about every video that's ever mentioned on Twitter, videos that are mentioned on
Facebook, and other social arenas.
All of that's coming in all kinds of different formats. It would be very expensive for us to have to
fully understand those formats, build schemas for them, and structure it just right.
So we have an open-ended system that goes into Hadoop and can process that in an open-ended
way. So to me, big data is really its volume plus very open-ended, unknown payloads in that
data.
Gardner: How do you know you're succeeding here? Clearly, going from 11 hours to 2 hours is
one metric. Are there other metrics of success that you look to -- they could be economic,
performance, or concurrent query?
Tell me what you define as successful analytics platform and what you’ve been finding using
Vertica?
Meisl: At the highest level, it's going to be revenue and margin. But in order to achieve the
revenue and margin goals that we have, obviously we need to have very efficient processes for
doing the campaign delivery and the measurement that we do.

5. As a measurement company, we measure ourselves and watch how long it takes to generate the
reports that we need or how responsive we are to our customers for any kind of ad-hoc queries
that they want or special custom reports that they want.
We're continuously looking at how well we optimize delivery of campaigns and we're
continuously improving that. We have corporate goals to improve our optimization quarter-over-
quarter.
In order to do that, you have to keep coming up with new things to measure and new ways to
interpret the data, so you can figure out exactly which video you want to deliver to the right
person, at the right time, in the right context.
Looking down the road
Gardner: Chris, we're here at the Big Data Conference for Vertica and its community. Looking
down the road a bit, what sort of requirements do you think you are going to need later? Are
there milestones or there is a road map that you would like to see Vertica and HP follow in order
to make sure that you don't run out of runaway again sometime?
Meisl: Obviously, we want HP and Vertica to continue to scale up, so that it is still a cost-
effective solution as the volume of data will inexorably rise. It's just going to get bigger and
bigger and bigger. There's no going back there.
In order to be able to do the kind of processing that we need to do without having to spend a
fortune on server farms, we would want Vertica, in particular, to be very efficient at the kinds of
queries that it needs to do and at loading the data and asking questions of it.
In addition to that, what's particularly interesting about Vertica is its analytic functions. It has a
very interesting suite of analytic functions that extends beyond the normal standard SQL analytic
functions based on time series and pattern matching. This is very important to us, because we do
fraud detection, for example. So you want to do pattern matching on that. We do pacing for
campaigns, so you want to do time series analysis for that.
We look forward to HP and Vertica really pushing forward on new analytic capabilities that can
be applied to real-time data as it flows into the Vertica platform.
Gardner: Well, great. I'm afraid we'll have to leave it there. We've been learning about how
Visible Measures measures and how they put together an analytic capability for video at some of
the highest scales I've heard. We've also heard about how they deployed HP Vertica as their
analytics platform to provide better analytics and deliver better insights to their customer.
So, a big thank you to our guest, Chris Meisl. He is the Chief Technology Officer at Visible
Measures. Thank you, sir.

6. Meisl: Thank you, Dana.
Gardner: And thanks also to our audience for joining us for this special HP discover
performance podcast, coming to you directly from the HP Vertica Big Data Conference in
Boston.
I'm Dana Gardner, Principal Analyst at Interarbor Solutions, your host for this ongoing series of
HP sponsored discussions. Thanks again for joining, and come back next time.
Listen to the podcast. Find it on iTunes. Sponsor: HP
Transcript of a BriefingsDirect podcast on how one company is able to track video viewing on
the Internet in real time, despite massive amounts of data flowing in continuously. Copyright
Interarbor Solutions, LLC, 2005-2013. All rights reserved.
You may also be interested in:
• Advanced IT monitoring Delivers Predictive Diagnostics Focus to United Airlines
• HP Vertica Architecture Gives Massive Performance Boost to Toughest BI Queries for
Infinity Insurance
• HP-Fueled Application Delivery Transformation Pays Ongoing Dividends for McKesson
• Podcast recap: HP Experts analyze and explain the HAVEn big data news from HP
Discover
• HP's Project HAVEn rationalizes HP's portfolio while giving businesses a path to total
data analysis
• Insurance leader AIG drives business transformation and IT service performance through
center of excellence model
• HP BSM software newly harnesses big-data analysis to better predict, prevent, and
respond to IT issues