JUNE 1, 2014 • Ninja Metrics Interview: How and why people spend money in games has taken on much greater importance as the free-to-play model has taken root worldwide. What is relatively novel is which players influence purchases by other players. The concept is not new, consumer electronics companies have been cultivating early adopters for decades – people who then go onto influence other consumers to purchase. Yet translating that concept to in-game purchases is often a much less clear-cut proposition. Tracking word of mouth can tell a company who helped get their friends to buy a video game, but tools to track in-game influence are rare.
That’s where a new analytics firm, Ninja Metrics, comes in. Coming out of the Big Data research movement with roots at the University of Southern California, Dmitri Williams saw how game data could provide a Social Value that could illuminate player influence. That insight led to the launching of Ninja Metrics last year. Wanting to know more about how his Katana commercial product was rolling out to game developers and publishers, DFC Intelligence spoke further with Williams about Ninja Metrics.
DFC: As an academician you have been researching gamers for 15 years now. At what point did the concept for Ninja Metrics present itself, and how long has the company and Katana been in development?
Dmitri: I’ve been doing research on games for a long time, but I’ve been a player for longer. I also started doing consulting work about 10 years ago in an effort to get out of the classroom. Around the same time, we formed our large research team, and started to do work on large-scale predictive analytics under grants from the Army and intelligence services. This was “Big Data” before we’d heard the term. We were dealing with terabytes of data in a way no social scientist had ever tackled, and we generated a ton of research from it.
About halfway into that, we discovered that the social graph was a really powerful tool for improving the performance of our models. In other words, when we went from “what did someone do?” to “what did their friends do?” we got a lot better at our tasks.
DFC: What inspired you to go commercial?
Dmitri: After the initial research had found the social angle to be a big boost, one of our researchers formalized that in an algorithm. We could look at what one person did and then actually predict what their friends would do. This was a pretty big breakthrough and we sat around wondering what we should do with it.
That lead to us taking commercialization seriously, and we founded Ninja Metrics not too long after – about four and half years ago. We took the best and brightest from our research team and made them our first employees. Another reason was that I achieved tenure at USC, and that allowed for more risk taking.
DFC: How many people now work at Ninja Metrics and what are their roles in the company?
Dmitri: We’re 16 today, and adding a few more over the coming weeks. The majority of the company is engineering, ranging from front-end to back-end to database and operations to algorithm development. One thing we do that’s probably unusual for a startup is to devote resources to ongoing R&D as we have PhDs on staff who are better suited to discovery and invention than writing production code. So, we look at that as a feature rather than as a bug.
Aside from the engineers, there are the usual finance, marketing, design and product roles to round out the company.
DFC: What has been the biggest learning point in branching out from academic research into running an analytics business?
Dmitri: The expected answer would be something about translating ivory tower ideas into practical applications, but that hasn’t been a problem. We’ve always been applied, and so to go from an outcome variable like “voting” to “spending” is not a big shift. The real learning curves have been in understanding how funding works, licensing, patents, and all of the mechanics of starting up a business. There’s no school for professors for that sort of thing, and so the last four years have been amazing – and frequently humbling – on-the-job training that’s not a whole lot different than any first-time entrepreneur would go through.
DFC: How is Ninja Metrics different from other mobile analytics companies and products? Who are you competing against to get your message out and acquire clients?
Dmitri: If you look at the way we take in data, the answer is nothing. If you look at what we do with it, the answer is everything. Basic analytics – things like DAU, k-factor, and how many players are at level six – that’s pretty well commoditized at this point. We’re differentiated in two ways right now.
First, we do automated predictive analytics. That’s worth parsing out. We tell you what players are going to do, with a reported degree of certainty. And we do it via a machine learning system that gets better and reports fresh results daily. That means no extra legwork for a one-off report, or no need to pay more for access to data or help, etc.
Second, we do Social Value. That’s our core IP, and truly unique.
DFC: When most people hear about “social analytics” they think working through social media data, such as mining Facebook and Twitter. But we understand Katana has nothing to do with traditional social media mining. Where does Katana get its data?
Dmitri: When we say “social,” we’re talking about the social graph itself. It’s not whether you “liked” something on Facebook or tweeted a comment. It’s the graph of actual connections between people, combined with the records of the in-app actions they take. We get the data to populate that graph and those actions directly from our clients via SDK in their app or via API from their server.
Everyone collects things like “the user logged in at time X.” We do that too. Then we also collect things like “this user played a match with that user.” That builds the graph, which is where really powerful analytics are possible.
DFC: The big value proposition, as we understand it, that Ninja Metrics offers over other games analytics companies is predicting the social value of a user. The essence of the example is that with Katana the lifetime predicted value of a customer can be understood from expected direct payments from that customer, which you call “individual LTV,” or iLTV. Therefore you can predict the social influence that player has on others in the game, and hence predict how that player will influence said others to also spend. That influenced spending is what you call “social value” that should be added to the individual iLTV. But if Katana can predict the iLTV for all customers, wouldn’t the social value of each customer be double counted in the sum of the iLTV for all customers, even ones that have not spent any money yet?
Dmitri: No, we’re very careful to avoid any double counting and to make sure the math is transparent. Let’s back up and clarify the basic component parts.
First, there’s Asocial LTV. That’s the portion of spending or activity that’s driven by the app, and not at all by social ties.
Then, there’s Influenceability. This is how much you are doing over and above Asocial LTV because of the presence of your friends. Imagine you are playing League of Legends and you play two matches a day because it’s a great game, then another two to hang out with friends.
Last is Social Value. This is how much you influence your friends. If you have five friends and your interactions cause them each to play two more matches – or spend $2 more – than they would have, then you have a Social Value of 10 matches, or $10.
What you’ve asked is whether there’s double counting here, and the answer is no. The total spending in our metrics are no different than in anyone else’s. What we’ve done is to properly attribute the dollars between these three values. You may spend $20, but $5 of that spending might be driven by your friend. So for you it’s $5 of Influenceability plus $15 driven by the app for $20 total, just as before. For your friend it’s $5 of Social Value to add to their total.
As you can see, the total amount of Social Value will always equal the total amount of Influenceability. In other words, the total amount going out will equal the total amount coming in. So when we say how much money there is in the system overall, we only count the social part one time, not twice. The bottom line remains unchanged, but now it’s attributed exactly where and how it should be.
DFC: Predictive analytics is still a relative new area of study. How solid is the methodology? Are you sure you have everything nailed down, or are you still finding surprises during implementation?
Dmitri: It’s fairly off-the-shelf now in academia, where the methodology is very solid. What varies is performance, and that’s something that any responsible scientist (or vendor) will give along with the results.
Imagine I tell you that this list of people are going to churn, or convert, or spend, or that they won’t do those things. We then compare the prediction to what actually happened, and report that as accuracy, or “confidence.” These start lousy on day 1, and then get better as the system “learns.” Our dashboards mechanically report this number – we use an F-score, for the statistically inclined – and the customer can draw their own conclusions about safety and actionability. In practice, our numbers are typically over .80 and sometimes over .90.
DFC: Which is more important to an online game publisher: a user who spends a lot, or a user that influences a larger group of players, and why?
Dmitri: Right now publishers only care about the user who spends a lot because that’s tangible. If they could see the Social Value alongside the actual spend, they’d quickly see that there is much more going on below the surface. Plenty of players who spend a lot are, in fact, driven by friends. If those friends went away, so would the spending. And many who cause spending don’t spend much at all themselves.
Our term for these big influencers is “Social Whales.” They are around 10% of most game populations, and they drive about 60-70% of the Social Value in the system. What we find pretty reliably is that these Social Whales are almost never the biggest whales themselves. If you think about this in daily life, it should ring true: the people who spend a lot to enjoy themselves are rarely the people who impact others most.
DFC: When you launched Ninja Metrics your data showed you that 25% of a person’s actions are driven by others. Is that percentage holding?
Dmitri: There’s a very large range. We see everything from 5% to 40%, with a lot of the range explained by how social the game itself is. Asocial, asynchronous, single-player games have very small amounts – but not zero because players still impact each other outside of play. Social, synchronous, multiplayer games have higher totals.
Whatever the percentage is it’s a pretty valuable baseline for the publisher to use to see improvements or declines from that point on. And, when it’s used in a portfolio, it’s really stark to see when Title A is killing it socially and Title B is sucking wind.
Also, even when there are small percentages overall, there are always Social Whales who would otherwise go unseen. Those are key players to protect and leverage.
DFC: Does Ninja Metrics see other analytics companies getting into tracking and predicting social value? If the integration of Katana into a game is relatively straight forward, what sustainable advantage does Ninja Metrics have with Katana if other analytics tools can track social data in games as well?
Dmitri: We invented Social Value, and thus far we haven’t seen anyone try to copy us. I’m sure it’ll happen eventually, and what they’ll find is that it’s really hard to stand up. This took a team of 25 researchers a long time to perfect, and then a company a long time to automate and launch.
I’d like to see platforms and companies come and license the tech from us so it becomes a shared standard.
DFC: Is analyzing social value patented, and if not, can it be patented?
Dmitri: We were awarded the patent last month.
DFC: What is a Key Performance Indicator and why should a game developer care?
Dmitri: This is pure jargon. Actually, when I first started and someone said they wanted KPIs I had no idea what they meant. It’s just a term for the basic metrics one would expect – DAU, ARPU, etc. – plus whatever the publisher finds to be “Key.”
DFC: Some of your more recent clients are Nerd Kingdom and Gamzio. Who else is on your client roster? We understand you are working with more than a dozen developers.
Dmitri: At this time we can’t provide a complete client list but we are working with developers along the spectrum from those starting out to some of the largest developers in the ecosystem.
DFC: A month after launching Katana you introduced gambling industry services. What led you to target casino gaming so quickly?
Dmitri: There was no pivot necessary. Since gambling is just another form of gaming, and since we already collect all the payment and action data required to power it, the original form of Katana actually could handle it at launch. We wanted to make sure the casino game operators knew that our tech also applied to them.
DFC: What is similar and dissimilar about predicting player actions between a consumer mobile game and casino gaming?
Dmitri: It’s very similar. In each, there are game mechanics, incentive schemes, feedback loops, sparkly objects and the appearance of money. The gambling industry has a longer history of perfecting the Skinner box-like mechanics to spur pulling that handle or pushing that button, but it’s not terribly unlike what game developers do. So, the predictive models behave very similarly.
The differences are more likely to appear between genres, not between real-money gaming vs. non-money gambling. As various firms try to add gambling aspects to more standard game genres, I’d expect little difference to appear. But, that’s the great thing about transparent analytics – my guess doesn’t matter because we’ll see it. Hopefully I can get up on the GDC stage every year and give census-type reports on trends and numbers. As a former researcher focused on publishing results, I really like to report!
DFC: How are you marketing Katana internationally, and are you getting the same level of interest as in North America?
Dmitri: We’re doing some keyword purchases, but most of our growth is coming from organic sources like search engines (SEO), press coverage, public talks and word of mouth. We’re early and still fine-tuning. For example, we have a lot of unanticipated interest from Central and Eastern Europe, and less than expected from Asia.
DFC: Katana was launched based on a service model, but in March you followed up with an enterprise version. What was the need that led you toward an enterprise product, and how is it priced?
Dmitri: The difference between the two is essentially whether it’s on site or in the cloud, plus how much support the customer wants. Our system can be a black box that lives up in Amazon, or behind someone’s firewall. Different customers have different needs for location. Then, they also have different needs for consulting support. Katana does a lot, and so to take advantage of all of it, it can be helpful to have extra guidance. In a big organization there are some extra challenges to navigate, and we can help there. For example, how do you get marketing and development to work well together using metrics? We designed Katana to essentially have different buttons for each group, but to use the same language. Still, it sometimes helps for us to come in and make these things explicit, or to listen to a customer’s unique needs, or to hear feedback that helps us refine and create new features.
Pricing for the basic service is very similar, but for enterprise clients there is usually a larger up-front cost to cover us coming in and doing an installation. Then we add on whatever support they’re looking for.
DFC: You have said that your algorithms can be applied in other areas such as ecommerce and motion picture distribution. With each new segment, are you creating a whole new consumer database, or utilizing the user data already acquired through video games?
Dmitri: The algorithm measures influence and doesn’t know what units it’s measuring in. So, it can be dollars spent, or hours, or sessions, or whatever. And it can be subdivisions within those, so things like foreign films versus superhero movies. As we add customers in other segments, it gives us the opportunity to add to the profile of an individual.
Let’s say you are a regular whale in FPS games, have no activity in shoe buying, but are a Social Whale in restaurants. Knowing all three things about you makes that profile very valuable. We get to know that, and advertisers of course want to know that. Most importantly, our customers get to know it, so long as they are willing to share abstracted data themselves. This is an opt-in service we’re building out.
DFC: Electronic Arts brought you in early on to analyze players of Star Wars: The Old Republic. What was EA looking for, and what did you discover for them?
Dmitri: SWTOR was a subscription model at the time, so they were almost entirely focused on retention. The things they cared about were Social Value and churn likelihood. We gave them both so they could know things like “this player is worth $30 and is 60% likely to churn” and “this player is worth $30, but an additional $240 in Social Value and is 60% likely to churn.” That extra social information changes everything.
Since then we’ve been seeing more free-to-play models, of course, but the idea of retaining the most important players doesn’t go away. We’ve added tools for smarter acquisition (where did the Social Whales come from and what was the ad spend ROI?) and monetization (I have Social Whales, now how much lift can I get through them?).
DFC: At the same time you launched Katana, Ninja Metrics raised $2.8 million in series A funding from the Harvard Business School Angels of Los Angeles and the Tech Coast Angels. At the time you foresaw another series of funding during the first half of 2014. Is that still the case, and if so, how is that funding going? If we remember correctly you had to knock on a lot of doors before getting that first round completed. Is the job easier now that you have clients?
Dmitri: We’re just closing our B round now, actually. And yes, it’s a whole lot easier to raise money once you have a team, a working product, and sales traction. The first round took 100-plus pitches over 10 months and this one took about 10 pitches over two months.
DFC: You were a speaker at GDC this year. What kind of response did Katana get from attendees? How did they respond to your product, and did they surprise you with ideas for additional applications of your product?
Dmitri: I gave two talks at GDC and another at the IGDA User Research Summit off site, but I didn’t pitch or describe Katana in any of them. GDC hates that, and I actually declined to get into our product despite some audience questions. My message was just about what Social Value is, and how developers should think about leveraging their social graphs. Attendees then did approach us afterward and we’ve had a lot of sales traction following those sessions. Normally up there I get 100 people in an audience and at the two talks I had 450-plus each time. It was really energizing.
As to additional applications, yes, we’ve had some great ones. There is a lot of attention right now on advertising attribution, and our tech can play very well in that space. Our current and prospective customers have had a lot of “what if you added a ___ to the UI so we could do a better job with our ____?” kind of ideas. We love those, of course, and will be building a lot of them in.
I suspect we’re going to be kind of “permanently beta” in that respect – always adding and refining. I’m the product owner, so hearing feedback is rewarding and refocusing. Sometimes they love things we didn’t think mattered, hate things we thought were key, and suggest things that weren’t on our radar. That is all incredibly valuable to us.
DFC: If Google came knocking tomorrow, how much would they have to pay for Ninja Metrics?
Dmitri: I bet we’d then have a pretty interesting conversation with smart people, just like we do with our customers.
Ultimately, I want to see this technology used broadly and to disrupt marketing, advertising and acquisition. As good as data science is getting, it’s missing both transparency and the human condition. It bugs the hell out of me that 50% of advertising works, but no one knows which 50%. We can know, and should know, and we should be able to factor in social impacts to boot.
I want to see a world where ads are delivered to those who create value, and to reinforce friendships. After the “let’s give the Social Whale a discount” phase has passed, big firms are going to see that the value lies in giving relationships a boost. When friends have a good time together, they spend. So the upshot of our tech should be to get businesses to support the right friendships, and to make money as a nice side effect. If there’s a big company that truly groks this, I’d love to talk to them.
