# Analytics for startups 

**Author:** Ilya Volodarsky
**Type:** Video
**URL:** https://www.ycombinator.com/library/8H-analytics-for-startups
**YouTube ID:** LLerCc7MOQo

---

Ilya Volodarsky, Co-founder of Segment, presents his tactics for setting up analytics to build your MVP and measure your
metrics.

## Transcript

hi everyone my name is Ilya I'm one of the co-founders at segment and I'm here to talk to you about how to set up analytics and the analytics foundation to build your MVP and to measure these primary and secondary metrics so this is going to be a little bit more of a tactical guide around what tools are there and then oolitic space in the marketing space which one should we actually be using how do I set them up cool just before I get started a lot of you may not know what segment is it's an analytics API where you can send analytics data and then toggle on different tools so we send the cool market position where we can actually see what tools startups are using and we

can bring that information to you secondly we've been around for about six years and so we have information about a lot of different startups as they grew and what tools they used so we're gonna be sharing a lot of those today cool so why talk about why even focus on the analytics obviously primary and secondary metrics drive the MVP and product market fit process and you're using that to actually test product market fit you're also using it once you get out of product market fit strict search to actually focus the team so

maybe there's gonna be an acquisition issue in the company that's preventing your growth or maybe the users that you're getting aren't as engaged or maybe you're having some monetization issues and so the funnel actually is a forcing function to understand your business and where founders should be actually spending their time and then finally all the way from you know to three-person team to Google with 1 million employees you're actually using metrics to operate and drive teams so eventually you have an engineering team you have a marketing team and so what gold do you set in front of the marketing team use analytics for that ok

so today we're gonna cover a few different things so first you always start with the funnel when you're thinking about analytics so that's the sequential series of steps so your users go through to actually get value and then pay you as well then we're gonna talk about collecting data for your analytics tools then we're going to talk about the top three metrics so this will include primary and secondary metrics that work for most products and then a product market fit methodology that you can apply on top of that and then finally we'll make recommendations about what tools are the best in the market right now that help the product market

fit journey ok so to start where we started the funnel so we'll make an example funnel for Netflix which is a company super familiar with any b2b product or b2c product actually has this type of funnel where you acquire a user you engage a user over a period of time that loop is called retention and then finally you monetize the user and then metrics both primary and secondary are performance indicators on top of each stage in the funnel so I on top of acquisition you can ask yourself how

many new users did I get this week versus last week and what's my growth rate there for engagement you take a cohort of users so from you know Sunday to the following Monday you you have you know 16 people sign up and then you can track that cohort of users week over week and see what percentage of them are still using the product four weeks later which is a good example of how to track retention and then we talked about monetization which is how much that new revenue did I make this week versus last week okay and then you apply your own custom business funnel to this so if

your Netflix we're all familiar the users sign up for Netflix then they play videos in a loop the Netflix is obviously very sticky watching it a lot and then finally when the trial runs out you do subscription upgraded and you get access to more content okay so how do you collect data once you have this funnel so there's analytics API is out there I'm using segments as an example you basically want to say user user 1 2 3 in this case has done user signup event and they happen to be an organic user which means they're not invited by someone

else then if your Netflix you might say the users video played and eventually subscription upgraded and so this is how you instrument your tracking and your mobile app or your or your web app then you think about event properties so imagine your Netflix and you're holding one of these video played events in your hands and you're wondering questions about it so you know what video art is the user actually playing how long is the video how far did the person get inside of the video right equivalent if you're holding a subscription upgraded event you're going to want to derive monetization as a Northstar metric so if

you're a subscription business you want to send your monthly recurring revenue if you're a transactional business like e-commerce or retail you want to send the actual value of the transaction okay so then you push this out into your web app your mobile app then you start seeing the data come in you look at the debugger you see okay user signup is here every good you add your first analytics tool will use amplitude as a good example here amplitude and Mixpanel are pretty awesome analytics tools out in the market right now and then you start seeing data flow inside of one of these analytics tools so this is amplitude you

can start seeing user signups growing as soon as you launched the real like mobile or web app okay so now that you have analytics set up it's time to focus on three different metrics the first one is the acquisition metrics signups per week it's really nice if you're a b2b business to cut this by the invite type so I you have organic users which are just signing up from coming to your website direct sign up and then some users are inviting other users so those are invite type right so when you're thinking about growth it's really important to think about the organic user in that case right another example

why if I'm properties are important the way you create this is you go to amplitude you say event segmentation report user sign up next here's your graph right so it's as easy as that just website some data gets to amplitude and then you can see the amount of organic users every week and then if you're working on the acquisition step as a secondary metric you can basically say today 218 users organically signed up in the last week but by the end of the month we want that to be at 300 and we're gonna execute projects a B and C

this month and then we're gonna watch this graph every single day on a TV dashboard in our office or apartment wherever we work and then we're gonna see if our efforts are actually driving this right so that's an idea of data driven operation of a team you set a metric and a goal and then you drive towards that every day okay the second one is retention to cohorts so someone recently asked about retention what we'll talk about that right now so with retention you want to think about cohorts of users so you want to say Monday to Sunday let's say December 10th through the 7th December 17th 16

users signed up and then you look at those 16 users as they use your product on a week 0 which is their signup week week 1 a week 2 week 3 and week 4 and the general idea here is like you can convince your mom or your grandma to use your product once but even your mom or your grandma won't continue to come back and use your product over time every single week right and so if you see users that are addicted that are coming back week over week that's a really good sign of product market fit so this business can see that the December tenth cohort only 6.25% of those users are still around on a week

four and that's a pretty low amount right so you probably want that to be somewhere between 20 or 30 at least and so you can set a goal saying I'm going to talk to a bunch of these target users and try to figure out you know why they're not getting value out of the product and then make some changes as well okay so what metric do you actually pick this is taken from one of gustaf slides pretty awesome you think about what value your company is giving to your users so Airbnb gives you value by letting you stay at different rental properties around the world right and they want you to do that at least one time a year otherwise they consider you

a churned user equivalently Facebook gives you value by letting you look at the newsfeed and you know connect with your friends and they want you to do that at least daily or monthly once and so when you think about product market fit you basically have these two different curves that happen so we have that cohort of 16 users that signed up in one a week and we track them over time and so what ends up happening is for products that don't have product market fit they end up tending to go to zero because people just don't care about the product right and that's definition product market fit for those tools and

the products that do have product market fit you'll see some kind of natural plateau don't mind this axis it should be somewhere you know between 20 and 30 percent at least okay so how do you create this graph it seems kind of complicated right luckily both Mixpanel amplitude have really awesome reports for this so in amplitude it's a retention analysis report you say users enter the cohort with user signup and then they return with the video player to the subscription upgraded event that's the value event and then you press next and out comes this graph and then you could look at for week

retention for cohorts improve the product and watch as new cohorts that strike the four week mark do better or worse right and that shows you whether your changes week over week are actually improving okay finally a revenue so this is the primary metric they want to be thinking about you'll use for a subscription business the subscription upgraded event you'll do a property sum over new plan monthly recurring revenue you press next and outcomes your weekly that your revenue graph and then you

could set monthly goals on top of this to make sure you're growing the at the rates that you want to be okay finally I if you have a founding team it's really good to basically put all the stuff on a dashboard and then put this dashboard on a TV in your office it's incredibly incredibly important basically this is kind of the the difference between being a data-driven team and not a data-driven team a lot of founders actually set up their analytics but then don't look at them ever again because it can be painful right while a data-driven team will put it on a TV and talk about projects talk about what

those projects are actually changing the metrics that they're trying to drive and then just completely understand the business every single day right and this kind of company in this kind of founder will actually scale to build better high performance companies because the next team of employees they hire will also be looking at that same TV dashboard and be driven off those same metrics so really important get a TV next what you want to do is have some kind of social accountability around your metrics so if you have your friends your parents your advisors your investors package up how your business is doing into an email this helps you synthesize what is

actually happening and then send it out to to those advisers and tell them where the business is struggling and what your plan is to fix it this allows the advisors to quickly understand the business and then respond back with much more appropriate advice cool and now we'll go into the startup stack so these are tools that we recommend that help this kind of tactical process of setting up these metrics so I'm gonna talk a little bit about that MVP business business workflow that that Michael talked about earlier so initially you're

building an MVP segment built about seven different MVPs before we actually found segments and all of those failed and eventually we found we found one that worked and the process of actually building that BP is incredibly important so once you have that little experiment built you want to enter private beta which basically just means getting like 10 20 30 customers to actually try this product and then having very direct lines of communication open with them what segment does nowadays every new product we ship we open slack channels with each one of our customers and we have the product managers sit in those

slack channels and talk with the customers for the products that don't get product market fit the customers just stop responding and we're asking asking asking they're not responding right and for the products that do have product market fit the customers are immediately being like oh we don't why don't you have this feature this is broken might I tread inviting my team and this is network not working so instead of you kind of pulling at the customer the customer starts pulling at you that's that's a good feeling of product market fit okay so at some point the private beta is going well you feel like people really care about this you

understand your target customer then you want to get a larger market segment to use it that's the launch that we talked about earlier try to get there as quickly as possible and then a launch is just more users that you get to test product market fit on and so if you feel product market fit there then you can start scaling the company right and you hire salespeople and you start doing paid paid marketing and things like that so different tools will guide you throughout this throughout this process so as you're building an MVP and you're about to give it to the first group of customers install Google Analytics install amplitude Google Analytics will

tell you who's coming from the Internet to your website and an amplitude will tell you which features are they using how engaged are they with that feature set unless you're able to stand over the shoulders of all of your users 100% of the time analytics is the next best alternative for that we also install live chat on the page so either slack with your customers or if you can't do that then maybe have a live chat available in the beginning of segment customers would ping us you know day and night and that's where we got the most valuable feedback from them so just as many open channels of

communication as possible next data warehouse this is something that we recommend it used to be expensive is no longer expensive today basically if you have a non technical co-founder on your team but want to ask questions around the data and they'll always ask the technical co-founder we'll have to provide the answers so data warehouse kind of democratizes the data not only for the non-technical co-founders but for everyone else in the company that you hire after company dashboards obviously I we should probably move that to the left email and push tools so as soon as you as soon as

user sign up you want to send them an email I'll talk about that in a second and then and then a help desk so at some point you'll have so much support tickets if you start feeling product market fit and if they're all going to your Gmail one founder will just get overwhelmed and not be able to answer them so you want to have a shared inbox where multiple founders can I can respond okay now I'm gonna go through a few different recipes of these different tools there we found really helpful in product market fit so the first one is improving product usability almost every product that's launched is unusable or highly

unusable for the first three months while you have the kinks and we see this with every single product no matter how much effort we put into it ahead of time as soon as customers hit it they start using it in ways that you just don't expect and so there's this tool called full story which helps you look at sessions of customers as they use your website so I'll tell a quick story on this this is a new feature in personas which is one of segments products and we launched them the metrics looked horrible so like customers are coming in but they weren't actually completing it they weren't they weren't using the product and we thought oh god you know

this likely doesn't have product market fair we have to go back to the drawing board then one of the designers and her team had this amazing idea let's look at the full story and so we see this user going in about to start this creation workflow they find this button they clearly don't understand what the button does they get so frustrated they just exit the page and so we saw this with a multiple with multiple different customers coming in and so we're like okay we just have to fix that button we fixed that button immediately all the metrics got better right and so that's why it's important

to have this type of viewing either to stand over your customers shoulders which is great what the stripe co-founders did or get a full story which is more scalable way to do that okay I call this the 43 minute founder email so when we launch a segment we would wait about 43 minutes and we would email the code or email the customer and say hey I'm Ilya thanks so much for signing up for segment your next step here is to add a source to segment and if you have any questions at all please email me or call me anytime I'm available for you since we launched that email in 2013 we've had hundreds of

thousands of responses to it so it's it's the connection between you and the customer over email that if they get confused they'll respond to it what you can use is a tool called customer IO it's a behavioral email tool which will say every time a user signs up wait you know 30 minutes 40 minutes 50 minutes whatever and then automatically send them this content and you could template the first name the company name and so forth based off of your analytics data so huge recipe recommend and then finally for democratizing data access I say this is more advanced so this is

after you're in your MVP stage you're feeling good about product market fit you might want to install data warehouse like Google bigquery and then mode analytics is a bi tool that works on top of it this lets you ask questions on top of the raw data that you might not be able to do an amplitude and mix panel just any kind of question you can ask for sequel and then even the non-technical co-founders will eventually pick up sequel and then start asking these questions themselves okay one common failure mode that we see with customers is trying to pick the perfect tool mix panel or amplitude you know bigquery or redshift and spending way

way way too long thinking about that the truth of the matter is you shouldn't optimize for picking the right tool right now both amplitude and mix panel will give you exactly the same result at your stage instead just get through that decision as quickly as possible but set yourself up for change in the future so this is an example diagram that shows a customer of segments that use different tools over a period of about three years and so you can see that they used about eight different tools between 2015 and 2017 and they either hired someone or they decided that their tools are no

longer doing the job and they switch from one set of tools to another so best-in-class tools change every two years just be prepared for a change and don't spend too much time trying to perfect your choice right now okay this is my recommendation of what tools are the best for the MVP process I'll walk you through them right now so Google Analytics is for understanding what users are coming to your website amplitude is for feature analytics Google bigquery is to democratize data access with a data warehouse which is just a database of data mode is the the

tool use on top of that to ask questions on top of bigquery intercom is like a list of all of your customers they're really good CRM for early-stage folks full story is for improving product usability in customer iOS for emailing your customers once you get big enough you can start using Google Ads Facebook Ads to actually do the data acquisition okay so I'll just jump straight to the slide when we were really young we when we were in 2011-2012 we didn't have a lot of money and so we didn't want to use all these different tools and so we

used Google spreadsheet for a CRM we didn't use asana or Trello because they were too expensive so we used emails and the only things we paid for we're basically github and AWS to host our products right so today happy to say the segment is now free for early-stage startups so for all you folks there's a bitly link down here that will send that afterwards and it's in the deals and then also segments went out and we did a bunch of deals with these customers so that if you're early-stage you can now get all of these tools for free so enjoy that and start using them to basically accelerate your product market fit

process thank you [Music] you're asking about growth rate so you're asking it if you're starting with

a very small base of users what growth rate you need to be on to make to make growth meaningful yeah it completely depends on the product right so if you're an enterprise b2b company then your closing deals for you know 10 15 20 thousand dollars then only a few of those customers can make I can give you enough capital to hire more people to do more marketing entire larger sales team if you are selling you know shirts and

pants then you probably need a higher volume so totally depends on it so it depends the kind of company you're building yep so the question is what kind of what

kind of retention period do you want to look at if I if you're renting properties like an Airbnb yeah that's a really good question it really those kind of periods depend on the kind of company you're building so I'm giving one presentation but there's hundreds of different companies that that need to apply the information slightly differently so if your Airbnb you probably would expect people to at least come back to your app you know like how often do people travel right they travel you know once a month once a quarter right and so that's the that's

the period that you want to make sure that they come back at yeah so it's not like week is correct for everyone but it's correct for the majority of companies exactly keep a pulse yep awesome all right well thank you so much [Applause] you
