05. Tracking Metrics

Estimated Time

  • Reading: ~10 minutes
  • Video: ~17 minutes
  • Activities: to be completed prior to the next week

Insights

  • Clearly define metrics that can be measured and analyzed
  • Don't focus on vanity metrics
  • Use cohort analysis to improve retention and product development

Episode Date: Sep 7, 2016 — Link to Video

Jason Calacanis | TWiST | Twitter | LinkedIn

  • As a startup founder, you have limited time and money to make your company a success
  • You can not afford to waste either of these resources
  • One of the best ways to work efficiently is to track key metrics
  • Tracking metrics will help you identify what is working and what isn't as quickly as possible
  • We mentioned in a previous lesson the importance of creating a "Start of Day" and "End of Day" report
    • This documentation is a nice start but to get the most out of it you need to take it a step further
    • Which items are you working on daily have your main goals - your single value proposition - in mind?
      • Are your tasks well defined? Are they measurable?
      • If you can't measure the impact of your work how will you know if you should continue working on it?
    • As a small team, you may have a general understanding of how your inputs are driving outputs
    • But the sooner you can get that process out of your head, and document it — the better
      • As your team grows every member of the team, and every project, should have key metrics to be tracked
      • If you have well-documented metrics you can communicate results more efficiently with your team
        • When results are impressive other members of your team can copy the input process
        • When results are not what you had hoped you can identify why and adjust
  • LAUNCH focuses on Amazon's method of working backwards to track inputs that drive outputs
    • The book Working Backwards speaks to how Amazon thinks about tracking metrics
    • They use the acronym DMAIC (copied from Six Sigma) for all metrics they track
      • Define, Measure, Analyze, Improve, and Control
      • Define
        • When defining a metric it must fall into one of the portions of Amazon's flywheel that Jeff Bezos famously developed on the back of a napkin (see image)
        • image
        • When defining the metrics the top priority is to clearly state the controllable inputs
          • As the leader of the project what variables are completely in your control?
          • Which of these inputs will drive the outputs?
          • Amazon thinks so deeply "working backwards" that they have their leaders write the press release for a new product first and then figure out how to deliver it
          • Defining what will be measured is vital and will dictate the success of the project
      • Measure
        • To put a little spin on Peter Drucker's famous quote... "If you can measure it, you can manage it"
        • Being able to measure results and compare them over time will help you make educated decisions going forward
        • One of the keys to using metrics is to ensure there is no bias in the data
          • If you and your team can not trust the data then you might as well not have it
        • Conduct regular reviews of the metrics to ensure you are measuring what you intended to measure
      • Analyze
        • After you measure the metrics and present them it is important to analyze them
        • What is the data you are gathering telling you?
          • Was your hypothesis correct?
          • Do you need to make changes to your inputs?
          • What could be dictating the results?
        • The goal of the analysis is to get to the root cause
        • Amazon implements the 5 whys process to surface the root cause while analyzing the results
      • Improve
        • You must have a solid foundation to begin improving
        • It is so important to have well-defined metrics, that can be measured, with a clear process to analyze the results without bias
        • Once that is in place you can begin to make decisions on what to change in order to improve
        • As much as you might want to jump right to improving the process, you mustn't skip steps
          • If you don't understand the root cause you may spend a lot of time addressing a symptom rather than the problem
          • With limited time and money, this can be detrimental
      • Control
        • Once improvements are made and the process is stable you have control
        • You can quickly look for abnormalities once you understand exactly what to expect
        • Eventually, you can look to automate parts of this process and only focus on it when there are issues
    • It is important to keep in mind that you can only directly change inputs, you can't directly change outputs
      • The outputs are the results of the inputs
      • Identify the main goal and what it will take to get there
      • Then focus on what you can control
        • Measure the results, analyze what they are telling you, and improve until the process is at a point that it can essentially run itself
    • Read, Working Backwards for a deeper dive into this concept implemented at Amazon and adopted by the LAUNCH team

  • So what metrics should you measure?
    • It really depends on your company, what you value, and what your goals are but there are a few things everyone can focus on depending on:
      • The stage of your startup
        • Creating an MVP: The Art of Small Steps mentions a few of the common metrics for any business early on
        • image
        • How many people are signing up and ultimately continue using your product?
        • How much does it cost to acquire them?
        • How much revenue do they bring to your company over time?
        • Are you delighting them?
        • As your company matures these metrics are still at the foundation, but you may prioritize additional metrics
        • Lolita Taub tweeted the chart below as a guide to which metrics VCs care about most
        • image
        • Justin Kan posted about the importance of being honest with your metrics
        • As we will mention in a moment, Justin highlights how leading with vanity metrics can be a red flag, and although not technically lying gives the impression that you are misleading
      • Also what metrics you track depends on your business model
        • Consumer Subscription
          • Since this model relies on individual users using your product it is important to track
          • Daily and weekly active users (DAU and WAU)
          • Measure the retention of their usage from day-to-day
          • It is always a good idea to understand your customer acquisition cost (CAC)
            • Acquiring customers through marketing is often one of the biggest spends
            • You need to understand how much it costs you to get a new customer
          • Once you have a customer you need to understand the lifetime value (LTV) of that customer
            • If you understand your CAC and LTV, as Jason points out, you can then identify a winning formula
          • It is important to track churn as a consumer subscription since the individual is only responsible for their own license and they can churn quickly
        • Enterprise SaaS
          • Some of the metrics are similar to Consumer SaaS
            • You always want to know your CAC and LTV
          • Monthly Recurring Revenue (MRR) is important as an enterprise SaaS provider
          • Another metric to track is the number of land and expand deals
            • After you get a large enterprise customer how does your product expand within the organization?
            • For example, how many new licenses are sold after landing the initial license in the enterprise?
          • Churn is usually lower at the enterprise level but is always worth noting as a founder
            • It doesn't play as big of a role as it does for consumer subscriptions in the eyes of an investor however
        • Marketplace
          • The frequency of transactions is a key metric in a marketplace where money is generally made off of each transaction
            • If the transactions are high revenue, like an Airbnb rental they don't need to happen as often
            • With lower revenue transactions, like taking an Uber, the number of transactions needs to be much higher
          • It is important to know the total amount of product that is sold over a given time period in a marketplace; thus the need to track gross merchandise value (GMV)
          • The take rate, or how much you getting from each of the transactions, is another important metric to be aware of
          • And as with the other business models being aware of CAC, LTV, and the retention rate of users is something you'll want to know
        • A Founder's Guide to Startup Metrics is a great reference if you're looking for more insight on how to go about calculating any of these metrics

Episode Date Apr 11, 2019 — Link to Video

Jason Calacanis | TWiST | Twitter | LinkedIn

  • Cohort analysis
    • Eric Ries believes cohort analysis is one of the single most important metrics to track
    • Cohort analysis is the grouping of users with similar characteristics to better track and analyze their behavior
      • This is often done by the date they begin using your product
      • Typically represented in a graph over time as a percent to give better insight
      • image
    • Grouping your users into cohorts provides insight over the lifetime of the user as well as data on the retention of your product over time
      • These insights allow you to make decisions based on the data
      • Graphing cohorts will allow you to understand user activity at a glance
        • You can anticipate where usage of future groups will drop off and be proactive
        • This helps with limiting churn and developing a more engaging product
      • Product development insights
        • The initial days of the cohort typically represent the onboarding process
        • If you are seeing cohorts leave your product in these early days then there is a problem with understanding what your product does and the simplicity of use from the start
        • This same concept can be used around key deployments or the addition of new features
          • Cohort analysis will quickly highlight if cohorts stick with your product as you release these features
          • It will show you both a positive and negative impact on the user's behavior
    • Cohort analysis provides feedback from several independent groups
      • You can understand which customers leave and when
      • Based on recent activities you may even understand why they leave
      • This analysis also provides actionable insights
        • If every group is experiencing the same drop off at the same time then you know there is a problem
        • You can also run split-tests (tests run at the same time with two different cohorts) and track outcomes
    • Cohort analysis is tracking the outputs that allow you to adjust your actionable inputs
    • You can dive deeper into Cohort Analysis with the blog: Cohort Analysis: Beginners Guide to Improving Retention

  • When thinking about metrics to track Ries discusses "The Value of the Three A's"
    • Actionable
      • The metric needs to have a cause and effect
      • Don't get caught up in vanity metrics
        • These are metrics that make you feel good but don't drive action
        • Things like total downloads of your application
          • This is a metric that can only go up
          • Tracking a metric like this does not give you any idea how many people are using it or if you should have had more based on previous trends
        • As Justin Kan tweeted, don't try to tell a story that the data doesn't support
      • Actionable metrics allow you to learn from the actions and improve
      • This goes back to controllable inputs that drive the outputs
      • Focus on the inputs!
    • Accessible
      • The metric reports need to be understood by your team
        • They need to be easy to understand
        • Everyone should have access to the key results
      • The metrics are supposed to help you make decisions
      • If it is hard to understand the report it is hard to make informed decisions
      • When this happens people often make the data tell them what they want to hear rather than listening to what it is telling them
    • Auditable
      • In order for metrics to have a meaningful impact employees need to trust the data
        • Can the data be easily tested and confirmed?
        • Is the report generation process simple?
      • Have conversations about the results and analyze and improve the inputs
  • Report with Graphs
    • There is going to be a lot of data to consume, use graphs to tell the story
      • The reason you are tracking data is to identify trends or patterns
        • When you have hundreds of data points across several projects and key metrics this can be quite hard to interpret without a visual representation
      • The graphs will allow you to more accurately analyze metrics over time
        • For example, simply saying you added 100 customers last week is nice, but if you had added 300+ the previous weeks the data is now telling you something different
          • You need to see the whole picture
        • Don't focus on vanity metrics — if they only go up, or don't provide you actionable insights they aren't as meaningful
    • Track metrics that are important to you based on your goals
    • Have regular discussions about what is represented on the graphs
    • Have the hard conversations if you notice that things are trending in the wrong direction
    • Your company will evolve and the metrics you prioritize should too
  • There are some general metrics that all startups should track which aligns with our startup flywheel
    • Are you delighting customers?
      • Are you adding customers at a meaningful rate?
      • Are you losing customers?
    • Do you have a great product?
      • Do customers use your product regularly?
      • Are your new features improving retention?
    • Do you have a great team?
      • Does your team track their controllable inputs?
      • Do they analyze the key metrics for feedback?
    • Tracking meaningful metrics can greatly impact customer retention, product development, and the overall success of your company
    • Startups have limited time and money, leverage the data to make informed decisions
    • It all starts with identifying and tracking the metrics that are meaningful to your company and doing so as early as possible

Additional Resources

Activities

Week 8

🔲 Define key metrics

🔲 Begin to group users by cohorts for analysis

Week 9

🔲 Analyze metrics

🔲 Analyze cohorts