05. Tracking Metrics

Estimated Time

• Reading: ~10 minutes
• Video: ~17 minutes
• Activities: to be completed prior to the next week

Insights

• Clearly define metrics that can be measured and analyzed
• Don't focus on vanity metrics
• Use cohort analysis to improve retention and product development

• As a startup founder, you have limited time and money to make your company a success
• You can not afford to waste either of these resources
• One of the best ways to work efficiently is to track key metrics
• Tracking metrics will help you identify what is working and what isn't as quickly as possible
• We mentioned in a previous lesson the importance of creating a "Start of Day" and "End of Day" report
• This documentation is a nice start but to get the most out of it you need to take it a step further
• Which items are you working on daily have your main goals - your single value proposition - in mind?
• Are your tasks well defined? Are they measurable?
• If you can't measure the impact of your work how will you know if you should continue working on it?
• As a small team, you may have a general understanding of how your inputs are driving outputs
• But the sooner you can get that process out of your head, and document it — the better
• As your team grows every member of the team, and every project, should have key metrics to be tracked
• If you have well-documented metrics you can communicate results more efficiently with your team
• When results are impressive other members of your team can copy the input process
• When results are not what you had hoped you can identify why and adjust
• LAUNCH focuses on Amazon's method of working backwards to track inputs that drive outputs
• The book Working Backwards speaks to how Amazon thinks about tracking metrics
• They use the acronym DMAIC (copied from Six Sigma) for all metrics they track
• Define, Measure, Analyze, Improve, and Control
• Define
• When defining a metric it must fall into one of the portions of Amazon's flywheel that Jeff Bezos famously developed on the back of a napkin (see image)

• When defining the metrics the top priority is to clearly state the controllable inputs
• As the leader of the project what variables are completely in your control?
• Which of these inputs will drive the outputs?
• Amazon thinks so deeply "working backwards" that they have their leaders write the press release for a new product first and then figure out how to deliver it
• Defining what will be measured is vital and will dictate the success of the project
• Measure
• To put a little spin on Peter Drucker's famous quote... "If you can measure it, you can manage it"
• Being able to measure results and compare them over time will help you make educated decisions going forward
• One of the keys to using metrics is to ensure there is no bias in the data
• If you and your team can not trust the data then you might as well not have it
• Conduct regular reviews of the metrics to ensure you are measuring what you intended to measure
• Analyze
• After you measure the metrics and present them it is important to analyze them
• What is the data you are gathering telling you?
• Was your hypothesis correct?
• Do you need to make changes to your inputs?
• What could be dictating the results?
• The goal of the analysis is to get to the root cause
• Amazon implements the 5 whys process to surface the root cause while analyzing the results
• Improve
• You must have a solid foundation to begin improving
• It is so important to have well-defined metrics, that can be measured, with a clear process to analyze the results without bias
• Once that is in place you can begin to make decisions on what to change in order to improve
• As much as you might want to jump right to improving the process, you mustn't skip steps
• If you don't understand the root cause you may spend a lot of time addressing a symptom rather than the problem
• With limited time and money, this can be detrimental
• Control
• Once improvements are made and the process is stable you have control
• You can quickly look for abnormalities once you understand exactly what to expect
• Eventually, you can look to automate parts of this process and only focus on it when there are issues
• It is important to keep in mind that you can only directly change inputs, you can't directly change outputs
• The outputs are the results of the inputs
• Identify the main goal and what it will take to get there
• Then focus on what you can control
• Measure the results, analyze what they are telling you, and improve until the process is at a point that it can essentially run itself
• Read, Working Backwards for a deeper dive into this concept implemented at Amazon and adopted by the LAUNCH team

• So what metrics should you measure?
• It really depends on your company, what you value, and what your goals are but there are a few things everyone can focus on depending on:
• The stage of your startup
• Creating an MVP: The Art of Small Steps mentions a few of the common metrics for any business early on

• How many people are signing up and ultimately continue using your product?
• How much does it cost to acquire them?
• How much revenue do they bring to your company over time?
• Are you delighting them?
• As your company matures these metrics are still at the foundation, but you may prioritize additional metrics
• Lolita Taub tweeted the chart below as a guide to which metrics VCs care about most

• Justin Kan posted about the importance of being honest with your metrics
• As we will mention in a moment, Justin highlights how leading with vanity metrics can be a red flag, and although not technically lying gives the impression that you are misleading
• It is obviously a bad idea to mislead investors
• But it is also dangerous to mislead yourself by tracking metrics that make you feel good but don't provide value (more on this in a moment)
• https://twitter.com/justinkan/status/1466554184568737793?s=20
• Also what metrics you track depends on your business model
• Consumer Subscription
• Since this model relies on individual users using your product it is important to track
• Daily and weekly active users (DAU and WAU)
• Measure the retention of their usage from day-to-day
• It is always a good idea to understand your customer acquisition cost (CAC)
• Acquiring customers through marketing is often one of the biggest spends
• You need to understand how much it costs you to get a new customer
• Once you have a customer you need to understand the lifetime value (LTV) of that customer
• If you understand your CAC and LTV, as Jason points out, you can then identify a winning formula
• It is important to track churn as a consumer subscription since the individual is only responsible for their own license and they can churn quickly
• Enterprise SaaS
• Some of the metrics are similar to Consumer SaaS
• You always want to know your CAC and LTV
• Monthly Recurring Revenue (MRR) is important as an enterprise SaaS provider
• Another metric to track is the number of land and expand deals
• After you get a large enterprise customer how does your product expand within the organization?
• For example, how many new licenses are sold after landing the initial license in the enterprise?
• Churn is usually lower at the enterprise level but is always worth noting as a founder
• It doesn't play as big of a role as it does for consumer subscriptions in the eyes of an investor however
• Marketplace
• The frequency of transactions is a key metric in a marketplace where money is generally made off of each transaction
• If the transactions are high revenue, like an Airbnb rental they don't need to happen as often
• With lower revenue transactions, like taking an Uber, the number of transactions needs to be much higher
• It is important to know the total amount of product that is sold over a given time period in a marketplace; thus the need to track gross merchandise value (GMV)
• The take rate, or how much you getting from each of the transactions, is another important metric to be aware of
• And as with the other business models being aware of CAC, LTV, and the retention rate of users is something you'll want to know
• A Founder's Guide to Startup Metrics is a great reference if you're looking for more insight on how to go about calculating any of these metrics

• Cohort analysis
• Eric Ries believes cohort analysis is one of the single most important metrics to track
• Cohort analysis is the grouping of users with similar characteristics to better track and analyze their behavior
• This is often done by the date they begin using your product
• Typically represented in a graph over time as a percent to give better insight

• Grouping your users into cohorts provides insight over the lifetime of the user as well as data on the retention of your product over time
• These insights allow you to make decisions based on the data
• Graphing cohorts will allow you to understand user activity at a glance
• You can anticipate where usage of future groups will drop off and be proactive
• This helps with limiting churn and developing a more engaging product
• Product development insights
• The initial days of the cohort typically represent the onboarding process
• If you are seeing cohorts leave your product in these early days then there is a problem with understanding what your product does and the simplicity of use from the start
• This same concept can be used around key deployments or the addition of new features
• Cohort analysis will quickly highlight if cohorts stick with your product as you release these features
• It will show you both a positive and negative impact on the user's behavior
• Cohort analysis provides feedback from several independent groups
• You can understand which customers leave and when
• Based on recent activities you may even understand why they leave
• This analysis also provides actionable insights
• If every group is experiencing the same drop off at the same time then you know there is a problem
• You can also run split-tests (tests run at the same time with two different cohorts) and track outcomes
• Cohort analysis is tracking the outputs that allow you to adjust your actionable inputs
• You can dive deeper into Cohort Analysis with the blog: Cohort Analysis: Beginners Guide to Improving Retention

• When thinking about metrics to track Ries discusses "The Value of the Three A's"
• Actionable
• The metric needs to have a cause and effect
• Don't get caught up in vanity metrics
• These are metrics that make you feel good but don't drive action
• Things like total downloads of your application
• This is a metric that can only go up
• Tracking a metric like this does not give you any idea how many people are using it or if you should have had more based on previous trends
• As Justin Kan tweeted, don't try to tell a story that the data doesn't support
• Actionable metrics allow you to learn from the actions and improve
• This goes back to controllable inputs that drive the outputs
• Focus on the inputs!
• Accessible
• The metric reports need to be understood by your team
• They need to be easy to understand
• Everyone should have access to the key results
• The metrics are supposed to help you make decisions
• If it is hard to understand the report it is hard to make informed decisions
• When this happens people often make the data tell them what they want to hear rather than listening to what it is telling them
• Auditable
• In order for metrics to have a meaningful impact employees need to trust the data
• Can the data be easily tested and confirmed?
• Is the report generation process simple?
• Have conversations about the results and analyze and improve the inputs
• Report with Graphs
• There is going to be a lot of data to consume, use graphs to tell the story
• The reason you are tracking data is to identify trends or patterns
• When you have hundreds of data points across several projects and key metrics this can be quite hard to interpret without a visual representation
• The graphs will allow you to more accurately analyze metrics over time
• For example, simply saying you added 100 customers last week is nice, but if you had added 300+ the previous weeks the data is now telling you something different
• You need to see the whole picture
• Don't focus on vanity metrics — if they only go up, or don't provide you actionable insights they aren't as meaningful
• Track metrics that are important to you based on your goals
• Have regular discussions about what is represented on the graphs
• Have the hard conversations if you notice that things are trending in the wrong direction
• Your company will evolve and the metrics you prioritize should too
• There are some general metrics that all startups should track which aligns with our startup flywheel
• Are you delighting customers?
• Are you adding customers at a meaningful rate?
• Are you losing customers?
• Do you have a great product?
• Do customers use your product regularly?
• Are your new features improving retention?
• Do you have a great team?
• Does your team track their controllable inputs?
• Do they analyze the key metrics for feedback?
• Tracking meaningful metrics can greatly impact customer retention, product development, and the overall success of your company
• Startups have limited time and money, leverage the data to make informed decisions
• It all starts with identifying and tracking the metrics that are meaningful to your company and doing so as early as possible

Additional Resources

• Startup Metrics You Need to Monitor
• SaaS Metrics that Matter
• Marketplace metrics
• Breaking Down The Three Engines Of Growth
• Cohort Analysis Explained with an Excel example (with template)
• How to build a simple growth dashboard (with template)
• The A-Z guide to startup metrics
• SaaS KPIs: Tracking The Right Metrics On Your Website And Grow Your Sales
• Metrics to Help You Make Wise Decisions About Your Start-Up
• Measure What Matters

Activities