Estimated Time
- Reading: ~10 minutes
- Video: ~17 minutes
- Activities: to be completed prior to the next week
Insights
- Clearly define metrics that can be measured and analyzed
- Don't focus on vanity metrics
- Use cohort analysis to improve retention and product development
Episode Date: Sep 7, 2016 — Link to Video
- As a startup founder, you have limited time and money to make your company a success
- You can not afford to waste either of these resources
- One of the best ways to work efficiently is to track key metrics
- Tracking metrics will help you identify what is working and what isn't as quickly as possible
- We mentioned in a previous lesson the importance of creating a "Start of Day" and "End of Day" report
- This documentation is a nice start but to get the most out of it you need to take it a step further
- Which items are you working on daily have your main goals - your single value proposition - in mind?
- Are your tasks well defined? Are they measurable?
- If you can't measure the impact of your work how will you know if you should continue working on it?
- As a small team, you may have a general understanding of how your inputs are driving outputs
- But the sooner you can get that process out of your head, and document it — the better
- As your team grows every member of the team, and every project, should have key metrics to be tracked
- If you have well-documented metrics you can communicate results more efficiently with your team
- When results are impressive other members of your team can copy the input process
- When results are not what you had hoped you can identify why and adjust
- LAUNCH focuses on Amazon's method of working backwards to track inputs that drive outputs
- The book Working Backwards speaks to how Amazon thinks about tracking metrics
- They use the acronym DMAIC (copied from Six Sigma) for all metrics they track
- Define, Measure, Analyze, Improve, and Control
- Define
- When defining a metric it must fall into one of the portions of Amazon's flywheel that Jeff Bezos famously developed on the back of a napkin (see image)
- When defining the metrics the top priority is to clearly state the controllable inputs
- As the leader of the project what variables are completely in your control?
- Which of these inputs will drive the outputs?
- Amazon thinks so deeply "working backwards" that they have their leaders write the press release for a new product first and then figure out how to deliver it
- Defining what will be measured is vital and will dictate the success of the project
- Measure
- To put a little spin on Peter Drucker's famous quote... "If you can measure it, you can manage it"
- Being able to measure results and compare them over time will help you make educated decisions going forward
- One of the keys to using metrics is to ensure there is no bias in the data
- If you and your team can not trust the data then you might as well not have it
- Conduct regular reviews of the metrics to ensure you are measuring what you intended to measure
- Analyze
- After you measure the metrics and present them it is important to analyze them
- What is the data you are gathering telling you?
- Was your hypothesis correct?
- Do you need to make changes to your inputs?
- What could be dictating the results?
- The goal of the analysis is to get to the root cause
- Amazon implements the 5 whys process to surface the root cause while analyzing the results
- Improve
- You must have a solid foundation to begin improving
- It is so important to have well-defined metrics, that can be measured, with a clear process to analyze the results without bias
- Once that is in place you can begin to make decisions on what to change in order to improve
- As much as you might want to jump right to improving the process, you mustn't skip steps
- If you don't understand the root cause you may spend a lot of time addressing a symptom rather than the problem
- With limited time and money, this can be detrimental
- Control
- Once improvements are made and the process is stable you have control
- You can quickly look for abnormalities once you understand exactly what to expect
- Eventually, you can look to automate parts of this process and only focus on it when there are issues
- It is important to keep in mind that you can only directly change inputs, you can't directly change outputs
- The outputs are the results of the inputs
- Identify the main goal and what it will take to get there
- Then focus on what you can control
- Measure the results, analyze what they are telling you, and improve until the process is at a point that it can essentially run itself
- Read, Working Backwards for a deeper dive into this concept implemented at Amazon and adopted by the LAUNCH team
- So what metrics should you measure?
- It really depends on your company, what you value, and what your goals are but there are a few things everyone can focus on depending on:
- The stage of your startup
- Creating an MVP: The Art of Small Steps mentions a few of the common metrics for any business early on
- How many people are signing up and ultimately continue using your product?
- How much does it cost to acquire them?
- How much revenue do they bring to your company over time?
- Are you delighting them?
- As your company matures these metrics are still at the foundation, but you may prioritize additional metrics
- Lolita Taub tweeted the chart below as a guide to which metrics VCs care about most
- Justin Kan posted about the importance of being honest with your metrics
- As we will mention in a moment, Justin highlights how leading with vanity metrics can be a red flag, and although not technically lying gives the impression that you are misleading
- It is obviously a bad idea to mislead investors
- But it is also dangerous to mislead yourself by tracking metrics that make you feel good but don't provide value (more on this in a moment)
- https://twitter.com/justinkan/status/1466554184568737793?s=20
- Also what metrics you track depends on your business model
- Consumer Subscription
- Since this model relies on individual users using your product it is important to track
- Daily and weekly active users (DAU and WAU)
- Measure the retention of their usage from day-to-day
- It is always a good idea to understand your customer acquisition cost (CAC)
- Acquiring customers through marketing is often one of the biggest spends
- You need to understand how much it costs you to get a new customer
- Once you have a customer you need to understand the lifetime value (LTV) of that customer
- If you understand your CAC and LTV, as Jason points out, you can then identify a winning formula
- It is important to track churn as a consumer subscription since the individual is only responsible for their own license and they can churn quickly
- Enterprise SaaS
- Some of the metrics are similar to Consumer SaaS
- You always want to know your CAC and LTV
- Monthly Recurring Revenue (MRR) is important as an enterprise SaaS provider
- Another metric to track is the number of land and expand deals
- After you get a large enterprise customer how does your product expand within the organization?
- For example, how many new licenses are sold after landing the initial license in the enterprise?
- Churn is usually lower at the enterprise level but is always worth noting as a founder
- It doesn't play as big of a role as it does for consumer subscriptions in the eyes of an investor however
- Marketplace
- The frequency of transactions is a key metric in a marketplace where money is generally made off of each transaction
- If the transactions are high revenue, like an Airbnb rental they don't need to happen as often
- With lower revenue transactions, like taking an Uber, the number of transactions needs to be much higher
- It is important to know the total amount of product that is sold over a given time period in a marketplace; thus the need to track gross merchandise value (GMV)
- The take rate, or how much you getting from each of the transactions, is another important metric to be aware of
- And as with the other business models being aware of CAC, LTV, and the retention rate of users is something you'll want to know
- A Founder's Guide to Startup Metrics is a great reference if you're looking for more insight on how to go about calculating any of these metrics
Episode Date Apr 11, 2019 — Link to Video
- Cohort analysis
- Eric Ries believes cohort analysis is one of the single most important metrics to track
- Cohort analysis is the grouping of users with similar characteristics to better track and analyze their behavior
- This is often done by the date they begin using your product
- Typically represented in a graph over time as a percent to give better insight
- Grouping your users into cohorts provides insight over the lifetime of the user as well as data on the retention of your product over time
- These insights allow you to make decisions based on the data
- Graphing cohorts will allow you to understand user activity at a glance
- You can anticipate where usage of future groups will drop off and be proactive
- This helps with limiting churn and developing a more engaging product
- Product development insights
- The initial days of the cohort typically represent the onboarding process
- If you are seeing cohorts leave your product in these early days then there is a problem with understanding what your product does and the simplicity of use from the start
- This same concept can be used around key deployments or the addition of new features
- Cohort analysis will quickly highlight if cohorts stick with your product as you release these features
- It will show you both a positive and negative impact on the user's behavior
- Cohort analysis provides feedback from several independent groups
- You can understand which customers leave and when
- Based on recent activities you may even understand why they leave
- This analysis also provides actionable insights
- If every group is experiencing the same drop off at the same time then you know there is a problem
- You can also run split-tests (tests run at the same time with two different cohorts) and track outcomes
- Cohort analysis is tracking the outputs that allow you to adjust your actionable inputs
- You can dive deeper into Cohort Analysis with the blog: Cohort Analysis: Beginners Guide to Improving Retention
- When thinking about metrics to track Ries discusses "The Value of the Three A's"
- Actionable
- The metric needs to have a cause and effect
- Don't get caught up in vanity metrics
- These are metrics that make you feel good but don't drive action
- Things like total downloads of your application
- This is a metric that can only go up
- Tracking a metric like this does not give you any idea how many people are using it or if you should have had more based on previous trends
- As Justin Kan tweeted, don't try to tell a story that the data doesn't support
- Actionable metrics allow you to learn from the actions and improve
- This goes back to controllable inputs that drive the outputs
- Focus on the inputs!
- Accessible
- The metric reports need to be understood by your team
- They need to be easy to understand
- Everyone should have access to the key results
- The metrics are supposed to help you make decisions
- If it is hard to understand the report it is hard to make informed decisions
- When this happens people often make the data tell them what they want to hear rather than listening to what it is telling them
- Auditable
- In order for metrics to have a meaningful impact employees need to trust the data
- Can the data be easily tested and confirmed?
- Is the report generation process simple?
- Have conversations about the results and analyze and improve the inputs
- Report with Graphs
- There is going to be a lot of data to consume, use graphs to tell the story
- The reason you are tracking data is to identify trends or patterns
- When you have hundreds of data points across several projects and key metrics this can be quite hard to interpret without a visual representation
- The graphs will allow you to more accurately analyze metrics over time
- For example, simply saying you added 100 customers last week is nice, but if you had added 300+ the previous weeks the data is now telling you something different
- You need to see the whole picture
- Don't focus on vanity metrics — if they only go up, or don't provide you actionable insights they aren't as meaningful
- Track metrics that are important to you based on your goals
- Have regular discussions about what is represented on the graphs
- Have the hard conversations if you notice that things are trending in the wrong direction
- Your company will evolve and the metrics you prioritize should too
- There are some general metrics that all startups should track which aligns with our startup flywheel
- Are you delighting customers?
- Are you adding customers at a meaningful rate?
- Are you losing customers?
- Do you have a great product?
- Do customers use your product regularly?
- Are your new features improving retention?
- Do you have a great team?
- Does your team track their controllable inputs?
- Do they analyze the key metrics for feedback?
- Tracking meaningful metrics can greatly impact customer retention, product development, and the overall success of your company
- Startups have limited time and money, leverage the data to make informed decisions
- It all starts with identifying and tracking the metrics that are meaningful to your company and doing so as early as possible
Additional Resources
- Startup Metrics You Need to Monitor
- SaaS Metrics that Matter
- Marketplace metrics
- Breaking Down The Three Engines Of Growth
- Cohort Analysis Explained with an Excel example (with template)
- How to build a simple growth dashboard (with template)
- The A-Z guide to startup metrics
- SaaS KPIs: Tracking The Right Metrics On Your Website And Grow Your Sales
- Metrics to Help You Make Wise Decisions About Your Start-Up
- Measure What Matters
Activities
Week 8
🔲 Define key metrics
🔲 Begin to group users by cohorts for analysis
Week 9
🔲 Analyze metrics
🔲 Analyze cohorts