Edu: Evidence Guided Product Management-- Don't Let Your Gut Lead You Astray w/ Itamar Gilad

Episode 65

Itamar Gilad: I think the best founders are both visionaries. They really have deep sense of taste and a belief in a certain future and a self conviction, but they're willing to hold on to their exact implementation lightly. So given new data, they're willing to pivot, they're willing to change. They still hold onto their big vision, but they're not convinced that there's only one way together.

Speaker: If you're at a startup, you know how essential it is to preserve cash. The last thing you want to do is blow a bunch of your precious capital on expensive laptops, phones, and tablets for you and your team. You have better things to do with that cash, like making a dent in the universe. That's why I'm so, so excited to share circular's new. Circular for business offering. Get your hardware on affordable, flexible, hassle-free monthly payments, and spend your cash on what really matters. For listeners of the Startup podcast, I have a special offer. Once you've signed up and been approved, use the code “TSP10” to get 10% off all subscriptions on our 12 month flexi plan. We're currently live in Australia and Singapore, go to now.circular.com.au/business or now.circular.sg/business to learn more and to sign up. I wouldn't be doing this if I didn't think it was a fantastic deal for startups. So check it out, use the code, and make me look like a hero to my co-founders. Now, circular.com.au/business or now.circular.sg/business and use the code TSP 10 for 10% off any device on a 12 month flexi plan. hey, I'm Chris.

Yaniv: I'm Yaniv

Itamar Gilad: And I’m Itamar.

Yaniv: That's right. This week it's a pleasure to have a former colleague of mine from Google and evangelist of Evidence guided product Management, Itamar Gilad joining us on the show, Itamar spent 15 years as a product manager across Microsoft, Google, and several startups in Israel. He has used his vast experience to create a number of frameworks to help product managers across the world improving their craft tomorrow.

Yaniv: It's a pleasure to have you on the show. Welcome.

Itamar Gilad: Pleasure to be here. Thank you for inviting me.

Yaniv: So, you specialize in evidence guided product management, and I've followed you for quite a while on, social media and really enjoyed your content. So tell us a little bit about how you came to specialize in this and what exactly is evidence guided product management.

Itamar Gilad: So, I spent quite a bit of time in various companies. Before this, I was engineer altogether. I'm about 25 years in the industry, and what I've noticed is that most of what I worked on, most of the stuff I developed and launched, Didn't matter, didn't really make any difference. No business metrics improved, no users felt a big difference.

Itamar Gilad: It's as if nothing happened. And occasionally from time to time you work on a project that is really moving the needle, that is really doing something. And I got to do one of those in Google as well. And Google is also historically a company that is able to produce success. And we're all using the products today.

Itamar Gilad: So I got really curious over the years, what's the difference between those projects that are successful, those that are not, those companies that are able to generate success repetitively. And I came to the conclusion that there is a guiding principle. There are a number of guiding principles, but one that is very important and is very common to all the kind of modern methodologies of product development.

Itamar Gilad: Design thinking, agile development, product, discovery, link startup. And that's the use of evidence. Evidence is key in a lot of, other realms of our life in law, in medicine, in science. We don't just rely on our opinions. We don't just rely on gut instinct. We use evidence. We don't trust even the experts to just call out what is the truth?

Itamar Gilad: And that's because our minds play tricks with us. They construct a version of reality that is not necessarily the real reality. We might think that the customers are like us than we think, than we might think that our idea is beautiful and the best ever while it's not really. So we have all these cognitive biases and all these opinions and ways to convince ourself that we are right,

Yaniv: So quite often you hear about people talking about being data-driven and you haven't said data-driven. You've said evidence guided, and I assume you chose those words carefully. So tell me a little bit about why evidence guided.

Itamar Gilad: Great question. I mean, it's very important to be data-driven. For me, data-driven means something else. Data-driven means you are collecting a lot of data. You're instrumenting your products, you are tracking your users, you're tracking your business. You have a lot of data to work with, and then you build dashboards and you analyze them and you make decisions.

Itamar Gilad: That's wonderful. That's being data driven, and I think it's very important for a modern company to do that. Even b2b. A lot of B2B companies are actually coming short on that. But that's not the same as evidence. Evidence is a bit of data that tells a clear story. It either confirms your or rejects them.

Itamar Gilad: It helps you move forward. And most of the data we encounter on the day-to-day and the dashboard doesn't do that for us. It doesn't tell a story, it's just information. so that's the difference. Evidence is actually seeking out that data that confirms or fuels your version of reality.

Chris: So Ima do you have a, great example of this, perhaps from your own career where you had a hypothesis and you soughted out some very specific evidence and, the evidence either confirmed or, rejected your hypothesis and it guided you to a, certain outcome.

Itamar Gilad: Yeah, absolutely. So, one case is client I was helping who had this idea for a product, and they were actually building this product, this SaaS product that was there to help companies run social campaigns in Twitter and Instagram and elsewhere. And they've been building this thing for years they had a base set of assumptions that the real problem is that it's really hard to run a campaign, you know, on Facebook.

Itamar Gilad: The interface really complicated and they needed a system to simplify this for the account manager. So I said, we have a bunch of account managers actually here in the company. Why don't we call them in and do what's called value proposition canvas?

Itamar Gilad: you guys are probably familiar with it. It's a brainstorming technique where we kind of surface what are the real needs of these people? What are the jobs, what are the pains associated with the jobs? What are the gains they, wish to achieve? And in that discussion, it turned out that the complexity, yeah, it's there, but that's not the main problem.

Itamar Gilad: The main problem was actually dealing with their customer with the brand, the big fashion companies or sports brands or whatever. Cause these people kept changing their minds and the meetings were very long and arguous and there was, a very inefficient process actually managing the campaigns from the client side.

Itamar Gilad: And they really needed a solution to address that. It's sad that this realization came so late, but it really was helpful for us moving on. So this just an example of how we might construct a complete picture of what's the problem that needs to be solved, but then it turns out to be completely false or partially false in this case.

Yaniv: you mentioned cognitive biases and, I love cognitive biases and, you know, all the study around that. decision science and behavioral economics and all of these fallacies, and, probably one of the biggest ones I'd love to get your thoughts on this.

Yaniv: One of the biggest ones is probably confirmation bias, right? I even see it in myself sometimes when you want to believe in something, sometimes you go out of your way to avoid collecting evidence just in case that evidence disconfirms the hypothesis that you really want to believe.

Yaniv: when you do that, obviously you're acting very much against your own best interests, and yet there is this human desire to hold onto an idea that's dear to you.

Itamar Gilad: I think it's, one aspect of, a much bigger problem. In big corporations. We have people, they're called managers who've built their entire career on sounding confidence on coming up optimistically, saying, my big idea is the best and only idea, everyone needs to rally behind it.

Itamar Gilad: have you've seen this in Google at all?

Yaniv: Oh my God.

Itamar Gilad: We've all seen this, and then confirmation bias plays a role and, logic and consensus the whole hierarchy structure of the division of power and all sort of, other ways for us to convince ourselves just prevents us from actually challenging the idea.

Itamar Gilad: We're just, yeah, that's the best idea. The senior VP of, engineering or of product decided we just need to go ahead and do it. And from there on, it's all execution. So one of my goals is to try to break this patterns and to provide framework for people to say, you know, if you fix these four areas, if you've managed to inject evidence into these four areas, it doesn't have to be perfect.

Itamar Gilad: Just improve. You will see a tremendous improvement in the quality and, the value that you create for your customers and for your business.

Chris: you know, you mentioned your passion for these various biases and I think the audience is familiar with some of the most popular ones, right? Confirmation bias and so on. Itamar, have you noticed biases that people maybe are less familiar with that play a really big role in leading people down the wrong path?

Itamar Gilad: There is something called narrow framing where, instead of looking at the whole picture, the wide picture, the complex reality, we just focus on a narrow version of it, a subset of the information, usually the most available information.

Itamar Gilad: And we say this is actually the picture of the world. And sometimes what I see is like the CEO is talking to some customers, usually the most problematic or the most demanding after three or four of these discussions. Alright. Here's the problem we need to solve this is, the pattern. This is anecdotal evidence because a few customers telling you the same thing doesn't really create a major pattern.

Itamar Gilad: You need to speak to far more to actually know. so this is one problem. It's also associated with halo effects. Like the last piece of information you got affects your thinking. The other problems I see are sometimes group biases. So either politics, internal company, politics, or politics with your investors or group think where the group tends to gravitate towards consensus and sometimes the.

Itamar Gilad: people who have, different opinion are kind of kept quiet, not necessarily intentionally, but you don't want to disrupt the, great feeling of consensus. So the group in itself creates more bias, becomes more positive or negative towards certain ideas just because of this group dynamic.

Chris: The narrow focus one is one that I find really trips people up and I'm often fond of encouraging teams to take a giant step back is the way I kind of like, segue into this idea that it's like, let's take a step back and look at the bigger picture or rethink this assumption or rethink this rat hole or this problem domain we've gotten ourselves into.

Chris: it's a really important. Mindset to continue to remind yourself to get into the group. Think one is an interesting one too, which is people tend to be conflict averse. They tend to be concerned perhaps about their job or their position, or about appearing dumb or out of step, or maybe they're disempowered in some way.

Chris: a lot of these tend to be part a category of things that I like to call cognitive inertia, where the, the line of thinking is moving in a certain direction and to force a reset either because of your lack of power, lack of voice, lack of confidence, lack of, wanting to get into a conflict.

Chris: these things are really hard to break out of, especially in medium to large groups. Right.

Yaniv: another, cognitive bias that I see very often, and again, one that I think I fall victim to is what's called the availability heuristic, there's this old joke where someone's looking for their lost key and they're walking under the street lamp, like looking very carefully for their lost key under the street lamp.

Yaniv: And then someone else comes up to me and they say, what are you doing? And they say I'm looking for my lost key. And the person said, are you sure you lost it over here under the street lamp? And the person said, no, I actually think I lost it under the hedge, but the light's much better over here.

Yaniv: it's a bad joke, sorry. But point here is that people often. Choose to act on evidence that is easy to acquire rather than evidence that has strong power, people who are shy don't like doing user interviews, so they won't, they'll just look at dashboards cause they're more comfortable looking at dashboards.

Yaniv: Or again, to your point, loud people with opinions, have more availability than quiet people who you have to ask. And so it's just this sense of whatever is in front of me, whatever evidence is easy to gather, that is the evidence I will use. but the problem is that that evidence might come from a biased source and be quite misleading.

Chris: last few episodes we've been talking about this idea of falling into the trap of doing b2b because that's the most direct path of getting revenue, for example. we will find founders who pre compromised their vision of a B2C disruptive full stack, internet service, because there's some bias towards what is easier to start with or easier to do, as well.

Chris: So often you will gravitate towards evidence that's easier to collect, and then strategies or tactics that feel easier to start with.

Itamar Gilad: When I read Thinking Fast and Slow by Daniel Carman, one observation I really liked he said, when people are faced with too much information or too little information, time pressure, Any question that requires deep cognitive analysis. and most of the product questions are actually one of those, like the question of which of these 20 ideas should we develop or what are the most important goals to achieve this quarter or this year, are very deep analytical questions.

Itamar Gilad: People tend to replace the question, what do I think about this with the question, what do I feel about this? And that opens the door to whatever biases you have. If you're risk averse, you will choose the least risky solution. If you like solutions that are technology heavy, you will choose those.

Itamar Gilad: pPeople are very compelling, especially the more senior people, they can be very compelling in explaining why this idea is better than the other. And we can spend. Hours, days, months, sitting in conference rooms and just convincing each other with opinions and bits of data. So that's the real risk, and that really diverts companies and startups to really bad directions that are not helpful.

Yaniv: you've brought me onto another topic I wanted to explore. Before we, get into sort of the solution space, the frameworks and the tools, but especially as founders, We talk a lot about conviction. Chris and I have had a whole episode about conviction and the importance of that.

Yaniv: And we talk about experience and gut feelings and the importance of taste and trusting your judgment and all of these things, which, I guess in the, Kahneman model would be like, system one thinking, more how do I feel about it? And these things have value, you don't wanna be at a startup where there's no conviction, where there's no mission. But at the same time, as you say, trusting your feelings too much can get you in a lot of trouble. So how do you think that we marry these two things so that we can kind of get the best of both worlds?

Itamar Gilad: I think the best founders are both visionaries. They really have deep sense of taste and a belief in a certain future and a self conviction, but they're willing to hold on to their exact implementation. Lightly. So given new data, they're willing to pivot, they're willing to change. They still hold onto their big vision, but they're not convinced that there's only one way together. They're not command and control managers that just force everyone to follow both their vision and the way to implement it. The second part, if you look at Bezos, if you look at Steve Jobs, if you look at Bill Gates, if you look at all the people that really achieved a lot, they're also very deep thinkers.

Itamar Gilad: They take the time to, stop and think and analyze things. is very famous for taking week vacations just to read books and contemplate. Steve Jobs would sit in his kitchen and just stare at the floor for hours. He would not allow anyone to interrupt him. He was just processing. He was just thinking, and that's true for all the great visionaries out there.

Itamar Gilad: So, If you want to be one of these visionary leaders, I would highly recommend hold onto your vision. Be optimistic. Be motivational, but also hold onto your opinions lightly and be willing to change them given new information. And think really deeply. And that's something a lot of founders I meet are kind of averse to.

Itamar Gilad: They just want to keep doing. The thinking part is not there as much as I would like to see. And some of these systems and some of these frameworks that we're introducing are forcing people to think. They're them to confront some deep questions that sometimes take weeks to answer, sometimes months, but that's perfectly fine. That's how it should be.

Chris: the problem is in the extremes, right? There are founders who are deep thinkers, but fail to execute. And there are founders who are executors, but fail to take a step back and think whether they're searching for those keys in the right place. magic almost in every way in life is at the intersection or the balancing of, the leverage point of the two dualities or polarities.

Chris: here it's this understanding of. the role of vision to animate and motivate people. The role of intuition to understand where you may have dropped those keys and where to start looking. And then the role of data and evidence in order for you to actually dig deep and look for those, those keys and maybe create a grid pattern search and actually get to the right result.

Itamar Gilad: if you have one of the genesis as your ceo, as your founder, you build the entire company around them. Cause they're so smart, they're so analytical, they're so wise, actually. they have such passion, such vision that you just want to empower them to. To drive forward to create this wonderful product.

Itamar Gilad: But most of us, unfortunately, are not like that. The alternative is to create an intelligent organization around you, where you distribute the thinking, you allow other people to make decisions. You don't funnel all decisions to yourself. And there's a lot of knowledge distributed inside the company, right? The people are actually building, the people are selling. they know so much more sometimes than the founder. So you need from very early on, I think, to create this decentralized intelligence in your organization.

Yaniv: one of my pet topics, I guess, is this sense of people trying to learn from the true outliers, from the truly exceptional people. it's a fools errand, right? Because the truly exceptional people They're exceptions. You might be one of them, but odds are you're not. trying to learn from their habits when you don't have their genius necessarily, is not a good way, it's not a scalable way of getting good results. that's sort of where you get the jokes about people wearing black polo necks and yelling at people and hoping that they're building the next apple.

Yaniv: you mistake perhaps some of the. Superficial aspects of how these people lead and assume that is where their genius is, but that's not where their genius is. And for the rest of us, it's better to think what is a relatively repeatable, relatively scientific approach building a great startup.

Itamar Gilad: Steve Jobs, while he was being interviewed by his biographer, I think one of the questions was, what are you most proud of? what is the product you are most proud of? And he said, apple, the company that I created. Cause he was very proud of the mechanism of creativity and innovation that is repeatable within Apple.

Itamar Gilad: And when you read the backstory about how the iPhone was invented and some of these other stuff, it wasn't just Steve Jobs, like dreaming up these things. He was actually against developing a phone. Initially he was against the app store and people managed to convince him to Adopt these ideas.

Itamar Gilad: But once he was willing to change his mind, and that's again the onto your opinions lightly, he was able to coalesce the different ideas into a concrete vision and to kind create a consistent vision of a product that kept evolving. But he was the visionary behind it. And I think that's the beauty of it, and that's why we need visionary in every startup.

Itamar Gilad: But don't assume that because you're the visionary, you also need to tell every person what to do and Exactly. Dictate every little detail. That's not of Steve Jobs work. This is not how any of these big, famous leaders work.

Chris: many visionaries are actually really, really bad at the execution, really bad. And they, they need to have great operational partners around them to, in some ways reign them in or at least, find a pragmatic path from here to their vision. but in all of that said, you know, I do want to put a little bit of an ad in here for the visionaries for having a vision.

Chris: I just got off the phone with a founder I work with where I was literally talking about this you know, how do we go raise the kind of money needed to execute on the business we want to build? it starts with having a, clear vision. for something big, bold, and ambitious.

Chris: And then it's about describing that in vivid terms, terms that animate and motivate the people around you that bring it to life, that get people excited about it. I find all too often founders give up on that too quickly, or they get too pragmatic or they, become myopic.

Chris: And, this is such an important part of the startup journey, is the storytelling is the visioneering. then, Ima we want to rotate to this now very quickly in the episode is, alright now how do we rotate to this evidence-based decision making to build the great products that get us from here to there?

Chris: So let's talk about some of those frameworks you've developed and how founders can get pragmatic.

Itamar Gilad: Cool. what I teach most companies, and that actually started my early clients work startups. So this actually started from startups, but it was inspired by what I learned at Google is that you need to break the problem into four parts

Chris: This is what you call your GIST framework, is that right? Goals, ideas, steps, and tasks.

Itamar Gilad: This framework or this model goals, which state what you want to achieve, ideas which are hypothetical ways to achieve the goals.

Itamar Gilad: And hypothetical is the key word here because most ideas actually don't work. Steps, which are build measure, learn loops where you develop the idea a little bit initially just in concept or you know, mock-ups and gradually with code and you test it and you pivot it or pocket it based on your, evidence that you gained.

Itamar Gilad: And then the last layer, which I call tasks, is actually how to bring this to the reality of agile teams. Cause even in startups, correct me if I'm wrong today, everything is agile. Everything is either scrum or hanban. And there are certain rules and there are certain ways that engineers like to work. So how do you connect all these experimental agile, truly agile form of thinking where we might change ideas and directions we might keep, iterating on experimentation.

Itamar Gilad: Into an agile mindset where it's, kind of pushes people towards delivery, unfortunately. So that's the fall layer. So with these four changes, I think a lot of companies can really improve, not necessarily give up on their vision or, be less forward thinking or opinionated.

Itamar Gilad: Absolutely. These are important things, but just supercharging the decision making with evidence.

Chris: Let's maybe go through each of these super quickly and give some precise examples of what each looks like, right? So what is a good example of a goal and maybe a bad example of a goal?

Itamar Gilad: Let me start from the bed. The bed is usually either no goals, all we are here is to make money, or all we are here is to ship the magical product. on the flip side, you might find this in much larger organization, it's like, let's shoot in every direction, every conceivable metric.

Itamar Gilad: Let's improve it. Let's create 10 OKRs each one with six key results. And let's just try to do our best. And none of this is actually goals. None of this is actually working. What I suggest is for a company to start from the top, it should have a mission, which is basically the top objective of the company.

Itamar Gilad: If you use OKR language objectives and key results, and then affix to this mission, a couple of metrics. One is what I call the top business metric. is it that you need to grow business wise? Is it revenue? Is it market share? Is it paying customers? Pick one. Be very specific. And the other is the North Star metric, which is a metric that measures the total amount of value you want to deliver to the market, or you are delivering to the market.

Itamar Gilad: I'll give you a couple of examples. For WhatsApp, it was always messages sent because every message sent a little increment of value for the sender and the receiver. And if you compare it to SMS at the time, it's rich media, it's free, you can send it from anywhere in the world, et cetera. this concept works also for, B2B companies.

Itamar Gilad: So Amplitude, for example, is measuring learning users, which are the number of users using Amplitude, that found some insight in the tool that was so important that they shared it with at least two other people. So that's a really great metric to show how much value they're creating

Chris: just to be very clear for the audience, the reason that's valuable is they're not just measuring active users, there's people just logging in, right? They're not just measuring registered users, which is even worse, right?

Chris: Just people who ever created an account just means absolutely nothing. they're measuring is the point of value, the volume of messages sent. So if you create a really great onboarding funnel, or if you're dumping money on growth, and you're just driving registrations, you can't cheat like metric still doesn't move.

Chris: And so there's no way to cheat that metric of that weekly learning user or that WhatsApp message. And so it really forces all your teams to think about. What do we need to do to get people to send that message and to send another one and to send another one? I love the one with the amplitude. It's like, how do we get them to find an insight that is so valuable at they need to share it?

Chris: And that is such a, such a high bar for your product. um, encourages great decisions about what to build and how to build it and what it should look like, and whether you're succeeding or failing,

Chris: I once had a, startup where I, was being introduced at an all hands, actually, and I didn't know this, but the all hands prior to my joining, they were celebrating their registered user count, and it was like a big milestone and everyone was cheering for that. And then they introduced me and I, I got onto the subject of registered users.

Chris: And I was like, now of course, registered users are irrelevant. the real question is active use, and I didn't know this, but a hush fell over the room and they're like, oh, we just celebrated our registered user count. Whoops.

Itamar Gilad: I would argue before you find product market fit as a startup, you still don't know how you're going to create value. So at that point, number of active users is still valued. It's like we don't know exactly how we will create value, but at least we're getting people to use the product the way we expect.

Itamar Gilad: So pre-product market fit, active users still okay, probably you don't need a business metric at this point. You're not trying to make money, you're just trying to create value. And later on you'll, find your business model. but after product market fit, you should know what is the core value.

Itamar Gilad: And how you create value and then you need to get more specific and you can adjust your optometric year by year. It's, perfectly okay, but one of the things I work with my clients on is getting to that point of what is the core value? What is really the value experience that you need to measure, not just as you say, top of the funnel stuff that is easy to measure, but the real thing, the real hard things that the people are coming for.

Itamar Gilad: Having a North Star metric also encourages customer focus. Cause in many companies, all they can think of is revenue, profit, and then nps. NPS is like, I'm covered on the customer front nps, they seem satisfied. they gave us, this score in nps. So no, absolutely not. You should put NPS in a much lower level, in your metrics hierarchy and start focusing on your North Star.

Chris: Yeah, NPS is a little bit like asking your customers what product they want, right? they have no idea how happy they should be. their behavior tells you whether they're actually generating those moments of value that you or yourself have tried to get them to do.

Chris: and so, yeah, it's something that needs to be further down on that list.

Itamar Gilad: Just to elaborate on goals, of course, in a company later on, you need to break those down to further to sub. Cause it's really hard to work just on your north star metric and top business metric. And you can use OKRs, you can use whatever you want, but you need to create lower level goals too. Once you have the goals in place, and there shouldn't be many, it should be very concise, very small set.

Itamar Gilad: I mean, product team of maybe up to 10 engineers cannot work on more than four key results per quarter. It's impossible. Even that is very hard to achieve. So be minimal. Then there's a question, okay, what do we do? We have this goal in front of us, what do we do? then again, it's the better of opinions.

Itamar Gilad: We can lock ourself in a conference room and convince each other with opinions, or the highest paid person will decide. But a better option is to start evaluating the goals a little bit more scientifically. And. I use ice, which was invented by a growth expert, Sean Ellis. You might be familiar with him, but, it's based on a very classic, product management framework, which is let's analyze the impact of the idea and the effort and the high impact low effort ideas are the ones we want to invest in.

Itamar Gilad: So that's all good in theory, but actually we're very bad at predicting impact and effort. so that's why Sean introduced confidence. Confidence is kind of the counterbalance and it says, how should, are we really that this idea is going to have that impact? And the impact is on the goals.

Itamar Gilad: you always have to have a key metric in mind. And how should are we? It's going to take four weeks and not 14 weeks. And that's where confidence is such a game changer in my mind. Lots of teams fall into the trap of its signing really high impact and low effort to their, ideas. There's a name for this.

Itamar Gilad: This is not a cognitive bias. This is called the planning fallacy. People tend to be overly optimistic about their ideas. but when you ask, alright, what's the confidence? That's a bit of a complicated question. What do you mean confidence? So here, I think there's only one way to answer the question, and it is, what evidence do you have that this is actually going to be a high impact idea or medium impact idea, and it will take four weeks.

Itamar Gilad: that's really the core of the G system, at the evidence, I found there's a lot of forms of evidence from my own self opinion. I'm a smart guy. My own self conviction that this is a great idea. Down to maybe I connected to some theme in the industry, you know, the current buzzword, whatever it is, web three or, generative machine learning.

Itamar Gilad: So that makes my idea good. Not necessarily down to more analytical ways of analyzing the idea, building a model, seeing if it works on paper, and then starting to dig for data, either anecdotal or. Bigger amount of data and then testing the ideas. So these different categories of evidence have different weights, and that's why I created the confidence meter.

Chris: Itamar is about to show us a, diagram and step us through it all. Now, we'll describe it really clearly so that you can follow along if you're just listening to the podcast on audio.

Chris: is a great chance for us to give a bit of a plug for our new YouTube channel, where you'll be able to see the diagram right on the screen. So find us on YouTube or on Spotify video, and you'll be able to follow us along on the slide.

Chris: So,

This episode of the Startup Podcast is brought to you by Circular. Yes, that's my startup. Subscribe to your startups tech with affordable monthly payments and save your cash for something more important, like changing the face of an entire industry. Go to now, circular.com.au/business in Australia, or now circular.sg/business in Singapore to learn more and to sign up. And don't forget that code TSP 10. That's TSP for the startup podcast. And the number 10.

Yaniv: this confidence meter was one of the things that really blew my mind a little bit.

Yaniv: I've been using it internally with my team. what's cool about it is it's a real cold shower, I think. as you say, eye scoring is valuable, but if you don't have any way of enforcing that measure of confidence, then you still fall prey to the planning fallacy. I can say, how confident are you?

Yaniv: And people are like, oh yeah, I'm, I'm, I'm quite confident. I'm very confident. the question is, what gives them that confidence? types of evidence. Can justify what degree of confidence, and that's exactly what the confidence meter does. And I think what it does, like I say, is, it suggests to you that you probably should be less confident than you think you should be.

Itamar Gilad: I fell for this problem many times. I was overly confident. I was trying to push my ideas as if there were a sure thing, over time I realized I need to be more humble. And this tool was really designed for me initially. So basically I designed it to work a little like a therme, so if you are just listening, it looks like a circle.

Itamar Gilad: It goes from blue, which is the really low level confidence. It's a zero, basically all the way to red where you have high confidence. And along the way there are different buckets of of evidence that you might find. So the blue area is mostly about opinions. It could be your own self conviction.

Itamar Gilad: It could be that you created a shiny pitch deck and that makes you feel good about it. But guess what? Behind every terrible idea, you can create a shiny pitch deck that sells it. I talked about thematic support, so that trips a lot of people. This idea is good because it aligns with the strategy of the company or, vision.

Itamar Gilad: This idea is good because it aligns with the buzzword. This is really low level confidence. Thousands of ideas are being built right now. Terrible ideas based on themes. You need to find much stronger evidence before you can convince yourself this idea works. Then there are other people's opinions, so you can review your idea with your colleagues or with experts or with stakeholders or wherever you want.

Itamar Gilad: And these people can actually find flaws in their, idea you didn't see. So it's a harder test, but even if they tell you for sure this is a good idea, or for sure this is a bad idea, they don't know. They don't have a, crystal ball either. So it's just opinions and, and as we said, group thinking and politics is a problem too.

Itamar Gilad: So be cautious on relying too much on this

Yaniv: think This is the bit that's mind blowing. So far, we've only barely left the near zero conviction area of the thermometer.

Yaniv: We are now in very low. And when you say the team management, external expert investor press thinks it's a good idea at a lot of startups that is like, I am hella confident, right? This is a great idea. Everyone it's a great idea. I have high confidence in this thing. And you tomorrow with this confidence meter, you, pouring a big cold bucket of water on it and saying it's still just a bunch of people's opinion.

Yaniv: and the truth is, a bunch of people's opinion doesn't count for all that much. It should not give you a high degree of confidence that what you are doing is likely to be impactful.

Itamar Gilad: As you pointed out in if, the tool also spits out numbers, so it's supposed to generate the confidence score between zero and 10. And as you rightfully pointed out, thematic support will give you maybe 0.05 out of, 10, and other people's opinion may be 0.1. you don't have to remember these numbers.

Itamar Gilad: There's also a spreadsheet you can download. it has all these weights and it'll help you calculate your confidence score. So that's available on my site and I assume we will link from the podcast to it.

Chris: we'll put a link in the show notes.

Itamar Gilad: next category is, what I call estimates and plans. So just on paper, you do some sort of model, maybe it's like a funnel analysis, maybe it's a business model canvas. Maybe your team goes ahead and breaks the effort into its pieces, and you come up with a better estimate of how much this thing is going to move the needle.

Itamar Gilad: And sometimes just on paper, many ideas die. You realize they're not going to be business viable or they're not going to affect that many users. And you just saved yourself a ton of effort because you did this analysis. Moving forward, you start looking for data. and that's usually happens as you start building your model.

Itamar Gilad: You realize you, you're missing some data and that data can come from your logs, from customer interviews, competitive analysis, from surveys, but it can come in small amounts. We talked about the CEO just talked to three customers and decided there's a big opportunity here. That's what I call anecdotal evidence.

Itamar Gilad: If only if three customers ask for it, it's still not a big opportunity unless you're a very, very large B2B enterprise kind of company. in most cases, this is a weak signal.

Chris: even then, Itamar, there are a lot of companies who are big, heavy duty enterprise companies who shouldn't be, right? They're there because of long sales cycles. And the one or two customers who ask for something and they've went off and built it, and now they have a very complicated product with a very complicated sales cycle, with very complicated deployment.

Chris: And what they're really trying to do is find something that can scale to thousands and tens of thousands of customers, but they're stuck. because of this low confidence of anecdotal evidence, decision making,

Itamar Gilad: I agree. The discussion internally is this important customer is asking for this, so we must make it a priority. So that's why I wanted to make this here and at least drive the discussion is this really such a huge opportunity that we're willing to put aside building the general purpose product that might serve many more customers in favor of this one off request from this one customer.

Itamar Gilad: Then you move to what I call market data. Market data comes from surveys, from deep analysis of your competitors and from smoke tests where you direct people to, uh, landing page or you pop up something in the ui, a call to action. But behind it, there's nothing, when people click on it, it's just a test whether or not they're interested and you tell them, sorry, we're not ready, but would you like to sign up?

Yaniv: for those who are listening and not visual, we're still on this thermometer. We're in the medium to low. Confidence level. And again, I just wanna reinforce this. What I really love about this meter is how long it takes to get out of low. in some organizations would be considered gold standard data.

Itamar Gilad: Totally. I mean, I built entire product based on this we're on a scale below, two out of 10 right now. so it's a bit disheartening. A lot of companies, once they introduce this, they come back to me and say, we never cross 1.5. We realize we never have confidence more than that. And said, all right, great.

Itamar Gilad: Build your experimentation framework and start experimenting, man.

Chris: the confidence scores along the inner edge of this diagram, they're exponential, so they start at 0.01. you go all the way down to anecdotal evidence. You're now at 0.5 to one. Still, we're trying to get to 10 here, guys. If, you're listening we're talking about market data and smoke tests, we're still only at three out of 10.

Chris: it's a really beautiful framework or metaphor that you've created here. I love it very much.

Itamar Gilad: we're still like, a hundred times more than self conviction, which is where we started. But I wanted to create a framework where actual hard evidence that comes from testing and experiments will be about a thousand times more influential than your personal opinion, about hundreds times more influential or have heavier weight than the team's opinion or the group thing.

Itamar Gilad: So that's where we're headed. Now, the real area where you start having medium and high confidence is by testing your idea. There's various forms of tests. Of course, you can do a usability test. You can do, a longitudinal test. You can build MVPs. There's. Tons and tons of stuff. That's actually the next layer

Itamar Gilad: and then you can go into really high confidence tests when you have the data, you can do ab experiments, et cetera, and alphas and betas and early adopter, programs. All these things give you various forms of confidence and you are really trying to validate the hypothesis, the core things that must be true for the idea to succeed.

Itamar Gilad: Now there's a question. When do you stop? I don't think every idea needs to go all the way up the scale up to 3, 5, 7. That will be crazy. You can't test every idea to that level. You need to know when to stop. And I think if you're just making a minor design week or you're changing the order of the settings in your settings page, fine.

Itamar Gilad: You can do this even based on expert opinion. You don't need to go and test this in a very strong way, although a company like Netflix will test even that. But once you start going into more expensive and more risky ideas, you need to venture further and further. And almost all ideas need to go into at least one form of test.

Itamar Gilad: It could be just a usability study. After that, we feel good. but you should definitely ask yourself, are we convinced that this is a good idea? Once you are, once you feel we found sufficient evidence, we are confident enough you can switch gears, you can stop testing, you can switch into delivery, and then it's all hands on deck shipping this idea, and it's much more likely to be a good idea.

Itamar Gilad: Cause along the way, you, improved it, right? Based on the evidence you got, you pivoted it, you changed it. So the idea we will launch will be much more profound than the one you started with.

Chris: I really like that you have user research, even though that's on the, left side of the dial here it's still only a three out of 10 in terms of confidence and I like that very much because, I'm often fonder saying users that you're asking questions of, want to be helpful.

Chris: They want to give some feedback like, Hey, that, yeah, that does look great, or maybe my cousin John could use that. Or they want to appear smart, so they want to be skeptical in some way that's unnecessary. And, so these, focus groups have limited efficacy. They're, as you've got it here, like a three outta 10, whereas you've really privileged, real results and product telemetry and actual outcomes of user behavior up around the sevens and tens up there, which is I think very, very appropriate.

Yaniv: even going further back to about 0.4 where we've got a business model, I love this one very much because, I think it's actually very common, especially at big companies, but also sometimes, smaller companies that think it's gonna help them raise capital or something where you have a big fancy business model and you're like, look, it's in the model.

Yaniv: we are guaranteed to succeed. The model says so, models are incredibly valuable. Don't get me wrong, and shout out to my co-founder Nick, who is the best modeler I've ever met. But are there to help you map out different scenarios. They do not constitute a high confidence source of evidence.

Yaniv: And yet so often that somehow the authority of the spreadsheet and the variables and all of this makes it treated as this artifact, that needs to be treated reverently as this very. High confidence piece of data, and yeah, folks, it just isn't, you can put whatever numbers in there you like, which goes to show how much confidence you should have in the outcome.

Itamar Gilad: Yeah, we can let the evidence bias us towards the direction we want. Very easily. Pat and I see a lot is, there's some idea it's really hot, everyone loves it. it aligns with some theme. It's very buzzy in a review. Everyone's in favor. The management is very bullish. Let's just build this and there's an anecdotal evidence.

Itamar Gilad: The leading competitor has launched that same feature or similar feature. That's it. That's all the validation we need. Let's just go ahead and build the whole thing. I don't know how many times I've seen products fail because of this pattern, including in my own product.

Chris: It's the blind leading the blind. Right? the competitor did it cause some random customer had it or some highly paid person thought of it. now you are copying their. Low confidence decision

Itamar Gilad: assuming that the competitor actually knows what the future holds and that this is a great idea.

Itamar Gilad: I mean, Yaniv and I worked at Google and we know how many things Google does that actually is not very well based, but someone from the outside might say, if Google does this, it must make,

Itamar Gilad: sense.

Yaniv: actually think Itamar, if I remember the timelines correctly, the period when we worked together coincided with the Google Plus period, and that was turning the entire company upside down because the CEO of Google at the time was afraid of Facebook and he was like, Facebook is doing certain things, therefore we must do the same things.

Yaniv: some of the younger folks listening to this would never have even heard of Google Plus. That's how much of a comprehensive failure it ended up being.

Chris: Google Plus is an interesting case study because there's. Two aspects to this confidence scoring and the decision making you have to do there is what is the right thing to be afraid of, or the right problem to go solve. And then there is what is the right way to solve it.

Chris: So I would actually argue was appropriately afraid of Facebook. And I think Google is now living with, having missed the boat on social media. The problem I believe about Google Plus was the way they chose to solve it. And, the question there becomes how do you apply this confidence score to, implementation decisions or product strategies, and doing that the right way at the right time.

Itamar Gilad: Yeah, I think there was quite a lot of evidence coming out after the launch of Google Plus that not all is well. And, I think Google Plus, instead of pivoting or instead of really asking the hard questions, persisted with the plan And it was always a feeling if, we just launched this next big integration or this next big feature, it'll work.

Itamar Gilad: This was a very formative experience for me, by the way, Google Plus and also some very positive experiences I had in other projects, in Google.

Itamar Gilad: And in fact, this year, I'm, intending to publish my book called Evidence Guided, I've been writing for four years and promising to launch every one of these years.

Yaniv: Have you tested the content along the way?

Yaniv: evidence do you have that it's a good book.

Itamar Gilad: it just comes to show that consultants are just the worst users of their own, theories.

Itamar Gilad: This was a complete waterfall project, I have to say.

Itamar Gilad: So the beginning of the book, chapter one, the first story I tell is Google plus my experience of Google Plus. just to give it a context that even the best companies sometimes fall into these traps.

Chris: My sense was, and I wrote a blog post about this at the time, the problem with Google Plus is that it didn't leverage any of Google's existing assets, in particular Google's existing attention surfaces, whether it was Google search or Gmail or Docs or what have you.

Chris: They created a new destination

Chris: newsfeed

Yaniv: Oh, YouTube, Chris, like again, Itamar and I were at YouTube at the time. It happened. And even then, which is, it's still true today. Even then YouTube was, depending on how you measured it, the most popular social network amongst American teenagers. There was research on this and we're like, hi, we're, over here.

Yaniv: We, kind of have something that's nearly a social network and it was just ignored,

Itamar Gilad: the second largest social property in the world. We had it

Yaniv: still do, right? But it's never been as leveraged as it should have been as a social property.

Chris: so what struck me was what Google was missing was certain core pillars of a social network that needed to be enmeshed in the existing Google Properties notification profile, newsfeeds that could appear on Google Plus, or even on the side of Google Drive, or of Gmail, or of search rather than some other different thing with circles and its own video conferencing and its own what have you.

Yaniv: It was classic Google.

Chris: I'd be curious, a, your guys' experience from the inside of that and B, how this framework might have helped.

Itamar Gilad: the first time I heard about Google Plus was from, we were having a status meeting with our manager, and Yaniv was complaining that the secretive projects happening we're kind of, departmentalized out of it. and that's the first time I heard about this thing while it was being built. I think it's worth, a really nice case study and we need to look at it in context.

Itamar Gilad: What happened before Google Plus is that, Google launched Android very successfully. It was a big bet. Project worked on it for years, and then when it came out, it was just a home run. And that was a pet project of, Larry Page, the CEO of, Google. not at that time. He became the ceo, during the time of Google Plus.

Itamar Gilad: And also Apple launched in 2007, the iPhone, and a few years later, the iPod, two big bet projects that were just home runs again. So the whole concept back then in the industry was, let's make big bets. That's the way to innovate. Let's just go all in on this big idea and just work our butts off and make it a success.

Itamar Gilad: Right. do you remember the big bet discussion that happened within Google? the narrative?

Yaniv: Definitely.

Itamar Gilad: and actually the two founders sent us an email and warned us against incrementalism and against just doing small things. Just do the big things guys. So here was this immense threat to good core business.

Itamar Gilad: This Facebook thing, were spending hours and hours back then on Facebook and they had the, possibility to target ads, which Google doesn't have as much, not on a personal level. They can target to your interests based on search, but we don't know enough about you. So the idea was we need to build one of those for ourself.

Itamar Gilad: We need to build a social network. That's my interpretation by the way. I wasn't an insider on Google Plus or any of that. and we need to plus one it, which is. Ironically, one of the actions that Google Plus offers plus one, we need to do a slightly better, Facebook. Facebook that provides circles and gives you more control over your privacy.

Itamar Gilad: a social network that does a bunch of other things, video conferencing, you remember that was a key feature back in the

Itamar Gilad: day. I think what Google missed here was a core assumption that people actually need another social network that is very similar to Facebook and just does a few things better.

Itamar Gilad: And that's a core trap. A lot of founders, a lot of companies fall for this idea that if we just take an existing idea that is successful and just improve it a little bit or make it a little bit cheaper, we'll have a home run. And again, it's a case of building confidence where there isn't none. The fact that the leading competitor has one doesn't mean you can succeed, So Google went into overdrive and we actually moved entire groups of people into Google Plus remember that? And if all of a sudden it was the,

Itamar Gilad: don't remember the internal name, but it was like a tornado inside Google

Yaniv: Emerald. C it was called.

Itamar Gilad: lc. Do you know why it's called Al c

Yaniv: some crappy old book I can't even remember.

Itamar Gilad: I think it's a painting. It's actually a, a painting where you see the al c, like a tsunami wave, capsizing a boat. And that was kind of the, the of Facebook. So the concept now was we have this one big idea. And the idea was not just to launch a social network, but to connect it to the fabric of the rest of Google.

Itamar Gilad: So to search, to YouTube, to Gmail where I worked on integration with, Google Plus. that's the strategy. That's what Google's going to do. We're going to turn Google into one Unity, one product that is connected by, Google Plus. Very aspirational, very convincing. And a lot of people were convinced Google would succeed because Google really had a lot of unfair advantages, deep pockets, huge talent pool.

Itamar Gilad: but the idea just didn't work. And what happened is Google tried to iterate for years trying to make this idea succeed. And in the meantime, not far from Google's headquarters, there was a minor startup, I think 50 people when they were acquired, called WhatsApp,

Itamar Gilad: very boring chat app that started threatening, Facebook much more than Google Plus ever did.

Itamar Gilad: WhatsApp became very popular and people started socializing within WhatsApp and Instagram at the same time, took photos and created this very social experience around them. Something that Google definitely could have done because we had Picasa and we had a lot of infrastructure around photos.

Itamar Gilad: We also had Google Chat, of course. so the concept idea that cannot fail, prevented Google from seeing these other alternatives.

Itamar Gilad: I, I see here two things that I would change and it's really easy to criticize in hindsight, right? one is set the goals, right? The goal was to ship Google Plus and to ship the strategy. And I think the goal should have been to make Google more social or to enable social interactions through Google product.

Itamar Gilad: If you made this goal, there are so many creative people inside Google and they would've come with wonderful ideas, including things like Instagram, including things like WhatsApp, and I'm sure Google could have done a lot better. So by setting the goals wrong already, Google set itself up a little bit for failure.

Itamar Gilad: And then there was the lack willingness to look at multiple ideas. Especially ideas that divert from the core narrative of we need to have this core Google Plus thing. if the goals were set up right and then there was a question to the organization, what are some ideas? And some of these ideas might have been big, of course they could have grown, but Google could have gone to its origins and say, all right, let a thousand flowers bloom.

Itamar Gilad: Let's try a lot of ideas. Think big, but start small, iterate, fell fast. Eventually you'll find amazing ideas that will now, put Google in a much better position in social, in my mind. so that's where I kind of started thinking about gist as a framework, honestly, in that period in my life.

Yaniv: I think that makes a lot of sense. And it was a great trip down memory lane. It's been, a long time since I even thought about it until recording this episode. And yes, I think it was one of those grand failures it was absolutely a failure, but I think all of us who were there probably learned a lot of valuable things from it.

Yaniv: And so, in that way it's really fun to talk about it again.

Itamar Gilad: And we need to be fair to Google. Google has done amazing things and so many good products and so many good features. this was maybe a low point in, terms of innovation, but it still doesn't paint the entire company black,

Itamar Gilad: right?

Yaniv: course not.

Chris: Yeah, I'd hate for the lesson to be, you know, incrementalism wins the day, right? It's, you need to be able to make big, bold bets. and, fail spectacularly. That's sometimes part of the process. But in making those bets, making them smart, making sure that the whole company comes along and, also, iterating your way to, innovation as well is key to the whole thing.

Chris: Ia, thank you so much for taking the time to walk us through your frameworks and walk us through this evidence-based product management. I think it's, absolutely critical and, revelatory for, founders and product managers everywhere. How can people find you, find your stuff and, connect with you if they'd like to learn more?

Itamar Gilad: the best place is my website, itamargilad.com. You will find my resources page where you can download basically everything we talked about plus a lot more. So there's the confidence meter there.

Itamar Gilad: slides about gist or articles about gist, some videos and some other frameworks we didn't cover for experimentations and for other things. and a number of eBooks. So all of this is available, you just need to sign up for my newsletter. But, apart from that, there's no hidden strings.

Chris: Awesome. And yeah, of course. We'll put that in the show notes and you can, find your way from there.

Yaniv: Okay folks, if you are listening to this and finding this episode and previous episodes of the Startup podcast valuable, don't forget that you have signed up for the Startup Podcast pact. That means that in return for the value that we've given you, please follow, rate and review us in your listening app.

Yaniv: our channel on YouTube and give us a shout out on LinkedIn or wherever you have an audience. It really helps more people find us, takes a couple of minutes of your time and it means a world to us. So thanks very much for doing that.

Yaniv: Now, Chris, if people have been listening to this and are also thinking maybe I'd like to work directly with you, how can they find you, and how can they work with you?

Chris: Yeah, absolutely. so I help founders and startups avoid the dead ends and wasted time of building a high scale Silicon Valley style company. So you can learn more about that over at csad.com/advisory.

Yaniv: Fantastic. And we're both quite active on LinkedIn and occasionally on Twitter, so we're pretty easy to find on both platforms on Twitter, it's @ChrisSaad and @yBernstein. So we always love getting feedback and suggested topics for future episodes. Thanks again so much, Ima for joining us. It's been an absolute pleasure.

Itamar Gilad: The pleasure was mine. Thank you guys.