Blake Beus  0:00  So we talked a lot about split tests. Yep. And we've talked about it a lot in the past and everything. And I was listening to a Freakonomics podcast episode recently called, is Google Search getting worse? And in there, they talked with some actual representatives from Google, about some of the things and I wanted to bring up because I thought it was relevant in the fact that the interpretation of the test results is incorrect. And even Google did this. Yeah. And so I kind of wanted to point this pointed out. So it's, it's I don't know if you feel this way. But I literally feel that Google search is getting

Greg Marshall  0:40  worse. As far as being accurate. Yeah, what

Blake Beus  0:43  you're looking at feels harder for me to find what I want. I don't know. Have you noticed? Is that been similar to you or your, your good,

Greg Marshall  0:51  we know what I'm not. When it comes to Google search, and like, maybe I'm looking for somewhere to eat or products or something like that. I don't use it a ton outside of like researching marketing stuff. But it does feel a little bit less like integrated. Like, it does feel like I do have to, like, look a little bit deeper for what I want. Part of it could be the ads above six or eight or 12. Yeah, search ads before he can even get to the camera. And so yeah, you know, I probably haven't been paying attention as close because I'll just keep scrolling. So I find when I look, right. But if you're talking about immediacy, like I'm typing something, and it's not right there, I would say yes, it has gotten worse, because I'm not, I didn't notice I am scrolling more on Google. Yeah, what's the cause of that? So

Blake Beus  1:41  there's, there's a lot of different causes for that. One of the things I do for a lot of my clients as I help find solutions for them, right, so, so, right now I have quite a few clients where we have data integration issues, right. So they want to bring their marketing and advertising data, and pair that and merge that with their email marketing CRM, like HubSpot, or or any of those and merge those things together. So they can actually have actionable data. And so, so I deal with a lot of these kinds of data integration things, but I have to do a lot of searching to find, maybe they need a tool or something like that. So I'll search for a tool or an integration system or whatever. And I would say over 50% of the articles that I find these days, are something along the lines of, you know, the top 10 tools for this. Yeah, and it's just regurgitated content. And then at the end, the paragraph is almost identical on all of them, it says something like, so as you can tell, no matter what you choose, you're gonna find a good solution. And it really depends on your needs. And I'm like, I want an opinion. Yeah, I want you to and you could tell that whoever danced, take a stance, and you can tell it, whoever wrote the article, didn't write the article to actually help you make a decision, they probably haven't even used any of the software, they're just kind of regurgitating things. And you're getting a lot more of that content on Google. In addition, you're getting Google adding things like they have, they have, depending on what you're searching, they'll have like a box along the top with some information, or they'll have the sidebar box, like a little widget on there. You see that if I search for like a movie star or something like a little box, there, they'll have like, if you're searching for a restaurant or something, they'll have like a widget with placements and things like that. So what kind of the end result is that when you search for something, you could have widgets, ads, another widget, a thing on the side, and then your organic search results, you literally have to scroll down under everything. Now, I don't hate ads. So obviously, we talk about ads all the time. I think ads are a very, very important part of running a business and marketing and everything. But Google has done a lot of different things to try to maximize their profits, get more people to stay on their system, as opposed to like bouncing off on to something else, and whatever. So that's a this is some foundational knowledge. On this podcast, they interview the person at Google and they shared this person had been with Google for a long time, 15 years or whatever. And, and essentially, they ran an accidental split test, okay, for 10 years. So they had 10 years of Google search data. And here's here's what the test was they had a test where some people would not get anything other than just the search results, and a few ads on top and then everyone else would get all the new bells and whistles. And as they added new bells and whistles that got busier and everything. The person who wrote that test was the person they were actually interviewing saying she had moved on been promoted whatever. And then 10 years down the road, someone comes to her and says, Hey, we noticed a glitch. And she's, they start talking about it. And it was her test that she wrote the code for and never got turned off. She got promoted. moved on, and everybody forgot about it. Yep. And so she said, Well, before we actually turn that off, yeah. Let's analyze the data. Yep. And in the podcast, she's she, the, because the question asked was, by the podcast interviewer was, do all of those widgets and things actually make Google better? Yeah. And this was where and when, when they asked better? Are they referring to user experience or business better for the user? of Google? I want to clarify, yes, yeah, that's important, better for the people who are actually using Google, because those are the people you're best paying for those of your customers, right. And so they did an analysis of 10 years worth of data. Now, that's trillions and trillions of data points and searches and things. That's a lot of data. And the conclusion they came to was that there was a significant, let's see exactly how she put it. And this is where I was like, No, you're wrong. She said that, we found that people that had all of the widgets, and everything did, I think it was like three to 9% more searches in Google than people who didn't. And the conclusion she came to was that, therefore, people like the widgets, and all of that more than they like not having them.

Greg Marshall  6:46  So now it just sounds like you got lost, so you just keep searching,

Blake Beus  6:51  right? That was my exact thing. I said, as soon as I heard her conclusion, I was like, or, you know, people were frustrated and couldn't find what they were looking for. So they had they were forced to search for. And so the whole reason I wanted to tell this story and bring all of this up is that we need to make sure we're interpreting, interpreting the results of our split test correctly. And we need to follow those up with other tests to confirm what we believe, you know, we need to think about that.

Greg Marshall  7:25  Here's a couple of questions I have for you, since you're heavy in the data world. Yeah. What is like? And this is, like a personal question from me, as far as I would like to know, your opinion? What's the statistical relevance? What's like the minimum amount, that you really should be able to say, this is relevant, right? Like if you run an AB test, and it's like, you had, you know, five purchases? And maybe the ads been seen 2000 times, right? That, that seems a little bit less accurate than if you were like, you had 50 purchases. But the ads we're seeing 400,000 times, right, right, is there like what's kind of that threshold of like, where you where you can actually take the data and say, this means something.

Blake Beus  8:14  So the there are actual math equations to measure this, and it's called the standard deviations. It's a statistical thing. And it's been a long time since I've taken my stand, and actually done the math. But statisticians actually use a mathematical formula to determine the relevance of a data point. But essentially, it takes, it takes into consideration the sample size. So in the ads case, how many impressions it got, and how many people it was put in front of, because those are two different things, impressions versus number of unique impressions or number of unique people. So you would measure it based on impressions versus based on people and run the analysis that confirm compared to how many conversions or objectives were met. And then you you run some statistical analysis. And then if it's within a certain standard deviation of the, everything's measured on like a bell curve, right? So the relevance is right there in the middle of the bell curve, and the standard deviations are measured off to the side and the relevance is determined based on how far you are, how far you are from the center, but also how quick the bell curve goes up and down. If it's a slope that's like this, the relevance you need a wider standard deviation in order for you to say okay, within the standard deviation, we have a 90% relevance because we're getting 90% of the people but we have more variation that relevance right. Whereas if you have a very sharp curve within within like a half deviation, you get a 90 or a 95% row relevance because as that gets 95% of God's statistical significance, I'm going to I'm going to take out to answer your question as a real, central really quick to answer your question is, with Facebook ads and things like that, unless you're running very, very large dollar amounts, what I would do is kind of go on gut feelings and then follow up with a test. So for example, 4000 impressions with five conversions is more significant than 400 impressions with five conversions. Yep. Even though you have the same number of conversions, that might just be you happen to hit the right people at the right time, right up front. Which can be misleading. It can be mislim. So if you were to shut that off and be like, Oh, okay, let's just bump the budget up. 10x. Yeah. It might not scale appropriately. So you got to use your gut, and then run follow up tests on whatever you're testing.

Greg Marshall  10:48  So basically, to what I'm gathering is, the more kind of data you have, the better as far as like the sample size, right? It's going to be more accurate when you can actually make better decisions versus small decisions. And another. Here's another point, the reason the reason why I asked you that I so a lady that I really respect in the marketing world, she heard she had put out a tweet the other day that I was like, that's an interesting point, because I have seen that, but I've never been able to, because you taught the opposite. So I've never been able to like, figure it out why that is. And she had mentioned that click through rate really isn't as important as people think, especially as you scale. As you scale, the click through rate goes down. Yeah, but you're still getting the returns and purchases, she used the sample size, where she had a client that had a click through rate of like maybe 1%, or like point eight, zero, right. And I was getting a two to three row as at I think she said $500 a day in span. And she had another client spending 15 grand a day click the rate was like, point oh, eight or like really low. But still, they were maintaining a two to three return on adspend. So she had used that data point to basically determine this should give you an idea that click through rate is probably not the real metric to measure the success of an app. Yeah. And I thought, well, that's interesting, because I have seen like ads that scale really well that are below the 1%. Click through rate.

Blake Beus  12:25  Yeah, I would definitely say that. You what you need to do is look at the metric that makes the most sense, given the circumstances. And this is where I want to say gut or intuition. But But think about it. So for example, if you have a campaign that is new and doesn't have really any conversions, or the conversions are pretty low. Yep. conversions are a difficult metric to determine the successful of the success of that ad, because there's some bottlenecks elsewhere. So that's when you kind of move closer to the top of the funnel. And say, okay, maybe click through rate is the most important metric for us to look at right now between these two ads, because our spend is a little low, maybe it's a new ad account or so we don't have an established audiences or anything yet. So let's kind of look closer up. But as the campaigns get more mature, as the ad account gets more mature, as the offers get more mature, as the offer alignment with the audience gets more mature, yep, then you can start looking saying okay, click through rate is now maybe not the most significant thing, because we have a consistent record of sales and conversions. Now, the conversions is the most important thing. So we can kind of ignore click through rate, as long as our conversions are where we want them to be or improving.

Greg Marshall  13:47  Yep. So I think with that, with that being said, it just helps me understand because I've always been taught, click through rates important. And I have seen some level of correlation, but I always do like to challenge my initial beliefs to see if Do they still hold true? Or are they true, up to a certain point or economy. And so one of the things that I picked up from that too, was when she's discussing the click through rate, because I always thought like, well, you can kind of manipulate that if you want. Yeah, just put a cat video up with something really funny or something that's like, you know, you can you can almost

Blake Beus  14:26  force a click or a clickbait or something or whatever you can, you can game that number,

Greg Marshall  14:32  but that doesn't mean I'm getting the end business result that I want. Yeah, so that can be misleading. And I think that what she's mentioning is it's almost you could, I mean, obviously, you'd have to test this with relevance. But I want I would love to see the correlation between what if you Your goal should be the opposite. The click through rate would actually be you want it to be lower? Because you're pre qualifying people more If you are getting the conversion, right, like,

Blake Beus  15:03  Yeah, but you could manipulate that number to by just having the wrong audience. That's just so my click through rate is low, because I have the wrong audience,

Greg Marshall  15:11  assuming, right, that you're targeting the right person, right.

Blake Beus  15:15  And so I would say, that would be an interesting thing to look at, given, you're very comfortable and confident with the alignment of the audience, the messaging in the ad, the messaging on the landing page, and the offer that they get at the end, if you're confident in that, in all of those things being aligned, then I would say, Yeah,

Greg Marshall  15:37  well, and based on you saying that I was making the assumption, yeah, that the funnel is proven. Yeah, we've seen it work over and over and over again. And then you would test the click through rate and add to see, that would be something interesting, because I wonder if there is any, like, reverse correlation or something, you know,

Blake Beus  15:58  very likely could be definitely have to

Greg Marshall  16:01  take it out. You know, that's, that's where you start geeking out about like, well, you know, what, if you, you're looking at a different metric to try to prequalify more, because you would think your click through rates, assuming everything else is working, and it is a good offer, and you're targeting the right person, your click through rates would be lower if you're pre qualified, more meaning, let's say I'm going after an audience of entrepreneurs, and I say, my product, you know, you would say it's an apple, you say, like my product is for if you're making 10 million or more a year, and you're looking to spend, you know, 100 grand a month on ads, right, click this to talk to me, you're gonna have a really low click through rate because there's not as because they're very, very small. But if you're delivering

Blake Beus  16:46  on that product and offer and the people in that group are really criteria, that criteria, you can be wildly profitable on the kind of an app.

Greg Marshall  16:57  So that's kind of what I mean is I wonder if like the level of pre qualifying in the app, like makes the click through rate go down, but the return go up? Right? You got me? Yeah,

Blake Beus  17:08  absolutely. And I think I think some of these things are, some of the ideas and concepts we have with which metrics are important, are kind of old school carryovers, from the golden days of Facebook ads, when you had so much visible data on everything, which has been shut down or hidden behind algorithms for privacy reasons. And I think that's a good thing. But you know, seven, eight years ago, with Facebook ads, you could literally target people whose income was over a million dollars a year and lived in this place. And in their first name was Adam, and they had a dog named Gary, right? Like, you could target it was almost creepy levels of targeting, which

Greg Marshall  17:50  isn't that crazy to think like, now, when you think about that? Doesn't that sound crazy? Yeah, like, that's probably a little bit too much knowledge of an individual, but at one point,

Blake Beus  18:01  but yeah, we were run out a lot of gurus, for lack of a better term come out of that space. Yep. And, and teach people a lot of their concepts and ideas. But as things have changed, some of those concepts and ideas are still carrying over like the click through rate. Of course, I could have a 20% click through rate when I could target literally the exact person in my ad, I could say, hey, hey, Adam, with a dog named Gary, this ads for you?

Greg Marshall  18:31  Yep, click here. You're gonna get way higher click through rates.

Blake Beus  18:35  But, you know, now we can think okay, click through rate maybe is not the most important metric. We need to think about our campaigns and our business business, and our offers and everything more holistically, which I think is a good move, saying has so many people abusing that system?

Greg Marshall  18:53  Well, I also think to looking at your business holistically, will make you have a better business and actual business. Yeah, because, like, I don't remember who I think actually perpetual traffic. And I'm a big fan of they mentioned that if a lot of people don't have business, they have an offer. Right? So they have like one thing, they're selling it but then there's no upsells and down sells and like continuation right? And I feel like that back in the day, you could really just have an offer. And it worked just fine. Granted, the targeting was very tight. Yeah. But now you do have to like think of, you know, as Dan Kennedy mentioned, now, acquisition costs are more realistic. Yeah. How they really should have been just there was this golden age of where you could get like $5 Commission.

Blake Beus  19:45  Well, and it's it's important to note that every business starts somewhere. So if you have a side hustle or something like that, you're gonna have it's very likely that you're gonna go through the My business is literally just one awful Yeah, ads. And you're gonna go through that you have a you have you have to get there and you have to figure out okay, how do I turn this single offer business into an actual business that's going to last 2030 4050 years. And that's part of being an entrepreneur is figuring out okay, how do we go from from this little tiny thing to something that's more substantial that has some foundational elements that can grow and stay around long term?

Greg Marshall  20:30  Also, just to clarify, by no means? Am I saying having that first offer is not valuable? Yeah. Or it's part of the stairsteps. I think what he meant, and in the podcast also was not that that's bad, like you're less than you have it, but that there were a lot of businesses that maybe he probably ran into that were running. Maybe not so morally correct, or Yeah. Just bad offers, that they just were building these offers and taking advantage of all of that privacy that? Yeah, that we did not have back. Yeah, right. And so, but yeah, when you first start, every business has to begin with, well, what's the first thing I'm going to sell? And who is that customer? And why do they need this right? And you and you build on top of that. And I think if your goal is to only stick to like just, you know, because you see marketers out there that they just do like random launches or wherever they're not really like trying to grow a business or just doing like quick revenue. He's trying to do sales, right, quick revenue hits, I'm in and I'm out. Right. I think that with all this privacy gone is been mostly eliminated.

Blake Beus  21:45  I think so most, I definitely think so. Because it's hard.

Greg Marshall  21:48  I mean, how do you do that? Now you actually have to build relationships with people. That group I think, was not really interested in the note. And

Blake Beus  21:56  you have to actually, like, get to the point where you're hiring actual staff, right? Because there was a lot of those businesses that had an offer, and it was like them and a couple of vas. Yep. And a Facebook page and a landing page software. Yep. And that was it. And they had an app to create some PDFs or whatever. And, and it's okay, that that's where you start. But if that was their entire end goal, yeah. It's not I yeah, very, I see very few of those ads anymore. I just don't think they're working anymore.

Greg Marshall  22:27  I know. The types of ads I see are much different. Like I remember, and I'm sure even on TV, I remember just getting hammered by Google ads, like 24 hours a day, like every time I open a Facebook. Yep. Is the new guys like math or see that guy? But I don't see as many of those in our and so I think, I think yeah, the future. I do believe that based off the topic of stats, there will be a new set of metrics that you'll have to measure things off of as when the game changes. You need to evolve with it right? And we no longer have that precise targeting that you did before. That's not a bad thing. You just need to work around it. Yep.

Blake Beus  23:09  All right. Well, let's let's wrap this up. Greg. How can people get in touch with you?

Greg Marshall  23:13  Greg marshall.co. You can book a free strategy call and like bs.com Alright, so next time, we'll see you later. Okay, bye.