[MUSIC] So, you have to be rather familiar with A/B testing that we studied on the last week. So we will talk about control groups in A/B testing and how they can help us to restore that a non target events within the rejected clients portion. Well, usually aA/B testing implies some control group. What does it mean? Cultural group allows us to restore and get some representative data about the customer otherwise, we cannot observe. Well and there are different situations and different scenarios how the cultural group can be built, for example, for credit scoring. That would be a small portion of clients that are granted long, regardless of model estimates off their risk. It helps us Thio grant loan for every client considerably small portion off our client base, and to understand, would or not they default in marketing and response modeling, you can build too small random groups, which one of them will receive communication and other will be not. It means that therefore, you can compare one versus another to evaluate the effect of your communication. And say in collection processes when we deal with bad clients which are already not credit worthy we decide which channel to use to you communicate? Would it be a call? Would it be an SMS, or would it be meeting with our employees? So therefore, we build a random group of clients, and we apply random channels of communications to those clients. All these things happens regardless of model predictions that we have. It's the key point, because we have to get the rial, the whole picture off client base we deal with, and we have to get the data off. Are all actions towards those clients not only those actions that are prescribed by our model? Well again, return to our example here. You have, say, a small portion of blue font clients. They're called control group. We can grant alone regardless off our model estimate off their risk and collect the target event for them. Mention that in the lower part of the table, we still have a number of clients which are unknown from the point of view of target variable. Why does this happen? Naturally, we try to make our country group as small as possible because we understand that the actions we undertake for those clients. They are not supported by our model, and they can be too risky and therefore lead to severe losses. This situation can happen for credit scoring. Well, as soon as we have this control group, we can use this as a proxy for the general population off all rejected clients. But we have to mention that we have to weight them what is the weight and why we should use it? The reason for that is that our control group is small, and in its proportion, it's kind of a small part of that rejected population of clients. So when we will upend that information with our non target event and with clients with non target event, we have to kind of conserve. And you have the upkeep that proportion of clients between accepted ones and projected ones. Let's see precisely how this happens. Well, for initial clients with known target event, we keep our weight equal one. And for clients within control group, we'll come up with some larger weight. Because there are only small portion of clients that represents the whole population of rejected clients, and the weight is calculated rather simply. It's just a ratio between number off rejected clients and the number of clients within control group. So each client within control group will have larger weight. In our calculations you can see that if we some all the weights, we will have the total number off clients within our client based both accepted and rejected ones. Well, now we can kind of derive formulas for false, positive and false negative that use the informations from our control group in the upper formula. You have formal for false, positive, frayed, and it looks a little bit scary, but really simple. You have all those clients which our model predicted as default ones, and at the same time they were not. So it's false positive to multiplication of two indicated functions give this result. We also multiplied by wait for this client and with some of this in new moderator in denominator, we have just the number of clients with negative label, so it means that they're not default clients. They're good clients, and we also apply away to them. Therefore, we kind of calculate not the number of clients, but with sum their weights. Pretty the same happens for the formula with false negative rate. And a soon, as we assume as we calculate this false negative and false positives. We could go to the benefit curve constructions as we did in the week to, and our estimates would be unbiased this case. Well, just to make these formulas kind of look more familiar to you, we will just test them on these two example. We see these two clients, and we understand that the number of rejected clients is for it's all red rose that were previously in this lower part of table. At the same time, there are only two clients within control group, so our ratio is 4 to 2, which makes two. So each observations from the control group receives weight 2. The same time we can input this in tow, false, positive and false negative rate formulas and get the following results. The false positive is racial between 1 and 4, which is 25% and for force false negative creation. We big always of false negative cases within our accepted clients and our control group. Which means one plus, one plus two, and we divided by the total numbers like four plus two, plus two and we get 50% estimate for false negative oration. This is a good way to handle the problem of unobserved model errors. But there is a certain problem with constructing control groups. This maybe too expensive sometimes, for instance, in credit scoring, it means that we give alone to random clients random borrowers, regardless of their financial situation. Regardless, their risk estimates of course, we try to make this group as small as possible. But in fact we kind of expose ourselves to great credit risk. Therefore, large losses can a cure. From the other point, if we don't take credit scoring there. IT infrastructure cannot be really that ready for smooth A /B testing in order to run to process. In the parallel, you have to kind of have that infrastructure and have that IT culture within your company or your startup. And not always the company kind of companies are easy about it. So therefore they're going to be just technical problems by collecting and gathering all the data from A/B tests and control groups simultaneously and merging them for further analysis. Drip up designing control groups can help us to restore and gain representative data about rejected clients and help us co-create false positives and false negatives. Errors of our model you have to keep in mind that you have to wait observations from control group in order to increase their weight as soon as usually there only small portion off that rejected population well. At the same time, running control groups can be a problem. Either it's too expensive or it's just unavailable from the 80 standpoint. Well, what should we do when we cannot use control groups? Is there any other option? Indeed, the reason we will see that in next episode.