Currently, the credit score problem has two datasets: Set V01, including users with label 0 is the High label of TS, and V1 is a set of 600K labels of 0 including 200K Low, 200K Medium and 200K High labels of TS. In addition, there are 54K users with bad debt, which will be used to uniformly blend into both sets to ensure an even distribution of bad labels.

The question is which set should be taken out to train the model. Comparing 2 volumes V01 and V1 when removing label 1, have the following characteristics: Volume V01 has a more standard label than V1 (because in fact, after visualize, there are some L labels of V1 that belong to the bad debt group, label H. This does not happen at all. This means that the label H of TS as label 0 will have higher confidence). However, the set V1 has a distribution closer to the actual distribution of the dataset, because our dataset includes not only users with label 1 with High score as in V01, but also labels with M, L. Choosing 1 in Two sets V01 or V1 make training sets, so they have the following characteristics:

• 1. If V01 is selected, the label is more standard, but the distribution of the dataset is much different from the real distribution, so theoretically, the results will be much down on the back test set.
• 2. If V1, a non-standard label is selected, it can be dangerous that the model will have the RECALL index completely down, because many bad labels are classified as good. However, the results on the back test set may not be down much
• Frequency and average value of TV bill transactions in the month.
• Frequency and average value of loan transactions during the month.
• Number of different transaction types used
• Behavioral data on telecommunications, Viettelpay Pro, Appsflyer can be added added during model building.

If we choose V01 as the train set, and predict on the set V01, then the result on V01 is quite reliable, because the distribution of this set V01 is closer to the real distribution of the whole dataset. After mixing well according to the ratio of bad labels on both sets V1 and V01, the train model on V01 gives the following results:

Train result, test on V01, gini result down because label 1 is rarer, due to subdivision to set V1 for testing
Model results, predict on set V1, gini doesn't down too much

If we delete all attributes with more than 1 month of inactive, we get the following results:

Figure 3 trains on the set V1, the first test is the test set on V1, the last test is the set V01. Bad labels are uniformly divided among episodes
Note, the down results are also partly due to too few labels 1 in the training set, with a rate of about 2% label 1 in the training set.

If we train on V01, we get the result:

Train on V01, TEST is the set spit from V01, extra test is episode V1

Deleting users with many months of inactive months results in a drop, which is understandable because bad debt users have a higher inactive rate than good debt users. However, this is advisable, because it is not advisable to predict users with too few observations. In addition, deleting these users also helps the model not to put too much weight on the null property, so many other more meaningful behaviors will be detected and included in the model, from which the model will classify well. than.

In both models, training on the set V01 or training on V1, the accuracy on the test set extracted from V01 is always better than the test set extracted from V1. This is quite reasonable since the set V1 has a more complex distribution, the separation between the two types of labels is quite small. The result on the test set extracted from V1 is 0.638 if train on V01 and 0.674 if train on V1. The interesting thing is that even when training on set V1, the accuracy on the test extracted from this set is smaller than the accuracy on set V01, which proves that the model is quite stable and performs quite well on datasets with different distributions.

Proposals:

• Get the results on the test set extracted from V1 as the results for the reportss
• Need to get more Recall score is another indicator to evaluate the model, instead of just observing the gini index. In the problem under consideration, mistakenly guessing label 1 to 0 (Faux negative) is much more dangerous than guessing wrong label 0 to 1 (Faux Positive). In terms of the Recall index, the model in Figure 4 is much better than the model in Figure 3 (although the Gini index has the opposite result).
• Considering the set of users with the number of inactive months greater than 1, there are 35% of bad debts, while the distribution of bad debts in general is only 1.85%. The distribution is as follows:

Figure 6 Cases of bad debt are usually either inactive, or active a number of months later inactive, the ability to avoid debt.s

What is the relationship between high inactive months and bad debt?

• There is a high probability that the bad debt affects active month, and not vice versa, because before bad debt, the user still uses that phone number normally, so there will be a low number of inactive months. However, when bad debt occurs, this user also gives up this phone number to avoid debt, leading to an increase in the number of inactive months observed afterwards.
• In fact, this factor needs to be removed from the model, because it only happens after the time of predict. Therefore, it is necessary to remove all users that have inactive>=1. This removal will certainly bring down the results of the model, but is necessary to prevent data leakage.s
• Hypothesis: User's Bihavior before and after bad debt occurs is different. So, ideally, the data should be obtained before the bad debt occurs (even before the loan is taken, if possible). It is very likely that this model will give poor test set and back test results, but it is highly likely that it will perform better in practice, compared to current models.
• However, it is still recommended to keep this feat to predict MB's back test result, because this feat still predicts bad debt good when bad debt event has occurred (and this is the case back test MB, and not is the case for lending decisions).