This repository is about the reimplementation of the result of the paper "Credit risk evaluation model with textual features from loan descriptions for P2P lending", Zhang et al. (2020). It has been made within my Master Thesis topic research at NTNU (Uncertainty Quantification for Language Models in Safety-Critical Systems). The model specificities are given bellow.

The dataset is from LendingClub, [2] loans data from the American market.

Since LendingClub does not provide anymore this data, the dataset used have been taken from Kaggle.

In this section the dataset used will be presented, the feature selection -as in the original paper- will be described, and its characteristics will be compared to the one used in the paper.

In this section the variables names and its description are given. There are few variables that has been computed from existing variables in the dataset, but where not in the orignal dataset. The code for this process can be found in src/data/features.py.

[*]: Malekipirbazari et al. (2015)

The data selection has been done the closer to the one done in the original paper.

Hence, as mentioned in the article, I used the loans issued during the period of 2007–2014. After performing a data cleaning operation (discard loans with missing values and with a description length of less than 20 words), I have obtained 69,276 (70,488 in the paper) loan records, 10,151 (against 10,534) (14.65% against 14.94%) of which were ultimately placed in a default status.

On each cells the data is given like this: {paper} / {processed}. The results provided can be runned in read_csv.ipynb. Pbc means Point-biserial correlation.

The first thing was the feature selection for the model. Hence the hard features (not textual features) are processed by one hot encoding them. Hence a splitting is done: 80% for training set, 10% for dev set and 10% for test set. A resampling strategy on the training set is done to address the data imbalance problem: I have performed an over-sampling strategy for the loans in default and an under-sampling strategy for the positive sample. Due to implementation issues, advanced techniques such SMOTE Chawla et al. (2002) for oversampling or Tomek's link Tomek (1975) for undersampling could not been processed here because of the textual features and that I have not found a way to track the ids. Because in the origial paper they do not mention a specific technique, I have decided to use both a random over sampling and under sampling Brownlee (2021). Once the training set is balanced, I normalize the hard features by proceeding with standardization. Meanwhile, the textual features are processed as described in the Textual features section. Finally, the model is trained on both train and dev sets.

The textual features preprocessing is basically processed in two parts. First, the segmentation and then converting the segmented words into embeddings.

According to the paper I used CoreNLP model from Manning et al. (2014) for the segmentation of textual features. However, I understand that, from that time, the technology evolved and as been incorporated to the python package stanza. So instead of using the Java server provided by the authors of the original paper of CoreNLP, I have chosen to use stanza.Pipeline which is more optimized for Python execution. Moreover, the authors of the paper that I tried to replicate do not mention which option (called annotations) of the CoreNLP.Pipeline they used. So I felt free to use some different types of them. I have also added an option to prune words which are included in the nltk.stopwords set. The results are provided is the Results section.

After tokenizing the words, I used a pre-trained word vectors model named GloVe, from Pennington et al. (2014) to create the embeddings of the tokens, as they did in the original paper. GloVe provide several pre-trained word vectors models which has for the embedding dimensions, only 3 sizes: 100, 200 and 300. As described in Model section, and since my goal is to replicate the best model from the paper, the number of heads for the multi-head attention mechanism, from Vaswani et al. (2017), is 8. Hence, I have chosen the 200 dimensions model since the article does not mention the dimension of the Transformer Encoder (TE) model. The implementation itself have been done with spacy, the most recent official package from Stanford for GloVe embeddings.

The base of the model used is a Transformer Encoder (TE) model from Vaswani et al. (2017). The textual features processed are then fed into the TE. As in the original paper, the TE is composed of 1 layers, with 8 heads for the multi-head attention mechanism, and feed-forward networks (FFN) has a size of 50 neurons and the activation function used is the rectified linear unit (ReLU) function. Then, as the TE is considered here as a feature exctractor, only the first sequence of the output of the TE is used. It's concatenated with the hard features and fed into a feed-forward neural network (FNN) with 2 hidden layers of 10 neurons each, separated by a ReLU activation function. The output layer consists of two neurons passed through a Softmax layer.

The model is trained with a binary cross-entropy loss function and the Adam optimizer with a learning rate of 0.0001 and weight decay of 0.0001 to avoid over-fitting. An early-stopping strategy is applied on training. The model is evaluated on the test set using the ROC-AUC and G-MEAN metrics, developped in Metrics section.

A dropout strategy of 0.3 (value not precised in the article) is used for the TE and after each layer of the FNN to avoid overfitting.

The model is evaluated on the test-set using the ROC-AUC and G-MEAN metrics. The ROC-AUC is the area under the receiver operating characteristic curve, which is a plot of the true positive rate against the false positive rate. The G-MEAN is the geometric mean of the true positive rate and the true negative rate. The ROC-AUC is a measure of the model's ability to distinguish between the positive and negative classes, while the G-MEAN is a measure of the model's ability to balance the true positive and true negative rates. Both are effective metrics for imbalanced data sets (Gong et al., 2017).

The G-mean is defined as follows:

$$G_{mean} = \sqrt{\left(\frac{TP}{TP + FN}\right)\left(\frac{TN}{TN + FP}\right)}$$

1. Clone the repository:

   git clone [email protected]:art-test-stack/credit_risk_eval_model.git

2. Navigate to the project directory:

   cd credit_risk_eval_model

3. Create a virtual environment

   For example I use virtualenv:

   virtualenv -p python 3.10 venv

4. Install Python depedencies:

   pip install -r requirements.txt

5. Download the GloVe on Glove's website and put the 200d model file in the model/glove.6B/ folder (more details on that README.md).

6. Download the LendingClub dataset from Kaggle and put the .csv file in the data/ folder.

Once all the installation steps are done, you can do the preprocessing, train, and evaluate the model by running the following command:

python main.py

You can then add different arguments to the command line to change the model's parameters. For example, if you already done a preprocessing, you can skip it, if you want to train another model from the same data, by running:

python main.py --skip_preprocessing

Here the list of the different arguments:

• --no_stopwords (flag): Disable the stopwords from the nltk package.

• --not_balance_dataset (flag): To not balance the training set.

• --skip_preprocessing (flag): Skip the preprocessing step.

• --not_verbose (flag): Disable print information during the training.

• --epochs: Number of epochs to train the model. Default: 500

• --batch_size: Batch size for training. Default: 1024

• --dropout: Dropout rate for the model. Default: 0.3

• --early_stopping_patience: Number of epochs with no improvement after which training will be stopped. Default: 100

• --early_stopping_min_delta: Minimum value to improve model in early stopping. Default: 0.0001

• --model_name: Name of the trained model to find its .pt file and its training loss. Default: 'model.pt'

1. Zhang, et al. (2020). "Credit risk evaluation model with textual features from loan descriptions for P2P lending." Electronic Commerce Research and Applications, 39, 100989. pdf

2. Milad Malekipirbazari, Vural Aksakalli, "Risk assessment in social lending via random forests," Expert Systems with Applications, Volume 42, Issue 10, 2015, Pages 4621-4631, ISSN 0957-4174, pdf.

3. Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. (2002). "SMOTE: Synthetic Minority Over-sampling Technique." Journal of Artificial Intelligence Research, 16, 321–357.

4. Ivan Tomek. (1976). "Two modifications of cnn." IEEE Transactions on Systems, Man, and Cybernetics, 6, 769–772.

5. Jason Brownlee. (2021). "Random Oversampling and Undersampling for Imbalanced Classification." Retrieved from article.

6. Manning, Christopher D., Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. (2014). "The Stanford CoreNLP Natural Language Processing Toolkit." Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 55-60. pdf.

7. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. (2014). "GloVe: Global Vectors for Word Representation." pdf

8. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. (2017). "Attention is All You Need." Advances in Neural Information Processing Systems, 30, 5998-6008. pdf

9. Gong, Joonho, Kim, Hyunjoong. (2017). "Rhsboost: Improving classification performance in imbalance data." Computational Statistics & Data Analysis, 111, 1–13. pdf

This project is licensed under the MIT License - see the LICENSE file for details.