ISWC2020 Semantic Web Challenge - Product Classification Top1 Solution

1. Train 17 different BERT base models with dynamic masked softmax
2. Adopt a two-level ensemble strategy to combine the single models
3. Utilize pseudo labeling for data augmentation

• BERT-PO uses the pooler output of BERT as the product representation.
• BERT-K-hidden concatenates the first hidden state from the last $K$ hidden layers of BERT as the product representation.
• BERT-K-hidden-PO concatenates the first hidden state from the last $K$ hidden layers as well as the pooler output of BERT as the product representation.
• BERT-seq uses the hidden states from the last hidden layer of BERT as the input of another sequence layer, and then concatenates the pooler output of BERT, with the last hidden output as well as the max-pooling and mean-pooling over the hidden states of sequence layer, as the final product representation.

1. Devise a mask matrix for each sub-level based on the category hierarchy
   $$ M^{l} \in{0,1}^{N^{l-1} * N^{l}} $$
2. Adopt dynamic masked softmax to filter out the unrelated categories
   $$ P\left(y_{v}^{l} \mid s, \theta\right)=\frac{\exp \left(O_{v}^{l}\right) * M_{u, v}^{l}+\exp (-8)}{\sum_{v^{\prime}=1}^{N} \exp \left(O_{v^{\prime}}^{l}\right) * M_{u, v^{\prime}}^{l}+\exp (-8)} $$

1. Averaging ensemble to the single models with the same architecture but trained on different folds of data
2. Voting ensemble to 17 different single models

pip install virtualenv
virtualenv tf2
source tf2/bin/activate
pip install -r requirements.txt

1. dataset
   Download the dataset below, put them in the raw_data dir:

   • train.json
   • validation.json
   • test_public.json
   • task2_testset_with_labels.json

2. pre-trained BERT model Download the files of BERT model, put them in the raw_data/embeddings/bert-base-uncased dir:

   • bert-base-uncased-vocab.txt
   • bert-base-uncased-config.json
   • bert-base-uncased-tf_model.h5

python preprocess.py

python train.py

python ensemble.py

python train_pseudo_label.py

python ensemble_pseudo_label.py