Long Short-Term Memory Models
Results and Discussion
This section compares performances of state-of-the-art (SOTA) models in terms of accuracy, time, and loss.
Time Analysis
The training time comparisons of SOTA models are indicated in Table 1. The results indicated that most DL models provide reasonable training time except the transformer-based model. Models that use LSTM, BoT, and CNN performed an epoch per minute, whereas the BoT-based model achieves 13 s per epoch in contrast to 28 min in the case of the transformer model. In the testing phase, results are aligned with the training phase. Even though only time analysis does not give a concrete interpretation of a model, we see a considerable time efficiency difference between BERT and other models.
Table 1. Training and testing time comparison of SOTA models
|
Epoch/Test |
LSTM |
BoT |
CNN |
Transformer |
|---|---|---|---|---|
|
1 |
1 m 41 s |
0 m 14 s |
0 m 30 s |
28 m 4 s |
|
2 |
1 m 40 s |
0 m 13 s |
0 m 30 s |
28 m 7 s |
|
3 |
1 m 41 s |
0 m 13 s |
0 m 30 s |
28 m 6 s |
|
4 |
1 m 40 s |
0 m 13 s |
0 m 30 s |
28 m 7 s |
|
5 |
1 m 40 s |
0 m 13 s |
0 m 30 s |
28 m 7 s |
|
6 |
1 m 41 s |
0 m 13 s |
0 m 30 s |
28 m 6 s |
|
7 |
1 m 41 s |
0 m 13 s |
0 m 30 s |
28 m 4 s |
|
8 |
1 m 40 s |
0 m 13 s |
0 m 30 s |
28 m 7 s |
|
9 |
1 m 40 s |
0 m 13 s |
0 m 30 s |
27 m 56 s |
|
10 |
1 m 40 s |
0 m 13 s |
0 m 30 s |
27 m 58 s |
|
Testing |
15 ms |
9 ms |
10 ms |
35 ms |
Validation and Test Losses
Validation loss is another critical metric to evaluate how a model fits new data. Validation loss is also a good indicator of overfitting. The models’ validation, training, and test losses are shown in Fig. 1 and Table 2.
Fig. 1. Validation and training losses of the models.
The loss graph of the transformer-based model indicates that it could converge faster than other models with fewer training epochs. This will be a result of pre-training of the transformer model.
Table 2. Test losses of the models.
|
Models |
LSTM |
BoT |
CNN |
Transformer |
|---|---|---|---|---|
|
Loss |
0.323 |
0.391 |
0.344 |
0.209 |
Validation Accuracy
Validation accuracy in combination with validation loss could be used to determine the model's generalization ability. The validation and testing accuracies of the models are given in Table 3. Validation accuracy reveals that five epochs of training are enough to get good results which are also in line with the validation loss. Testing accuracy is aligned with the validation accuracy where the transformer-based model achieves the best performance.
Table 3. Validation and testing accuracies of the models.
|
Epoch/Test |
LSTM |
BoT |
CNN |
Transformer |
|---|---|---|---|---|
|
1 |
73.28% |
72.08% |
77.03% |
91.93% |
|
2 |
82.19% |
76.29% |
84.56% |
91.76% |
|
3 |
79.64% |
80.50% |
86.06% |
92.02% |
|
4 |
87.71% |
83.55% |
86.78% |
90.74% |
|
5 |
87.81% |
85.47% |
86.99% |
91.31% |
|
6 |
89.27% |
85.47% |
87.23% |
91.31% |
|
7 |
89.65% |
87.09% |
87.16% |
90.89% |
|
8 |
91.52% |
87.68% |
87.30% |
91.19% |
|
9 |
88.06% |
88.07% |
86.96% |
92.15% |
|
10 |
89.69% |
88.46% |
87.40% |
91.85% |
|
Testing |
86.96% |
85.18% |
85.04% |
91.58% |
Observations derived from the performance comparisons are outlined below.
Observation 1: BoT-based model is faster than other DL models.
Observation 2: Transformer-based model takes a long time to train and predict.
Observation 3: Optimum epoch number could be determined using accuracy and loss of training and validation phases. Five epochs of training provide optimum training.
Observation 4: Transformer-based model converges faster than other models.