Self-attention on RNN-based text classification

Dailun Li; Zeying Tian; Yining Duan

doi:10.1117/12.2641031

3 October 2022 Self-attention on RNN-based text classification

Dailun Li, Zeying Tian, Yining Duan

Proceedings Volume 12290, International Conference on Computer Network Security and Software Engineering (CNSSE 2022); 1229005 (2022) https://doi.org/10.1117/12.2641031
Event: International Conference on Computer Network Security and Software Engineering (CNSSE 2022), 2022, Zhuhai, China

Abstract

Nowadays, many researchers claim that the self-attention mechanism generates a better performance regardless of circumstances. Having doubted such opinion, we examine different models with scalar dot-product self-attention mechanisms under various experimental settings to develop a comprehensive understanding of such technique. In conclusion, we have verified that the performance of the attention mechanism grows as the dataset input increases in size. In addition, we have observed that our attention layer can impact the model’s performance negatively on small datasets. Moreover, we demonstrate that the attention mechanism is model-dependent: opposite effects may be obtained on the same dataset with different model.

Citation Download Citation

Dailun Li, Zeying Tian, and Yining Duan "Self-attention on RNN-based text classification", Proc. SPIE 12290, International Conference on Computer Network Security and Software Engineering (CNSSE 2022), 1229005 (3 October 2022); https://doi.org/10.1117/12.2641031

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available