R-DEIM Net: An Efficient Rationale-Augmented Dual-Expert Interaction Model for Paraphrase Detection

Problem

This work addresses the gap in capability between the high accuracy of large language models and the practical scalability of efficient Siamese-BERT variants in paraphrase detection. The authors highlight the need for a model that can achieve competitive performance while maintaining efficiency, particularly in real-world applications.

Method

The proposed architecture, R-DEIM Net, consists of 76 million parameters and features a dual-expert design:

• Interaction Expert: This component captures token-level similarity patterns through the use of multi-scale 2D convolutions combined with an attention head, enabling it to handle variable input lengths effectively.
• Reasoning Expert: This expert employs a Flan-T5-small decoder to generate rationales, which serve as auxiliary supervision for the model. This approach aims to enhance the interpretability and robustness of the predictions.
• Feature Extraction: The model extracts and pools the hidden states from the decoder, utilizing these as complementary features for the final classification task.
• Data: The model is trained and evaluated on the Quora Question Pairs dataset, a standard benchmark for paraphrase detection tasks.
• Training Compute: The paper does not specify the training compute resources utilized for model training.

Results

The R-DEIM Net achieves an accuracy of 90.07% on the Quora Question Pairs dataset, which is slightly lower than the MFAE BERT model's accuracy of 90.54%. The F1-score reported for R-DEIM Net is 90.16%, although no baseline for this metric is provided in the paper. The available text does not report quantitative results for other performance metrics or comparisons against additional baselines.

Limitations

The authors do not report any limitations in their work. However, the lack of detailed training compute specifications and the absence of a broader range of baseline comparisons may limit the understanding of the model's performance in various contexts.

Why it matters

The development of R-DEIM Net has significant implications for downstream work in paraphrase detection and related tasks. By combining dual-expert mechanisms with rationale generation, this model not only aims to improve accuracy but also enhances interpretability, which is crucial for applications requiring trust and transparency in AI systems. The balance between efficiency and performance could facilitate the deployment of advanced paraphrase detection systems in resource-constrained environments.