Adaptive Ensemble Thresholding for OOD Intent Detection

Akbari, Masoud; Mohades, Ali; Shirali-Shahreza, M.Hassan

doi:10.22133/ijwr.2025.528472.1295

Adaptive Ensemble Thresholding for OOD Intent Detection

Document Type : Original Article

Authors

Masoud Akbari

Ali Mohades

M.Hassan Shirali-Shahreza

Department of Mathematics and Computer Science, Amirkabir University of Technology, Tehran, Iran;

10.22133/ijwr.2025.528472.1295

Abstract

Out-of-domain intent detection in natural language understanding systems faces significant challenges from suboptimal threshold selection and signal degradation through inappropriate normalization techniques. This paper presents an adaptive ensemble thresholding framework that substantially extends our previous conference work by addressing fundamental limitations in existing variational autoencoder-based detection methods. Our approach combines reconstruction loss from variational autoencoders with classifier confidence scores to create a unified detection signal that captures both semantic deviation and prediction uncertainty. The framework incorporates a novel smart scaling strategy that preserves natural separation ratios between in-domain and out-of-domain samples, preventing the signal destruction caused by standard normalization approaches. Through systematic parameter optimization using grid search techniques, the method adaptively determines optimal ensemble weights and threshold selection strategies tailored to specific dataset characteristics. We evaluate our framework across multiple datasets with varying semantic complexity and domain structures, demonstrating consistent performance improvements over baseline variational autoencoder approaches and recent state-of-the-art methods. Compared to our previous VAE-based approach, the framework demonstrates an average performance gain of 3.15 percentage points across all evaluation metrics. Our analysis reveals that ensemble scaling strategy significantly impacts detection performance, with proper signal preservation being more critical than sophisticated threshold selection methods. This work provides a principled approach to adaptive ensemble learning for out-of-domain detection, offering a robust solution that generalizes effectively across diverse datasets and linguistic contexts including low-resource languages like Persian.

Keywords

Natural Language Understanding

Out-of-Domain Intent Detection – Adaptive Thresholding – Ensemble Learning

Subjects

Web Software Platforms & Tools

A. Gupta, P. Zhang, G. Lalwani and M. Diab, “Context-aware self-attentive natural language understanding for task-oriented chatbots,” Amazon, 2019. https://www.amazon.science/publications/context-aware-self-attentive-natural-language-understanding-for-task-oriented-chatbots

B. Galitsky, “Chatbot Components and Architectures,” in Developing Enterprise Chatbots: Learning Linguistic Structures, Springer International Publishing, 2019, pp. 13-51. https://doi.org/10.1007/978-3-030-04299-8_2

Z. Zhang, R. Takanobu, Q. Zhu, M. Huang and X. Zhu, “Recent advances and challenges in task-oriented dialog systems,” Science China Technological Sciences, vol. 63, no. 10, pp. 2011-2027, 2020. https://doi.org/10.1007/s11431-020-1692-3

A. G. Khoee, Y. Yu, R. Feldt, A. Freimanis, P. A. Rhodin and D. Parthasarathy, “GoNoGo: An Efficient LLM-Based Multi-agent System for Streamlining Automotive Software Release Decision-Making,” in 36th International Conference on Testing Software and Systems (ICTSS), Cham: Springer Nature Switzerland, 2025, pp. 30-45. https://doi.org/10.1007/978-3-031-80889-0_3

A. G. Khoee, S. Wang, Y. Yu, R. Feldt and D. Parthasarathy, “GateLens: A Reasoning-Enhanced LLM Agent for Automotive Software Release Analytics,” arXiv preprint arXiv:2503.21735, 2025. https://doi.org/10.48550/arXiv.2503.21735

P. Wang et al., “Beyond the Known: Investigating {LLM}s Performance on Out-of-Domain Intent Detection,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Torino, Italia, 2024, pp. 2354-2364. https://aclanthology.org/2024.lrec-main.210.pdf

G. Arora, S. Jain and S. Merugu, “Intent Detection in the Age of LLMs,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, Miami, Florida, pp. 1559-1570, 2024. https://aclanthology.org/2024.emnlp-industry.114.pdf

S. Lu, Y. Wang, L. Sheng, L. He, A. Zheng and J. Liang, “Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advances,” arXiv preprint arXiv:2409.11884, 2025. https://doi.org/10.48550/arXiv.2409.11884

H. Lang, Y. Zheng, B. Hui, F. Huang and Y. Li, “Out-of-Domain Intent Detection Considering Multi-Turn Dialogue Contexts,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Torino, Italia, pp. 12539–12552, 2024. https://aclanthology.org/2024.lrec-main.1097.pdf

D. Yang, K. Mai Ngoc, I. Shin, K.H. Lee and M. Hwang, “Ensemble-Based Out-of-Distribution Detection,” Electronics, vol. 10, no. 5, p. 567, 2021. https://doi.org/10.3390/electronics10050567

A. G. Khoee, S. Wang, Y. Yu, R. Feldt and D. Parthasarathy, “GateLens: A Reasoning-Enhanced LLM Agent for Automotive Software Release Analytics,” arXiv preprint, 2025.

J. Yang, K. Zhou, Y. Li and Z. Liu, “Generalized Out-of-Distribution Detection: A Survey,” International Journal of Computer Vision, vol. 132, no. 12, pp. 5635-5662, 2024. https://doi.org/10.1007/s11263-024-02117-4

P. Cui and J. Wang, “Out-of-Distribution (OOD) Detection Based on Deep Learning: A Review,” Electronics, vol. 11, no. 21, p. 3500, https://doi.org/10.3390/electronics11213500

X. Ran, M. Xu, L. Mei, Q. Xu and Q. Liu, “Detecting out-of-distribution samples via variational auto-encoder with reliable uncertainty estimation,” Neural Networks, vol. 145, pp. 199-208, 2022. https://doi.org/10.1016/j.neunet.2021.10.020

Z. Xiao, Q. Yan and Y. Amit, “Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder,” Advances in neural information processing systems, vol. 33, pp. 20685-20696. 2020.

M. Guarrera, B. Jin, T.-W. Lin, M. A. Zuluaga, Y. Chen and A. Sangiovanni-Vincentelli, “Class-Wise Thresholding for Robust Out-of-Distribution Detection,” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), New Orleans, LA, USA, 2022, pp. 2836–2845. https://doi.org/10.1109/CVPRW56347.2022.00321

S. Ando and T. Kounoike, “An Ensemble OOD Detection with Norm-enhancing Representation Learning,” in Proceedings of the 2024 8th International Conference on Information System and Data Mining, 2024, pp. 90-94, https://doi.org/10.1145/3686397.3686412

F. Ataeiasad, D. Elizondo, S. Calderón Ramírez, S. Greenfield and L. Deka, “Out-of-Distribution Detection with Memory-Augmented Variational Autoencoder,” Mathematics, vol. 12, no. 19, p. 3153, 2024. https://doi.org/10.3390/math12193153

Z. Zeng and B. Liu, “Unsupervised out-of-distribution detection by restoring lossy inputs with variational autoencoder,” arXiv preprint arXiv:2309.02084, 2024.

https://doi.org/10.48550/arXiv.2309.02084

T. Denouden, R. Salay, K. Czarnecki, V. Abdelzad, B. Phan and S. Vernekar, “Improving reconstruction autoencoder out-of-distribution detection with mahalanobis distance,” arXiv preprint arXiv:1812.02765, 2018.

https://doi.org/10.48550/arXiv.1812.02765

H. Torabi, S. L. Mirtaheri and S. Greco, “Practical autoencoder based anomaly detection by using vector reconstruction error,” Cybersecurity, vol. 6, no. 1, p. 1, 2023. https://doi.org/10.1186/s42400-022-00134-9

Y. Yu, S. Shin, S. Lee, C. Jun and K. Lee, “Block Selection Method for Using Feature Norm in Out-of-Distribution Detection, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 15701-15711. https://doi.org/10.1109/CVPR52729.2023.01507

D. Khurana, A. Koli, K. Khatter and S. Singh, “Natural language processing: state of the art, current trends and challenges,” Multimedia Tools and Applications, vol. 82, pp. 3713-3744, 2023. https://doi.org/10.1007/s11042-022-13428-4

M. Akbari, A. Mohades and M. H. Shirali-Shahreza, “A Hybrid Architecture for Out of Domain Intent Detection and Intent Discovery,” in 2025 11th International Conference on Web Research (ICWR), Tehran, Islamic Republic of Iran, 2025, pp. 137-144. https://doi.org/10.1109/ICWR65219.2025.11006168

Y. Yang, R. Gao and Q. Xu, “Out-of-Distribution Detection with Semantic Mismatch Under Masking,” Cham: Springer Nature Switzerland, 2022, pp. 373-390. https://doi.org/10.1007/978-3-031-20053-3_22

Y. Zhou, “Rethinking Reconstruction Autoencoder-Based Out-of-Distribution Detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 7369-7377. https://doi.org/10.1109/CVPR52688.2022.00723

J. Li, P. Chen, Z. He, S. Yu, S. Liu and J. Jia, “Rethinking Out-of-Distribution (OOD) Detection: Masked Image Modeling Is All You Need,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 2023, pp. 11578-11589. https://doi.org/10.1109/CVPR52729.2023.01114

X. Du, G. Gozum, Y. Ming and Y. Li, “SIREN: Shaping Representations for Detecting Out-of-Distribution Objects,” in Advances in Neural Information Processing Systems, vol. 35, pp. 20434-20449, 2025.

S. Pei, “Image background serves as good proxy for out-of-distribution data,” arXiv preprint arXiv:2307.00519, 2023.

https://doi.org/10.48550/arXiv.2307.00519

Z. Liu, J. P. Zhou, Y. Wang and K. Q. Weinberger, “Unsupervised Out-of-Distribution Detection with Diffusion Inpainting,” in Proceedings of the 40th International Conference on Machine Learning, PMLR, 2023, pp. 22528-22538. https://proceedings.mlr.press/v202/liu23bd.html

R. Gao, C. Zhao, L. Hong and Q. Xu, “DIFFGUARD: Semantic Mismatch-Guided Out-of-Distribution Detection Using Pre-Trained Diffusion Models,” 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, 2023, pp. 1579-1589, https://doi.org/10.1109/ICCV51070.2023.00152

H. Wei, R. Xie, H. Cheng, L. Feng, B. An and Y. Li, “Mitigating Neural Network Overconfidence with Logit Normalization,” in Proceedings of the 39th International Conference on Machine Learning, PMLR, 2022, pp. 23631-23644. https://proceedings.mlr.press/v162/wei22d.html

L. Tao, X. Du, X. Zhu and Y. Li, “Non-Parametric Outlier Synthesis,” arXiv preprint arXiv:2303.02966, 2023. https://doi.org/10.48550/arXiv.2303.02966

Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong and S. X. Yu, “Large-Scale Long-Tailed Recognition in an Open World,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019. pp. 2537-2546. https://doi.org/10.1109/CVPR.2019.00264

H. Lu, D. Gong, S. Wang, J. Xue, L. Yao and K. Moore, “Learning with Mixture of Prototypes for Out-of-Distribution Detection,” arXiv preprint arXiv:2402.02653, 2024. https://doi.org/10.48550/arXiv.2402.02653

S. Regmi, B. Panthi, Y. Ming, P. K. Gyawali, D. Stoyanov and B. Bhattarai, “ReweightOOD: Loss Reweighting for Distance-based OOD Detection,” 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA, 2024, pp. 131-141, https://doi.rg/10.1109/CVPRW63382.2024.00018

A. Rahimi and H. Veisi, “Integrating Model-Agnostic Meta-Learning with Advanced Language Embeddings for Few-Shot Intent Classification,” in 2024 32nd International Conference on Electrical Engineering (ICEE), Tehran, Islamic Republic of Iran, 2024, pp. 1-5, https://doi.org/10.1109/ICEE63041.2024.10667921

J. An and S. Cho, “Variational autoencoder based anomaly detection using reconstruction probability,” Special lecture on IE, vol. 2, pp. 1-18, 2015. http://dm.snu.ac.kr/static/docs/TR/SNUDM-TR-2015-03.pdf

Y. Li et al.,“AVOID: Alleviating VAE's Overestimation in Unsupervised OOD Detection,” 2024. https://openreview.net/forum?id=3a505tMjGE

A. Bansal, M. Yuhas and A. Easwaran, “Compressing VAE-Based Out-of-Distribution Detectors for Embedded Deployment,” in 2024 IEEE 30th International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA), Sokcho, Republic of Korea, 2024, pp. 37-42, https://doi.org/10.1109/RTCSA62462.2024.00015

S. Ramakrishna, Z. Rahiminasab, G. Karsai, A. Easwaran and A. Dubey, “Efficient Out-of-Distribution Detection Using Latent Space of β-VAE for Cyber-Physical Systems,” vol. 6, no. 2, pp. 1-34, 2022. https://doi.org/10.1145/349124

Huang, H. J. Sicong and K. Y.-C. Lui, “Inference, Fast and Slow: Reinterpreting VAEs for OOD Detection,” 2025. https://openreview.net/forum?id=K1VpgaYPnX

Y. Zheng, G. Chen and M. Huang, “Out-of-Domain Detection for Natural Language Understanding in Dialog Systems,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 1198-1209, 2020. https://doi.org/10.1109/TASLP.2020.2983593

T. L. Molloy, J. J. Ford and L. Mejias, “Adaptive detection threshold selection for vision-based sense and avoid,” in 2017 International Conference on Unmanned Aircraft Systems (ICUAS), 2017.

B. Magaz, A. Belouchrani and M. Hamadouche, ”Automatic Threshold Selection in Os-CFAR Radar Detection Using Information Theoretic Criteria,” Progress In Electromagnetics Research B, vol. 30, pp. 157-175, 2011. https://doi.org/10.2528/PIERB10122502

H. Lin, H. Vishwakarma and R. K. Vinayak, “Adaptive Out-of-Distribution Detection with Human-in-the-Loop,” ICML 2022 Workshop on Human-Machine Collaboration and Teaming, Baltimore, Maryland, USA, 2022. https://ramyakv.github.io/Adaptive-OOD-Detection-Human-in-the-loop.pdf

X. Wu, J. Lu, Z. Fang and G. Zhang, “Meta OOD Learning For Continuously Adaptive OOD Detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. https://doi.org/10.1109/ICCV51070.2023.01773

K. Fang, Q. Tao, X. Huang and J. Yang, “Revisiting Deep Ensemble for Out-of-Distribution Detection: A Loss Landscape Perspective,” International Journal of Computer Vision, pp. 6107-6126, 2024. https://doi.org/10.1007/s11263-024-02156-x

L. E. Hogeweg, R. Gangireddy, D. Brunink, V. J. Kalkman, L. Cornelissen and J. W. Kamminga, “COOD: Combined out-of-distribution detection using multiple measures for anomaly & novel class detection in large-scale hierarchical classification,” in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2024.

E. A. Abyaneh, R. Zolfaghari and A. A. Abyaneh, “User Intent Detection in Persian Text-Based Chatbots: A Comprehensive Review of Methods and Challenges,” in 2025 11th International Conference on Web Research (ICWR), Tehran, Islamic Republic of Iran, 2025, pp. 243-249, https://doi.org/10.1109/ICWR65219.2025.11006173

M. Farahani, M. Gharachorloo, M. Farahani and M. Manthouri, “ParsBERT: Transformer-based Model for Persian Language Understanding,” Neural Processing Letters, vol. 53, pp. 3831-3847, 2021. https://doi.org/10.1007/s11063-021-10528-4

R. Zadkamali, S. Momtazi and H. Zeinali, “Intent detection and slot filling for Persian: Cross-lingual training for low-resource languages,” Natural Language Processing, vol. 31, no. 2, pp. 559-574, 2025. https://doi.org/10.1017/nlp.2024.17

A. Abaskohi et al.,“Benchmarking Large Language Models for Persian: A Preliminary Study Focusing on ChatGPT,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024. https://aclanthology.org/2024.lrec-main.197.pdf

M. Akbari et al., “A Persian Benchmark for Joint Intent Detection and Slot Filling,” arXiv preprint arXiv:2303.00408, 2023.

https://doi.org/10.48550/arXiv.2303.00408

T.-E. Lin and H. Xu, “Deep Unknown Intent Detection with Margin Loss,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. https://aclanthology.org/P19-1548.pdf

L. Shu, H. Xu and B. Liu, “DOC: Deep Open Classification of Text Documents,” in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017. https://aclanthology.org/D17-1314.pdf

H. Zhang, H. Xu and T. E. Lin, “Deep Open Intent Classification with Adaptive Decision Boundary,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2021, Vol. 35, No. 16, pp. 14374-14382. https://doi.org/10.1609/aaai.v35i16.17690

X. Liu, Y. Lochman and C. Zach, “GEN: Pushing the Limits of Softmax-Based Out-of-Distribution Detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 23946-23955.

C. T. Hemphill, J. J. Godfrey and G. R. Doddington, “The ATIS Spoken Language Systems Pilot Corpus,” in Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990, 1990. https://aclanthology.org/H90-1021.pdf

A. Coucke et al., “Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces,” arXiv preprint arXiv:1805.10190, 2018. https://doi.org/10.48550/arXiv.1805.10190