Transforming Data Overload into Value: AI-Based Enhancements to the Information Gain Methodology
Abstract: Organizations keep far more data than they use, and low-value records raise storage cost and cause information overload. This paper asks how to decide automatically whether a new record belongs in the primary database or in an archive, without deleting data. We correct an earlier entropy-based information gain (IG) method that counted keywords, redefine IG as the Kullback?Leibler divergence between a record?s topic distribution and the corpus distribution, and add relevance, novelty, and supervised scoring with SHAP explanations. A simulation with 30 seeds of a synthetic corpus and 20 seeds of real PyPI package descriptions compared the methods at a 95% recall target, with paired significance tests. Keyword, TF-IDF, and BM25 scoring had to retain every synthetic record. Semantic IG with relevance reached F1 0.860 against 0.439 and archived 65% of records. On real text the gain was small: F1 0.418 against 0.397. A MinHash duplicate filter outperformed embedding novelty for near-exact copies, and neither handled heavily edited copies. The encoder is latent semantic analysis, so pretrained encoders remain untested.
Keywords: Information gain; Entropy; Data retention; Sentence embeddings; Explainable AI
How to Cite?: Tanushree Gupta, "Transforming Data Overload into Value: AI-Based Enhancements to the Information Gain Methodology", Volume 15 Issue 4, April 2026, International Journal of Science and Research (IJSR), Pages: 1966-1981, https://www.ijsr.net/getabstract.php?paperid=SR260422112400, DOI: https://dx.doi.org/10.21275/SR260422112400