Intelligent Data Processing and Automation in End-to-End Machine Learning Pipelines
Abstract: Data processing plays an essential role in the development and performance of machine learning and deep learning models. Traditional machine learning relies on separate, manually designed stages for data preprocessing, feature extraction, feature selection, and model optimization. Deep learning offers a more integrated approach by using neural networks to learn useful features directly from data. However, the quality, quantity, and complexity of training data continue to influence model performance. Data preprocessing, augmentation, and feature engineering remain necessary to address missing values, noisy data, class imbalance, and limited training samples. Feature engineering also helps improve model efficiency and interpretability, particularly in applications such as healthcare, fraud detection, and industrial fault diagnosis. Automated Machine Learning (AutoML) provides methods for combining these tasks into a unified process, reducing manual intervention in data preparation, model selection, and hyperparameter optimization. Neural Architecture Search further supports the automatic design of suitable network structures. The discussion examines the differences between traditional and deep learning pipelines, the continuing need for data processing, and the role of automation in developing efficient, adaptable, and reliable machine learning systems.
Keywords: Machine Learning, Deep Learning, Data Preprocessing, Feature Engineering, Automated Machine Learning
How to Cite?: Putty Srividya, Dr N. Subramanyam, "Intelligent Data Processing and Automation in End-to-End Machine Learning Pipelines", Volume 13 Issue 10, October 2024, International Journal of Science and Research (IJSR), Pages: 2076-2079, https://www.ijsr.net/getabstract.php?paperid=SR241012105719, DOI: https://dx.doi.org/10.21275/SR241012105719