Efficient and Secure Transformation of Log Files using ADLS GEN2 Architecture
International Journal of Science and Research (IJSR)

International Journal of Science and Research (IJSR)
Call for Papers | Fully Refereed | Open Access | Double Blind Peer Reviewed

ISSN: 2319-7064


Downloads: 2 | Views: 116 | Weekly Hits: ⮙1 | Monthly Hits: ⮙2

Research Paper | Computer Engineering | India | Volume 13 Issue 11, November 2024 | Popularity: 4.7 / 10


     

Efficient and Secure Transformation of Log Files using ADLS GEN2 Architecture

Sheik Syed Sulaiman M


Abstract: Web server log encompasses a document where all the activities transpiring on a web server are carefully recorded. It captures each request made to the server like request type, IP address of the device, what files where requested, date & time of request, name and location of requested file, file size etc. These files are stored in their raw form for a long time for analysis and taking high quality decisions which causes storage problem and security threat as data present in the log files are in raw format. This paper focuses on providing an economical, secure and high - performance solution for the storage of large amount of raw log files. The proposed system includes Azure Data Lake Storage Gen2 which allows large volumes of data to be stored in their raw form as well as they are subjected to transformation and advanced analysis processes without the need of a structured writing scheme. This paper mainly provides solution that is affordable and more accessible to perform web server log data ingestion, storage and transformation over Data Lake. This paper also proposes the use of Azure Trigger Function that transforms the log files into parquet files which reduces the storage space compared to their original size. A hierarchical data storage model has also been proposed for shared access to data over different layers in the Data Lake architecture, on top of which Data Lifecycle Management rules have been proposed for storage cost efficiency. The aim is to maintain this data in the long term to be used in future advanced analytics processes by cross - referencing with other organizational or external data which could bring important benefits.


Keywords: Cloud Data Lake, Azure Data Lake Storage Gen2 (ADLS GEN2), Data Lake architecture, Web Server log, Azure Trigger Function


Edition: Volume 13 Issue 11, November 2024


Pages: 1074 - 1079


DOI: https://www.doi.org/10.21275/SR241118115433


Please Disable the Pop-Up Blocker of Web Browser

Verification Code will appear in 2 Seconds ... Wait



Text copied to Clipboard!
Sheik Syed Sulaiman M, "Efficient and Secure Transformation of Log Files using ADLS GEN2 Architecture", International Journal of Science and Research (IJSR), Volume 13 Issue 11, November 2024, pp. 1074-1079, https://www.ijsr.net/getabstract.php?paperid=SR241118115433, DOI: https://www.doi.org/10.21275/SR241118115433

Top