Processing a Web Server Log in Apache Hadoop

Abstract

The project is intended to perform a Big Data task on the data set of a website named Udacity, which provides online education to the millions of people around the globe. The website runs a web forum for the students to have conversation regarding their curriculum. Students post various queries on the forum. The aim of the project is to monitor and analyze those queries and sentiments of the students to improve the demographics of the web forum. For this purpose the Udacity web server log is processed on Apache Hadoop and various insights from the data set has been explored. Such type of analysis and information which is obtained from the students records is very useful for the web forums to generate business profits and further the extracted information helps the forums to decide the direction for their future marketing strategies. It also helps in generating more viewership in order to generate the advertisement revenue from the forum. Similarly the Hadoop Map Reduce methodology can be used to build other systems like recommendation systems, item classification systems and fraud detection systems. Such type of systems have some similar characteristics i.e., huge amount of data and work can be parallelized. Therefore processing the data in Apache Hadoop using map reduce methodology can be really worthwhile.