WEB CONTENT MINING USING GRAPH THEORETICAL MODEL AND GENETIC ALGORITHM

Abstract

The World Wide Web (www) is arguably the largest repository of data and continues to expand in size and complexity daily. The growth in size, however poses a lot of challenges to web users. Retrieving required web pages and information efficiently and effectively has become a herculean task for web users due to information overload and limitations of some information retrieval tools. Web mining adopts data mining techniques to automatically discover and retrieve information from web documents and services. Web content mining gathers, categorizes, organizes and presents accurate information available online to the user of the web. The research work designed a technique for performing data mining on web documents which utilize graph representations of document content and it addressed the problems of initialization, convergence to local minimal and failure to handle large datasets which are the limitations of reviewed works. Genetic algorithm is adapted to work with the graph representation of the web documents while the graph serves as data items to the genetic algorithm. The design of the system is in three phases namely: contents extraction, preprocessing and database of mined data. Maximum common sub graph (MCS) was used to calculate the distance between clusters. The research was conducted using a Computer System with 2GHz processor and 1GB RAM running Window 7 operating system, web scraper (import.io), PHP 6 and MySQL5 as the database.