Focused Crawler Strategy Based on Multi-Objective Ant Colony Algorithm
摘要
The traditional search engine based on keyword matching retrieval often fail to ensure the completeness and accuracy of the scraped data,but using the focused crawler method based on semantic retrieval tend to deviate from the focuse and fall into local optimum.To solve the problem,this paper proposes a focused crawler method based on multi-objective ant colony optimization algorithm.The method constructs a focused crawler domain ontology and focused vector.Then whether the link is relevant to the focuse is determined based on the anchor text relevance of the link,the focused relevance of the Web page where the link is located and the focused relevance of the page pointed to by the link.The multi-objective optimization model for the focused relevance degree of the link is established.The ant colony algorithm based on multi-objective optimization is introduced into the link selection proess of the focused crawler,and the non-dominated sorting and the Nearest and Farthest Candidate Solution(NFCS) is adopted to select the Pareto optimal link in order to guide the search direction of the focused crawler and improve the global search performance.Experimental results show that compared with FCSA,WSE and other traditional focused crawler methods,the proposed method improves the completeness of scraped data and can capture the Web pages with high relevance to the focuse more quickly.