|File size:||535 KB|
Editor's review LexstwebSpider
You can index 10 billion pages by using web crawler jointly with Search engine builder -lexst-SEA. It supports Multi threads and distributed free for both internet and interanet. It allows you to control the spider that is crawls your pages as well as you can save the pages locally or sent to a search-engine applicant.