21)
Easy Web Leech 1.70
This easy to use program scans and downloads content from multipe web pages. You can specify any file extension(s) that you want to search for. 22)
Robot-Manager 3.1
Manage and track visits by robots, spiders and crawlers to your website. Identify which spiders visited, when they visited, and what pages they indexed. Direct spiders to your most important pages, and exclude them from non-public pages. Easy to use 23)
Web Bot Programming Library 1.0.0
Crawl the web with this free ActiveX web bot component. Advanced features include caching, "avoid" patterns, and robots.txt compliance. 24)
Mobile-Hunt 1.2.4.0
Mobile-Hunt is a professional tool for searching new or used cars in the most used automobile portals from Germany and Europe. 25)
Subject Search Spider 5.01
A personal information retrieval intelligent Web engine that automates searching for information on the Internet. Its reports include both links ranked by relevancy and text extracts from the pages that are found by a powerful built-in search engine. 26)
Subject Search Siter 5.0
Your personal intelligent Web agent which investigates Web sites looking for your search phrase in 37 languages and finds information that is buried in it. SSSiter provides you with the report with a ranked list of page links and text extracts. 27)
DRKSpider 2.23
DRKSpider will navigate a web site from a given URL, checking internal (and external) links, style sheets, images and other files. 28)
SoftPerfect Network Search Engine 1.1
A self-contained index-based network search engine that indexes all shared files in your local area network and provides users with the ability to locate necessary files within a few seconds. Built-in web-server and database ensure easy deployment.