There's no arguing that the Internet lets us access amazing volumes of information on virtually any subject. However, if you're like us, you may have found it difficult to filter out unnecessary information from this enormous repository. Gathering specific facts can be time consuming, with data usually scattered across many sites. Search engines like Yahoo!, HotBot, and even Ask Jeeves, have attempted to fill this void, but have been only partially successful. A recent study found that search engines have indexed less than 55 percent of the Web. The same study predicted that this percentage would in fact continue to shrink as the number of new pages on the Internet grows.

In the future, people will probably turn to personal, automated search programs to find what they need. These Web-bots provide more targeted and thorough searches. In this article, we'll look at the Web-bot shown in Figure A, which lets you research any topic on the Internet. Then, we'll cover a few of the basics you'll need to create a Web-bot fit to rival Jeeves himself!

To boldly go where no Web-bot has gone before

Next, indicate how thorough a search you want the bot to conduct in the Search Depth text box. High numbers make for in-depth searches, but take longer to complete. Lower numbers are less thorough but complete much quicker. If you have a slow Internet connection and only a few minutes to run the Web-bot, consider entering a 2 or 3. If you have a fast Internet connection or have a lot of time (for example, you may be running the program over-night), enter a higher number like 9 or 10. The Web-bot doesn't care how high you make this number. As you can see in Figure A, we entered 3 for our search depth.

Full speed ahead, botty

Finally, tell the Web-bot where to start. Search engines can be good launching points, so if you want to start with one of these, choose the corresponding option button. If you want to start at a custom URL, click the Custom URL option button, and then enter the URL in the text box.

Now that we've set the Web-bot's options, we're ready to launch it. To do so, click Start Search, and then click Yes when the program asks if you're conducting a new search. That done, the Web-bot races ahead at warp speed, looking for the information you requested. (OK, that's the last of the Star Trek references, promise!)

At any time, if you wish to take a closer look at a URL, just click the Pause button. Then, find a URL in the treeview and right-click on it. Doing so transports the page into the browser on the right side. The program also logs email addresses, as well as the URLs, in a local Access 97 database for your later perusal. We called this database WebAgent.mdb.

The anatomy of a Web-bot

In addition to the controls visible at runtime, Figure B shows the few controls not visible. As you can see, we've placed an ImageList and Inet control on the form. Also, the larger box at the very bottom is an RTFTextbox control. Finally, note that in the main body of the Web-bot, we used a Treeview to list the Web sites and email addresses, and a Browser control to display the pages. Now, let's take a look at the more complex features.

Figure B: We'll import HTML pages into the RTFTextbox control, and then use its Find method to search the HTML for the selected topic.

Navigating to a Web page

Listing A: Navigating to a URL

Function mNavigateToURL(ByRef rIntInternetControl _

As Inet, ByRef rbrwsBrowserControl As WebBrowser, _

End Function

Displaying Web pages

rbrwsBrowserControl.Navigate vstrURL

Analyzing a page

• The email addresses located on the page.
• The links that exit the page, so the Web-bot can continue its journey.

Listing B: The mExtractHTML function

Function mExtractHTML(ByVal vstrStartDelimiter _

As String, ByVal vstrEndDelimiter As String, _

End Function

The functions mcolGetAllUrlsInPage and mcolExtractAllEmailAddressesOnPage build on the previous function and return the links or email addresses (respectively) back to the calling routine via a collection. These functions are smart enough to remove links and email addresses that might appear valid to a less sophisticated Web-bot, but really wouldn't be applicable. For example, most email addresses to mailing lists are of the format subscribe@somedomain.com. The routine weeds these out. Other examples of screened email addresses include sales@somedomain.com and support@somedomain.com.

Avoiding infinite loops

Second, before going to any new URL, it determines if it already visited the page. To do so, it calls mblnAlreadyVisiting, shown in Listing C. If the database contains the URL, then the Web-bot skips the page, thus short-circuiting the infinite loop.

Listing C: Code to detect duplicate URL

Function mblnAlreadyVisiting(ByVal vstrURL As String)

Dim objConnection As ADODB.Connection

End Function

Resuming operation after stopping

To allow the user to resume his search, the Web-bot uses the same URL log that protects against infinite loops to keep track of the currently visited URL. If the application gets prematurely shut down, it will simply pick up where it left off.

Conclusion