This project I post today really has not much of a practical or functional value to anyone. (alto I think they are pretty cool to web designers and developers. I am sure there will be detractors out there.) It is just to showcase the use of some ASP.NET objects and how easy it is to use them and also do some simple string manipulation.

This project is a HTML Content Parser. It gets a stream of HTML Content from a specified URL Web Page. Then it sets to go through whole stream extracted and picks out the HTML HyperLinks and Images and displays them in an HTML Table in a hyperlink format for users to click on directly to get there. This will be particularly useful for uses who are interested in some images on websites and finds it tedious to look through the view source of the pages to extract out the image sources of the page.

Please do check out the live version of this project from my website. Please click here to get there now. Please feel free to post any comments or criticisms on my project and articles.

Lets make use of MS.NET's more intuitive OOP features to separate encapsulate and group different functions into classes and assemblies for easier maintanance.

This code here goes into a Class called
HTMLContentParser.vb
'///////////////////////////
Imports System.IO
Imports System.Net
Imports System
Imports System.Text
Imports System.Text.RegularExpressions
Public Class HTMLContentParser
Function Return_HTMLContent(ByVal sURL As String)
Dim sStream As Stream
Dim URLReq As HttpWebRequest
Dim URLRes As HttpWebResponse
Try
URLReq = WebRequest.Create(sURL)
URLRes = URLReq.GetResponse()
sStream = URLRes.GetResponseStream()
Return New StreamReader(sStream).ReadToEnd()
Catch ex As Exception
Return ex.Message
End Try
End Function
Function ParseHTMLLinks(ByVal sHTMLContent As String, ByVal sURL As String) As ArrayList
Dim rRegEx As Regex
Dim mMatch As Match
Dim aMatch As New ArrayList()
rRegEx = New Regex("a.*href\s*=\s*(?:""(?<1>[^""]*)""|(?<1>\S+))", _ RegexOptions.IgnoreCase Or RegexOptions.Compiled)
mMatch = rRegEx.Match(sHTMLContent)
While mMatch.Success
Dim sMatch As String
sMatch = ProcessURL(mMatch.Groups(1).ToString, sURL)
aMatch.Add(sMatch)
mMatch = mMatch.NextMatch()
End While
Return aMatch
End Function
Function ParseHTMLImages(ByVal sHTMLContent As String, ByVal sURL As String) As ArrayList
Dim rRegEx As Regex
Dim mMatch As Match
Dim aMatch As New ArrayList()
rRegEx = New Regex("img.*src\s*=\s*(?:""(?<1>[^""]*)""|(?<1>\S+))", _ RegexOptions.IgnoreCase Or RegexOptions.Compiled)
mMatch = rRegEx.Match(sHTMLContent)
While mMatch.Success
Dim sMatch As String
sMatch = ProcessURL(mMatch.Groups(1).ToString, sURL)
aMatch.Add(sMatch)
mMatch = mMatch.NextMatch()
End While
Return aMatch
End Function
Private Function ProcessURL(ByVal sInput As String, ByVal sURL As String)
'Find out if the sURL has a "/" after the Domain Name 'If not, give a "/" at the end 'First, check out for any slash after the 'Double Dashes of the http:// 'If there is NO slash, then end the sURL string with a SLASH If InStr(8, sURL, "/") = 0 Then
sURL += "/"
End If
'FILTERING
'Filter down to the Domain Name Directory from the Right
Dim iCount As Integer
For iCount = sURL.Length To 1 Step -1
If Mid(sURL, iCount, 1) = "/" Then
sURL = Left(sURL, iCount)
Exit For
End If
Next
'Filter out the ">" from the Left
For iCount = 1 To sInput.Length
If Mid(sInput, iCount, 4) = ">" Then
sInput = Left(sInput, iCount - 1) 'Stop and Take the Char before
Exit For
End If
Next
'Filter out unnecessary Characters
sInput = sInput.Replace("<", Chr(39))
sInput = sInput.Replace(">", Chr(39))
sInput = sInput.Replace(""", "")
sInput = sInput.Replace("'", "")
If (sInput.IndexOf("http://") < 0) Then
If (Not (sInput.StartsWith("/")) And Not (sURL.EndsWith("/"))) Then
Return sURL & "/" & sInput
Else
If (sInput.StartsWith("/")) And (sURL.EndsWith("/")) Then
Return sURL.Substring(0, sURL.Length - 1) + sInput
Else
Return sURL + sInput
End If
End If
Else
Return sInput
End If
End Function
End Class
'///////////////////////////////////////////////////////

The Function getHTMLContent requires a URL parameter input in a string format. From there we use the HTTPWebRequest and HTTPWebResponse objects to send a request to the specified URL and get their HTML Content as a Response. Note the structured error handling implemented here. This structured error handling is explained in a different topic altogether. The returned value should be placed and displayed in a HTML TextBox for retrieval purposes later.

The ParseHTMLLinks and Images Functions make use of a Regex object that should be very familiar to Java and C# Developers and would look alien to VB Developers. They are actually a pattern matching object and can be used together with the Match Object. These are all objects of the System.Text.RegularExpressions Namespaces and therefore MUST be imported and declared into the VB.NET class. What they do is essentially a Regex pattern match into the Match object with the HTML Content (retrieved from an earlier HTML TextBox we use to display the retrieved HTML Content) as the source. As and when it finds the matched pattern specified by Regex, it returns the string containing the pattern, process it with ProcessURL Function and then adds it to an ArrayList. The ArrayList class is essentially the Collection class of the classic VB. It has the ability to add and remove from the collection which is far more intuitive and easier to use than the array class. Both the ParseHTMLLinks and Images return an arrayList of Links and Images.

The ProcessURL Function here essentially uses some very intrinsic VB functions and some new VB.NET ones (of which I am a developer of for years and therefore am familiar with). I also realized that some detractors out there will propose the use of the stringBuilder class as an immutable class to manipulate strings in this function. What the stringBuilder class differs from the String class is that the stringBuilder class is immutable which means it doesnt create a new instance of itself any time it is referred to. It is therefore more efficient on the machine's resources. The String class creates a new instance of itself whenever it is assigned an expression. and you can imagine the strain on resources when it the same string is manipulated 10 times, it will create 10 new instances of itself. Hardly efficient. I use the string class here because although its much more inefficient, its much more familiar to the VB Developers who are on transition to VB.NET and this topic here, more or less, focus on the ASP.NET HTTPWebRequest and HTTPWebResponse objects. I will save the Stringbuilder class topic for later articles. But I am sure other developers and authors here will and already have explained the stringBuilder class aready.

This code here goes to an ASP.NET ASPX page '////////////////////////////////////////////////////////////////////////////// Private objParser As HTMLContentParser Private Sub cmdGetHTML_ServerClick(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles cmdGetHTML.ServerClick
Dim sURL As String = "http://" & txtURL.Value
txtHTMLContent.EnableViewState = False
txtHTMLContent.Value = objParser.Return_HTMLContent(sURL)
End Sub
Private Sub cmdParse_ServerClick(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles cmdParse.ServerClick
Call PopulatetblParsedContent()
End Sub
Private Sub PopulatetblParsedContent() 'Populate Links Table
Dim sURL As String = "http://" & txtURL.Value
Dim myAnchor As HtmlAnchor
Dim intRows As Integer
Dim intRowCount As Integer
Dim objRow As HtmlTableRow
Dim objCell As HtmlTableCell
Dim sLinks As String
Dim sImage As String
Dim lstLinks As ArrayList = objParser.ParseHTMLLinks(txtHTMLContent.Value, sURL)
Dim lstImages As ArrayList = objParser.ParseHTMLImages(txtHTMLContent.Value, sURL)
tblParsedContent = Me.tblParsedContent
tblParsedContent.EnableViewState = False
For Each sLinks In lstLinks
objRow = New HtmlTableRow()
objCell = New HtmlTableCell()
myAnchor = New HtmlAnchor()
myAnchor.Target = "_blank"
myAnchor.InnerText = "Link: " & sLinks.ToString
myAnchor.HRef = sLinks.ToString
objCell.NoWrap = False
objCell.Controls.Add(myAnchor)
objRow.Cells.Add(objCell)
tblParsedContent.Rows.Add(objRow)
Next
For Each sImage In lstImages
objRow = New HtmlTableRow()
objCell = New HtmlTableCell()
myAnchor = New HtmlAnchor()
myAnchor.Target = "_blank"
myAnchor.InnerText = "Img: " & sImage.ToString
myAnchor.HRef = sImage.ToString
objCell.NoWrap = False
objCell.Controls.Add(myAnchor)
objRow.Cells.Add(objCell)
tblParsedContent.Rows.Add(objRow)
Next
End Sub
'/////////////////////////////////////////////////////////////////////////////////

We then go now to focus on how to extract the HTML Content and parse them. Design your ASPX page and have
1) A HTMLTextbox for users to specify the URL for processing
2) A HTMLButton called cmdGetHTML with a ServerClick event handler to handle to Click event. The event will trigger a routine that uses the HTMLContentParser class that we had coded earlier and use the getHTMLContent function to return a string of HTMLContent for display into a txtHTMLContent HTMLTextBox.
3) A HTMLTextBox called txtHTMLContent to hold the returned HTML Content
4) A HTMLButton called cmdParse with a ServerClick event handler that calls the PopulatetblParseContent
5) A HTMLTable called tblParsedContent

Personally, I think the HTMLTable server control is amazing. I have used this example here to show its intuitiveness to add cells and rows to it. Again, my detractors out there may question the routine to populate the tables. I say that this is just an article to explain one of the ways to populate a HTMLTable. It is very intuitive and I am sure most developers out there can just understand it without much explanation. It makes use of the HTMLTableRow and HTMLTableCell to add cells into rows and rows into the HTMLTable. Note the use of the For each ... in the ArrayList Collection to extract each link and image, assign them to another server control HTMLAnchor, add this HTMLAnchor to a HTMLTableCell, add the HTMLTableCell to a HTMLTableRow and finally add the HTMLTableRow to a HTMLTable. Very intuitive to program and code !

In the world of software development, the HolyGrail is seldom achieved as there is No One Right Way to do things, however there are Many Wrong Ways. This article here is more or less a tutorial on certain ASP.NET Objects and the intuitiveness of the programmatically of the web server controls. Feel free to modify them at your own pace and leisure to suite your learning curve transition from VB to VB.NET/ASP.NET. Use the StringBuilder class instead of the StringClass in the ProcessURL Function of the HTMLContentParser.vb class and after you are familiar with the program structure of the HTMLTable, use the DataSource and DataBind Techniques of the HTMLTable, the ASPDataList and the ASPDataGrid Server Controls. There is only one way to learn properly and that is from SCRATCH. In that sense, you can fully appreciate why and how you do things. After all, aint the whole world of MS.NET developed from SCRATCH which is much better than patches and add-ons to the imperfections of yesterday.