VB.Net versus VB6 performance tests
Author: Jack Hoxley
Written: 9th July 2002
Contact: [EMail]
Download: Complete Archive (54kb)

Contents of this lesson
1. Introduction
2. Maths Performance
3. Memory Performance
4. Further Performance Tests

1. Introduction

As many of you will be aware already - in February 2002 I wrote a review of Visual Studio .Net for this website (you can read it here). In this review I discussed the performance of the new compiler, and came to the following conclusion:

"I will be happy to hold up my hand and admit my fault should my values be proven to be wrong - I still find it surprising that I got those results, and would quite like to be proven wrong! .Net isn't universally slower than the previous versions, as I have found a few cases where it is a bit faster, but that doesn't hold much in it."

At the time of writing that review I'd only been using the .Net languages for a couple of weeks - not really enough time to become particularly skilled with it. However, in the months since then I've done a little more work and a little more research (thanks to Patrice Scribe for getting the ball rolling lately!) and found that there were flaws in the code I used and that the times quoted were therefore misleading.

The main problem with my previous quoted times was that I was using VB.Net to program in a VB6 way - I've used VB6 for 4 years now, and because of obvious parallels I continued writing my code in this way. This is not a good strategy for using .Net; to really get the most (performance and features) out of the .Net frameworks you can't always program in the same way as you would have done before.

So, it's come to this. I decided to write this article to set the record straight - I got it wrong first time around, and hopefully I won't get it wrong this time around. There is no way to conclusively prove and/or test all aspects of performance, but I shall instead cover a few examples that I think are representative of multimedia programming.

2. Maths Performance

Maths is critical to how a multimedia program will perform, the reliance on maths is huge - and therefore even minor speed differences in this area can have a significant impact over all. This section is divided into two sections, the first analyzing the core maths functions (those built into the standard libraries), and the second being more complex algorithms using these functions / performing more complex functions.

2.1 Core Math Functions:
The core math functions to be tested are shown in the following listing:

Each of these functions was taken in turn, and put into a simple framework looking like that in the following listing. Each test was iterated 1 million times, and timed using the high performance counter. This overall time was then divided by 1 million to get an average time for each call - in the case of maths functions these are quoted in nanoseconds. Input variables for the maths functions were the loop variables, using a constant would have given the compiler too much opportunity to optimize away the entire test, and using random numbers proved to be misleading (the times represented the performance of the random number generator AND the math function).

Both programs were then compiled using the highest level of optimization possible (optimize for speed), and the system was restarted (to eliminate any overheads). Each program was then run 10 times to generate a list of times for each function. For reference, the test system used was a 700mhz AMD Athlon Thunderbird with 288mb PC100 RAM and Windows XP Professional.

The times recorded (and averaged over the 10 tests) are shown in the following table:

As you can see straight away its a fairly mixed set of results, I've highlighted the tests in red where .Net is slower, and in green where .Net is faster. Currently it's 4 against and 4 for - tiebreak. Three of the tests where .Net wins can be explained quite easily - they are the three function that had to be derived in VB6. This means that there is the function-calling overhead to consider, which will generate some slowdown. However, if you were to perform some high-level optimization of the VB6 code, you can change the times to look like this:

In doing some rather trivial optimizations on the code (making them inline, adding constants where applicable) you can reduce the times quite significantly (although Inverse Sine is an exception). However, whilst inverse cosine (for example) got 1.4x faster, it was not enough to catch up with VB.Net's initial performance value.

2.2 Applied Mathematics
This next section of maths related statistics covers applied maths - that is, helper functions / algorithms that are highly mathematically orientated. Anyone who's done any 3D graphics programming (Using Direct3D for example) will know that in every frame alone there can be a huge number of complicated equations and algorithms to process. These aren't necessarily all related to using the core maths functions shown above, but they will feature.

The reason for including this section is that while the core maths functions are regularly used, it's these real world examples that are more common, and any speed difference here is likely to have a far bigger impact on real-time performance than anything else.

Creating a 4x4 transformation matrix
Essential to all 3D graphics is the transformation matrix. This is typically constructed using a series of helper functions (exposed through the D3DX libraries in DirectX8). A while back (in this tutorial) I proposed a faster function for generating a complete transformation matrix. Basically, the source code takes in rotation, scaling and translation parameters and returns a properly generated 4x4 matrix. The source code is fairly boring, so I didn't bother inserting it in the main article text, but you can find it in the downloadable archive.

The times for creating a complete matrix:

The times are still biased in VB.Net's favor, but the speed difference is much less than that shown in earlier tests. However, the difference shown here of 51.3727 nanoseconds would translate into VB.Net being able to process the function 33,435 times more per-second than VB6 could. Small difference, big result.

Finding the inverse of a 4x4 transformation matrix
Inverse matrices aren't used as heavily as normal transformation matrices, but they are the only method for doing certain very useful techniques (such as 'picking' 3D objects based on a 2D mouse coordinate). Inverse Matrices by their very nature are quite slow; so this speed test is crucial.

As with the CreateMatrix() test, the source code is not presented here for clarity, you can find it in the downloadable archive. Thanks to Eric Coleman for letting me use his optimized inverse matrix function.

The times shown here are very much in VB6's favor. This particular function uses only *, /, + and - operations which would eliminate any of the performance issues with core maths functions shown above.

Normalizing a Vector
This is a rather trivial piece of code - but is often essential to many other vector-based calculations. This test represents the normalization of a 3D vector generated using random numbers in the interval [-5,+5] for each component (x,y,z). The list of vectors were generated separately so as not to introduce any additional variables into the final, timed, test. The second time represents an inline'd version of the code - so as to remove any function-calling overheads.

What is most odd about the tabulated results above is that VB.Net produces a slower time for the inline version of the code. It is traditionally assumed that inline functions (of a similar, small, size) will run faster - as is shown by VB6. Overall, VB.Net has produced the fastest possible time - 165ns is considerably faster than anything VB6 can offer.

Ray-Triangle Intersection Tests
This is a very useful test to perform - essentially this can be considered as a ray-tracing algorithm for triangulated/tessellated geometry. It can also be used quite effectively for collision detection between two 3D meshes. Ray-traced graphics often produce the best lighting/rendering but cannot as yet be used in real-time purposes - instead people often use ray-traced light maps in games.

The code involved in this algorithm is very complicated - and makes use of several nested-functions. It would appear though, that VB6 has the upper hand again.

2.3 Summary

This next table is a final summary of all the times collected for the math functions, this time including a percentage difference to compare the two compilers.

There are the final results - VB.Net wins on 5 counts, VB6 wins on 8 counts. Whilst that does make VB6 the "winner" in most senses, there isn't a huge amount in it - VB.Net is on average 1.034x slower (it performs at 96.7% of VB6's speed). As you'll find later on, you can get the source code used here for your own inspection should you not believe the data presented.

Whilst this is not the focus of this article, I've included a quick summary of the above data but for C++; in particular VC++6 and VC++.Net compilers.

Bare in mind that the C++ language is an international standard - therefore the source code used for each compiler was identical. The timing differences here really should be only down to the differences in compilers. Given these results it would imply that the VisualC++.Net compiler is capable of producing marginally faster code. The above results show, that on average, the .Net compiler generates code 1.33x faster.

3. Memory Performance

Memory allocation and processing.
Lets take an arbitrary example whereby a function must allocate a significant amount of memory and then perform a process on it. Similar in many respects to directly accessing the memory behind a surface or texture in Direct3D/Draw. The test code looks like this:

The times for the memory allocation and processing tests:

VB.Net's version of the code is 2.33x faster than VB6. This would indicate that VB.Net, the CLR and/or the .Net frameworks have a far superior memory manager.

In real-world terms, should you wish to use this process/technique to generate lighting in your games the two times listed above correspond to ~24fps and ~58fps (maximum possible). The former being VB6's time, which while interactive would prove to be quite jerky; the latter being VB.Net's which would be very smooth and pose no real problem to interactivity.

The next table shows the C++ compiler performance for the same code.

4. Further Performance Tests

The tests shown in this article can only represent a small percentage of real-world applications. For multimedia work, the best possible test would be to compare the performance of DirectX through .Net and VB6 - real frame rates and per-frame timing can then be extracted, and actually mean something.

However, this test is not going to happen any time soon. DirectX 9 (at time of writing) is far from being ready for release, but does include a managed-code core, such that it can be properly tested using the .Net languages. BUT, DirectX 9 will not support VB6, so it is impossible to directly compare VB6 and .Net's DirectX performance. The closest we can get is to compare DirectX 8 and DirectX 9 - but this adds additional variables into the system, the biggest one being that test results will reflect any difference in DirectX 9 and DirectX 8 and not the actual language differences.

Ignoring DirectX for a moment - the difference between both languages would be best judged by comparing a much larger program. One example that I would like to have done is a landscape ray-tracer. This type of algorithm can take upwards of an hour to complete, and relies extremely heavily on maths and memory manipulation. Unfortunately this is a rather complex program that would take quite a while to get working properly and comparably in both languages - something I didn't have time to do when writing this article. Maybe in the future.

As mentioned throughout this article, you can download the source code and examine it in your own spare time - generate results specific to you computer etc... The source code is available here, or from the top of the page.

Finally, if you think you have significantly sized code that you think makes a good comparison (regardless of the overall results) between VB.Net and VB6 please email me about it - Jack.Hoxley@DirectX4VB.Com , even if you don't want to share the code!