Now that we have the codecs and VirtualDub set up all nicely, it's time for the meat and potatoes of this solution: The capturing. Open up VirtualDub, and choose file, Capture AVI. This will change VirtualDub's layout entirely, but don't worry. Everything's under control.
First and foremost, everytime you come into Capture AVI, you'll need to choose File -> Set Capture File. This tells Virtualdub where to save the AVI we're going to capture. Next, is the Device menu. One of the default options with VirtualDub is ScreenCapture, and what an option it is, too.
The moment you check ScreenCapture as the device type, you will see an infinite feedback of your screen inside VirtualDub, like a hall of mirrors. If that's not the case, make sure that Video -> Overlay is checked, and get ready for some fun.
Head down to Video -> Set Custom Format, and set your size. I recommend checking the 'Use Custom Size' box at the bottom. I set mine to 1280 x 1024, because that's my monitor's resolution. Leave the Data Format at whatever it defaults to, and hit OK.
Next, let's go to Video -> Source, not to be confused with Video -> Video Source. Make sure 'Use Fixed Offset' is selected, and the X and Y coordinates are both 0. For Panning Area, select 'Entire Desktop'. This is the exploit part.
If you're using XP, checking the "Use OpenGL Hardware Acceleration" option will
dramatically increase the framerate you can record at. With a 7750 Dual Core Athlon, and a GeForce GTX 260, I can record at my monitor's maximum resolution at a full 30 fps in any game, including Crysis. Mileage may vary a bit, but this is the fastest possible recording setup on XP.
If you are using Windows Vista or 7, this option will give you nothing but a black screen recorded. This is because Microsoft 'fixed' the OpenGL exploit in the DirectX architecture with DirectX 10 and up. Shame on you Microsoft.
We're almost ready to record (these settings will save themselves next time, so you won't have to do as much work.), but first we need to choose Video -> Compression, and select HuffYUV. For those of you that do not have a powerful enough PC to encode to HuffYUV in real time, I might recommend PicVideo's MJPEG codec. It is not free, but it performs very quickly without too much quality loss.
Second to last, the audio capture settings. Choose Audio -> Audio Input, and select either 'What U Hear' 'Loopback' 'Master Record' or 'Stereo Mix' from the options available to you. If none of these options are available,
Citrinate's thread explains how to use
Virtual Audio Cable quite well, and it's more or less the same setup here.
Last but not least, head to Capture -> Settings to set your Framerate (I recommend 30 FPS), and Capture -> Timing to finish up your setup. Once again, all these settings (Except the Capture File) will be saved automatically for future use, if VirtualDub is closed normally.
Check the box marked 'Drop Frames when...' , Uncheck 'Insert Null Frames', and set Resync mode to 'Synch video to audio by adjusting video timing. Set Audio Latency to automatic, with 1 block, and hit OK.
Presto! You're done with the setup! That only took forever, right? Don't worry, you won't have to do any of that setup again.