D3D Vertex Shader 1.1 Glitch

[moeller]

I have added simple vertex shader 1.1 support to my direct3d implementation of my render system. My goal is to add shadowing with shadow volume and therefor vertex shaders could be handy. My framework is a bit complicated, using C# on .Net or Mono, Linux or Windows, and OpenGL or Direct3D9. But everything works nicely so far except for the Direct3D vertex shader part.

The general implementation seems to work ok. I read the shader code in Cg, compile it to vs_1_1 and load it on the device. Connect a vertex buffer and index buffer together with a vertex declaration and set the shader constants. I got a few sample shaders working 100% correct.

Now when it comes to the shadow volume extrusion shader I get some glitches. If I render the scene with the exact same configuration I get different looking frames at times. A correct one, and at times a wrong one. If I reduce the frame rate with a delay in my main loop I only get correct frames. If I use software vertex processing I only get correct frames. But using hardware vertex processing on my Geforce4 Ti 4200 I can get two different frames from the same render instructions.

What I debuged so far (in many sleepless nights) is that the visible glitches (shadows where no shadows should be, different renderings of the same data) are caused by the "slt" instruction. If the float x I compare (x < 0) is near 0, the shader decides not always the same, even if it is the same data I provide. I could reproduce this problem with different shader codes.

The fact that I get two different frames from the same data looked to me at first as if it is a problem with wrong data (index buffer) or not initialized data. But I have triple checked all now. So I really do not have any idea what might be wrong, beside some difference how the shader is executed.

Btw. compiling the same Cg code and using the same data buffers on my OpenGl implementation works 100% correct as well. So only the direct3d hardware case causes problems.

I can't believe that there are "different" vertex shader units on my card, processing the data in a different way. Or that directx processes half in software and half in hardware, while they are not 100% identical.

Anyone has ever had a similar problem? Or anyone got an idea how I could track down the problem.

Thank's for any ideas or comments.

Michael

[Reedbeta]

Try using an epsilon value, like instead of comparing x < 0, compare x < 0.001 (or -0.001, whichever is appropriate for the application). This is a common problem with floating point numbers although I don't know why you would sometimes get different results for the same input (are you sure it's exactly the same input, bit-for-bit?)

[moeller]

with a small epsilon I guess I could make the problem invisible on the current frame but not actually solve it. And basically my problem is not that the content of the screen is wrong, but that it is not the same each frame.

that I get different frames can have two reasons

A) the data beeing processed for the frame is not exactly the same each frame
B) the way the data is processed is not the same each frame

I can't really believe that (B) is my problem, because that would be a bug on the graphics hardware or drivers. If no one has ever heard about a problem like this than it's very very unlikely.

The problem about (A) is that the data I provide to the pipline is 100% constant. I use only buffer wich I do not modifiy during the render loop. All transformation matrices are constant as well. So the only real source of data beeing not 100% identical each frame is some not initialized or undefined data, where just random values enter the processing.

But there are not many places I can think of where I might introduce some undefined data. And the frames look basically correct beside the frames with glitches. At times all data I put into the pipeline gets to the right place and gets rendered correct.

Maybe I will try to reduce the problem to some very simple sample. But that is quite difficult, because already a small delay in the main render loop as well as switching to software vertex processing removes the problem.

Michael

[moeller]

after too many hours of endless searching, without finding a bug in my code, I switched back to some (good old?) version 53.3 nvidia driver I have been using for years (almost). An the with the old driver the problem is solved. I had installed the version 78.01 nvidia driver to use the NVPerfHUD. But since the NVPerfHUD isn't of much use to me atm, and the version 78.01 nvidia driver seems to cause the glitch I will stay with the old driver.

Michael