D3DXIntersect() Problem

[durban]

I'm simply trying to be able to pick an object with the mouse and change its render properties when its clicked.

Problem : The results aren't correct, the mesh seems to be moved over. When I click the mesh nothing happens, however, if I click in the region that D3DXIntersect "thinks" the mesh is, I get the proper result.

Anyone see a problem in this code? :

Pick() - returns true if the user clicked the mesh :


bool CPick::Pick( LPDIRECT3DDEVICE9 device, LPD3DXMESH mesh, D3DXMATRIX matProj, D3DXMATRIX matView, HWND hWnd )
{
D3DXVECTOR3 vecRay, vecDir, v;
D3DXMATRIX m;

D3DXMatrixInverse(&m, NULL, &matView);

POINT ptCursor;
GetCursorPos( &ptCursor );
ScreenToClient( hWnd, &ptCursor );

v.x = (((2.0f * ptCursor.x) / 400) - 1) / matProj._11;
v.y = (((2.0f * ptCursor.y) / 300) - 1) / matProj._22;
v.z = 1.0f;

vecRay.x = m._41;
vecRay.y = m._42;
vecRay.z = m._43;
vecDir.x = v.x*m._11 + v.y*m._21 + v.z*m._31;
vecDir.y = v.x*m._12 + v.y*m._22 + v.z*m._32;
vecDir.z = v.x*m._13 + v.y*m._23 + v.z*m._33;


BOOL Hit;
DWORD Face;
float u, w, Dist;

D3DXIntersect(mesh, &vecRay, &vecDir, &Hit, &Face, &u, &w, &Dist, NULL, NULL);

if( Hit )
return true;
else
return false;
}



Code Snippet from the Render() function where the call to Pick() is made :


if(drawMesh == TRUE )
{
sMesh* temp = m_CObj->GetStartPtr();
HWND hwnd;
if( hwnd = GetCapture())
{
do
{
if( m_CPick->Pick( _device, temp->_mesh, matProj, matView, hwnd))
{
temp->bHit = TRUE;
MessageBox(GetMainWndHandle(), "bHit=TRUE", "",MB_OK);
break;
}
temp = temp->Next;
}while( temp->Next != NULL );

temp = m_CObj->GetStartPtr();
}

do
{
if( temp->bHit == TRUE )
{
_device->SetMaterial(&temp->_materials[0]);
_device->SetRenderState(D3DRS_FILLMODE, D3DFILL_SOLID);
_device->SetRenderState(D3DRS_AMBIENT, 0x00FF00FF);
temp->_mesh->DrawSubset(0);
_device->SetRenderState(D3DRS_FILLMODE, D3DFILL_WIREFRAME);
_device->SetRenderState(D3DRS_AMBIENT, 0x00000000);
}
else
{
for( DWORD i=0; i<temp->_numMaterials; i++)
{
_device->SetMaterial(&temp->_materials[i]);
_device->SetTexture(0, temp->_textures[i]);

temp->_mesh->DrawSubset(i);
}
}
temp = temp->Next;
}while( temp->Next != NULL );

temp = m_CObj->GetStartPtr();
}



The World/View/Projection matrices were defined earlier in the Render() function*

I will appreciate any help/comments given, Thanks in advance

[bladder]

You are retrieving the first vector correctly through the projection matrix - though I'm assuming that your client area is indeed 400 by 300. Also, it so happens that even if you create a window that is 400 by 300, the actual client area is smaller becaue of the window borders and title bar, so if you havent taken that into account then the ray picking will be a little off. make sure your back buffer really is 400 by 300

Also, is your model vertices in world space or in model space? If you are storing them in model space, then you have to convert the ray into model space as well - to do that you'll need the WorldMatrix of your object, take the inverse, and multiply it with your ray direction and ray starting point, and use the coverted ray instead.

[durban]

The backbuffer is 400x300, and the function ScreenToClient() that I used should transfer the Client space accordingly.

It's still shifted after I transformed the rays with the inverse of the world matrix tho. I noticed that where D3DXIntersect thinks the mesh is, is actually a reflection of the mesh on the -y axis, so that's an improvement considering it was off in oblivion before. I'm using 4 different windows to show each perspective of the mesh with swap chains but that shouldn't matter since I'm passing the window's world/view/proj matrices to the function right? Here's the code I added:


D3DXMatrixInverse(&world, NULL, &matWorld);

D3DXVec3TransformCoord( &vecRay, &vecRay, &world );
D3DXVec3TransformCoord( &vecDir, &vecDir, &world );

[bladder]

I noticed that where D3DXIntersect thinks the mesh is, is actually a reflection of the mesh on the -y axis. I'm still stumped, any ideas?
20960


ah... change this:
v.y = (((2.0f * ptCursor.y) / 300) - 1) / matProj._22;

to this:
v.y = -(((2.0f * ptCursor.y) / 300) - 1) / matProj._22;

(notice the negative)

One more thing, you don't want to be using D3DXVec3TransformCoord for a direction vector, because then the result will be afftected by the translation component of the matrix, and you don't want to be translating the direction vector - just need to rotate it appropriately. So you need to use D3DXVec3TransformNormal instead.


Also, instead of getting the world matrix and inverting that, and transforming the ray to model space you can instead concatenate the world matrix with the view matrix (in that order) and then invert and calaulate the ray diration and position.

[durban]

That was it, works like a charm now. Still shifted down maybe 3 pixels or so but its still damn close. Thanks for the help bladder :D

[bladder]

That was it, works like a charm now. Still shifted down maybe 3 pixels or so but its still damn close. Thanks for the help bladder :D
20973


no problem.

As for the remaining error - Try changing your width and height values to see if that small error can get fixed. Instead of using 400 and 300, use the IDirect3DDevice9::GetViewport() function to get the viewport and use Viewport.Width and Viewport.Height instead.