OpenGL Color Keying

An old proven DirectDraw API has a good feature, useful for drawing sprites. It could blit graphics using specified color, that becomes transparent. I wonder if OpenGL has the same feature?

The most obvious way is to use glBlendFunc and specify pixel alpha value. But my goal is to add video texture support to an existing OpenGL framework. While my library functions do video loading they don't change alpha channels of the pixels. Scanning the entire image manually and setting transparency for every individual pixels could be time consuming because it's a video and frames change often.

Some Color Keying technique would be most useful. Does anybody know, how to switch it on in OpenGL?

Thanks in Advance

SDL does it btw

sure... just enable alpha testing and set the alpha value of the invisble pixels to 0

or do it in a fragmen program... ok... overkill but it would work

Originally Posted by Ed Mack

SDL does it btw

[snapback]9433[/snapback]

Interesting...and how fast?

Originally Posted by anubis

sure... just enable alpha testing and set the alpha value of the invisble pixels to 0

[snapback]9434[/snapback]

Invisible pixels are determined by transparent color in a video stream. Let's assume that every poison-green area (RGB[0,255,0]) will be transparent. I have to change alpha value every time the frame changes. Because new frame most likely has another configuration of poison-green ares. If it were DirectDraw with ColorKey, my video would take care of transparency itself.

...ok...overkill but what is "fragmen" program?

Originally Posted by Decibit

Scanning the entire image manually and setting transparency for every individual pixels could be time consuming because it's a video and frames change often.

I shouldn't think the performance hit would be too bad, but if it is then you could always do this in advance and save the modifed video with alpha values.

Just a thought, I don't know how you do colour keying although I'm sure it can be done.

baldurk : i think the only way is to use the alpha channel or to filter out the pixels in the shader

decibit : i meant to say fragment programm, which you might know under the name pixel shader

A lot of videos have already been prepared for the project and saved in Windows AVI format. An average size of a single file is 20 Mbs. Moreover, the framework should seamlessly switch between several videos. I use memory mapped streaming files. Trying to save modified video beforehand would take up to 150 MBs more memory.
Is there a way to store alpha values in the video (AVI) file? That would be an excellent solution.

i'm curious... what are you doing that you need transparent areas in a video ???

anyway, if you can, and that i don't know, store an alpha channel in the video that would be the best option.
if not it should be easy to add the alpha channel at runtime. that might depend on what else your application has to do at the same time but it should be no problem in general

In an AVI file definetely no but i beleive it's possible with an MPEG since it recognizes 'objects'.

Originally Posted by anubis

i'm curious... what are you doing that you need transparent areas in a video ???

[snapback]9442[/snapback]

Real time video mixer with multiple overlays.

Originally Posted by anubis

...it should be easy to add the alpha channel at runtime. that might depend on what else your application has to do at the same time but it should be no problem in general

[snapback]9442[/snapback]

Currently I do color keying rather straightforward. Look at my code.

for(int i = 0; i<width*height; ++i)
  if(pixels[i]==colorkey)
    pixels[i].alpha=0xff;

This runs too slow on P3-800 with 5 video streams. Can this be optimized?

Originally Posted by Mihail121

In an AVI file definetely no but i beleive it's possible with an MPEG since it recognizes 'objects'.

[snapback]9443[/snapback]

Could you advise a quick way to implement MPEGs. Recoding all my AVI files to MPEGs could take a long time, but that really doesn't matter, because it would be done once.

PS. btw. what editor could save alpha chanel to video?

There is NO quick way to implement MPEGs or AVIs 'cause there's tones of codecs/decodes, etc. involved, sorry to dissapoint you. Plus not sure if any video editor will save alpha channels cause it adds ~25% more to the size.

The best solution is probably to think of something on your own in this case althought it will cost some time...

if you want to require the user to have a 3d card you could speed it up (i take it you don't do anything besides video editing) by a big margin by doing the color keying on the card. this is fairly easy to do using a pixel shader since you would just have to check for the desired transparent color and add the aplha channel as the output of the pixel shader (fragment programm). since you might not want to get down with the shader assembler language you should take a look at nvidia's cg high level shader language. doing it this way will give you sweet performance, since such an operation comes nearly for free on the 3d card. also this way you could keep your AVIs

Originally Posted by anubis

if you want to require the user to have a 3d card you could speed it up (i take it you don't do anything besides video editing) by a big margin by doing the color keying on the card. this is fairly easy to do using a pixel shader since you would just have to check for the desired transparent color and add the aplha channel as the output of the pixel shader (fragment programm). since you might not want to get down with the shader assembler language you should take a look at nvidia's cg high level shader language. doing it this way will give you sweet performance, since such an operation comes nearly for free on the 3d card. also this way you could keep your AVIs

[snapback]9447[/snapback]

That requires a Geforce FX or higher though.

Could you filter & record the alpha channel to a b&w avi stream? Using one of the many media handling frameworks (eg Gstreamer) this shouldn't be more than a day or two's work.

Then at image time, you can do a two pass alpha blending, first drawing the b&w, then drawing the video with the alpha test to only fill white areas.. Although, for 5 overlayed videos that might be sticky..

i thought of that, too, mainly because that's what i did once for a texture. anyway, can't cg translate to lower forms of shaders ??? if so, you don't need a card that supports ARB_fragment_program

Originally Posted by Ed Mack

Could you filter & record the alpha channel to a b&w avi stream? Using one of the many media handling frameworks (eg Gstreamer) this shouldn't be more than a day or two's work.

Then at image time, you can do a two pass alpha blending, first drawing the b&w, then drawing the video with the alpha test to only fill white areas.. Although, for 5 overlayed videos that might be sticky..

[snapback]9451[/snapback]

Perhaps I'll do that, because the program should be run on GeForce 2 GTS - 64Mb, not on GeforceFX. Thanks for the idea.

I will still be looking for a method to do true color keying in OpenGL. But now I hope it is possible to release the project without it.

Originally Posted by anubis

i thought of that, too, mainly because that's what i did once for a texture. anyway, can't cg translate to lower forms of shaders ??? if so, you don't need a card that supports ARB_fragment_program

[snapback]9456[/snapback]

The only thing on lower hardware was ATI_fragment_shader and NV_fragment_shader, I believe. I would imagine Cg only converts to the latter.

Lower than that, and you have to use NV_register_combiner which I don't think Cg converts to, because it is so different.

you don't need a gffx. shader capabilities exist since gf1 and with register combiners you are actually able to do the colour keying if you want (but its not easy).

on the other hand, a radeon9550 se, or a gffx5200, is very cheap nowadays, and could handle the full shaders..