Been working and searching on this for about a week. I have two image objects: a mask (I have tried three of these, half back/half transparent, half white/half transparent and half white/half black), and an object (a heart in this case) that has transparent components (png). I also have a background that is made up of lots of Sprites (quads moving through 3d space). I'm handling all the depth management on my own.
I don't really understand how these blending algorithms work, so I created an app where you can cycle through all the combinations. Here is the code I'm using:
Code:
glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
//render all background objects
[self setBlend:before1 dstNum:before2 doTrace:NO tf:nil];
//eg glBlendFunc(GL_ZERO,GL_ZERO);
//render mask image
[self setBlend:after1 dstNum:after2 doTrace:NO tf:nil];
//render masked image
//render the rest of the scene
There are 11 parameters that I know of:
GL_ZERO (0, 0, 0, 0)
GL_ONE (1, 1, 1, 1)
GL_SRC_COLOR (Rs/kR, Gs/kG, Bs/kB, As/kA)
GL_ONE_MINUS_SRC_COLOR (1, 1, 1, 1) - (Rs/kR, Gs/kG, Bs/kB, As/kA )
GL_DST_COLOR (Rd/kR, Gd/kG, Bd/kB, Ad/kA)
GL_ONE_MINUS_DST_COLOR (1, 1, 1, 1) - (Rd/kR, Gd/kG, Bd/kB, Ad/kA )
GL_SRC_ALPHA (As/kA, As/kA, As/kA, As/kA)
GL_ONE_MINUS_SRC_ALPHA (1, 1, 1, 1) - (As/kA, As/kA, As/kA, As/kA )
GL_DST_ALPHA (Ad/kA, Ad/kA, Ad/kA, Ad/kA)
GL_ONE_MINUS_DST_ALPHA (1, 1, 1, 1) - (Ad/kA, Ad/kA, Ad/kA, Ad/kA )
GL_SRC_ALPHA_SATURATE (i, i, i, 1)
The first test I did, I had the first 10 on this list and went through all 10,000 combinations manually (with the black/transparent mask). I did not find what I needed there. So then I found the GL_SRC_ALPHA_SATURATE. With that, there are 14,641 combinations. I have not gone through all of those yet. In this second version, I put in all three mask types. It is a nice application, but I'm still not finding what I need. Here are the closest ones:

glBlendFunc(GL_ONE_MINUS_DST_COLOR, GL_ONE_MINUS_SRC_COLOR);
glBlendFunc(GL_SRC_ALPHA_SATURATE, GL_ONE);

glBlendFunc(GL_ONE_MINUS_DST_COLOR, GL_ONE_MINUS_SRC_COLOR);
glBlendFunc(GL_SRC_ALPHA_SATURATE, GL_DST_ALPHA);
I obviously need the shape of the first one, but the color of the second one. You know, just a plain old mask. I get really turned around thinking about all these combinations. And of course there are lots of duplicates. It would take the better part of a day to go through all the combinations with all 11 parameters. And I'm starting to lose hope that I'll find one that works. I've researched other methods, but haven't found anything else that comes this close.
Any ideas?