Problem solved... The buffer used should be unsigned byte
GLubyte pColor[4];
glReadPixels(X,Y,1,1,GL_RGBA,GL_UNSIGNED_BYTE,&pCo lor[0]);
Earlier it was:
Byte pColor[4];
glReadPixels(X,Y,1,1,GL_RGBA,GL_BYTE,&pColor[0]);
I don't have an explanation why this doesn't work though...