You need to do Bayer filtering,colour correction, colour balancing, white level balancing, Gamma correction in order to properly handle the colour data out of the sensor. You also have to probably do colour space conversion somewhere along the way.Thats just the colour ,you also have to deal with any other image processing. All this adds up to a lot of processing which is usually done in dedicated ASIC devices.
It is possible to do in an FPGA, we do much the same thing for our image processing, I cannot let you have any code for the FPGA as it is pro propriety but i can give you a few hints if you have any specific questions.
Just to give you an idea of the resources we use in an FPGA to do all the following takes about 40% of the logic elements, 19 hardware multipliers which are 18 bits wide and quite a few bits of block ram in an xilinx spartan x3c1500 running at 113.5 mhz
We do tend to overclock the image sensors so we can get the frame rate we need at the resolution we need, you'll find that most sensors will overclock quite well, with the attendant reduction in sensitivity.
We tend to do most of the processing in YCrCb colour space as it has a lower bandwidth requirement than RGB but some things such as white balancing are better done in RGB The standard texts on things such as colour correction,colour balancing and white balancing tend to be overly simplified and there are quite a few complications to trip the unwary.