There can be some reasons for noisy readings: Noisy reference voltage, poor ground routing intoducing hum loops, injected hum due to a mains transformer in close distance to ICL7135, signal noise at input, etc.
Mostly the noise comes from input signal noise, which can be rejected to some degree by a suited low pass filter. Standardly you should use a low pass filter consisting of 100kOhm and 100nF, as it's suggested in datasheet. But if this does not present enough rejection, then corner frequency should be lowered. Unfortunately, settling time increases then, means it takes longer to get a stable reading and for big changes of input signal settling time can become unacceptably high.
The following circuit offers both, fast settling time for big changes of input signal and at the same time improved noise filtering due to lowered corner frequency:


The circuit works as follows: If a big change of input signal arrives, then one of the Schottky diodes becomes conducting whenever the difference between input and output voltage of the filter is higher than about 0.2V. In this case only the 1kOhm resistor forms the time constant of low pass filter. So, the capacitor is charged up much faster, than with the 100kOhm resistor alone.
If the difference between input and output voltage of filter falls under 0.2V, the Schottky stops conducting and now the 100kOhm resistor forms the much bigger time constant of low pass filter, resulting in improved noise filtering.

Take care, only polypropylene capacitors should be used for the caps at ICL7135!!

Kai