From the photo, it is clearly visible that it is not just a naked laser diode. You seem to have a transistor and a trim potentiometer. And there may be other components on the other side of the board.

The thing is that a laser diode is a constant-current device. It may have a Vf of 3V, but you control the energy by adjusting the current - not the voltage. A tiny change in voltage will result in a huge change in current (and hence energy).

A laser module, on the other hand, may be constant voltage. You supply 3V and makes sure that you have enough current, and the electronics on the board then makes sure that the laser gets the correct amount of current. Another thing is that the Vf of a LED can vary a lot between batches and also depending on temperature which makes it impossible to just carefully try to find the exact voltage to supply - you need a feedback loop where the supplied voltage is adjusted up or down depending on the current. That is why you always have a series resistor or a constant-current driver for a LED.

With your PCB, it seems like there is a trim potentiometer, which seems to indicate that the electronics needs to be tuned to give a suitable amount of current to the laser. However, since the module is cheap and the components are few, I would expect that the electronics can only handle a quite small variation in input voltage and with a limited line regulation. I would expect that you need to use an external regulator to produce a known voltage similar to the two batteries.

Just one little thing here is that there are lasers that are pumped, in which case they may instead have electronics on the outside that produces a square wave. A pumped laser may need a lot of energy to light, but more energy than it will survive to be constantly fed with...

As already noted earlier, the buffer chip has much too high (and unknown) voltage drop so it is unsuitable for the task. You should fix your voltage with a voltage regulator and then have a MOS-FET transistor to drive the module.