After lots of more experiments I couldn't find the source of this weird behavior and I was about to give up! Finally, I measured the USB cable. To my surprise, I found at that both of the cables I've purchased as USB 2.0 cable were not shielded! The metal plated USB plugs at both end of the cable were connected to nowhere!

Using a shielded USB cable highly improved the EMI resistance of my device. With a shielded cable, the PC chassis ground is connected to my PCB ground and EMI can find a path to discharge.

Though using a shielded cable highly improved my device, it doesn't make my device EMI proof. I still need to make some more changes in PCB design and other things.

Now I wonder why someone would ever need a non-shielded USB cable and why these things are still sold in the market!