Why did he seem to like the idea that I don't have much confidence?

He's messing with your head. Ignore him, and start being a little more firm in your "NO you cannot come over' stance.