Why does he seem to like the idea I have little confidence?

Well, people with low self esteem and no confidence tend to be push overs and easily manipulated to do whatever other people want. So my guess is, he thinks you will eventually cave in and then he'll have you to use as a play toy.