Sex: Why can't I make up my mind?!?

The first time I had sex was with my first bf and first love, things ended badly and I was depressed for over year and didn't have sex for over a year, until last night. I had sex with a guy I've been hanging out with, he cooks for me, takes me to fun places, always opens the car door, he's such a good friend to me and I love his personality, but I don' think I want to date him because he has kids (which he is really, really good to). The sex was good, and I don't feel like I did anything wrong, but I can't decide if sex before marriage is wrong. Growing up I wanted to wait, but both times I did it it was because I felt comfortable and safe, so how can that be wrong? I'm 22, and I don't want a serious relationship right now, and yes I tried masturbating for over a year but after the first couple of months I felt depressed because I was horny and suppressing my urges and a vibrator just wasn't cutting it, and I really did want to wait to be in a more serious relationship but I just couldn't take the wait anymore. So I guess I'm just trying to see what everyone's thoughts and opinions are on the subject. I personally don't think sex before marriage is wrong, but because of the way I grew up part of me can't let go of thinking it is.