Yep you totally get it. Sex appeal is so different from beauty. It's a sort of charisma that some men have and it has nothing to do with looks.
And yes, the lust at first sight is rubbish. It's why I always go on a second date with someone if there was a cerebral connection. Many many times I have not been attracted to someone at first.
But the opposite happens too. Sometimes I can go on one or two dates with someone and start getting an inkling that they are unkind. That's when attraction dies for me.