For commuting (when I get the chance to - typically have to drive kids around so I take the car) or going for a leisurely ride with the kids, or cycling just to get around, I generally wear normal clothes, too. It’s just more practical for a ten, twenty minute ride at an easy pace. My office is also 5 minutes of mostly downhill from my house.
However, going for a “proper” ride - be it a weekday training ride, the long weekend ride, a group ride or race, getting dressed up in proper kit is just so much better.
I was kind of hesitant myself when I started riding, too, but you try one thing, then another, and soon you’re clad in lycra head to toe and wondering why didn’t you do it sooner