This is very fair. Math has always come fairly easily to me. So math intuition plays a part in my interest and ability to learn to program.
I think most people, even smart people, assume they couldn't do it though because I'm some kind of genius, which only a few programmers actually are.
Like stoly said above, I think programmers are probably slightly above average intelligence overall, so don't sell yourself short there. But yeah. We're not geniuses