Well, I have a lot to say about this! I think it is human nature to fear what we don't know. Once we see what a thing is and how it works, we don't fear it any more. I really think that the only way to overcome math fear is to want to understand it. Many students say, 'I'm terrible at math,' as an excuse not to learn it. That's like saying, "I'm terrible at laundry so I have no responsibility to get my clothes clean." Or, "I don't have the talent to be an olympic swimmer, so I won't bother learning to swim."
We are all better at some things than at others. I'm better at speaking than I am at drawing, but so what? I still draw. I still try.
All it takes to learn mathematics is an interest in learning it. It might not come as easily to you as to some other students, but so what? So work harder at it! You really can do it if you really want to. But if you convince yourself that you hate it (even if you don't know what math is, and math is most definitely not just arithmetic and algebra) then you have closed off all possibility of ever allowing yourself to enjoy it and see how neat it is. And you have closed off a whole world of beauty. It is like saying that you have no interest in the alphabet, and so you never learn to read. What a terrible loss that would be.
Bottom line: just because a subject might not be easy for you doesn't mean that you can't learn it. If you don't do well in math, it is because you have chosen not to.