I'm currently in the midst of my 4th year of Chem Eng. Coming out of high school, calculus was actually my worst grade in all of high school...and yet I still figured that without an A in calc I'd still be able to manage fine. I didn't take any advanced placement classes in high school, and sort of choose chem eng. on a whim since I liked chemistry and had the prerequisites.
First year hit me like a truck (nearly flunked out), but the playing field started to level out by the midpoint of second year for me because nobody could rely solely on what they learned in high school, and the disparities based on which region/school system everyone had come from started to disappear. I had one semester of physics and was done with it, thankfully. However, math is still my weak spot and it definitely brings me down on in the class both in how quickly I learn things and how well I can execute on exams.
I guess my point is that you don't have to be a math whiz if it's something you're passionate about, but it helps a lot. I've had to work really hard learning even small concepts sometimes to bridge the gap, but it's 4th year and I'm still around with the people I started with, and dozens of my original classmates (no exaggeration) are not.
Good luck!