First of all, why did you bring US dollars? Merchants here don't take USD. There isn't really much difference in USD and CAD, for example as of today, 1 USD = 1.053 CAD. So not really much difference. Yes, any bank you go to for conversion will charge fee and there will always be difference in their buying and selling rate. Buying from them will always be higher and selling to them will always have lower rate Plus the fee. Try contacting some local desi traveling agent, you should get a better rate than bank there but only do so if you have huge amount of USD, otherwise don't waste your time.
Hope you get the worth of your money at the best rate.