I'm not too sure about that. Having a work visa in the US is very different from being an American citizen and therefore being exempt from needing a visa to visit Canada. If your husband's home country is not visa exempt for Canada, do not assume he can just cross into Canada simply because he holds a US work visa. The US work visa relates to his residence in the US, it doesn't give him any rights to visit Canada.
The reverse of this situation holds, and that is why I am not sure he can visit with you. A Canadian PR has no rights to visit the US without a visa by virtue of that status if he needed a visa to the US before he became a Canadian PR.
I would advise you, if your husband holds ONLY a US work visa, that you check with the consulate for his country to see if he needs a TRV in order to visit Canada.