I have heard of cases where they have gone into detail about who was accompanying whom but in most cases they don't. They will only look at the 5 year period before you apply for the travel document. If you meet the requirements in that 5 year period because of Canadian husband living with you, you should get it.
Worst case scenario, lets say they make a big deal about your husband accompanying you and not vice versa and you lose your PR, you husband can always sponsor you for PR again.
I believe that was only one case, Leon, where the judge worried about whether the PR was accompanying the Canadian spouse. In several other cases the judges ruled that it didn't matter; as long as the PR and Canadian spouse were together, those days counted toward the PR quota. In fact, this latter point is now written in the relevant Operational Manual. I wrote about this about a year ago; it could be found by using the search feature, I suppose.
I could look it up if someone is interested and can't find my post.