A permanent resident in Canada needs to be in Canada for at least 730 days total (2 yrs.) in any 5 year period. If you don't do that, you lose your PR status even though your card might not be expired. That goes for you and each of your dependants to fill the residency requirements. Once you get PR, it doesn't matter anymore who is the primary applicant.
