In legal perspective, yes the ex-spouse may pursue a legal action (concubinage) . But that is rarely the case specially if the marriage has broken before, and not because of the current relationship. (unless he has a bitter ex-spouse lol).
But, kidding aside, you are right. Didn't really...