Probably the 18 and 21 year olds will not be an issue - if they are considered to be past the age of majority in your country, they can immigrate without his permission. It's the 13 yr old where you could run into an issue. Unless you have court documents that give you full legal and physical custody of that child, your ex has to give written consent for him/her to immigrate. CIC will not issue a visa for a child to immigrate if it removes the child from the country where another parent has legal right to visitation, etc.
I don't know what the laws are where you live, but if your youngest - at 13 - is afforded the right to decide where and with whom s/he wants to reside, and considering that s/he does not see the father, it's possible that a Court will grant full custody to you based on that, or based on abandonment - if it can be shown that the father makes no effort to see the child and the child wants nothing to do with him. Whatever the direction you proceed - get it worked out BEFORE you submit that permanent residence application. You have another wrinkle, though, in that your eldest is getting close to the age where s/he will no longer be considered a "dependent" to be sponsored. If it seems this custody issue with the 13 yr old will go on long enough that the eldest will reach age 22 before it's resolved, then you have to be pro-active and get the 21 yr old into a continuous full-time program of accredited post secondary education BEFORE s/he turns 22, and s/he has to
stay in it until the visas are issued. Also, be prepared to prove that s/he is still financially dependent on you. This preserves the "dependent" designation so that s/he can still be included in the sponsorship. (See Item #3 in this
Dependent definition from the IRPA Regulations.)