Even if he only has access but you have sole physical custody, unless his parental rights are terminated for some reason you will not be able to leave the country with the child without his permission.
By taking the child away you are denying his right to access, whatever you think of him (and...