Yes, I agree to what you said. Speaking is just one sub-part of the whole test. I've never been living in an English speaking country, but just had experience working with some of those who did stay there for a number of years for their degree, and found that their spoken English was horrible when we had to deal with guests who could only speak English. That's why I can only speak from my experience concerning the speaking part.
Also, in the government's perspective, living in an English speaking country does not necessary mean a person has proficient English skills. If they have the power to choose, why not choose the best ones with good skills right? One way to do so is to test their skills through some standardized test to play safe. While I hate having to take those tests myself, I totally would have do so if I were the government.
May I ask what kind of informal English are you referring to in the above statement that cannot be used in IELTS? (I have never been living in an English speaking country so I think the differentiation between formal and informal English is a bit blurry to me and I can't really tell the difference.)