You can remove any agent from servicing you.
Is this "agent" a registered one ie Registered Immigration Consultant(RIC)?
If not, then he has never been representing you in the first place. Infact he should not be representing you. He was actually impersonating you.
If he is indeed a RIC, then you can ask IRCC that he should not represent you and revoke his representation permission and report him.
Now, more importantly, if you have your visa on your passport, it will list a lot many details including your UCI number.
Do you have your application confirmation letter that he might have sent while putting your application? That will have your Application number.
With your UCI and application number, you can send communication directly to IRCC using WebForm. Another way is to call IRCC and talk to a IRCC client support agent directly. You can update your email address with them so any further communication will be with you rather than that agent. I am not sure but you may request them to send your PoE letter as well.
Also, look at following answers from IRCC:
1.
https://www.cic.gc.ca/english/helpcentre/answer.asp?qnum=856&top=25
I have lost, deleted or damaged my POE Letter. What should I do?
Contact us using the
Web form and ask us to re-send your
Port of Entry (POE) Letter.
If you applied online, we will usually re-send your POE Letter to you through your account. If you cannot access or
link to your original application in your account, make sure you include that detail in your message.
2.
https://www.cic.gc.ca/english/helpcentre/answer.asp?qnum=850&top=23
I recently reset my account, but I don’t see my application. How can I get it back?
PoE letter is very simple one :
Basically it contains your UCI and Application number and a standard language :