The pasuk in last week's parsha (Shemos) says they brought a nursing woman from the Jews. The Medrash brought down by Rashi comments that Moshe refused to nurse from an Egyptian woman because he was going to grow up and be a Navi so how could he eat non-kosher food. In other words, the non-kosher food would have affected his soul and made it impossible for him to be the Navi that he became.
The Gra comments that this Medrash is the source of the din in Shulchan Aruch in Yoreh Deah Siman 81 that although Jewish baby is allowed to have a non-Jewish wet nurse, the Shulchan Aruch says that you shouldn't do it because the non-kosher food will harm the baby spiritually. Likewise the Rama there states that a Jewish woman who needs to eat non-kosher food for health reasons should not nurse her baby because the non-kosher will harm him.
We see from the above, that non-kosher is intrinsically harmful for a person's soul even if they eat it b'heter.
The Ran in his Drashos says this explicitly. In the 11th drasha he discusses why Halacha is decided by Chachamim and not Neviim. He asks the following question. He says that if a Chacham makes a mistake and permits a forbidden food it is like a doctor who makes a mistake and gives a patient poison instead of medicine. In both cases the person is harmed. If so, why don't we have Neviim decide halacha with nevua and there would be no mistakes? He answers that nevua is not always available (see the drasha for more details). He then explains that even though eating non-kosher food is harmful the mitzva of listening to Chachamim may counterbalance the harmful effects. In any case, we see clearly from the Ran that non-kosher is objectively poison and harmful even if you eat it b'heter.
The Abarbanel (Devarim 17,4) disagress with the Ran. He holds that a person is never harmed by following the Chachamim. If it is mutar then it cannot be harmful. It would seem that the Abarbanel holds that non-kosher is not objectively harmful, rather it is harmful because it is prohibited. If for you there is no prohibition then it is not harmful.
Rashi in Chullin(5a) seems to agree with the Abarbanel. The gemara there is discussing whether a mumar l'avoda zara can do shechita. The gemara tries to bring a proof from Eliyahu Hanavi. At 1 point Hashem tells Eliyahu to go live by himself and the Orvim (birds) brought him meat to eat from Achav's place and (almost)everyone in Achav's palace worshipped Avoda Zara. How could Eliyahu eat the meat if meat slaughtered by a mumar is not kosher? It must be that a mumar is allowed to slaughter. The Gemara answers that על פי דיבור שאני. Rashi explains that Hashem permitted him to eat non-kosher meat (a Navi is allowed to violate the Torah based on his nevua). The obvious question is how could Eliyahu Hanavi eat non-kosher food if it is spiritually harmful? It didn't seem to affect him as he continued being a Navi. The answer would seem to be like the Abarbanel that because he ate it b'heter it was not spiritually harmful.
In fact, the Ritva in Chullin as well as the Maharsha understand the gemara's answer differently. They understand that the Gemara answered that Eliyahu Hanavi knew b'nevua that the meat was kosher, that it had been slaughtered by Ovadia who did not worship Avoda Zara. According to them, Eliyahu Hanavi did not end up eating non-kosher food.
To sum up, we have a major dispute whether non-kosher food is objectively harmful to a person (even if he eats it b'heter like a nursing baby) or not.
R' Moshe (Orach Chaim 2:88) quotes a teshuva of the Chasam Sofer where he discusses a handicapped girl in the following situation. If she stayed at home she would never develop and remain an ayno bar daas. However, if the parents sent her to a special school they claimed that she would reach the potential of a 13-14 year old. However, the school was in a non-Jewish area and there was no possibility of providing kosher food. The Chassam Sofer said that al pi din it is muttar, however he recommends against sending her for the following reason. If they send her she will become a bar daas and be chayav in mitzvos. However, the non-kosher food will affect her lev and she will probably violate torah and mitzvos and therefore it is better for her to remain an ayno bar daas.
We see clearly that this idea that any non-kosher food is metamtem halev, even if eaten b'heter (the girl was a ketan and an ayno bar daas so there was no issur), is brought down l'halacha.