יבמות טז,ב:
רש"י ד"ה מעשרת שבטים הוא: שנשאו עובדות כוכבים וקסבר דעובדת כוכבים שילדה מישראל הולד ממזר וחוששין לקידושי ממזר
שאלה: ואמאי לא אמר שחוששין שבנים של בנות ישראל שנשבו - בת אחרי בת עד לאמו של הגוי שקדש?
וכן: יבמות יז,א: רש"י ד"ה לפסול: רובן ממזרים, שנטמעו עשרת השבטים בעובדות כוכבים, ובניהן הרי הן כישראלים ונוהג בהן איסור ממזרות והן לא נזהרו ונשאו זה אחותו או בתו והרי הן ממזרים:
ועוד זכור לי שהמאירי באיזה מקום אומר שילד שנולד להורים שהם עצמם המירו - אין לראות בו ישראל, ואם כן - מתי נקבעו ההלכות שאנו נוהגים?
בברכה ובתודה
ישעיהו הכהן הולנדר
TRANSLATION:
(a) RASHI (DH me'Aseres ha'Shevatim) says that according to the view that the child of a Jewish man and a Nochris is a Mamzer, we are afraid that this person (who seems to be a Nochri) is a Mamzer.
But why does Rashi not consider the fear that such a Nochri is really the descendant, daughter after daughter on his mother's side, of a Jewish woman who was captured by and married to a Nochri (in which case he is not a Mamzer)?
(b) RASHI (17a, DH Lifsol) says that "most of them are Mamzerim, for the Aseres ha'Shevatim became mixed with Ovdos Kochachim, and their children had a status of Jews to whom the Isurim of Mamzerus applies, and they did not observe the laws and they married their own sisters or daughters, and now their descendants are actual Mamzerim."
I recall that the ME'IRI says that the child born to parents who themselves apostatized is not to be considered Jewish. If so, when were the Halachos that we follow established?
(a) It seems clear that the reason why Rashi does not write that we should be afraid that the Nochri comes from a Jewish woman who was captured is because the Gemara says afterwards (17a) in explaining Shmuel that "we have a tradition that the Jewish women of that generation were unable to have children" (Rashi 17a, DH Itzteruyei), and it does not seem likely that Rav Kahana in the Gemara here is arguing with that tradition. Also, it seems from the words of Shmuel that he is only arguing that the son of an Ovedes Kochavim is called "her son" and not "your son."
As far as why Rashi on 17a had to say that they married relatives and that is why we are afraid that this person is a Mamzer, and he does not explain as he does here on 16b (that the child of a Jewish man and a Nochris is considered a Mamzer), the OR L'TZION explains that this is because Rashi is explaining the Gemara according to Rebbi Yochanan here, and the Amora'im earlier argue about the opinion of Rebbi Yochanan, whether such a child is a Mamzer or not. Therefore on 17a, Rashi has to explain that the reason they might be Mamzerim is because their forebears married their own relatives.
(b) We find in the ME'IRI (at the end of the Perek, DH u'm'Kan Hayu Ketzas Rabosai Chochechim Lomar) something very similar to what you write. He quotes "a few of our teachers" who discuss the possibility that a Jewish apostate (Meshumad) has the status of a Nochri ("Goi Gamur").
The CHASAM SOFER (Chidushim to Orach Chayim 39) writes that "the Jewish people have the power to remove from their ranks an apostate, giving him a status of a complete Nochri, even with regard to matters of leniency."
Regarding the specific case that you mention, RAV MOSHE FEINSTEIN (Dibros Moshe Yevamos 23, Igros Moshe EH I:8) mentions that those who maintain that a child born from a Nochri and a Jewish woman requires Gerus to be considered Jewish -- this applies only when the Jewish mother apostatized. He writes that l'Halachah, one should be concerned for that opinion in such a situation (and require Gerus for the child). (See also the MAHARIT ALGAZI, Hilchos Bechoros 8:64, 65, who says that it depends on how the child was raised, as a Jew or as a Nochri. In AVI EZRI, Hilchos Isurei Bi'ah 15:3, Rav Shach, shlit'a, writes that "such a thing should not be said," and it has no source in Halachah at all.)
Yosef Ben-Arza
See Be'er Heitev, Even ha'Ezer 4:23. With regard to Shitas Rashi, see Pischei Teshuvah, EH 4:1.
Kol Tuv
Yitzchok Zirkind