Gemara

(Rav Gidal citing Rav): If (on Shabbos) one forgot to mention Shabbos (he did not say Retzeh in Boneh Yerushalayim), he says "Baruch Asher Nasan Menuchah l'Amo Yisrael l'Os ul'Bris. Baruch Mekadesh ha'Shabbos";

If one forgot to mention Yom Tov (Ya'aleh v'Yavo), he says "Baruch Asher Nasan Yomim Tovim l'Amo Yisrael l'Simchah ul'Zikaron. Baruch Mekadesh Yisrael veha'Zemanim."

(Rav Gidal): If one forgot to mention Rosh Chodesh, he says "Baruch she'Nasan Roshei Chodashim...."

R. Zeira: I do not know whether or not he says "l'Simchah," whether or not he ends the Berachah, and whether Rav Gidal said this himself, or in the name of Rav.

Rav Nachman erred and returned to the beginning. Gidal bar Minyomi asked him why he did so.

Rav Nachman: Rav Shila cited Rav to say that one who erred must return to the beginning.

Gidal: Rav Huna cited Rav to say that if one erred, he says "Baruch she'Nasan..."!

Rav Nachman: Rav Menasiya bar Tachlifa cited Rav to say this is only if he did not begin ha'Tov veha'Metiv. If he began it, he must return to the beginning.

Rishonim

Rif (36a DH v'Heicha): If one blessed Birkas ha'Mazon on Shabbos or Yom Tov and erred and did not mention Shabbos, he says "Baruch Asher Nasan Shabbos li'Menuchah ... Baruch Atah Hash-m Mekadesh ha'Shabbos." If he forgot to mention Yom Tov, he says "Baruch Asher Nasan Yomim Tovim... Baruch Atah Hash-m Mekadesh Yisrael veha'Zemanim." This is if he remembered before beginning ha'Tov veha'Metiv. If he remembered after beginning ha'Tov veha'Metiv, he returns to the beginning.

R. Yonah (DH l'Os): After mentioning Asher Nasan Menuchah..., he adds l'Os ul'Bris. These are two matters, so this is like a long Berachah. Therefore, he concludes with Shem u'Malchus (Hash-m's name and kingship). Similarly, he is Chosem on Yom Tov because he adds (a second matter, i.e.) l'Sason ul'Simchah. If he remembered after beginning ha'Tov veha'Metiv, he returns to the beginning. Some say that this refers to the beginning of Birkas ha'Mazon. The Rambam says so. I disagree. Since he did not yet finish ha'Tov veha'Metiv, this is no worse than one who usually says supplications after Shemoneh Esre. If he forgot to mention Rosh Chodesh in Retzeh, as long as he did not finish his supplications, he returns to Retzeh, and not to the beginning. Also here, as long as he did not finish ha'Tov veha'Metiv, he returns only to Boneh Yerushalayim. Also the Ra'avad says so.

Rambam (Hilchos Berachos 2:12): If one forgot to mention on Shabbos or Yom Tov Kedushas ha'Yom, he says "Baruch Asher Nasan Menuchah l'Amo Yisrael l'Os ul'Bris Kodesh. Baruch Atah Hash-m Mekadesh ha'Shabbos." On Yom Tov, he says "Baruch Asher Nasan Yomim Tovim l'Amo Yisrael l'Sason ul'Simchah Baruch Atah Hash-m Mekadesh Yisrael veha'Zemanim." If he remembered after beginning ha'Tov veha'Metiv, he interrupts and returns to Birkas ha'Zan.

Rebuttal (Ra'avad): The fourth Berachah is no worse than one who usually says supplications after Shemoneh Esre. If he forgot to mention Rosh Chodesh in Retzeh, even if he finished Sim Shalom (and is saying supplications), he returns to Retzeh, and not to the beginning. Therefore, also here he returns to Boneh Yerushalayim.

Defense (Rosh 7:23): Shemoneh Esre is different, for the last three Berachos are considered like one. One who omitted (during the 10 days of Teshuvah) "b'Sefer Chayim" in Sim Shalom or "u'Chsov l'Chayim" in Modim, according to Rabbeinu Yitzchak, returns to the beginning of Retzeh. Also regarding Birkas ha'Mazon, the first three Berachos are like one, so one returns to the beginning.

Rosh (ibid.): Even though the compensation Berachah is Samuch l'Chavertah (right after another Berachah), it begins with Baruch, just like ha'Tov veha'Metiv. Also, it is said only on occasions (when one happened to forget. It was not enacted to regularly follow another Berachah.) If he already began ha'Tov veha'Metiv, he returns to Birkas ha'Zan. This is like we say about one who uprooted his feet after Shemoneh Esre. He returns to the beginning of Shemoneh Esre. Here, there is no uprooting of legs. Finishing the Berachos mid'Oraisa is like an absolute interruption.

Poskim

Shulchan Aruch (OC 188:6): If one forgot to mention Shabbos, he says "Baruch Atah Hash-m Elokeinu Melech ha'Olam she'Nasan Shabbasos... Baruch Atah Hash-m Mekadesh ha'Shabbos." If one forgot to mention Yom Tov, he says "Baruch Atah Hash-m Elokeinu Melech ha'Olam she'Nasan Yomim Tovim... Baruch Atah Hash-m Mekadesh Yisrael veha'Zemanim." If Yom Tov was on Shabbos, he says "...she'Nasan Shabbasos li'Menuchah... v'Yomim Tovim l'Sason... Baruch Atah Hash-m Mekadesh ha'Shabbos v'Yisrael veha'Zemanim."

Taz (4): If one does not know the text of the compensation Berachah, he must return to the beginning, since he does not know how to correct his omission. If one knows the beginning and closing, even if he does not know properly the text in the middle, he need not return due to this.

Bi'ur Halachah (DH she'Nasan): The Taz requires investigation. If one forgot to say Mashiv ha'Ru'ach, and remembered before beginning Atah Kadosh, he says it (before Atah Kadosh - 114:6). The same applies to one who remembered before Modim that he omitted Ya'aleh v'Yavo (422:1). Since he did not yet start the next Berachah, it is as if he did not yet finish this Berachah. If he does not know the text of the compensation Berachah, why can't he return to Retzeh or Ya'aleh v'Yavo? It is as if he is still engaged in Boneh Yerushalayim!

Suggestion: Birkas ha'Mazon is different. Chachamim consider the Berachah finished once he says Boneh Yerushalayim. They were lenient for one who says the compensation Berachah, so he will not need to return to the beginning.

Rejection (Bi'ur Halachah): They brought a proof from here, that one can fix the Berachah before beginning ha'Tov veha'Metiv. This shows that it is not considered finished before beginning ha'Tov veha'Metiv. According to the suggestion, there is no source (to say that he can fix the matter). This is difficult according to the Shulchan Aruch, who permits fixing an omission before beginning Atah Kadosh or Modim. However, that law is not simple. Several Rishonim disagree.

Mishnah Berurah (16): If one is unsure whether or not he mentioned Shabbos, we assume that surely, he did not mention it.

Mishnah Berurah (20): This is when he forgot both Retzei and Ya'aleh v'Yavo. Therefore, he must include both of them in the compensation Berachah. If he forgot only of Shabbos, he says only "Asher Nasan Shabbosos li'Menuchah... If he forgot only of Yom Tov, he says only Asher Nasan Yomim Tovim...

Shulchan Aruch (ibid.): All of these Berachos are with Shem u'Malchus.

Beis Yosef (DH v'Im): The Gemara did not mention Shem and Malchus. Therefore, Meforshim argue about this. R. Yonah says that one closes with Shem and Malchus. Orchos Chayim says that the Yerushalmi says so. The Rashba says 'presumably, all of these have Hash-m's name at the beginning and at the end, even though he would not have mentioned Hash-m's name if he said them in the proper place. This is because if he says them in a Berachah Samuch l'Chavertah (right after another Berachah), they are included in the (previous) Berachah that has Shem and Malchus. It seems that the Tosafists hold like this, and Semak, and also the Rif and Rambam connote a little like this.

Mishnah Berurah (21): If one remembered after saying Baruch Atah (at the end of ha'Tov veha'Metiv), surely he may return to say Retzei or Ya'aleh v'Yavo, and then Boneh Yerushalayim. If he remembered after saying Baruch Atah Hash-m, it is proper to conclude immediately Lamdeni Chukecha, so Hash-m's name will not be l'Vatalah. It is as if he did not finish the Berachah. He returns to Retzeh.

Kaf ha'Chayim (27): If he concluded Lamdeni Chukecha, all agree that he may return to Retzeh or Ya'aleh v'Yavo. However, the Birkei Yosef and others say that he should not conclude Lamdeni Chukecha. Rather, he concludes Boneh Yerushalayim, and says "Baruch she'Nasan Shabbosos..." The Chayei Adam, Ben Ish Chai and others prefer concluding Lamdeni Chukecha, so he will not add an extra Berachah.

Shulchan Aruch (ibid.): This suffices if he remembered before beginning ha'Tov veha'Metiv. If he remembered after beginning it, he repeats Birkas ha'Mazon.

Beis Yosef (DH u'Mah she'Chasav Rabeinu): The Rambam, Rosh, and Rashi hold that if one remembered after beginning ha'Tov veha'Metiv, he returns to the beginning. We follow them.

Magen Avraham (8): The three Berachos are considered like one, therefore anywhere that one erred, he returns to the beginning. If he erred in the fourth Berachah, he returns only to ha'Tov veha'Metiv.

Mishnah Berurah (23): The Chayei Adam says that saying Baruch Atah Hash-m Elokeinu Melech ha'Olam ha'Kel is considered starting the Berachah. If he said only Baruch Atah Hash-m Elokeinu Melech ha'Olam, he concludes "Asher Nasan Shabbosos...", and then says ha'Tov veha'Metiv.

Bi'ur Halachah (DH Ad): The other Acharonim say that once he began ha'Tov veha'Metiv, there is no solution. He must return to the beginning. They hold that once he started ha'Tov veha'Metiv, it is evident that he was Mesi'ach Da'as (diverted his mind) from the third Berachah. The Yerushalmi (7:4) connotes like this. The Or Zaru'a explicitly says so. Since he said Baruch Atah Hash-m with intent for ha'Tov veha'Metiv, how does it help to conclude "Asher Nasan Menuchah..."? He was Mesi'ach Da'as from the third Berachah! The Chayei Adam's law requires great investigation.

Kaf ha'Chayim (28): Some say that saying Baruch is considered starting ha'Tov veha'Metiv.

Mishnah Berurah (24): If one erred in ha'Tov veha'Metiv, he returns to it. It is a separate Berachah.