Gemara

30b (R. Yochanan): I saw R. Yanai pray (without a Tzibur) twice (in succession, i.e. Shacharis and Musaf).

Question (R. Yirmeyah): Perhaps he repeated the same prayer because he did not have proper intention the first time!

Answer (R. Zeira): A great Chacham testified about this. Surely he knew that he prayed Musaf!

R. Chiya bar Aba prayed twice (in succession).

i. R. Zeira: Surely this was not because you did not have good intent the first time. R. Elazar taught that one should estimate whether or not his mind is settled enough to have intent. If it is not, he should not pray!

34b (Beraisa): One who prays must concentrate in all the blessings. If he cannot do this, he should concentrate in one.

(R. Chiya): He should concentrate in Avos.

Bava Basra 164b (Rav Amram): (Most) people transgress every day Iyun (lack of concentration in) Tefilah.

Rishonim

Rosh (4:21): We suggested that R. Yanai prayed again because he did not have proper Kavanah the first time. This shows that if one did not concentrate at least in Avos, he must pray again.

Rosh (5:24): R. Elazar taught that one should pray only if he estimates that his mind is settled enough to have intent. This means that he can have intent in Avos, like it says here.

R. Yonah (20b Sof DH Bein): One must concentrate in Avos even b'Di'eved. If he did not concentrate in Avos, he prays again.

Tosfos (Rosh Hashanah 16b DH v'Iyun): The Gemara in Bava Basra means that one does not have proper intent in Tefilah, like it says in the Yerushalmi. R. Chiya said 'I never had intent. Only once, I wanted (or needed) to intend, and I thought in my heart 'who goes first in front of the king - the Arkavsah (an official above the Reish Galusa), or the Reish Galusa?' (I.e. do the more important officers come first, or last?) Shmuel said 'I counted birds.' R. Bun bar Chiya said 'I counted rows of the building.' R. Masniya said 'I am thankful to my head, for when I reach Modim, it bows by itself.'

Ran (Rosh Hashanah 3b DH v'Amar): No one has proper intent in Tefilah, like it says in the Yerushalmi. This is why Rav Yehudah said 'I will receive (reward, or help from Hash-m) for concentration in prayer (Shabbos 118b).

Poskim

Shulchan Aruch (OC 101:1): One who prays must concentrate in all the blessings. If he cannot do this, he should concentrate in Avos.

Beis Yosef (DH Tanu): R. Elazar did not distinguish between Avos and the other Berachos. However, Tosfos, the Rosh, Semag and the Tur say that he refers to Avos.

Question (Beis Yosef DH Kasuv): Semak says that if one cannot concentrate in all the Berachos, he should concentrate in Avos and Modim. If he cannot, he should concentrate in Avos. What is his source that Modim is more important than other Berachos?

Answer (Kaf ha'Chayim 3): Admas Yehudah says that Kavanah in Modim is more important because this saves one's spine from becoming a snake. (This is said about one who does not bow in Modim.)

Tur: R. Eliezer says that one should accustom himself to have Kavanah at least in the closing of each Berachah. They have a total of 113 words, like Chanah's Tefilah. The word "Lev" (heart) appears 113 times in the Chumash, to hint that one must have Kavanah in these 113 words.

Beis Yosef (DH Kasav ha'Rav): Also Hagahos Maimoniyos and the Roke'ach say so.

Kaf ha'Chayim (1): The closing is praise of Hash-m. It is more important than the middle of the Berachah, which is request.

Mishnah Berurah (2): One must have intent for the meaning of the words. One must know the meaning of the words in the entire Tefilah, and surely for the first Berachah!

Mishnah Berurah (3): Avos is the praise of Hash-m. It is improper for one's heart to wander while saying it! Some Poskim say that Modim is like Avos l'Chatchilah. If one thinks that he cannot have Kavanah even in Avos, he should not pray until his mind settles so he can have intent at least in Avos.

Kaf ha'Chayim (2): The Avudraham says that Avos is more important than the other Berachos because it has the greatness of Hash-m, merit of the Avos, and Ge'ulah of the Acharonim (i.e. the final redemption).

Shulchan Aruch (ibid.): If he did not concentrate in Avos, even if he concentrated in all the other Berachos, he prays again.

Mishnah Berurah (4): If one lacked Kavanah in Avos, letter of the law he must pray again. If he remembered (that he lacked Kavanah) before saying 'Hashem' in the end of the Brachah, he should return to say from Elokei Avraham.

Halichos Shlomo (ha'Gaon R. S. Z. Auerbach, Ztz"l, 8:8(30)): This is when he had no Kavanah. One returns to where his Kavanah ceased. One may do so in other Brachos, even if he will repeat Hash-m's name.

Nekiyus v'Chavod bi'Tfilah (p.202, citing ha'Gaon R. C. Kanievsky Shlita): One need not return if the meaning is understood even without the words that were said without Kavanah.

Ma'aseh Ish (1 p.129, citing the Chazon Ish) says that if one lacked Kavanah in Avos, it suffices to think now about the meaning of the words he said without Kavanah. This is if he did not yet begin the next Brachah.

Halichos Shlomo (8:9): One may do so at any point in Shemoneh Esre, and to strive to intend in Modim.

Halichos Shlomo (8(13)): The Roke'ach says that if one had Kavanah in Modim, he was Yotzei even if he did not have Kavanah in Avos. If one did not have Kavanah in Modim, he should mentally go back from the beginning of the Berachah (ha'Gaon R. C. Kanievsky Shlita, in Ishei Yisrael (the updated edition) p.798:48).

Rema: Nowadays we do not pray again due to lack of Kavanah, for also in the repetition it is likely that he will not have Kavanah. If so, why should he pray again?!

Magen Avraham (1): Since Keri'as Shema requires Kavanah in only one verse, (if he did not have Kavanah,) he repeats it (even nowadays - 63:4).

Gra (DH v'Ha'idna): See Tosfos, and all the more so, nowadays we may not rely that we will have intent the second time. The Yerushalmi explicitly says that if one prayed without Kavanah, he does not pray again, unless he is sure that now he will have Kavanah. R. Chiya and others said that they did not have Kavanah.

Damesek Eliezer: It is astounding how R. Chiya, the greatest of his generation, and all the Amora'im never had intent. Even nowadays, some people have Kavanah! Surely the Yerushalmi means that only once before praying, he expected to have proper intent. Perhaps this is what the Yerushalmi brought just before, that once he did not have intent, and he prayed again confident that he will have intent, and due to this he stumbled, and thought about other matters during the Tefilah.

Note: The question is most astounding regarding R. Chiya, for the Tefilah of R. Chiya and his sons was so powerful that Eliyahu needed to interrupt it, lest it bring Mashi'ach prematurely (Bava Metzi'a 85b)! Damesek Eliezer's answer answers this, but it is harder to give his answer for the other Amora'im. Pnei Moshe (on Yerushalmi Berachos 17b) says that the Amora'im lacked intent because they were distracted thinking about Torah. E.g. 'birds' can refer to Chatas ha'Of that flew amidst Olos ha'Of and became mixed with them. Ber Ratnar (the author of Ahavas Tziyon vi'Yerushalayim, cited in Torasan Shel Rishonim on the Yerushalmi) brings that Afruchiya means stars (and not birds. Shmuel was an expert astronomer.)

Bi'ur Halachah (DH ha'Mispalel): This is even according to the opinion that Mitzvos do not require Kavanah. That refers to intent to be Yotzei the Mitzvah. The Rashba (brought in Beis Yosef 63) connotes that one should be careful not to think about other things in the middle, even if he has Kavanah when saying the words. L'Chatchilah one should be stringent like this. It is not clear if b'Di'eved he was Yotzei.

Bi'ur Halachah (DH v'Im): The Shulchan Aruch connotes that l'Chatchilah he may pray, since due to Ones he cannot settle his mind. The Beraisa connotes like this. It is astounding that the Rambam mentioned only b'Di'eved if he had intent in Avos. Perhaps he agrees that it is better to pray than to totally omit praying. Surely, if he can wait until his mind settles, and not lose the time for Keri'as Shema or Tefilah, he must do so.

Kaf ha'Chayim (4): The Tur says that nowadays we do not pray again. The Mechaber holds that even nowadays one should pray again and strive to have Kavanah. The custom is not like the Mechaber. If one did not have Kavanah, the Berachos were not l'Vatalah, but this is very serious and one must strive to have Kavanah at least in Avos.