Gemara

(Mishnah): Inside from the Agaf, (the thickness of the doorway) is considered inside. Outside the Agaf is considered outside.

(Rav Yehudah): The same applies to prayer.

He argues with R. Yehoshua ben Levi;

(R. Yehoshua ben Levi): Even an iron wall cannot separate between Yisrael and Hash-m.

(Reisha): Inside from the Agaf is like inside.

Inference: The Agaf itself is like outside.

Contradiction (Seifa): Outside the Agaf is like outside [but the Agaf itself is like inside]!

Resolution: The Agaf of [most] gates of the Azarah is like inside, since the gate itself was Hukdash [like the Azarah];

The Agaf of a gate of Yerushalayim is like outside, since the gate itself was not Hukdash [like Yerushalayim];

(R. Shmuel bar Rav Yitzchak): The gates of Yerushalayim were not Hukdash, to allow Metzora'im [who may not enter the city] to take shelter underneath from sun and rain.

Eruvin 92b: If a Tzibur is in a wide [Chatzer that was breached to a narrow Chatzer], and the Shali'ach Tzibur is in the narrow one, they are Yotzei (Tefilah through listening to him);

If the Tzibur is in the narrow one, and the Shali'ach Tzibur is in the wide one, they are not Yoztei.

If nine are in the wide one, and one is in the narrow one, they join [to form a Minyan];

If nine are in the narrow one, and one is in the wide one, they do not join.

Sotah 38b - Question: Does a barrier [between the Kohanim and Yisre'elim] block Birkas Kohanim?

Answer: No. R. Yehoshua ben Levi taught that even an iron wall does not intervene between Yisrael and Hash-m.

Rishonim

Rambam (Hilchos Korban Pesach 9:1): One may not take Korban Pesach outside the house in which it is eaten. The Agaf, i.e. the thickness of the doorway, is considered outside.

Tosfos (85b DH v'Chen): Rashi says that we discuss joining for Tefilah. This is wrong. The Halachah follows R. Yehoshua ben Levi, who says that even an iron wall does not separate. The Stam Gemara in Eruvin connotes that a Mechitzah prevents joining. One in the small Chatzer does not join with nine in the big Chatzer. 'And similarly for Tefilah' cannot refer to hearing the Shofar or Kri'as ha'Megilah, for if one heard from outside the Beis ha'Keneses or in his house, he was Yotzei (Rosh Hashanah 27b)! The Ri says that we discuss an individual answering to Kedushah or Kaddish.

R. Yerucham (Nesiv 3, 7:27:4, cited in Beis Yosef OC 55 DH uvi'Gmara): It seems that people who stand in the threshold of the Beis ha'Keneses do not count towards a Minyan. This is not considered part of the house. We learn from Korban Pesach. The Torah forbids taking the meat outside the house, and we say that the Agaf is like outside.

Poskim

Shulchan Aruch (OC 55:13): All 10 must be in one place, and the Shali'ach Tzibur with them.

Gra (DH Tzarich): We learn from Eruvin 92b. Tosfos explains that Pesachim 85b discusses answering to Kaddish and Kedushah when there are 10 without him, like in Sa'if 20. Rosh Hashanah 27b discusses Shofar and Megilah, which do not require 10.

Mishnah Berurah (48): This is even if they do not see each other, since they are in one house. If they are in different rooms, they do not join even if the door between them is open, because there is not a breach, and the doorway itself is considered a Mechitzah. This is like two houses. Even when there is no door in between, whenever they are two Reshuyos and do not see each other, they do not join. Some are stringent even when they see each other.

Mishnah Berurah (49): If nine were in the house and one was in the Sukah, some say that he joins, and some disagree. If nine were in one place and one was in back of a curtain spread merely for privacy, they join. If there are Tefilin or a Sefer Torah, and a curtain was spread for a Mechitzah to permit relations, it is a Mechitzah also for joining to a Minyan, so they do not join (Pri Chodosh). The Pri Megadim leans to say that in all cases they join.

Kaf ha'Chayim (68,69): Regarding the Sukah and a curtain, one should be stringent to say that they do not join.

Shulchan Aruch (ibid.): One who stands in the doorway from the Agaf and outside, i.e. when they close the door, from the place of the inner edge of the thickness of the door, is like outside.

Beis Yosef (DH veha'Omed): Rashi says that the Mishnah teaches that Agaf is the entire place that the door occupies when it is closed. From the inner edge of the thickness of the doorway until where the door hits [what stops it], and [further] inside, i.e. in the city, is like inside. From the Agaf and outside is like outside, i.e. where the door hits. The Mishnah does not teach about the Agaf itself. The Gemara teaches about it. Tosfos says that a Mechitzah separates from joining for Tefilah, and all agree that inside from the Agaf is like inside regarding Tefilah.

Beis Yosef (DH uvi'Gmara): The Gemara connotes that regarding joining for Devar shebi'Kedushah, the Agaf itself is like inside. If it were not normally like inside, they would not have been Mekadesh it for gates of the Azarah! Also for gates of Yerushalayim, they would have been Mekadesh it, if not for Metzora'im. However, R. Yerucham says that it is like outside, and the Rambam considers it like outside regarding Korban Pesach. It seems that we learn from there to Tefilah, like R. Yerucham said.

Magen Avraham (11): Sefer ha'Tanya rules that it is like inside. The Gemara connotes like this. See the Beis Yosef.

Gra (DH mi'Makom): R. Yerucham holds that since there was a need to be Mekadesh the Agaf, this shows that it is not considered inside. The Rambam says similarly. It seems oppositely, that it is like inside, and they intentionally refrained from being Mekadesh it, like the Gemara says! The Beis Yosef says so. Even his Perush this requires investigation, for this is no worse than a small Chatzer totally breached to a big Chatzer. If the minority or Shali'ach Tzibur stands there, they join (Sa'if 16)! See the Magen Avraham, and mi'Ta'am Acher (for another reason), like mentioned above.

Damesek Eliezer: The Gra challenges R. Yerucham. The Gemara needed to give a reason why the gates of Yerushalayim were not Mekudash, but not why the gates of the Azarah were Mekudash. This shows that the Agaf is like inside! Therefore, since there was no reason for the Azarah one way or the other, they were Mekudash. The Gra further challenged R. Yerucham and the Shulchan Aruch. Surely one who stands in the Agaf should join. He is no worse than one in a small house [next to a big house]. It is Batel to the big house, even though the small house would not get Kedushah due to the big house. We say that the Bayis [ha'Mikdash] and the Azaros were Mekudashos, i.e. each by itself. It seems that the text of the Gra towards the end should say "mi'Ta'am Echad" (due to the first reason).

Mishnah Berurah (50): The Magen Avraham and Even ha'Ozer rule that when the minority stand in the Agaf, they join. This is no worse than a small Chatzer breached to a big Chatzer. The Gra agrees.

Mishnah Berurah (51): Yad Efrayim says that if the door is fixed inside, and it enters only part of the gap of the doorway, all agree that what is inside from the Agaf is like inside. One may rely on this. Even though the Pri Megadim connotes that R. Yerucham considers this like outside, in any case many argue with R. Yerucham.

Bi'ur Halachah (DH Shel): If the door is fixed inside, and it enters only part of the gap of the doorway, R. Yerucham and the Shulchan Aruch hold that one there does not join. According to the Tanya and Magen Avraham, the Pri Megadim holds that they join, but Yad Efrayim disagrees. According to the Gra, this is no worse than a small Chatzer breached to a big Chatzer, so one may be lenient also about this.

Bi'ur Halachah (DH ul'Chutz): If they see each other, they can join even if they are totally outside, like the Mechaber is lenient in Sa'if 14. L'Chatchilah it is better to enter inside. This leniency that seeing each other helps, like for Birkas ha'Mazon, is from the Rashba (Teshuvah 96), and he said only "perhaps".

Kaf ha'Chayim (72): Chikrei Lev was unsure about a doorway without a door. Perhaps it is like inside. Beis Yakov says that it is like outside. I say that due to the Safek, one there does not join with those inside, nor with those outside.