Gemara

(R. Yochanan): If one is told to transgress or else be killed, he may (or must) transgress, except for dolatry, Arayos, and murder.

(Rav Dimi): At a time of Shemad (a royal decree against Mitzvos), one may not transgress even a light Mitzvah. He must submit to be killed.

(Ravin): Even when there is no Shemad, he may not transgress in public (in front of at least 10 Yisre'elim). He must submit to be killed.

(Rava bar Rav Yitzchak): Wearing shoelaces like Yisraelim is an example of a light Mitzvah.

Question: Esther was married to Achashverosh in public (i.e. many Yisraelim knew about it - she should have refused and forfeited her life)!

Answer #1 (Abaye): She was Karka Olam (passive, until she went to the king on Mordechai's command. In such a case, one need not forfeit his life).

Answer #2 (Rava): When the Nochri wants the Yisrael to transgress for the Nochri's benefit, even in public, one need not forfeit his life.

Support (Rava, for himself): On their idolatrous festivals, Nochrim demand that we give them vessels with coals (for their rituals). Even though this pertains to idolatry, we allow them, we do not forfeit our lives!

Since they intend for their own pleasure, it is permitted.

Rishonim

Rif and Rosh (8:3): Nochrim used to wear red shoelaces, and Yisraelim wore black, to avoid Bigdei Nochrim. If a Nochri tells a Yisrael 'wear red shoelaces in public, or be killed' in such a place, he may not transgress.

Rashi (74b DH Arkesa): If Yisrael tie their shoelaces differently, for they are more modest, even if it is not a Mitzvah, rather, a custom, he must be Mekadesh Hash-m in front of his colleagues.

Rambam (Hilchos Yesodei ha'Torah 5:1,2): If a Nochri seeks to make the Yisrael transgress, the Yisrael should comply unless 10 Yisraelim are present, in which case he must submit to be killed.

Ba'al ha'Ma'or (Sanhedrin 17b DH Abaye): Mesiras Nefesh is never required when the Nochri intends for his own benefit. We learn from Esther. This was in public and it was Arayos, for she was "l'Vas" (a wife) to Mordechai. Similarly, when Nochrim demand coals (for their rituals) on their idolatrous festivals, we comply, for they intend for their own benefit. This applies even at the time of Shemad. However, regarding idolatry itself there is no case where the Nochri wants the Yisrael to serve for the Nochri's benefit. Also, the source to require Mesiras Nefesh for murder (you do not know who is more worthy to live) applies even if the Nochri intends for his own benefit. Abaye and Rava do not argue with each other; both of their answers are true. Even if they argue, the Halachah follows Rava, who permits when the Nochri intends for his own benefit.

Rebuttal (Milchamos Hash-m): R. Tzadok and Rav Kahana were ready to kill themselves rather than sin with Nochriyos who intended for their own benefit (Kidushin 40a). Surely they would not go beyond the letter of the law to kill themselves, which is forbidden. The Gemara (Yoma 82a) permits eating on Yom Kipur for Piku'ach Nefesh, and says that the same applies to anything except for idolatry, Arayos and murder, i.e. even though this is for benefit.

Rambam (ibid. 3): This is when the king did not decree to Mevatel Yisrael from Mitzvos. When there is such a decree, one must Moser Nefesh for any Mitzvah even if there are no Yisraelim around.

Kesef Mishneh: Rashi explains a light Mitzvah which one may not transgress in public to be tying shoelaces like the custom of Yisrael, which is more modest than the Nochrim custom. The Rambam does not require Mesiras Nefesh for customs of Yisrael. He holds like the Rif, who says that the light Mitzvah is not wearing Bigdei Nochrim, a Lav - "v'Lo Selchu b'Chukos ha'Goy..."

Rambam (ibid. 4): If Mesiras Nefesh was required and one submitted to be killed he was Mekadesh Hash-m, like Daniel, Chananyah, Misha'el and Azaryah. If he transgressed he was Mechalel Hash-m. If 10 Yisraelim were present, the Kidush or Chilul Hash-m was b'Rabim.

Poskim

Rema (YD 157:1): If a Me'anes threatens a Yisrael that he must transgress, and the Yisrael save himself by paying, even if he must give all his money, he must do so rather than transgress a Lav.

Source (Gra 4): One may not even transgress Shevus of Shevus to save money, e.g. to tell a Nochri to extinguish a fire on Shabbos.

Pischei Teshuvah (4): Some say that one must forfeit all his money to avoid transgressing a Lav through an action. The Pri Megadim was unsure if one must spend all his money to avoid a Lav mid'Rabanan, or only a fifth. The Chavas Ya'ir and Radvaz require him to spend all his money.

Rema (ibid.): Chachamim taught that one must protest. If not, he is held responsible. If this could be dangerous, he need not spend money for this.

Source (Gra 5): Chachamim did not reveal families with lineage problems because it would have been dangerous (Kidushin 71a).

Shulchan Aruch: If a Nochri threatens a Yisrael to transgress a Mitzvah or be killed in public, i.e. in front of 10 Yisraelim, if the Nochri's intent is to make the Yisrael transgress (even regarding shoelaces), the Yisrael must be Moser Nefesh. If the Nochri intends for his own benefit, the Yisrael should transgress and not be killed. When there is a decree (only regarding Yisrael), one must be Moser Nefesh even for shoelaces.

Gra (9): The Gemara does not ask why Esther transgressed at the time of a royal decree, because the decree was not limited to Yisrael.

Shach (4): The Yisraelim need not be there. If 10 know about it, it is called in public, like we find regarding Esther.

Taz (3): The Bach (DH v'Chosav Beis Yosef) says that at a time of Shemad one must be Moser Nefesh even if the Nochrim intend for their own benefit. This is not correct. Even then the Tur requires Mesiras Nefesh only if they intend for the Yisrael to transgress.

Bach (ibid. and DH v'Ika): SMaK says that in public one must be Moser Nefesh even if the Nochri intends for his own benefit. He holds that Abaye holds that Hana'as Atzman does not permit in public. This forced Abaye to find a different Heter for Esther, i.e. Karka Olam. Rava holds that even though the matter was known, since the Bi'ah was in private it is not considered in public. Rava agrees that Hana'as Atzman does not permit in public.

Rema: This is only if they want Yisraelim to transgress a Lav. One need not be Moser Nefesh for an Aseh.

Taz (4): One need not be Moser Nefesh for an Aseh because the Nochrim could imprison him and prevent him from fulfilling it.

Shulchan Aruch (ibid.): Regarding idolatry, Arayos and murder, even in private when there is no decree, even if the Nochri intends for his own benefit, one must be Moser Nefesh.

Rema: This is only if they command the Yisrael to do an act, e.g. to have Bi'ah with a woman or kill someone. If they forcibly rape a Yisraelis or throw a person on a baby to kill the baby, or if Reuven had an erection and they want to force his Ever onto an Ervah, Mesiras Nefesh is not required. One must be Moser Nefesh for any transgression pertaining to idolatry, Arayos and murder, even a Lav (without Misah), but not for "v'Lifnei Iver Lo Siten Michshol".

Shach (10-13): Even though the Rambam holds that one is lashed for Kirvah to Ervah (e.g. hugging) only regarding Chayavei Kerisos, he agrees that Mesiras Nefesh is required even for Arayos mid'Rabanan. Most Poskim hold that Bi'as Yisrael with a Nochris is Arayos only because Kana'im Pog'im Bo, which is only in public. However, the Nimukei Yosef says that it is Chayavei Kerisos and requires Mesiras Nefesh even in private. R. Tam says that Bi'as Nochri with a Bas Yisrael is never Arayos. However, all the Poskim say that if she is an Eshes Ish it is Arayos.

Avnei Nezer (YD 131:5): The Rambam explains that one must be Moser Nefesh even for shoelaces, due to 'uv'Chukoseihem Lo Selechu. The Maharik says that this Aveirah is only when one wants to mimic Nochrim! We must say that even though he does so only due to the coercion, the Nochri will not think so. Even though the Yisrael does not transgress, due to his intent, Kidush Hash-m applies. We cannot say that it is due to v'Ohavta, for that applies only to idolatry.

Avnei Nezer (128:7): One may not transgress even a Lav, due to Avizraihu (peripheral Isurim) of the three Aveiros. If a certain herb is needed to heal, and the only one available is of Avodah Zarah, the Shulchan Aruch (155:2) brings first (which shows that he favors) the opinion that permits, for the cure is not based on idolatry. The Rema did not comment! There, benefit from idolatry is a separate Isur "Lo Yidbak." It does not show that he believes in it, so it is not considered Avizraihu of idolatry.