Gemara

(R. Pinchas ben Ya'ir): "Guard yourself from every evil matter" - do not have thoughts (of women) by day, which lead to Tum'ah (emissions) at night.

Bava Basra 164b (Rav Amram): There are three Aveiros that a person is not saved from transgressing them every day: thought of (sexual) transgression, (lack of) Iyun (concentration) in prayer, and (the dust of) Lashon ha'Ra.

Chulin 37b: Yechezkel said about himself "I did not make my soul Tamei" - I did not have emissions at night due to Hirhurim during the day.

Yoma 29a (Rav Nachman): Thoughts of sin are worse than sin.

Sukah 26b (Abaye): R. Yosi forbids young men to sleep with Tefilin when their wives are there, lest they come to Hergal Davar (relations, while wearing them).

Shabbos 64a (R. Elazar): "Va'Nakrev Es Korban Hash-m... Taba'as Agil v'Chumaz Lechaper Al Nafshoseinu" - Chumaz is a girdle;

(Rav Nachman): Moshe suspected that the soldiers (who defeated Midyan) brought an atonement because they had sinned.

The soldiers: We did not do a sinful action, but perhaps we had sinful thoughts.

(Tana d'vei R. Yishmael): The soldiers needed (to bring the Kelim) to atone for getting pleasure from seeing forbidden women.

(Rav Sheshes): The Torah lists external Tachshitim (e.g. rings) with inner Tachshitim (girdles) to teach that one who looks (for pleasure) at a woman's pinky needs atonement like one who looks at her Ervah.

Nidah 13a (R. Yochanan): One who emits semen unnecessarily is Chayav Misah -- "va'Yera b'Einei Hash-m... va'Yames Gam Oso".

13b (R. Ami): One who brings himself to Hirhurim does not enter Hash-m's abode:

Avos (Mishnah 1:5): Whenever one speaks much with women, he causes evil to himself, is Batel froom Divrei Torah and will fall to Gehinom.

Rishonim

Rambam (Hilchos Isurei Bi'ah 21:19): It is forbidden for a man to intentionally cause Kishuy or to bring himself to lustful thoughts. Rather, if such thoughts come he should divert his mind from folly to thoughts of Torah.

Ba'alei ha'Nefesh (Sha'ar ha'Kedushah DH va'Afaresh): Just like one must separate from Aveiros, one must separate from thoughts of Aveiros. If a thought comes to him, he must force himself to expel it from his heart. "V'Hiskadashtem vi'Hyisem Kedoshim" commands about Kedushah of deeds and thoughts.

Tosfos (Avodah Zarah 20b DH she'Lo): R. Pinchas' Drashah is a proper Drashah. This is why R. Elazar did not learn the Lav of Motzi Shem Ra from this verse. It was a Chidush that Yechezkel avoided Hirhurim, for all transgress this every day.

Yere'im (45): Chachamim warned not to look at Arayos; an Asmachta is "Guard yourself from every evil matter." The soldiers needed Kaparah for Hirhurim.

Tosfos (Shabbos 64a DH mi'Dei): Even though even Bi'ah is permitted with a Yafes To'ar (a beautiful Nochris) in war, Hirhurim about other women are forbidden (one may take only one Yafes To'ar).

R. Yonah (Avos 1:5 DH mi'Kan): Speaking much with women increases the Yetzer Ra upon him. He is Batel from Divrei Torah because he cannot think about Torah while his heart is on a woman. The two thoughts cannot coexist.

Ramban (Igeres ha'Kodesh DH umi'Kan): Hash-m punishes for thoughts like for action only regarding idolatry. Thoughts of Aveirah at the time of Bi'ah are worse than Aveirah, for it sullies the semen and corrupts the resulting fetus.

Yad Ramah (Bava Basra 164:52): The Aveirah of Hirhurim is even concerning a single woman.

Drashas ha'Ran (6 DH v'Chen): Moreh Nevuchim (3:8) says that thoughts of sinAveirah are a rebellion against Hash-m with the most esteemed part of a person, i.e. his intellect. Aveiros of action are with the lower part of man, i.e. his Guf. I add that one who constantly sins (through Hirhurim) without doing Teshuvah demonstrates a wayward spirit in his heart.

Kolbo (64 DH ha'Megale'ach): One who transgressed Arayos or spilled his seed must be extra careful to avoid anything that causes Hirhurim, e.g. seclusion with a woman, looking at her face, or the areas near the Ervah or breasts. He should not look at her clothing even on the wall, or have any thoughts of women other than his wife. He should rejoice in love of his wife, for she saves him from sin.

Poskim

Shulchan Aruch (EH 23:3): One may not intentionally cause Kishuy or to bring himself to lustful thoughts. Rather, if such thoughts come, he should divert his mind from folly to thoughts of Torah.

Source #1 (Beis Yosef DH u'Mah she'Chosav v'Im): If you encounter the debased one (Yetzer Ra), pull him to the Beis Medrash (Sukah 52b).

Source #2 (Gra 9): Torah saves (from sin) when one engages in it (Sotah 21a).

Igros Moshe (EH 1:69 DH v'Hinei Shnei): There are two Isurim of Hirhurim. "V'Nishmarta mi'Kol Davar Ra" forbids Hirhurim lest it cause emissions. This is even if one thinks about his wife, or other things that will not arouse desire for Isur, e.g. to look at a pretty single girl, colored clothing of a woman he knows, or animals mating. "V'Lo Sasuru ... v'Acharei Eineichem" forbids Hirhurim of sin. Just like "Acharei Levavchem" forbids thoughts of idolatry even without an action, "v'Acharei Eineichem" forbids mere thoughts of Arayos. Kidushin 39b connotes that only for idolatry, one is punished for mere thought. This refers to Misah. R. Yochanan says that one is Chayav Misah for spilling seed. Even if this is literal, it is Misah b'Yedei Shamayim; it would not justify falling from a building (Kidushin 39a), which is like stoning. Also, R. Yochanan discusses causing emissions physically, not through Hirhurim. Regarding Hirhurim it says 'one who brings himself to Hirhurim...' "V'Nishmarta" does not apply to women, but "...v'Acharei Eineichem" does.

Shulchan Aruch (OC 38:4): One who wears Tefilin must be careful to avoid lustful thoughts of women.

Rema: If one cannot avoid Hirhurim, it is better not to wear them.

Beis Yosef (DH v'Chosav) Rabbeinu Meir says that Bochhurim who have Hirhurim even while reciting Shma should not wear Tefilin. Even one who cannot guard himself from gas may not wear Tefilin. All the more so one may not be lightheaded with lustful thoughts about women. This is difficult. Youths should not wear Tefilin lest they come to Hergal Davar, i.e. Bi'ah (Sukah 26b). Hirhurim are not a reason to forbid! Perhaps Rabbeinu Meir explains that Hergal Davar refers to Hirhurim. The Yerushalmi says that Tefilin require a body clean of evil thoughts. Rather than exempt people due to Hirhurim, we should train them to avoid Hirhurim and accept Ol Malchus Shomayim.

Magen Avraham (4): Perhaps even though one may not wear Tefilin when he has Hirhurim, we would not forbid wearing Tefilin if one is unsure if he will have such thoughts. (Shulchan Aruch ha'Rav says so explicitly.)

Support (Dameshek Eliezer, on Gra 5): People used to wear Tefilin all day, even though everyone transgresses Hirhurim every day!

Tur (OC 240): Ideally, one should extend Bi'ah so that his wife will be Mazri'a first, and he will be rewarded with sons. One who fears lest he have sinful thoughts at the time should have Bi'ah quickly, and he will receive the same reward.

Acharonim

Toras Chaim (Bava Basra 164b DH Shalosh): The Gemara did not say that a person transgresses the three Aveiros every day, rather, he is not saved, to teach that even one who wants to be careful will not succeed. Thoughts of Aveirah are worse than Aveirah, for one cannot save himself from thoughts.

Maharsha (Bava Basra 164b DH Gimel): If no one can avoid the three Aveiros, why are they considered Aveiros? Every 'Adam' transgresses these. Man was created from above and below; he can determine his standing. Nochrim did not merit, so they are below. They are not called 'Adam.' Mediocre Yisraelim are called 'Adam'; they transgress every day. 'One should seek a Rebbi like an angel, i.e. he merits to be above. Such a person can avoid even the three Aveiros.

Question (Maharsha Avodah Zarah 20b DH Tanu): How can we learn the Isur of Hirhurim from "V'Nishmarta mi'Kol Davar Ra"? Shelichus Yad requires speech, not mere thought, because it says "Al Kol Devar Pesha"!

Answer #1 (Har Tzvi 127 DH Agav): The Sifri learns about Lashon ha'Ra from "Davar", and about Hirhurim from "V'Nishmarta". Further, we can say that "Davar" refers to something forbidden elsewhere. Here, the Torah commands to be careful about it. 'Dibur' excludes thought only when it is used to express the (verb of the) Isur, e.g. "v'Daber Davar."

Answer #2 (Gilyonei ha'Shas Kesuvos 46a DH Sham v'Nishmarta): "V'Nishmarta mi'Kol Davar Ra" includes thought, for 'Kol' is extra. (Note: we must say that "Al Kol Devar Pesha" includes a Shali'ach, therefore we do not include thought.)