Rashi: They are those who rule over them. This is like "Yinun Shmo" (72:17), "v'Achariso Yihyeh Manon" (Mishlei 29:21). All their rulers, the early and latter, have one thought - to first combat the Guardian of Yisrael, and afterwards Yisrael. They burned all Mo'adei Kel in the land - all His meeting places. The Pelishtim destroyed Shilo, Nebuchadnetzar destroyed Bayis Rishon, and after them [Edom] destroyed Bayis Sheni.

Radak: Ninah (we will kill) Osam (them). This is like "Cherev ha'Yonah" (Yirmeyah 46:16). This expression also refers to monetary Ona'ah (affliction, e.g. overcharging) - "v'Lo Sonu Ish Es Amiso" (Vayikra 25:17) and bodily affliction - "v'Lo Yonu Od Nesi'ai Es Ami" (Yechezkel 45:8).

Malbim: Their descendants. Not only did the Mikdash remain destroyed in their days - their grandchildren said together in their hearts that they burned all Mo'adei Kel in it the land. They glorify themselves that Elokim no longer has a place to make His Shechinah dwell.