What does "Kevod Hashem" refer to?
Radak #1: It is when all Ma'aseh Bereishis was finished - "va'Yar Elokim Es Kol Asher Asah v'Hinei Tov Me'od" (Bereishis 1:31). All are His deeds and they are His honor; He should rejoice in them.
Radak #2: This refers to the time of the Ge'ulah. They are all His deeds, and He rejoices in them when they do His will, for everything was created for man. In this world, sometimes He is sad and sometimes happy. It says about Dor ha'Mabul "va'Yinachem Hashem?. va'Yis'atzev El Libo" (Bereishis 6:6). Also Dor ha'Flagah, Anshei Sedom and Amorah and other generations that did evil deeds, He did not rejoice in His creations. However, it says about Yemos ha'Mashi'ach "Ki Az Ehpoch El Amim Safah Berurah Likro Kulam b'Shem Hashem Le'avdo Shechem Echad" (Zecharyah 3:9). This will be after Resha'im are eradicated - "v'Hayu Chol Zedim v'Chol Ose Rish'ah Kash v'Lihat Osam ha'Yom ha'Ba" (Mal'achi 3:19).
Radak (35, citing Ibn Ezra): It is the creations, for they were made via His word.
Malbim: It is honor for Hashem when the creation constantly stands in its Chidush, like it was when Hashem created it.
Why does it say that Hashem rejoices?
Radak: It is an anthropomorphism to say so about Elokim, to help people understand. (Also, He is never sad.) This is a metaphor for one who is happy when his actions succeed, and sad when the opposite occurs.
Radak (35, citing Ibn Ezra): This refers to persistence of the Kelalim.
Malbim: If individuals lived forever, Hashem's honor for creating anew would not be recognized. They would rebel against His honor via thinking that they always existed, and they exist by themselves. Via death, they know that Hashem formed them, and they honor Him, so He rejoices in His deeds.