Why was he hesitant to give 20 loaves in front of 100 men? In famine years, this is a nice amount to eat!
Rashi, based on Kesuvos 106a 1 : He gave each loaf in front of 100 men.
The Gemara says that there were 22 loaves (Lechem Bikurim and Karmel in addition to the 20).