Soul Gems are one of the many collectibles scattered throughout the levels of the Skylanders games.
There is one Soul Gem for each Skylander. The Skylander's Soul Gem can be found in their first game. This means that there will be a Soul Gem for each Series 1 Skylander as well as any “gimmick” Skylanders (i.e. Giants, SWAP Force, Trap Masters).
So in Spyro's Adventure there are 32 Soul Gems because all 32 characters were new (Series 1) figures.
In Skylanders Giants, there were only 16 Soul Gems. One for each of the 8 Giants and one for each of the 8 new Series 1 Skylanders.
In SWAP Force there were once again 32 Soul Gems. One for each of the 16 swappable SWAP Force Skylanders and one for each of the 16 new Series 1 Skylanders.
In Trap Team there were 36 Soul Gems. One for each of the 18 Trap Masters and one for each for the 18 new Series 1 Skylanders.
Soul Gems have two basic purposes.
First, immediately after collecting a new Soul Gem you will have the option to watch a short preview video of the character to whom the Soul Gem belongs. If you choose not to watch it right away that's okay because you can always access these videos from the menu any time after you've unlocked them.
Secondly, collecting the Soul Gem will unlock the final ability upgrade for that character. Once you collect the Soul Gem you will still need to purchase the upgrade (from Persephone) with gold you've collected, but it will now be an option.
The Soul Gem ability is often the strongest attack your Skylander has so it will be to your advantage to find, unlock, and purchase this power.