Cold tap water in Iceland is both pure and clean. It is sourced from the same springs as Icelandic bottled water sold in stores in Iceland and abroad. It is perfectly safe to drink.
In Iceland the hot and cold water on the tap come from different sources. Usually the hot water comes from a geothermal source and has some sulfur smell to it, making it less feasible to drink, though it is considered perfectly safe for consumption. Some newer buildings have Thermal Exchange System, which uses geothermal water to heat cold spring water for consumption to protect faucets from building up of minerals from the geothermal water, which has the pleasant side effects of eliminating the smell of sulfur from the tap.