First of all, they're unique to Iceland. They are descendants of ponies brought to the country by Scandinavian settlers in the 9th and 10th centuries. Since then, the species suffered severe Natural Selection, given that the environmental conditions in Iceland are harsh. The "weaker ones" often died by starvation or cold. That made those horses (they can also be referred as ponies, because of their tiny size - but Icelanders don't have a word for "pony", so they call'em "horses" anyway) stronger and more resistant, and still today they are used to carry heavy things around the country. They are also used for leisure, racing and showing.
The breed has two extra and unique gaits ("gait" is a way a horse can move, either naturally or as a result of training by humans), called tölt and skeið. The other gaits they have are the common walk, trot, gallop/canter. The tölt is a lateral ambling gait, with a footfall pattern similar to the walk (left hind, left front, right hind, right front), but its different from it because the tölt can be performed in various speeds. The skeid , on the other hand, is a pace often used in races, since it's fast and smooth (left hind and left front, suspension, right hind and right front).
A curious thing about Icelandic horses is that they have very few diseases. This is accomplished by strict laws that prevent horses that leave the country from coming back, and other horses can not be imported. The equine equipment that arrives in Iceland also needs to be new or entirely disinfected before having contact with the horses. This results in almost no acquired immunity to diseases in those animals, and an outbreak on the island could probably devastate the breed. Let's hope that won't happen!