Actually, if you look at creatures in real life, the bigger ones move most slowly. Muscles are pretty much built the same for all mammals. Using the same machinery, larger animals need to move more mass around. Think how fast a mouse can run, and compare that to how fast an elephant can run.
Take a look at most Olympic athletes in running events. They're taller. Short people have to take more steps during 100m than a tall person does. Longer stride matters a lot.
In your example, a mouse only really appears to be moving faster due to it's size. an elephant gets from point A to point B faster.
Look at animals like the cheetah. they run faster, despite muscles being muscles, because of how their stride works, and how the layout of their skeleton allows for that stride. Look at fast running birds, and notice they have long legs, and longer strides.
In mabi, humans and elves are basically the same speed. But for mysterious reasons, humans just jog around, while elves actually sprint. Giants you will notice also tend to jog... but they move faster than humans because of their longer stride.