Quote from Justified;101418:
Four Swords Adventures has Ganon's appearance along with Vaati's death. It's the only game with both Vaati and Ganon.
All other games (except Adventure of Link) either have Ganon as the villain, mention an attempt to revive/unseal him, or occur after his death.
So either Four Swords Adventures is the first appearance of Ganon (thus, Minish Cap is absolute first), or for some reason Ganon just disappeared for a while Vaati wreaked havoc.
Also, you're saying Four Swords Adventures happened before Link to the Past, which happened after Ocarina of time. That says nothing about whether Swords Adventures or Ocarina came first, just that Link to the Past comes last out of the three. Likewise neither Minish, Swords Adventures, nor Ocarina have a backstory for Ganon (and while Minish Cap does have a legendary hero, it involved a hatless Link wielding the Picori Blade, not the Master Sword).
FSA does have a backstory for Ganon, who is implied to be a new one. There's some quotes from the Gerudo that back it up.
1.
The pyramid is sacred to our people. To have a criminal such as him enter... His presence stains its holy ground.
2.
Curse that Ganondorf! He broke our laws and left for the pyramid... I don't know what he plans, but I am worried...
OoT Ganon was always trying to get the Triforce, while this one has a different objective. The Gerudo don't know what his goal is, but OoT Ganon's was very clear; he wanted to get the Triforce and rule Hyrule. There's more quotes that prove it, but I really don't remember them...