The trouble with that theory is that scaling of things other than weapons doesn't quite work that way. At low levels a character will be doing 8 damage per second with the Flames spell. This is similar to a weapon attack but has some range and is easier to aim but costs magicka to cast. Overall a totally reasonable choice. Unfortunately in the endgame things don't work out quite so nicely. The most powerful spammable single target spell will deal roughly 108 damage per cast and at most one cast per second is reasonable. This is a big improvement in that it does 13x the damage of the low level spell and has much greater range.
Unfortunately one of the main things the game allows in theory is a hybrid caster / weapon user and in the end game spells have scaled 13x to the weapon's 100x. This is a problem because the hybrid character will either one shot everything with their sword and wonder why they even bother with spells or they will have a reasonable time with their sword and the spells won't even scratch the enemies. Neither is particularly good. It gets even more silly if you twink out harder and sneak attack the enemies - I got my dual sword damage up to around 30,000 for a single sneak attack which kills anything in the game pretty much and although setting up a sneak attack is much tougher than shooting fireballs from far away it is pretty hard to say no to doing 300x the damage.
What ends up shaking out of all of this is that Bethesda made some errors when scaling the various ways to attack. It is actually very much like Diablo 2 in that weapons scaling off of many enormous % bonuses eventually eclipsed spells in damage and only Static Field let a sorceress compete. Diablo 3 deliberately avoided this issue by not having spells do fixed damage but rather scaling them off of weapons just like weapon users. People criticized this for having bad feel but seeing how badly things get messed up when you scale weapons multiplicatively and spells additively in Skyrim I am inclined to support the decision of the D3 designers. (I supported it back in the beta days too!)
If in Skyrim the two styles were really independent then I wouldn't worry too much about it. Unfortunately the game really is meant to accommodate hybrids and it ends up not doing so all that well. Of course you don't have to twink out as hard as I did but even with just using the best weapon you find and drinking potions you find and putting points into the weapon trees it is clear that weapons rapidly make spells a futile venture.
I ended up wanting to up the difficulty level even beyond Legendary so I decided to simply refuse to use any potions that increase Enchanting or Smithing or any enchanted gear that increased Smithing or Alchemy. This dropped my damage by 75% and forced me to actually pay attention to armour and resists so it brought some level of challenge back into the game. I don't much like refusing to use all the options available to me but this does make the game feel a lot more reasonable.