I run with a primarily ACE setup, and I like to play in both Windowed and Fullscreen mode (windowed when i'm soloing and like to have wowhead open in a browser), and fullscreen when i'm raiding and need the max fps and detail.

However, some mods are very strange, in that they do not seem to scale well. For example, if I place a Bartender3 bar on the absolute right side of my screen in windowed mode, when I switch to fullscreen mode, the bar is floating a little bit inside. Also, I have my quartz bars lined up perfectly to my AgUF unitframes in fullscreen mode, but in windowed mode, they're all over the place.

Is there a way to get all my mods using the same positioning system?

The easiest solution is to use maximized windowed mode so that the screen size doesn't change between windowed and fullscreen mode.

If that isn't an option, there isn't any easy solution short of pestering the coders of the mods in question to make thier positioning handle window resizing correctly, which may be difficult depending on how the positioning is done. Realistically, you'll probably have to use something like Visor/MoveAnything/etc. to manually set the anchors for frames to the screen edge that they should move with.

Its become annoying to the point that it really ought to be a function in the Ace API that allows the user to choose which system he wants.

This isn't a permanent solution, but if you wanted, you could set the mod positions up for one instance, e.g. fullscreen, make a copy of your WTF folder and label it clearly, set the mods up for whichever windowed mode you like, make a second copy of your WTF folder, and then rsync your official WTF folder with the WTFsolo/WTFraid directories based upon whichever you plan on doing before you start the game.