The books by Diana Gabaldon involve a mysterious portal to get the main character some 200 years. After that, it's a straightforward historical romance. On the other hand, if magic is involved in the plot/story, then it crosses to fantasy.
Other elements of fantasy include a quest or heroic adventure, conflict between good and evil, often a made-up language. Sometimes portals are used in fantasy, rarely in s-f. Fantasy stories also often include some kind of "royal" family. They often take place on some other world entirely, not Earth.
Put it this way: legends of King Arthur are fantasy. There's the sword, there's Merlin the Magician.
Stories about Robin Hood are not, although they're not science fiction either. Everything that happens in Robin Hood stories are real-world, no magic of any kind.
Science fiction may also take place on other worlds, but again, what happens is real-world possible. Often advanced technology is assumed.
Alternate history is a genre of s-f that is not fantasy. Examples include many books by Harry Turtledove. Many alternate histories start with 'What if the South had won the Civil War" and take it from there. Or "What if the Nazis had won WWII."
I hope that clarifies it.
Oh, and it's "dichotomy" but I of course knew exactly what you intended.