Many Scots (including one of my ancestors) went to fight as mercenaries for the King of Sweden, so perhaps they married and had a family there.
Leith and Berwick were major ports for hundreds of years, it's fairly likely that some of the sailors would have been Swedish, whether on Swedish ships or on ships from Denmark, Estonia, Finland, Germany, Latvia, Lithuania, Poland, Russia participating in the Baltic Trade.
Perhaps some of those sailors decided to drop anchor and stay in the Scottish Borders or Northumberland or were stranded when their ship sailed without them; a few such sailors are mentioned in the Poor Law records.
Perhaps one of them was your ancestor.
To celebrate Genealogy Day in Sweden, which is on 21 March, ArkivDigital is offering free access to Swedish records in their collection at the weekend.
Sweden is one hour ahead of us so the free period is from 11pm on 20 March to 11pm on 22 March.
You'll need to register and download the ArkivDigital program.
There's also a useful guide to Swedish genealogical records.
If this helps you to widen your family history, please let us know.