I think it makes sense he came from the Wyvern Mountains, because that's Wildlands. Which is an important thing to consider because, regardless of the kingdoms you find yourself in, the character always refers to the inhabitants in a way that implies a distinction between the adventurer and them. "You Varsilians", "You Mercians", and so on and so forth.
That being said, the intro says than an inheritance letter summons you to "the capital". Now, if it refers to New Garand as "the capital" instead of "the capital of Varsilia", that can mean you actually come from Varsilia (thus not the Wildlands), or that you simply find yourself in Varsilia at the time.
I don't think you come from anywhere south in Varsilia because the character says he hasnt ever met Varannari the first time we meet Deep-Shadow. So, where is there no Varannari. The Wyvern Mountains would be a good guess.
We could be from New Anthur, I think, just hear me out. We are described in the following way:
"As an adventurer, you are more concerned about your lack of job and your empty purse"
The say I see it, we could have started with minimal New Anthur reputation (what a civilian there usually has), and lost it in out misadventures. At the start, we are laughably weak to the creatures in New Anthur Bay, which would explain why we lost that rep (tried to do a job, ended up biting more than we could chew, lost rep). That failure also explains your lack of job (you're known to be incompetent and unexperienced, so people dont hire you) and your lack of money.
I guess you're always free to imagine it however you please, though, because the vagueness is obviously intended. No matter where you are or who you talk to (Hell, even the lack of knowledge abut ANY PLACE IN THE MAP AT ALL), you are always treated, and act, as a foreigner.
You come from nowhere. Your home is the world.