Yeah. Remember when Xant said he got his ass handed to him in towns by common thugs. I didnt have that problem. Like in rpgs, patiently scavenge and do most of the secondary quests(those that dont have an absurd level requirement) in the world you are in before you move on to the next, otherwise get ready for an asswooping. Secondary quests given by people who are in the main story are a must-complete. Not only are they fun and lengthy quests, they also give bucketloads of xp, money, decent equip and most definately I think they contribute to the main story in the end aswell.
Tbh the game even putting those extremely important secondary quests right next to those common secondary quests given by peasants is a crime. Some of them probably have gigantic inpacts on the world. Like the one that has you kill the king, the guy whose massively related to the world and the lore in general. Accidentally overlooking something like that might suck.