If you removed S&D from the map, it would seriously slow down gold coming into economy if people were wasting days travelling to zero S&D fiefs and then having to guess a new one. Would hurt independants more than mercantile clans also.
Also, the current period is relatively peaceful, there's abit more banditry on EU, but no heavy warfare. Almost no banditry on NA, and no major wars either, so it's very easy to trade. Once things heat up, less people will be trading, people will be closer to their lands with armies, and also juicy caravans will be hunted down and picked off, so there should be more S&D on map.
Finally, go smuggler and poach S&D off hoarders who "close borders". If you have 200-300 heavily armed troops, and carrying less than 300-420 goods, even large factions will hesitate to attack you. You will cost a lot more to rub out than the value of your goods, which weakens them vis a vis their real enemies. The fact that you have a decent amount of well armed soldiers will make you attractive to mercs also.