Hehe, I've had to restart countless of times. Sometimes it's the fire doing a tactical strike at my storages and production building, once a tornado came and wiped out my whole 100 pop village, only 15 survived the disaster itself, but the next winter took care of the last ones.
Now I've managed to build properly to avoid fires from spreading (put the buildings in groups of max 5 + wells near important stuff), and the only tornado I've had wasn't so bad. Got a epidemy of measles, but that only killed like 20. Now almost year 100, I have a nice town of almost 500 citizens, and expanding. Started on hard difficulty.
Some tips:
Spread out a lot, this way tornadoes won't do as much damage. Place your buildings in small and separate groups, so they won't go down like dominoes when the first fire starts.
Try to farm as many different types of crops as possible, and never farm 2 fields of same crop type next to each other. This way parasites won't ruin your whole harvest. Same goes for orchards and pastures too.
As you expand, keep up a good network of Marketplaces to balance the resources between regions.