Raids are simply attacks against the garrison, they cannot capture the fief, but if they win all the gear and (?) gold in the fief will be taken by them. Any troops on persons in the fief will remain untouched, as will any gear or gold on persons in the fief.
Population cannot fight a raid, only the garrison can, which is why you have 0 troops despite your 1970 population.