As a ranged character, you're going to have less hit points on average than infantry characters. The more detailed links to armor penalty equation for ranged were already posted, but it's not the end of the world if you have a small penalty due to armor weight.
Generally body armor with weight 10-12 is as high as I prefer to go with my crossbow user. Light hand and leg armor, pick a helm that you like the look of (but try not to go for the heaviest possible ones). This won't save you from infantry guys with a fuckoff-huge 2 handed sword, but you can soak a few hits from enemy ranged or 1 handed weapons.
Even in medium armor like your blue tunic and mail, you will be faster than most infantry once you raise your agility to 24 or higher and max out athletics skill. If you wear really light weight armor you can zip all around the map to get the best shooting spots and run from anyone but cavalry. But you'll die in one hit from pretty much anything.
The regular crossbow is great for a hybrid build if you're going to use a good 1H weapon and a shield. If you use the heavy crossbow or arbalest consider using very light melee weapons and no shield if you're comfortable blocking manually. You will feel the armor penalty more heavily if you're using the top-end crossbows as well.
Find out which playstyle suits you best! You might prefer the hybrid jack of all trades, or the slow and steady sniper, or the annoying marathon runner peasant that can pick and choose what fights he gets into.
