Children's tales.
The wall is in crap state atm, the Night's Watch is nothing. Also they have begged forever for men, and nothing has ever happened, so some people may believe its the boy who cried wolf (the fable) all over again.
The Southron lords (and even some of the Northern ones) do not believe in legends and undead, that is all a load of bullcrap that has not happened for thousands of years, so they dismiss it as old folk's tales and superstition. So they play their Game of Thrones without realising that it is all for nought if the Others (White Walkers in the series) come and go over the wall. Even the Northern lords believe the Wall to be there to keep the wildlings out, and the tales of the Others have been banished to Old Nan's stories and other unreliable sources. Magic has been getting weaker and so the danger is dismissed. Further their armies went South to war, and are scattered and weak.
Also some lords may even believe that the Wildlings are only attacking because of the Civil War that led the armies of the North south, and the Southron kings have enough on their plate already (the Riverlands on fire, the Lannisters and Tyrells trying to consolidate their gains and plotting from King's Landing, the North in disarray, etc). It's a huge setup that is happening that is making Westeros ripe for invasion. The only armies still strong are the Dornish, those of the Vale, and the Ironborn fleet, the rest is a shadow of what it was.
About the recruiter, the travel times are large, and they are worried about that. He may be somewhere along the path, or have stopped at some halls in hopes of getting more recruits, or tons of other things may have hindered him, news don't travel like in modern times, and the Wall is very isolated. On top of that Lord Commander Mormont was killed, and there is no new Commander yet, which leads to all these issues becoming even bigger. They have Benjen Stark's disaparition, the attack of the Wildlings, the failure of the Great Ranging, etc all far more pressing than where the recruiter is.