Rolling Stone has a good interview with George R. R. Martin, author of the Game of Thrones series. You should read it here, but only if you don't want spoilers on the show so far.
One passage stood out:
How did you come up with the Wall?
The Wall predates anything else. I can trace back the inspiration for that to 1981. I was in England visiting a friend, and as we approached the border of England and Scotland, we stopped to see Hadrian's Wall. I stood up there and I tried to imagine what it was like to be a Roman legionary, standing on this wall, looking at these distant hills. It was a very profound feeling. For the Romans at that time, this was the end of civilization; it was the end of the world. We know that there were Scots beyond the hills, but they didn't know that. It could have been any kind of monster. It was the sense of this barrier against dark forces – it planted something in me. But when you write fantasy, everything is bigger and more colorful, so I took the Wall and made it three times as long and 700 feet high, and made it out of ice.
As it happens, I had this exact experience in 2006, albeit without coming up with a major plot element for a novel in the process. Being a big history guy, I drove out of my way to visit the wall, built by Emperor Hadrian in the first century, that kept out the marauding Picts out of the old Roman Empire. Despite the sheep milling about me, I felt just the same emotions as Martin. You're looking down into a valley and some dark woods that you must have known held all manner of treacherous, vicious barbarians, and you were the watcher on the wall, keeping civilization safe. The sense of historical import was weighty.
I thought I had a photo of the exact view, but it may not have turned out. This is from the site, though: