The talk about a physical or psychological 4-minute barrier has been repeated countless times over the decades, and it's nonsense.
For Bannister, in particular, the first four-minute mile wasn't a barrier; it was a multi-year obsession that focused not only on his own running and racing, but also on what all of his competitors were doing at any given time and how Bannister might be able to beat them out of the the big prize by scheduling some of the most ridiculous and unethical paced productions in the history of the sport. The nadir, I suppose, may have come when American Wes Santee was scheduled to run what might could have turned out to be the first sub-four mile on June 27, 1953, at a meet in Ohio. Bannister hastily arranged for a mile run for a few adults to be inserted into a schoolboy meet in the London area, with this adult mile to take place about five hours before Santee's race in Ohio. With his connections, Bannister arranged for Australian Olympian Don MacMillian (in London at the time) to pace the first half-mile, with Chris Brasher to jog the first lap in about two minutes so that he could take over as a fresh pacer for Bannister for the rest of the "race," even though Brasher was a lap down and would end up pacing Bannister to the end with his own four-minute 3/4 mile performance. Bannister ended up with a time of 4:02 for the mile, which would have a new British record, but the whole scheme was so obviously unethical that Bannister was simply disqualified for conduct unbecoming of the sport. After that embarrassment, Bannister and his athletic and journalistic supporters learned to tone down the crassness of their fraudulent "races" a bit, but if Bannister's May 1954 time trial had been as carefully scrutinized by the British athletic press as they would have for a sub-four by someone else -- like the American Santee, or the Australian Landy, or -- God forbid! -- an uneducated East African in a current or former British colony, I'm pretty sure that Bannister would not have become known as the first first four-minute miler.
I'll save my more general thoughts about Bannister, including his post-May 1954 behavior, for later. He's an interesting psychological study, whose reputation rests largely on a massive collection of exaggerations and outright falsehoods, and the central character in a myth that reflects much about the somewhat peculiar British class system and its underlying tenets.