The Norwegian influence is still discernible in place and personal names, in the dialect words used by the inhabitants, and in the store of myth and legend, most noticeably collected in the
Orkneyinga Saga, a sequence of tales written around 1200. Orkney also has extensive prehistoric remains, mostly of the Neolithic era. These rich resources form the foundation of all of Brown's work.
Brown was born on 17 October 1921 into a poor family: his father, John Brown, was a postman, and his mother, Mhairi Mackay Brown, worked in the local hotel. Brown attended the local school, Stromness Academy, where he discovered his talent for writing in the weekly "compositions" set by his English teacher. Brown writes engagingly in his autobiography, For the Islands I Sing (1997), of his voracious reading as a child, moving from the ripping yarns in comics he read as a young boy to his discovery as an adolescent of Romantic poetry. His school education ended when he contracted tuberculosis and was sent to a sanatorium. He never fully recovered his health, but the enforced leisure enabled him to read and develop his literary taste. The illness also meant he could not serve in the forces in World War II; moreover, he was virtually unemployable.