After Cambridge and ordination, Atlay served in parish ministry in Nottinghamshire and near Cambridge, but it was as a tutor at his old college, St John’s, that he showed his particular skills as an educator of the young. This skill he extended in his next post as vicar of Leeds. Here, Atlay continued the distinguished ministry of his predecessor, Walter Hook, training many curates and running his parish with huge care and efficiency. In 1867 he refused the bishopric of Calcutta, but the following year he accepted Disraeli’s offer of the bishopric of Hereford, in succession to Renn Dickson Hampden. Atlay brought to the diocese the same thoroughness which had been so successful at Leeds and Cambridge. Except for his times of duty in the House of Lords, he rarely left the diocese but spent all his time focused on his clergy and their parishes. He is said to have visited every church in the diocese and certainly worked tirelessly for the improvement of clergy incomes and accommodation. Education was of great importance to him – he supported its extension in the diocese and was a great champion of girls’ schooling – an area often neglected in his rural diocese. His own family – he had eleven children – gave him a great sympathy for young people and Archbishop Benson summed up the views of both young and old when he described Atlay as ‘the most beautiful combination of enthusiasm, manliness and modesty’. He is buried in the Lady Arbour of Hereford Cathedral and there is a fine marble recumbent effigy of him in the north transept of the cathedral.