What is the main conflict in The Sea, the Sea by Iris Murdoch?

A series of letters and uninvited visitors repeatedly disrupts Charles' refuge, the most disturbing of which involve previous lovers. After a particularly emotional encounter, Charles suddenly realizes that a woman almost run down by the departing visitor's car is in fact his childhood sweetheart, Hartley.