What does James remember as they all sit in the hot sun on the boat?

He remembers his father telling him that it would rain and he wouldn't be able to go to the lighthouse.