The Sun

The sun symbolizes the day-time. The speaker uses the sun to suggest that she loves her partner regardless of time and space, including by the natural light of the day.

Candle-Light

Candle-light symbolizes the evening. The speaker says that she loves her partner by the light of the sun (i.e., during the daytime hours) and by candle-light, suggesting that her love is constant and enduring.

Lost Saints

The lost saints to which the speaker alludes in Line 12 symbolize a wavering religious faith. The speaker says about her love, "I love thee with a love I seemed to lose /
With my lost saints" (11-12), suggesting that the religious faith she had in her childhood was eventually challenged or lost. She argues that her romantic love has miraculously restored that faith.