The glass castle in the title (2005) can be seen as a promise Jeannette’s father breaks throughout their life together, symbolic of the neglect and suffering the children endure throughout their childhood, whether in the deserts of Arizona or a bleak mining town in Appalachia. Despite the broken dreams, the hunger, the cold, the flights from the law, their father’s alcoholism, they are none the less raised in an atmosphere of continuing love, a fact which may have enabled them to rise above their upbringing and make good lives for themselves.
I would say the glass castle is representative of Rex's engineering and mathematical genius - he was so far ahead of his time! Yet it is also representative of his inability to be a proper father-figure to his children and just as they gradually become disillusioned as to his ability to provide properly for them (physically, emotionally, in a nurturing manner), so too do they realize that his plans for the glass castle are simply another promise that he won't be able to keep. We talk of building 'castles' in the air, and speak of 'empty promises' --- that is what the glass castle is: a lovely dream, but ultimately empty.
The glass caslte is a symbol shown, yes, by Jeannette's poor providing father, as stated in previous answers. The plans for the glass castle gives the children the feeling that their father is an engenering genious, and an amazing person, but the glass castle is infact made of glass (which is very fragile) and proves how great and extravagent Rex Walls's promises were and how fragile they were, which means that you wouldn't be able to be one to take his word for anything. The glass castle also symbolizes hope for the children in the family. It symbolizes that one day their lives will become pure as glass and they will live as if in a fantasy castle built for only the finest of rullers.