Who did Maddie and Ellis argue with the day after the party?

The day after the disastrous party, Maddis and Ellis argue with his parents. At the end of the argument, Ellis brings up the photo and his father is furious. Maddie and Ellis are banished and they move to a tiny apartment, the best they can afford on their limited resources.