Jake enlists in the US army but deserts after it is clear that his role there is purely menial. After going to England and living with a white woman, he gets on a freighter to go back home. In Harlem, Jake picks up an attractive brown woman, who demands fifty dollars. After sleeping with this woman, outside Jake finds out that she has given him the money back. Jake, to his dismay, finds out that he has lost the woman's address.

Jake is with his old buddy from the army, Zeddy at the Congo cabaret. He is selected for attention by the attractive singer, Rose. Jake goes to live with Rose, but does not feel true love for her. Zeddy gets Jake a job on the docks, but Jake quits when he finds out that he is a strikebreaker. A couple of days later, at a gambling den, Zeddy is attacked by Nije for not paying his debt, and Zeddy almost knifes Nije before Jake stops Zeddy.

Zeddy gets his chance at Ginhead Susy's place on Myrtle Avenue. Zeddy ends up living with Susy, but soon resents that Susy has control over him because she pays his bills. Zeddy goes chasing after another woman, Susy finds out, and he is thrown out by her. Jake enjoys being with Rose, and gets the prestige to go to a buffet den connected to the owner of the Baltimore cabaret. Unfortunately, this illegal club is soon shut down by the Vice Squad, along with the Baltimore itself. Jake has his own problems, with evidence of Rose's lovers. Rose senses Jake's resentment and scratches at him. Jake slaps her and is disgusted by his own behavior, so he leaves Rose.

Jake gets a job on the railroad as a cook. There Jake meets the black Haitian intellectual, Ray who is now a waiter. Jake gets to have women in places like Pittsburgh, and also takes drugs, but Ray inadvertently overdoses and is sent to the hospital, though soon recovers. There is tension between Jake's chef and a pantryman. Ultimately, the pantryman gets the chef to have a hysterical breakdown in front of the white steward and be demoted from his post as chef. Ray tries to enjoy a stop at Madame Laura's in Philadelphia along with Jake, but cannot be with women that he sees as prostitutes. Back in New York, Jake becomes sick from drinking too much liquor. He drinks again one day and has a relapse. Jake gets visitors including Ray's girlfriend, and later a man called Yaller Prince, who is actually a pimp. Ray tells the story there of a pimp named Jerco who loved his woman and commits suicide after her death.

Jake is still on leave from the railroad and sees Yaller Prince assaulted by a man who backs up one of Yaller's former girls. Billy the Wolf tells Jake that he will give Jake a gun to defend himself. Then, Jake goes out to a cabaret, where the crowd dances to a wild jazz song. There to his surprise, Jake sees his long lost brown woman. Her name is Felice, and Jake and Felice spend a quiet week together. The next Saturday they go to a cabaret, but when Jake is busy, he hears Zeddy, her old boyfriend, trying to drag her away until Jake intervenes. Zeddy pulls out a razor, but is stopped by Jake's gun. Zeddy declares publicly that Jake is a deserter, and Felice is afraid that Jake will get in trouble. The reunited couple decides to immediately resettle in Chicago.