Joe Mitchell was born on April 24, 1930, and grew up in Northampton County, North Carolina. He graduated from Coach High School in nearby Seaboard and then moved to East Orange, New Jersey, where some of his brothers were living. He worked for an industrial plating company until the plant went on strike in 1957, and then returned to North Carolina. Mitchell moved to Greensboro in December 1958.
While seeking employment in Greensboro, Mitchell met North Carolina State A&T University professor W. M. Nelson, who encouraged him to get involved with civil rights organizations. Mitchell, who had been a member of the NAACP in since the age of 12, joined the Greensboro chapter in 1959, and was later appointed membership chairman. He joined the Greensboro Citizens Association in the mid-1960s. In 1962 Mitchell was appointed to the Voter Registration Commission and worked with that group until 1986. He was also active in efforts to elect Henry Frye to the state House of Representatives, the first African American from Guilford County to do so.