John Gilmore was born in Summit, Mississippi, but grew up in Chicago and originally played the clarinet, taking up the tenor saxophone while serving in the United States Air Force from 1948–1952.
Gilmore played briefly with pianist Earl Hines, and in 1957 then pursued a musical career, playing briefly with pianist Earl Hines before encountering Sun Ra in 1953.
In 1957 he and Clifford Jordan recorded a Blue Note date that is regarded as a hard bop classic: Blowing In from Chicago. In the mid-1960s Gilmore toured with the Jazz Messengers and recorded with Paul Bley, Andrew Hill (Andrew! and Compulsion), Pete La Roca (Turkish Women at the Bath), McCoy Tyner (Today and Tomorrow) and a handful of others.
His main focus for the next four decades remained with the Sun Ra Arkestra. Many who noted Gilmore's talent, and thought he could be a major star like John Coltrane or Sonny Rollins, were puzzled by his dedication to Sun Ra.
Gilmore's explained that his devotion to Sun Ra was because of Ra's use of harmony. He said that Sun Ra was "more stretched out than Monk."
In 1990, he told Graham Lock in The Wire that "I'm not gonna run across anybody who's moving as fast as Sun Ra ... So I just stay where I am."
After Sun Ra's death in 1993, Gilmore led Ra's "Arkestra" for a few years before his own death from emphysema in 1995.