Born in Chicago, Gorey came from a colourful family; his parents, Helen Dunham Garvey and Edward Lee Gorey, divorced in 1936 when he was 11, then remarried in 1952 when he was 27. One of his step-mothers was Corinna Mura, a cabaret singer who had a brief role in the classic film Casablanca. His father was briefly a journalist. Gorey's maternal great-grandmother, Helen St. John Garvey, was a popular 19th century greeting card writer/artist, from whom he claimed to have inherited his talents. He attended a variety of local grade schools and then the Francis W. Parker School. He spent 1944–1946 in the Army at Dugway Proving Ground in Utah, and then attended Harvard University from 1946 to 1950, where he studied French and roomed with future poet Frank O'Hara.

Although he would frequently state that his formal art training was "negligible", Gorey studied art for one semester at The School of The Art Institute of Chicago in 1943, eventually becoming a professional illustrator. From 1953 to 1960, he lived in New York City and worked for the Art Department of Doubleday Anchor, illustrating book covers and in some cases adding illustrations to the text. He has illustrated works as diverse as Dracula by Bram Stoker, The War of the Worlds by H. G. Wells, and Old Possum's Book of Practical Cats by T. S. Eliot. In later years he illustrated many children's books by John Bellairs, as well as books in several series begun by Bellairs and continued by other authors after his death.