Born Hyman Arluck in Buffalo, New York, the child of a Jewish cantor, he learned to play the piano as a youth, and formed a band as a young man. He achieved local success as a pianist and singer before moving to New York City in his early twenties, where he worked as an accompanist in vaudeville and changed his name to Harold Arlen. Between 1926 and about 1934, Arlen appeared occasionally as a band vocalist on records by artists, among others Eddie Duchin.

In 1929, Arlen composed his first well-known song: "Get Happy" (with lyrics by Ted Koehler). Throughout the early and mid-1930s, Arlen and Koehler wrote shows for the Cotton Club, a popular Harlem night club, as well as for Broadway musicals and Hollywood films. Arlen and Koehler's partnership resulted in a number of hit songs, including the familiar standards like "Stormy Weather.”

In the mid-1930s, Arlen married, and spent increasing time in California, writing for movie musicals. He began working with lyricist E. Y. "Yip" Harburg and in 1938, the team was hired by MGM to compose songs for The Wizard of Oz, the most famous of which is "Over the Rainbow", for which they won the Academy Award for Best Music, Original Song. He also wrote "The Man That Got Away”, both defining songs of Judy Garland's career.

In the 1940s, Arlen teamed up with lyricist Johnny Mercer, and continued to write hit songs like "That Old Black Magic", "Ac-Cent-Tchu-Ate the Positive", and "One for My Baby (and One More for the Road)."

Arlen died in 1986 of cancer at his Manhattan apartment at the age of eighty-one. Shortly before his death, Arlen adopted the 22 year old adult son of his brother Julius "Jerry" Arluck, so that his estate would have an heir in order to extend his copyright. Samuel Arlen runs the company that owns the rights to the Arlen catalog.