Jongen was born in Liège, where his parents had moved from Flanders. On the strength of an amazing precocity for music, he was admitted to the Liège Conservatoire at the extraordinarily young age of seven, and spent the next sixteen years there. Jongen won a First Prize for Fugue in 1895, an honors diploma in piano the next year, and another for organ in 1896. In 1897, he won the Belgian Prix de Rome, which allowed him to travel to Italy, Germany and France.
He began composing at the age of 13, and immediately exhibited exceptional talent in that field too. By the time he published his Opus 1, he already had dozens of works to his credit. His monumental and massive First String Quartet was composed in 1894 and was submitted for the annual competition for fine arts held by the Royal Academy of Belgium, where it was awarded the top prize by the jury.
In 1902, he returned to his native land, and in the following year he was named a professor of harmony and counterpoint at his old Liège college. With the outbreak of World War I, he and his family moved to England, where he founded a piano quartet. When peace returned, he came back to Belgium and was named professor of fugue at the Royal Conservatory of Brussels. From 1925 until 1939, he served as director of that institution; he was succeeded by his brother Léon Jongen. Fourteen years after leaving the directorship, Joseph Jongen died at Sart-lez-Spa, Belgium.