Bailudong Academy is located at the southern foot of Houping Mountain, 10 kilometers south of Wulao Peak at Lushan Mountain.

During the Tang Dynasty (AD 618-907), the poet Li Bo and his brother settled there in seclusion. After being appointed governor of Jiangzhou prefecture, Li revisited the site and ordered the construction of buildings and pavilions.

The site then became a tourist attraction named Bailudong. During the early Song Dynasty, it was extended to include an academy and was formally called Bailudong Academy.

As the first comprehensive academy in Chinese history, the Bailudong Academy was reputed as the leader of the Four Academies in China along with the Yuelu, Juyang and Shigu Academies. Many renowned scholars of the Song Dynasty, such as Zhu Xi, gave classes there.