Sumpu Castle was built on a plain in a contoured style surrounded by three moats: an inner, central and out moat. THe castle was composed of a main enclosure (the Honmaru), surrounded by a secondary enclosure (the Ninomaru) and tertiary enclosure (the Sannomaru). On the northwest corner of the main enclosure there was donjon (Tenshukaku) with five external roofs and seven internal levels.

The HIgashi Gomon was important gateway located on the eastern side of the Ninomaru of Sumpu Castle. This square-shapted gate is composed of the Higashi Gomon Bridge, the Korai Gate, the Yagura Gate, all of which cross the Ninomarubori (the center moat) and the South and West Tamon Yagura. The gate was also called the "Tatewaki Mae Gomon" after Tatewaki Ando, whose residence was located in front of the gate. It had yet another name, "Jokei GOmon", after kitchen for chief retainers.

The HIgashi Gomon was burn down along with the Tenshukaku, the Goten, the Tasumi Yagura, and other building in 1635 and was reconstructed in 1638. The present reconstruction was carried out following the blue prints from the 1638 reconstruction.

Original Sumpu Castle was built by Imagawa families around 1400's. After Ieyasu Tokugawa gave his shognate to his sun, Hidetada Tokugawa, he moved here and rebuilt one of the great castle for himself. Up until the death of Ieyasu, Sumpu Castle is also sub capital which has strong power for Tokugawa government as a principal. This castle was politically important to control all the west land masters.

The Tenshukaku, main tower, of Sumpu Castle have been built three times. In 1585, Ieyasu built the first castle. The second by Ieyasu in 1605 after he turned over the shogunate to Hidetada, and the thrid was two years later after it burned down in a fire.

The castle lasted in this state until Meiji Period. Sumpu Castle is most well known as the "retirement castle" of Ieyasu after he relinquished the shogunate to Hidetada.