The castle was originally built in 1050 by Gundulf, Bishop of Rochester and later entrusted to the Archbishop of Canterbury, who begun to build the keep (the tallest medieval building to survive in Europe). It stands on the east bank of the River Medway.