The castle has been dated to the late 12th century, when, as a result of the final loss of all Byzantine possessions in southern Italy, Corfu became the western border of the Byzantine Empire. However, another theory proposes an earlier construction phase in the 6th century, when residents of the city of Euroea in Epirus, fleeing barbaric invasions, settled in the area. Kassiopi Castle was permanently abandoned and deserted after the Ottoman raids of 1537 and 1571.

Before the Venetian conquest of the island and especially during the Venetian period, the harbour and town of Kassiopi frequently appeared in travel accounts, as it was a common stop for supplies. Especially from the 14th century onwards, the small church located southeast of the castle, dedicated to Panagia Kassopitra, became an important pilgrimage site, as mentioned in numerous pilgrim texts. According to their testimonies, the church housed a miraculous icon of the Virgin Mary, which at some point seems to have been attributed to Luke the Evangelist, along with an inextinguishable oil lamp whose oil was believed to cure fever and protect ships from rough seas. Many travellers also described Kassiopi Castle as abandoned or in ruins. In addition, they often recounted versions of a legend in which the city and castle were destroyed by a fire-breathing dragon.