The Palace of Ribeira Fortress was built by the Italian architect Filipo Terzi between 1581 and 1584, following orders from king Philip I of Portugal (II of Spain). It was a square building with three floors. At the top was the Ambassador's Room, thus designated as a reception area for important figures. It was a room with magnificent views of the river Tagus and decorated with a  coffered wooden dome painted with heraldic emblems, during the Archduke Albert's Government. In the reign of king João IV, this iconography was altered to mark the new reigning dynasty – the Braganzas. Episodes of the War of Restoration were recorded here, although the supporting evidence (canvas, tiles or tapestries) has yet to be identified. The second floor had several rooms, in which the Musical Bookstore of João IV was installed, transferred from the Palace of Vila Viçosa in 1647. Some rooms were designated for the conservation of books and manuscripts, others for reading and the practice of music. The former had oil panels with portraits of musicians commissioned in Flanders and the latter displayed a ceiling painted with frescoes, depicting musical instruments, by a well-known painter at the time, José de Avelar Rebelo.

The first floor had a very high ceiling, and so was divided into two floors. On the ground floor was situated the artillery, as this was the defensive area of the fortress. Above, the mezzanine corresponded to the lodging of the chief chamberlain of João IV – the Penaguião family, future Marquises of Abrantes.