"The castle was sited above the River Usk overlooking the river valley and the confluence of the River Gavenny with the River Usk on a site that would have been naturally defensible in earlier times and may well have been a fortified site from Bronze Age and Iron Age periods and was then also favoured by the Romans who built their Roman fort of Gobannium on the same hilltop just a little to the west of the site later developed into the castle. With steep slopes down to the river on three sides the remaining side was where the town under the protection of the castle developed, over what must have been the Roman fort and settlement site. The main castle gatehouse faced the town, which was later walled.
The castle has Norman origins: the early motte was recorded as being built by Hamelin de Balun as first holder of the feudal title Baron Bergavenny at the order of William the Conqueror in 1075. Hamelin is therefore thought of as the first holder of the title Baron Abergavenny.
In 1233 AD the castle was sacked by Richard Marshal, 3rd Earl of Pembroke during his alliance with the Welsh leader Llywelyn the Great.
In 1175 Abergavenny Castle was the scene of an infamous act: the Massacre of Abergavenny. Henry Fitzmiles, the third son of Miles de Gloucester, 1st Earl of Hereford had been killed by a Welsh leader, thought to be Seisill ap Dyfnwal of Castell Arnallt in 1165. As there were no other male heirs, his brothers all having died or been killed without issue, Abergavenny castle and lands in Brecknockshire and Upper Gwent passed via his sister Bertha, who was a daughter of Miles de Gloucester, 1st Earl of Hereford, to his apparent step-nephew William de Braose, 3rd Lord of Bramber.
William de Braose, 3rd Lord of Bramber Lord of Abergavenny decided to avenge the death of his uncle Henry. He summoned Seisill ap Dyfnwal, his eldest son Geoffrey, and a number of other local leading Welshmen from Gwent to Abergavenny Castle to hear a royal proclamation. He then had them put to death when they did not like the proclamation they heard. William was 'retired' from public life for this abuse of a royal safe-conduct and Abergavenny was taken over by his son and heir, another William de Braose, 4th Lord of Bramber.
In 1182 Abergavenny castle was attacked after William de Braose had been defeated in battle and Dingestow castle towards Monmouth destroyed.
Control of the castle passed back and forth during the turbulent years as the Welsh Marches changed hands in the 12th century between English and Welsh forces. During the thirteenth and 14th centuries a huge amount of building work was undertaken on the castle whilst it was in the hands of the Hastings family. The most prominent features that remain from this period are the towers in the western side of the castle.
During the rebellion of Owain Glyndwr in the early 15th century the town of Abergavenny was sacked and burned by Welsh forces in 1402, as were other English held settlements in Monmouthshire such as nearby Grosmont and Crickhowell as the rebellion gained momentum, however the castle itself didn't fall as it was capable of defence against an infantry attack and could have at this stage withstood a siege if necessary. A small garrison of archers and men-at-arms could defend a castle against an army and such an arrangement usually deterred any attack. Chepstow Castle, although a much larger castle, had a garrison at this time of twenty men-at-arms and sixty archers and was avoided by Glyndwr's forces altogether although they successfully sacked Newport Castle. The gatehouse of Abergavenny, and the youngest part of the castle, dates from 1400 and was probably a measure taken to further fortify the castle in the light of the rebellion.
The stone keep, along with most of the other internal castle buildings, shell keep and towers, was damaged badly in the Civil War when the castle suffered slighting to prevent it becoming a stronghold subsequently. The ruins you see today are the result of much of this planned damage . The English Civil War in Wales saw Raglan Castle also significantly damaged in this way.
In the 19th century, the present square 'keep' building - now housing the Abergavenny Museum - was constructed on top of the motte as a hunting lodge for the Marquess of Abergavenny in 1819. The grounds are laid to lawn and are used for events such as open air plays and historical re-enactments including the celebrations of the castles founding back in 1990 when 900 years of history was marked by the people of Abergavenny." - Wikipedia