An inscription in stone on the south west face of the building tells us:
This building was erected MDCCLXXXIV by the representative of the late
Sir William James, Bart
to commemorate that gallant officer's atchievements (sic) in the East Indies
during his command of the Company's Marine Forces in those seas
and in a particular manner to record the conquest
off the Coast of Malabar
which fell to his superior Valour and able Conduct
on the 2nd day of April MDCCLV
Wikipedia has an article about Severndroog Castle that tells us:
Severndroog Castle is a folly situated in Oxleas Wood, on Shooter's Hill in south-east London in the Royal Borough of Greenwich. It was designed by architect Richard Jupp, with the first stone laid on 2 April 1784.
It was built to commemorate Commodore Sir William James who, in April 1755, attacked and destroyed the island fortress of Suvarnadurg (then rendered in English: Severndroog) of the Maratha Empire on the western coast of India, between Mumbai and Goa. James died in 1783 and the castle was built as a memorial to him by his widow, Lady James of Eltham.
Designated a Grade II* listed building in 1954, the Gothic-style castle is 63 feet (19 m) high and triangular in section, with a hexagonal turret at each corner. From its elevated position, it offers views across London, with features in seven different counties visible on a clear day.
The tower was used by General William Roy in his trigonometric survey linking the nearby Royal Greenwich Observatory with the Paris Observatory; a 36-inch theodolite (now in London's Science Museum) was temporarily installed on its roof. This Anglo-French Survey (1784–1790) led to the formation of the Ordnance Survey. In 1848, the Royal Engineers used the castle for their survey of London.
Following Lady James' death in 1798, the building passed through the hands of various landowners, including John Blades, a former Sheriff of London, a Mr Barlow (ship owner) who built nearby Castle Wood House, and Thomas Jackson (a railway and docks contractor of Eltham Park). In 1922, the tower was purchased by London County Council and it became a local visitor attraction with a ground-floor tearoom serving drinks and cakes. In 1986, when the GLC was abolished, responsibility for Severndroog passed to Greenwich Council.
In 1988, the local council could no longer afford the building's upkeep and it was boarded up. In 2002, a community group, the Severndroog Castle Building Preservation Trust, was established. In 2004, it featured in the BBC TV series Restoration (presented by Griff Rhys Jones, Ptolemy Dean and Marianne Suhr, producer-director Paul Coueslant) - with the aim of gaining support for a programme of work to restore the building and open it to the public.
In July 2013 work began on renovating the castle, funded by a £595,000 Heritage Lottery Fund grant, and it was officially reopened to the public on 20 July 2014.
The Green Chain Walk and Capital Ring long-distance paths go through Eltham Common and Castle Wood and past the castle between Shooter's Hill and Eltham.
The entire castle, or just the William James room on the second floor, can be hired for weddings, functions, private events and meetings. The castle is open to the public.
As mentioned, the Castle is Grade II* listed with the entry at the Historic England website telling us:
1784. 3-storey triangular tower with hexagonal angle turrets of ½-storey height. Tower and turrets battlemented. Multicoloured stock brick with 1st floor band and plinth. Quatrefoil blocked and painted windows in turrets. Other windows have 2-centred, pointed arches, wide on the main faces and narrow in the 3 external faces of each turret. Moulded labels over windows, some of which are blocked and treated in trompe l'oeil fashion as in turret tops. 1st floor windows in wide, stuccoed recesses. Small door with alternating block surround in inner return of North-east turret. Later doors inserted in North-West and South turrets. Main, studded double door, in South-West face, has moulded surround in deep coving, with label over. Cornice head and fanlight with interlacing bars matching the window heads. On the other 2 sides the former entrances blocked, fanlights only remaining.
The PMSA website also tells us:
Severndroog Castle is a tall, three-sided battlemented tower, three storeys high, with a hexagonal turret on each corner and Gothic doorways and windows. Some windows, for example in the turrets, are false and painted in.
The tower is situated in Castle Woods high up on Shooters Hill. It is surrounded by trees now, which limit the view, but was originally used as a belvedere, and from the top of the tower there are still spectacular views right across London north and south, and across Essex and Kent.
James's widow had the tower built as a memorial to her husband and apparently it could at that time be seen from her home at Park Farm Place. It was designed by Richard Jupp, a surveyor to the East India Company, and built on land owned by the parish of Eltham. She gave £500 in return for the use of the land, of which the interest was to be used to provide coal for the poor of Eltham. It was put up in 1784 and became known as Severndroog Castle.
Modelled on Isaac Ware's Shrub Hill, Windsor, according to Pevsner.
Castlewood grounds were acquired by the London County Council in 1922 and are now owned by Greenwich Borough Council. It was open part-time to the public until around ten year ago, with a small cafe downstairs and access to the roof for the views. It has unfortunately been closed and empty for some years and subject to the usual neglect and vandalism. Current council plans are to sell it off as offices with occasional public access [April 2002].
The tower was built in memory of Sir William James (1721-83), who lived nearby at Park Farm Place. It commemorates his victory in 1755 at Severndroog Island off the Malabar Coast of India, which cleared the area of the pirates who were operating from their stronghold there. He was born around 1721 in Wales and went to sea firstly on coastal vessels and later to the West Indies. In 1747 he joined the East India Company, became a captain, and patrolled the coast between Bombay and Goa to protect trading vessels from Arab pirates. One notorious one was Tollagee Angria who held the island fortress of Severndroog. In 1755, as Commander of the East India Company's Bombay Marine ship Protector, and supported by the local ruler, he exceeded his orders to blockade the pirate's stronghold, and instead launched an attack and destroyed them. He was able to get in close enough to blow up the fortress because he had previously taken soundings along the rocky coastline. The Company, who had been spending considerable amounts of money providing protection from piracy, gave him only £100 as a reward, but he did gain a small fortune from prize money in his skirmishes with the French in the area, and helped consolidate the British position in India.
He built a house in Eltham when he returned to England. He became a Director of the East India Company and a Governor of Greenwich Hospital, and was made a member of the Royal Society for his contribution to navigation. He died 16 December 1783.