The Manor House overlooks a busy harbour which, for hundreds of years, has been steeped in fishing and sea trading. The original cellars of the manor house date back to the 1600's with the present manor house being built in the early 1800's. The cellars have been used as the kitchen and staff quarters. A tunnel surrounds the cellars and this may have been used for smuggling.

Originally called "Waterloo Villa", later shortened to "Waterlooville", the manor house had a connection with Wellington and the Battle of Waterloo. Looe had a thriving boat building industry up to and including World War Two. The port was a busy outlet, shipping tin and sand that was brought down by rail. The railway that runs from Liskeard to Looe extended right onto the quay, obviously for the shipping of raw materials, but now stops before the bridge over Looe River.