Above the central door was the Byng family crest - onwers of the farm, who were based in Wrotham Park, South Mimms.

In the 1861 census, the occupant of Bullbaiters Farm was 60 year old William King, who farmed 190 acres and employed 3 men and 2 boys. In addition to his family two farm labourers also lived in the farm. Thrift Farm, nearby, was occupied by a farm labourer according to the census - so it may have been used as a farm cottage. Quite often, especially involving what had been smaller tenanted farms, the fields would be combined into a larger farm and the 'redundant' farm house used as farm cottages.