For more than three hundred years Downing Street has held the official residences of the First Lord of the Treasury, an office now synonymous with that of Prime Minister, and the Second Lord of the Treasury, an office held by the Chancellor of the Exchequer. The Prime Minister’s official residence is 10 Downing Street; the Chancellor’s official residence is next door at Number 11. The government’s Chief Whip has an official residence at Number 12, although the current Chief Whip’s residence is at Number 9.

The street was built in the 1680s by Sir George Downing, 1st Baronet, on the site of a mansion, Hampden House. What was on the site before the mansion is vague, but there is evidence towards a brewhouse called The Axe, owned by the Abbey of Abingdon. Downing was a soldier and diplomat who served under Oliver Cromwell and King Charles II, and who invested in properties and acquired considerable wealth. In 1654, he purchased the lease on land east of Saint James’s Park, adjacent to the House at the Back, and within walking distance of parliament. Downing planned to build a row of townhouses "for persons of good quality to inhabit". However, the Hampden family had a lease which prevented their construction for 30 years. When the Hampden lease expired, Downing received permission to build further west to take advantage of recent developments. The new warrant issued in 1682 reads: "Sir George Downing ... is authorised to build new and more houses further westward on the grounds granted him by the patent of 1663/4 Feb. 23. The present grant is by reason that the said Cockpit or the greater part thereof is since demolished; but it is to be subject to the proviso that it be not built any nearer than 14 feet of the wall of the said Park at the West end thereof."

Between 1682 and 1684, Downing built the cul-de-sac of two-storey townhouses with coach-houses, stables and views of St James’s Park. How many he built is not clear; most historians say 15, others say 20. The addresses changed several times; Number 10 was numbered 5 for a while, and was renumbered in 1787. Downing employed Sir Christopher Wren to design the houses. Although large, they were put up quickly and cheaply on soft soil with shallow foundations. The fronts had facades with lines painted on the surface imitating brick mortar. Winston Churchill wrote that Number 10 was "shaky and lightly built by the profiteering contractor whose name they bear".

The upper end of the Downing Street cul-de-sac closed access to St James’s Park, making the street quiet and private. An advertisement in 1720 described it as "a pretty open Place, especially at the upper end, where are four or five very large and well-built Houses, fit for Persons of Honour and Quality; each House having a pleasant Prospect into St James’s Park, with a Tarras Walk". The houses had several distinguished residents. The Countess of Yarmouth lived at Number 10 between 1688 and 1689, Lord Lansdowne from 1692 to 1696 and the Earl of Grantham from 1699 to 1703. The diarist James Boswell took rooms in Downing Street during his stay in London during 1762–63 at a rent of £22 per annum. He records having dealings with prostitutes in the adjacent park.

Downing probably never lived in his townhouses. In 1675 he retired to Cambridge, where he died a few months after the houses were completed. His portrait hangs in the entrance foyer of the modern Number 10.

The Downing family also built Downing College, Cambridge, which was established in 1800, after its founder Sir George Downing, 3rd Baronet, left a portion of his estate to establish Downing College when the land became available. (A door from Number 10 is in use in the college).

The houses between Number 10 and Whitehall were acquired by the government and demolished in 1824 to allow the construction of the Privy Council Office, Board of Trade and Treasury offices. In 1861 the houses on the south side of Downing Street were replaced by purpose-built government offices for the Foreign Office, India Office, Colonial Office, and the Home Office.

Barriers were erected at the St James’s Park end of the street for the unveiling of the Cenotaph on 11 November 1920. They were a public safety measure intended to prevent the crowds in Whitehall becoming too dense.

When the movement for Irish independence became increasingly violent, it was decided to retain the barriers, which were raised and strengthened. On 26 November 1920 construction commenced on a wooden barricade, 8 feet (2.4 m) high at the end of the street. They were described as being of a "substantial character" mounted on proper foundations and incorporated vehicle gates. The barriers were taken down in 1922 when the Irish Free State was created.

Vehicle access was curtailed in 1973 when metal barriers were placed across the entrance to the street. In 1974, the Metropolitan Police proposed erecting a semi-permanent barrier between the pavement and carriageway on the Foreign Office side to keep pedestrians off the main part of the street. The proposal came with assurances that tourists would still be permitted to take photographs at the door of Number 10. The Prime Minister, Harold Wilson, rejected the proposal, feeling that it would appear to be an unacceptable restriction of the freedom of the public. Wilson’s private secretary wrote "I much regret this further erosion of the Englishman’s right to wander at will in Downing Street."

In 1982 access was further restricted by railings and a demountable gate. They were replaced by black steel gates in 1989. The increase in security was due to an increase in violence, particularly by the IRA during The Troubles.