All Hallows-by-the-Tower was first established in 675 by the Saxon Abbey at Barking and was for many years named after the abbey, as All Hallows Barking. The church was built on the site of a former Roman building, traces of which have been discovered in the crypt. It was expanded and rebuilt several times between the 11th and 15th centuries. Its proximity to the Tower of London meant that it acquired royal connections, with Edward IV making it a royal chantry and the beheaded victims of Tower executions being sent for temporary burial at All Hallows.

The heart of Richard I is said to be buried somewhere in the north part of All Hallows' churchyard where a chapel once stood that had been built by Richard in the 12th century.

The church was badly damaged by an explosion in 1650 caused when some barrels of gunpowder being stored in the churchyard exploded; its west tower and some 50 nearby houses were destroyed, and there were many fatalities. The tower was rebuilt in 1658, the only example of work carried out on a church during the Commonwealth era of 1649-1660. It only narrowly survived the Great Fire of London in 1666 and owes its survival to Admiral William Penn, father of William Penn of Pennsylvania fame, who had his men from a nearby naval yard demolish the surrounding buildings to create firebreaks. During the Great Fire, Samuel Pepys climbed the church's spire to watch the progress of the blaze and what he described as 'the saddest sight of desolation'.

Restored in the late 19th century, All Hallows was gutted by German bombers during the Blitz in World War II and required extensive reconstruction, only being rededicated in 1957.

Its outer walls are 15th-century, with a 7th-century Saxon arch doorway surviving from the original church, which is the oldest piece of church material in London. Many brasses remain in the interior (where one of London's brass rubbing centres is now located). Three outstanding wooden statues of saints dating from the 15th and 16th centuries can also be found in the church, as can an exquisite Baptismal font cover which was carved in 1682 by Grinling Gibbons for £12, and which is regarded as one of the finest pieces of carving in London. In 1999 the AOC Archaeology Group excavated the cemetery and made many significant discoveries.

The church has a museum called the Crypt Museum, containing portions of a Roman pavement which together with many artifacts was discovered many feet below the church in 1926. The exhibits focus on the history of the church and the City of London, and include Saxon and religious artefacts. Also on display are the church's registers dating back to the 16th century, and notable entries include the baptism of William Penn, the marriage of John Quincy Adams, and the burial of Archbishop William Laud. Laud remained buried in a vault in the chapel for over 20 years; it was moved during the Restoration to St. John's College, Oxford.