The constituency, which voted 60.9% for Leave in the 2016 referendum, had been regarded as potentially fertile ground for the new party.
But with a relatively high turnout for a by-election of 48.4%, it appears that they were unable to match Labour’s organisation on the ground.
Jeremy Corbyn warned voters to write off Labour "at your peril" after the party by-election.
Speaking alongside the newly elected Lisa Forbes, Mr Corbyn told supporters his party offered "hope".
He said: "What we did was offer the politics of hope, not the politics of fear.
"We offered the politics of hope to end austerity, to fund our schools properly, to employ our police properly, to give our young people a future in this country.
"Everybody, all the experts wrote Lisa off. All the experts wrote Labour off yesterday. Write Labour off at your peril."
Brexit Party leader Nigel Farage said the message to voters who wanted Brexit was to think tactically in future to ensure his party succeeded.
He admitted it would "take time" for people to vote for The Brexit Party.
Mr Farage, who posted a letter at 10 Downing Street this morning outlining his Brexit demands, said: "The thing we failed to do was to persuade enough Conservative voters that by voting Conservative, they got Labour.
"By voting Conservative, they got a non-Brexiteer to be elected. That's going to take people some time for people to understand that there are parts of the country where the Brexit Party is the only challenge to Labour."
Mr Farage made a brief appearance at the count in a sign that he may have been expecting victory - but was left disappointed.
But he later said the result showed that his party had shaken up British politics and called for Tories to vote tactically for the Brexit Party in future Westminster contests.
Watch the declaration of the Peterborough by-election result
Jeremy Corbyn said it was a “great win” and represented a rejection of the Conservatives’ “disastrous” handling of Brexit.
“In this key seat, the Conservatives have been pushed to the margins,” he said.
“This result shows that in spite of the divisions and deadlock over Brexit, when it comes to a vote on the issues that directly affect people’s lives, Labour’s case for real change has strong support across the country.”
Shadow chancellor John McDonnell tweeted that it was “great to send Farage sneaking out by the back door, puncturing the Brexit Party’s balloon. Well done Lisa Forbes.”
Labour secured 10,484 votes to the Brexit Party’s 9,801, with the Tories on 7,243 votes in a seat which has traditionally been a two-way Conservative-Labour marginal.
The Liberal Democrats were fourth with 4,159, while the Greens came fifth with 1,035 votes.
However, Labour’s vote share fell by more than 17 percentage points from the 2017 general election, while the Conservatives were down by more than 25 percentage points.
In her victory address, the winning Labour candidate Lisa Forbes hailed the result as a defeat for the Brexit Party’s “politics of division”.
“Despite the differing opinions across our city, the fact that the Brexit Party have been rejected here in Peterborough shows that the politics of division will not win,” she said.
The defeated Brexit Party candidate Mike Greene admitted they had been unable to match Labour’s established organisation in the constituency.
However, he insisted that they would continue to build on the momentum from their triumph in the European elections when they topped the poll.
“Two parties have been ruling this country for decades. That is not happening any more. We were ahead of the Tories, only 683 votes behind Labour,” he told Sky News.
“They have decades of data. We had nothing just four weeks ago. We did not have the vote numbers they had. We didn’t have the voting history they have.
“We will be back. Let’s see what does happen in the next general election.”
Onasanya was elected as a Labour MP in 2017 with a wafer-thin majority of 607 but was suspended from the party after she was sentenced.
Among local Tory activists there was frustration that the continuing turmoil over Brexit meant they were unable to take advantage of Labour’s difficulties.
The chairman of the Peterborough Conservative Association Wayne Fitzgerald said it was now essential the party committed to taking Britain out of the EU, even if it meant no deal.
“Parliamentarians in Westminster are against the will of the people,” he told Sky News.
“The Conservatives must de-select every MP who will not accept a WTO (World Trade Organisation) Brexit if it comes to that.
“If Boris (Johnson) or whoever is leader doesn’t do that, Mr Farage will sweep to 450 seats in the next general election.”
Former Brexit minister David Jones gave a succinct assessment of the by-election result.
The Tory MP tweeted: “Message from Peterborough: no Brexit = Corbyn.”
Analysis by ITV News Elections Analyst Professor Jane Green
Can the Brexit Party – just weeks old – get MPs in Parliament and threaten the governing chances of either Labour or the Conservatives?
This was the big question after the June European Parliament elections, when the Brexit Party gained the highest share of votes and MEPs. Could they translate that success to be an electoral threat in Westminster?
The Peterborough by-election was the first test of this question. The jury is still out, but there is considerable potential.
The Brexit Party failed to win in Peterborough, with Labour holding on instead. However, it was close. The Brexit Party vote share was 29%. Labour’s was 31%. Moreover, the Conservatives were down substantially; a full 25.5 percentage points on their performance in 2017. If we compare the Brexit Party’s success in Peterborough to the last time UKIP was riding high in the polls (2015), the Brexit Party almost doubled that share of the vote in the same constituency.
By-elections aren’t directly comparable. They are characterised by lower turnouts, a tendency to punish the party in power, and this one was fought just after the European Parliament elections in the context of a government and parliament paralysed by Brexit.
Nevertheless, British voters are showing a tendency to reach either for the main Brexit alternative, or – in other contests – for parties that support Remain. Over time, British voters have shown a growing tendency to reach for ‘other’ parties (than Labour and the Conservatives).
The major parties’ shares were declining considerably in general elections over time (except only in 2017), and also in European Parliament elections over time. The electorate is more volatile, more likely to reject the two main parties if they don’t like the choices on offer.
British Election Study data shows us that more and more people are switching their vote choices between elections than ever before, and fewer and fewer people now hold an attachment to political parties that make their votes more consistent over elections.
The lesson from the European Parliament election was that no political party can afford to take any of their voters for granted. The Peterborough by-election only serves to confirm that lesson further.