The Conservative party will win back Rochester and Strood in next year's general election "when the future of the country is at stake," William Hague vowed on Good Morning Britain.
The Leader of the House of Commons admitted "it was always disappointing" not to win but maintained Ukip won "by less than half" of the votes they were predicted.
Labour's candidate Naushabah Khan said after the vote that she was proud of the positive campaign that the party had fought in Rochester and Strood.
Labour polled 6,713 votes - 16.7%.
The Conservative candidate, Kelly Tolhurst, says she will continue to fight UKIP up to the General Election - and fears that support for Nigel Farage's party will help Labour into power.
UKIP's newest MP Mark Reckless says that the result in Rochester and Strood gives him a solid platform for the General Election in May. This was his reaction immediately after the result was declared.
Mark Reckless won the Rochester and Strood seat with 16,867 votes - 42.1% of the total votes cast. This was his reaction the as the result was declared.
Mark Reckless will be returning to Parliament as a UKIP MP after securing an historic victory in the Rochester and Strood by-election.Read the full story ›
The Green Party outpolled the Lib Dems by almost five to one to move into fourth place at the Rochester and Strood by-election.
Green candidate Clive Gregory won 1,692 votes - almost trebling the party's share of the vote.
The Liberal Democrats continued a humiliating string of by-election performances. Their candidate Geoff Juby finished fifth with just 349 votes and lost his deposit.
A total of 40,113 votes were cast - a turnout of 50.67%.
In his victory speech, Mark Reckless said the large number of Labour voters who switched their support to him showed "the radical tradition, which has stood and spoken for the working class, has found a new home in UKIP"
"As Labour represents those comfortable at the top of the public sector, it is not Ed Miliband but UKIP that represents the concerns of most working men and women," he declared from the stage after a six-hour count.
It came after the Labour leader was forced to step in after the MP Emily Thornberry resigned from the shadow cabinet for a tweet she posted about the by-election.
She sparked complaints by posting an image of a modern terraced house with three red and white Cross of St George flags and a white van in the drive along with the message "Image from Rochester".
After a social media and political backlash, she apologised after being given a dressing-down by leader Ed Miliband and Labour later announced that she had resigned her shadow cabinet position.
The owner of the house, Dan Ware, told The Sun: "I've not got a clue who she is, but she's a snob."
He said the flags had been left up since being raised when England played in the football World Cup in May.
Conservative Party chairman Grant Shapps said he was "disappointed with the result" of the Rochester and Strood by-election and added it would now be harder to do the things the party wanted on controlling immigration and the economy.
But he said the narrower-than-predicted margin of victory for UKIP meant Tory candidate Kelly Tolhurst was well-placed to win back the seat for the Tories in the General Election on May 7.
"Over the course of this campaign the gap has closed. They have ended up with about a 7% lead for Mark Reckless. They were predicting something over twice that level. So 2,900 is not a big majority now to try to win back in 170 days time," he said.
Mark Reckless has won the Rochester and Strood by-election for Ukip to become the country's second Ukip MP. He defected from the Tories last month.