Education Secretary Michael Gove has condemned holiday firms for trying to "fleece parents" by ramping up travel prices when schools are out.
His comments come as a poll for ITV News found a third of adults said they have gone on a family holiday during term time, when costs are significantly lower.
Mr Gove, though, said parents "should not take children out of school during the school term".
Instead, he said varying term dates across schools would help families benefit from off-peak travel prices.
More than two thirds of Britons have condemned the cost of travel during school breaks, an exclusive poll for ITV News has found.
The ComRes survey found only 13% of adults agreed that current travel prices during school holidays were "fair" - compared to 71% who disagreed with the statement.
However, the 2,052 Britons questioned were divided over whether parents should be able to take their children out of school at any time they like.
Half of Britons (49%) believed that was a step too far while just over a third (35%) thought parents should be free to travel with their children whenever they wanted.
A third of parents say they have taken their children out of school during term time for a family holiday, an exclusive poll for ITV News has found.
Half of parents of children under the age of 18 (51%) and 42% of the public in general also confirmed they would do the same to benefit from cheaper travel prices.
The ComRes poll of 2,052 adults was conducted as MPs prepare to debate whether holiday companies should do more to keep the price of family getaways down when school is out.
The poll found more than half of British adults (54%) disagreed that a £120 fine for taking a child out of school was reasonable, compared to 30% who agreed with the council-imposed penalty.
UK councils tell ITV News how many fines have been issued to parents for holidaying with their children during term time this school year.Read the full story ›
More than 166,000 people have backed the petition to stop travel companies raising their prices during school holidays which has triggered today's MPs debate.
The online campaign received a huge boost in support after a Facebook post by an angry parent complaining about being "ripped off" by steep rise in holiday prices outside term time went viral.
Paul Cookson, 41, said he was "sick to death" of the practice after being quoted £999 for a holiday that would have cost £699 during the school year.
Stewart Sutherland says people need to take children on holiday in term-time to save money and also because of the difficulty of taking leave during school holidays.
He says life is short and will be difficult for young people so they should have as much family time as they can.
The Government has come out in defence of the UK holiday industry, after a petition started by parents frustrated at the extra costs imposed on family getaways triggered a Parliamentary debate.
A spokesperson for the Department of Business said:
The UK holiday industry is an extremely competitive market and as a result the sector relies on the profits from peak periods to make sure it can trade throughout the year.
On top of this, the industry competes internationally against companies from other countries and as a result may need to increase prices during these periods.
If anyone suspects that companies are behaving anti-competitively, they should present the evidence to the Civil Aviation Authority (CAA) or the Office of Fair Trading (OFT).
MPs will debate the extra cost of a family holiday after an irate parent started a campaign to stop companies raising prices when school was out.
A petition inspired by Paul Cookson's initial rant climbed to 160,000 signatures earlier this month, triggering today's 4:30pm debate in Westminster Hall.
Father-of-one Cookson initially complained when he realised he would have to pay an extra £300 to take his family on holiday in the UK when term had ended.
In his complaint, which went viral, he said: “I am sick to death of being ripped off with this country.
I choose not to take my daughter out of school term away on holiday and stick to the set school holidays - but why should I be penalized by doing the correct thing as a parent??? [sic]
Rising inflation means that families will have to spend a little extra to book a getaway this half-term, the head of travel money at M&S Bank has said, as new figures reveal an average increase by 3.7% since this time last year.
Fraser Millar added:
Despite the euro strengthening against the pound over the past year, European destinations continue to prove relatively good value for money, and shorter flight times ensure the family has a little longer to enjoy the sun.
The average half-term trip has increased in price by 3.7% since this time last year, according to figures from M&S Bank.
How have some of the more popular destinations have managed to fare?
- An Egyptian trip has gone up by more than 6%
- France is up more than 5%
- Spain holidays have increased 4.9%
- The least-expensive trip is to Portugal where costs since last year have risen 3.08% to £576 or £2,303 for a family of four
- The most costly of the destinations surveyed is Mauritius - up 3.3% to £1,743 per person, or £6,972 for a family of four
- Only two of the most popular destinations - Greece (up 1.69%) and Cyprus (up 1.66%) have seen rises lower than the current UK inflation rate of 2.7%