Andy Burnham said the former health secretary's decision to appear on the show demonstrated "bad judgement".
It comes as Hancock avoided elimination for the sixth time, after radio DJ Chris Moyles was voted off on the show on Thursday 24 November.
Mr Burnham said: "He’s clearly better at showbiz than he is at politics.
"He’s not, in my view, a bad person, but it is a bad judgement because politicians should be about we – what’s good for us.
"But this is all about me, isn’t it, me, me and my situation and a cost-of-living crisis for his constituents.
"There are things that he should be here answering for, not in the jungle trying to curry favour with other celebrities and the British public."
Hancock held the post of health secretary during the majority of the coronavirus pandemic, during which time the vaccine was developed and tested.
After making his exit from the jungle, Chris Moyles told hosts Ant McPartlin and Declan Donnelly he was "gutted" that Hancock was more popular than him saying "what the bejesus is going on?"
Moyles admitted it was difficult for him living alongside the former health secretary in camp, saying: "I really struggled with it, to be fair.
"And then what I did was, because I slagged him a bit, I separated Matt Hancock with Matt because we were living with this guy, and he was doing his best to win us stars for food and everything and chipping in so I had to separate the two.
"Matt Hancock I’ve got major issues with but Matt – less issues."
Hancock has also faced criticism from across the political divide for joining the reality show at a time when Parliament is sitting.
Listen to the ITV News' arts and entertainment podcast, Unscripted...