He delighted tourists on New Year's Eve when he took a stroll through the Square, but he was taken by surprise as he was dragged back by a woman eager to hold his hand.
Francis exclaimed and then struck the woman's hand twice to free himself.
It is not the first time the Pope has reprimanded worshippers for their heavy-handedness.
He could be seen repeatedly pulling his hand away as people try to kiss his papal ring during a visit to an Italian pilgrimage site.
He later set the record straight, explaining why he refused to let the long line of worshippers kiss his ring - the reason was "very simple: hygiene", a spokesperson said.
Vatican spokesman Alessandro Gisotti explained the footage, saying the Pope was concerned about hygiene when he began pulling his hand away to discourage people from kissing his ring.
"The Holy Father told me that the motivation was very simple: hygiene," Gisotti said to reporters, adding: "He wants to avoid the risk of contagion for the people, not for him."
At the New Year's Eve Vespers service in St. Peter's Basilica, Francis urged people to practice more solidarity.
Since becoming pontiff in 2013, Francis has preached openness - a reform-minded agenda that has irritated a small but vocal group of ultra-conservatives in the church.
The Pope spoke of the city of Rome during his last homily of 2019 and warmly greeted Virginia Raggi, the mayor of the Italian capital at the end of the Vespers.
Vespers is a sunset evening prayer service for Roman Catholics.