Israel said its troops were defending its border and accused Hamas militants of trying to attack under the cover of the protest.
It was not immediately clear what might come out of the council session, which had been called for by Kuwait.
No joint statement or action followed a similar meeting after protests in March, and two UN diplomats said members could not reach unanimous agreement on Monday on issuing a proposed statement circulated by Kuwait.
The draft statement, obtained by The Associated Press, would have expressed “outrage and sorrow” at the killings, sought an “independent and transparent investigation”, and called on all sides to exercise restraint.
It also would have demanded that all countries comply with a decades-old Security Council resolution that called on them not to have embassies in contested Jerusalem. The United States inaugurated its embassy there on Monday.
The US vetoed a Security Council resolution in December that would have required president Donald Trump to rescind his declaration of Jerusalem as Israel’s capital.
Earlier on Monday, the Palestinian UN envoy urged the United Nations’ most powerful body to condemn the killings. Israel, meanwhile, called on the council to condemn Hamas, the Islamic militant group that rules the coastal strip and led the protests.
Israel is not a council member, nor are the Palestinians.
Monday’s violence came as the US and Israel celebrated the embassy opening, the first official recognition of Jerusalem as Israel’s capital after 70 years. The move infuriated Palestinians, who seek eastern Jerusalem as a future capital of their own.
“A great day for Israel!” Mr Trump tweeted.
“A tragic day for the Palestinians,” retorted their UN ambassador, Riyad Mansour. In remarks to reporters and a letter to the Security Council president, he called on the council to “make its voice heard to stop the carnage”, bring those responsible to justice and protect civilians.
Israeli Ambassador Danny Danon said the council needed to speak out against Hamas.
“Every casualty on the border is a victim of Hamas’ war crimes, every death is a result of Hamas’ terror activity, and these casualties are solely Hamas’ responsibility,” he said in a statement.
The demonstration came after a weeks-long Palestinian campaign against a blockade imposed by Israel and Egypt after Hamas seized control of Gaza in 2007. Protesters set tyres ablaze, sending thick plumes of black smoke into the air, and hurled firebombs and stones toward Israeli soldiers across the border.
The Israeli military said Hamas tried to carry out bombing and shooting attacks under the cover of the protests and released video of protesters ripping away parts of the barbed-wire border fence.
UN Secretary-General Antonio Guterres said in Vienna that he was “deeply concerned” about the developments in Gaza, and a spokesman later underscored a call for restraint.
“With tensions high and more demonstrations expected in the coming days, it is imperative that everyone show the utmost restraint to avoid further loss of life,” deputy spokesman Farhan Haq said.