At least four people were killed and 34 wounded, hospital sources said, as militants fought demonstrators on the streets of Benghazi. Gun shots could be heard in the area before the fighters were forced out.
Looters carried weapons out of the vacated Ansar al-Sharia military base compound as men clapped and chanted: "Say to Ansar al-Sharia, Benghazi will be your inferno."
Ansar al-Sharia has been linked to the attack on the U.S. consulate in Benghazi last week in which the U.S. ambassador to Libya and three other Americans died. It denies involvement.
Chanting "Libya, Libya", "No more al Qaeda!" and "The blood we shed for freedom shall not go in vain!" hundreds of men waving swords and even a meat cleaver, stormed Ansar al-Sharia's headquarters in Benghazi.
Demonstrators pulled down militia flags and set a vehicle on fire inside what was once the base of former leader Muammar Gaddafi's security forces who tried to put down the first protests that sparked 2011's uprising.
As protesters left Ansar al-Sharia's headquarters, the crowd swelled, reaching thousands as it headed toward the Islamists' military base, which was shared with another militia group.
Protesters said the militiamen opened fire as they arrived and several people were wounded.
Vigilantes armed with machetes and clubs blocked the highway leading away from the compound, stopping cars to prevent looters from driving off with heavy weapons.
The apparent defeat of Ansar al-Sharia across Benghazi and the huge outpouring of public support for the government marks a huge transformation in a country where the authorities had seemed largely powerless to curb the influence of militia groups armed with heavy weapons.
However Ansar al-Sharia and other Islamist militia have bases elsewhere in eastern Libya, notably around the coastal city of Derna, known across the region as a major recruitment centre for fighters who joined the wars in Iraq, Afghanistan and Syria.
Thousands of Libyans had marched in Benghazi on Friday in support of democracy and against the Islamist militias that Washington blames for the assault on its consulate.
Hundreds of Ansar al-Sharia supporters held their own protest.
Friday's "Rescue Benghazi day" demonstration called for the government to disband armed groups that have refused to give up their weapons since the NATO-backed revolution last year.
Although the main demands of the marchers did not mention the attack on the U.S. consulate, it seems to have provided the impetus for the authorities to rally support behind the country's weak government.