Swiss public broadcaster SRF reported after polls closed at noon that projections based on partial counts indicated most voters had rejected the measure, which was championed by the populist Swiss People’s Party.
It had proposed giving preferential access to jobs, social protection and benefits to people from Switzerland over those from the 27-nation bloc that surrounds it.
However, doing so would likely have severely dented the country's economy.
Voter Yann Grote said he did not approve of further limiting freedom of movement.
“I’m not at all in favour, and even more now, because it’s not a time to isolate Switzerland,” he said.
If passed, the proposal would have further strained the relationship between the Alpine country and the EU.
Switzerland is not a member of the EU, but is surrounded on all sides by member countries and is part of the Schengen Area allowing passport-free movement between many European countries.
If passed it could also lead to reciprocal disadvantages for millions of Swiss citizens if they want to live or work in the EU.
The EU would likely end Switzerland's access to the internal market.
Roughly 1.4 million EU citizens live in the country of about 8.2 million, while around 500,000 Swiss live in EU countries.
In a similar referendum in 2014, the Swiss narrowly voted in favor of limiting access of EU citizens to live and work in Switzerland.
The deal the government implemented only partly fulfilled the mandate of the referendum, prompting the People’s Party to get the issue back on the ballot again this year.
Since the last vote, Switzerland has witnessed the personal and professional turmoil that Britain’s 2016 referendum to leave the European Union has caused, especially for EU citizens in the UK and Britons living on the continent.
Voter Elisabeth Lopes said: “I’m a daughter of immigrants, so it is a matter that touches me.
"If Switzerland had to withdraw or reduce these agreements (with the EU) I think we would be the real losers.”