With almost all the results in from Scotland, the SNP seems likely to win three of the six Scottish MEP seats - with Scottish Labour losing its two MEPs.
Results from 31 of the 32 councils across Scotland showed the SNP, who fought the election campaign on a strong pro-European platform, had polled 37.7%
The party's previous best Euro election result was 32.6% in 1994.
Speaking during a visit to Dublin on Monday morning, First Minister and SNP leader Ms Sturgeon said: "It's an absolutely fantastic result from my party's perspective.
"We didn't just win the election, we are polling 20-plus points ahead of our nearest rivals, it's astonishing, astonishing for any party, but our party has now been in government for 12 years.
"So as well as being a very clear anti-Brexit vote, I think it's an endorsement of our party's record in government."
While Ms Sturgeon's party increased its vote - it had polled 28.9% in the last European elections in 2014 - Labour's vote slumped.
Five years ago it was a close second with 25.9% of the votes, but with all but one of the council areas having declared, that fell to 9.3%.
Ms Sturgeon said Scottish Labour had "paid the price" for its indecision on Brexit.
She said: "The lesson of that in this election is that they paid a price for sitting on the fence on Brexit.
"My party has taken a consistently anti-Brexit position and I think that's fair to say that was vindicated in the election so far."
Labour's David Martin - who had been the UK's longest-serving MEP having first been elected in 1984 - conceded he had lost his seat, with the party paying the price for its failure to take a clear stand on the crucial issue of Brexit.
Mr Martin tweeted: "While obviously personally disappointed to lose my seat, I'm devastated for the hundreds of good Labour people who put so much into the campaign. We lost not because of lack of effort but lack of clear message."
Labour Edinburgh South MP Ian Murray - who has campaigned for a second European referendum - hit out on Twitter, saying: "The Labour leadership refused to listen to me, their members & their supporters. Maybe now they'll listen to the voters?"
In a tweet on Monday morning, Scottish Labour leader Richard Leonard said: "I have known and campaigned with David Martin for 35 years. He has been a stalwart representative for Scotland and for Labour. I am extremely sad that he has not been returned to serve another term."
The Brexit Party, who came second across Scotland, are on course to secure one MEP, as are both the Conservatives and the Liberal Democrats.
The Brexit Party's Louis Stedman-Bryce said: "The result is pretty amazing, obviously five weeks ago the party didn't exist and we've gone from zero to second place in this election, which I think is phenomenal, and in the wider picture we are leading across the UK.
"This just demonstrates that there is a strong and valid number of people in Scotland who voted to leave the European Union."
The full results will be known later on Monday, when the figures from the Western Isles - which does not count votes on a Sunday - are declared.
Ms Sturgeon also tweeted about the "outstanding results" for her party, saying they were "on course for our best ever Euro election result".
She added: "Polling way ahead of any other party. Higher % in Scotland than Brexit Party's current share in rUK. Emphatically clear that Scotland doesn't want Brexit."
Meanwhile Alyn Smith, who was the lead SNP candidate, said it was "clear Scotland has endorsed Remain".
He added: "Our Remain vote can not be ignored any longer. Scotland's for Europe."
In the City of Edinburgh Council area, Labour was sixth - behind the SNP, the Liberal Democrats, the Scottish Greens, the Tories and the Brexit Party.
Lothian MSP, and former Scottish Labour leader Kezia Dugdale said her party should respond to the results by "backing a People's Vote unequivocally and committing wholeheartedly to remain".
Scottish Conservative leader Ruth Davidson said: "It was clear before the election the Conservatives were going to get a kicking across the UK for not delivering Brexit as promised.
"I will redouble my efforts to ensure all those across the country seeking to get rid of the nationalist government at Holyrood feel able to come together and support my team at future elections.
"Dividing the pro-union vote only benefits one party - the SNP."
The Liberal Democrats had lost their Scottish MEP in 2014, when the party paid the price at the polls for its coalition with Conservatives at Westminster, but its increased share in this election leaves the party in Scotland on course for a return to the European Parliament.
Speaking on the BBC's Good Morning Scotland programme on Monday, Scottish Lib Dem leader Willie Rennie said the result is a "big win" for the party.
He said: "This was quite some result and I think it shows that the Liberal Democrats have got the momentum.
"There's lots of people who voted for us this time that have never voted for us before and we need to work hard to keep their trust that's been shown in us.
"But we've got a real opportunity to represent the majority of people in Scotland who are pro-United Kingdom and pro-European Union."