Lethal Collin scored at the beginning and end of the game to take his tally for the season to 16 to spark delirious scenes of celebration at the final whistle.
Stevenage recovered from conceding a goal after only 108 seconds to have the better of the second half and a goal from Darius Charles gave them hope of avoiding a giant-killing only for Collin to strike again three minutes from time.
Isthmian League Maidstone went ahead in the most basic of manners, a long clearance from goalkeeper Lee Worgan was headed on by Alex Flisher into the path of Jay May. Veteran Stevenage goalkeeper Chris Day could only parry May's soft shot but the rebound fell for Collin to lash home from 10 yards.
Worgan made excellent saves from Lee Barnard and Roarie Deacon in the first half as Stevenage fought desperately to get back into the match.
But they had to wait until the 47th minute before Charles volleyed in a centre from Charlie Lee and for long spells it looked as if Graham Westley's side, 60 places above seventh tier Maidstone in the league pyramid, would take control.
But with extra time looming, substitute Ben Greenhalgh got clear on the left and his perfect centre was headed in by Collin, with Stevenage appealing for offside.
Southampton and Brighton are both through to the fifth round of the FA Cup. Saints beat Yeovil at St Mary's two nil while Brighton celebrated a 3-1 victory at Port Vale. There were hopes of an upset at Bournemouth.
But as Sarah Gomme now reports the Championship side couldn't quite overcome the might of Liverpool.
Liverpool have beaten Bournemouth 2-0 in the FA Cup fourth round.
And you can see the goals by clicking here
A 3-0 home defeat to Chelsea, an ugly spat with the Premier League over referee Clattenburg and a three-match ban for club-record signing Dani Osvaldo mean it has hardly been a happy new year to date for Saints fans.
However, the majority of those that braved the wet and windy conditions left St Mary's with a smile on their face after Mauricio Pochettino's side secured a thrilling 4-3 win against Sky Bet Championship promotion hopefuls Burnley.
Saints, winners in 1976 and finalists 11 years ago, recovered from an uneasy start to go ahead in style midway through the first half, with Nathaniel Clyne smashing home an exceptional volley from the edge of the box.
Rickie Lambert followed that up with a fine strike of his own six minutes later, only for two Southampton fans to put Burnley level at the start of the second half.
Sam Vokes and Danny Ings both spent time with the club as schoolboys and showed them just what they were missing, with a header from the former followed by the latter's low strike from an acute angle.
However, Burnley-born Rodriguez - given a warm reception by both sets of fans after replacing Lambert - put Saints back ahead, choosing not to celebrate in respect to the club he left in 2012.
There was no holding back when fellow substitute Lallana put the off-field issues behind him to net from distance - a strike that meant Kevin Long's late strike was merely a consolation.
Aldershot met Middlesbrough for the first time in the FA Cup - but it wasn't a happy meeting for the Hampshire side.Read the full story ›
Aldershot put on a convincing performance to book their place in the fourth round of the FA Cup. Danny Hylton struck a hat-trick and was then sent off in the closing stages as Aldershot progressed to the fourth round of the FA Cup with a 3-1 victory over Rotherham.
Full report on the game can be found here on ITV's Sport website.
Crawley had reached the 5th round in the last two seasons & they made the perfect start at the Broadfield Stadium with Nicky Adams' strike.Read the full story ›
Middlesbrough boss Tony Mowbray paid tribute to non-league Hastings United after his side were forced to survive a few scares.Read the full story ›