Defence Secretary Philip Hammond told ITV News it was not for him to judge whether Sgt Danny Nightingale should have been sent to prison.
He added that the Court of Appeal "acted swiftly" in this case which showed the system was "robust".
Julian Brazier, MP for Canterbury and Whitstable and a former SAS reservist officer, who has campaigned on behalf of Sgt Danny Nightingale, welcomed his release.
I was delighted to hear the news.
While firearms offences are normally very serious, given the very unusual circumstances of the case, and Sgt Nightingale's remarkable record of service, the original sentence was a serious miscarriage of justice.
I am delighted that Danny will be going home to his family for Christmas.
SAS sniper Sgt Danny Nightingale thanked his wife and supporters for their "trust and support" after walking from the Court of Appeal.
Sgt Danny Nightingale said outside court after being released: "Thank you to the world for being so kind".
His wife Sally said she was "very, very happy" and that she had "fought for this and got justice today".
Security staff say Danny Nightingale is in the main hall of the court waiting to walk out.
A flag with a message is being held up behind the cameras waiting for Danny Nightingale to leave court. It reads: "SAS hero, not criminal".
Sgt Danny Nightingale's wife Sally, 38, was lost for words after the verdict but nodded when asked if she was "thrilled".
The crowd outside the Royal Courts, where a SAS sniper had his sentence reduced to a suspended 12 months, is chanting: "There's only one Danny Nightingale".