Deputy Commons Speaker Nigel has vehemently denied allegations of rape and sexual assault made against him, saying the accusations were "completely false".
The 55-year-old Tory MP described his arrest yesterday at his home in Pendleton, Lancashire, and subsequent questioning by police as "the worst 24 hours of my life".
Mr Evans made clear through his solicitor that he intended to carry on as Deputy Speaker while police continue their investigation.
ITV News Political Correspondent Simon Harris reports from Lancashire:
However, Speaker John Bercow has agreed he should be excused from his duties chairing the Queen's Speech debate, which begins on Wednesday.
Following his release on bail, Mr Evans delivered a brief statement outside his local pub, expressing his "incredulity" at what had happened.
– Conservative MP Nigel Evans
Yesterday I was interviewed by the police concerning two complaints, one of which dates back four years, made by two people who are well known to each other and until yesterday, I regarded as friends.
The complaints are completely false and I cannot understand why they have been made, especially as I have continued to socialise with one as recently as last week.
I appreciate the way the police have handled this in such a sensitive manner and I would like to thank my colleagues, friends and members of the public who have expressed their support and, like me, a sense of incredulity at these events.
Lancashire Police said Mr Evans, who has been MP for the Ribble Valley constituency since 1992, was arrested on suspicion of raping one man and sexually assaulting another between July 2009 and March 2013.
He was released on bail late last night until 19 June.
Defence Secretary Philip Hammond questioned whether Mr Evans could continue as Deputy Speaker while he was under investigation.
"I stick rigidly to the view that we should treat people as innocent until they are proven guilty but it is quite difficult to carry out a sensitive and high profile role while being under this kind of scrutiny", he told BBC1's The Andrew Marr Show.
However, overwhelmingly the mood among MPs and residents in the village appeared strongly supportive of Mr Evans, who has long been a popular figure at Westminster.
Foreign Secretary William Hague described him as a "long-standing friend" and said that MPs of all parties would be "very sorry to see this situation".