Video report by ITV News Correspondent Rupert Evelyn
Builders and workers in the construction industry say they feel "angry and unprotected" turning up to building sites, despite strengthened stay-at-home measures.
The measures also banned public gatherings of more than two people and ordered the closure of non-essential shops.
But Downing Street said that construction work should continue if it can be done following Public Health England (PHE) and industry guidance.
On Tuesday, the Prime Minister's official spokesman said that the Construction Leadership Council had issued guidance to the industry.
The spokesman added: "We urge employers to use their common sense when managing live projects and ensuring that employees can follow the Government guidance and practice safe social distancing on site."
He told ITV's Good Morning Britain: "My view is no, and I made that point quite forcibly at yesterday's Cobra, I made that point quite clearly to the Prime Minister.
But added: "According to the Government’s advice the answer is yes."
Housing, Communities and Local Government Secretary Robert Jenrick said work on building sites can continue but workers should practise social distancing.
It comes as housebuilder Taylor Wimpey said on Tuesday that it has closed its construction sites, show homes and sale sites due to coronavirus.
The company said: "In the interest of customer and employee safety, we have taken the decision to close all of our show homes, sales centres, and construction sites for all work except that needed to make the sites safe and secure."
Earlier on Tuesday, rival Redrow said its sites remain open with "strict precautions in place including enhanced levels of cleaning, additional hygiene facilities and social distancing".
A crane operator at a London construction site with 400 to 500 workers told the PA News Agency: "Everyone on site at the minute feels angry and unprotected."
The man, who asked not to be named, said if he opted not to go in he "wouldn’t get paid or even could lose my job".
Adding: "(It's) scary really - I've got a family and kids at home. I'm in London around all these people, bringing it all back into an isolated home."
Electrician Dan Dobson said the Government must provide support to self-employed workers, including about a million in the construction industry, and then shut down building sites.
He added: "Everyone on site at the minute feels angry and unprotected.
"None of them want to go to work, everyone is worried about taking it (coronavirus) home to their families."
British Safety Council chairman Lawrence Waterman said that all non-essential construction work should be stopped so that workers can stay safe.
He added: "The construction sector needs clarity from the Government – on most sites social distancing will be impossible or simply unsafe.
"All non-essential construction should end now so that construction workers can go home and stay home like everyone else."
While some companies said they were closing sites down and suspending work, others said they would remain open with "strict precautions" in place.
The Conservative MP tweeted a letter sent to the project CEO following, what he called, "shocking reports" of failure "to comply with social distancing & even coughing over local people".
HS2 said its "joint venture partners" are responsible for their sites and the health, safety and well-being of their workers.
A spokesman added: "Over the course of this week and into next, we will be reviewing the majority of works on our construction sites in-line with government advice on dealing with Covid-19."