Members of Bristol's Disability and Equality Forum say people are finding it hard to understand and trust government advice.
Forum Manager Laura Welti says advice keeps changing and is poorly communicated.
One minute people are being told to shield because they are extremely vulnerable and then suddenly the Government turns around and says oh it is fine you can go out now.
“If the Government can change as to whether you are vulnerable or not on the head of a pin, then how much can you trust that advice that you’re getting.”
Isolation is a key problem for people who have been told they are vulnerable to catching coronavirus.
For the past three years Georgina Moore has been running a Boccia club in Yate for people with physical and learning disabilities.
Since March she has only been able to hold sessions virtually for fear of any member becoming infected.
The 36-year-old has been staying broadly positive but says it is hard not to feel isolated because of her disability.
I just don't think people understand how anxious it makes us feel.
"My friends who've all got kids they don't want me to see [them] because if they gave me [Covid-19] they would never forgive themselves. So I am being left out of things, not on purpose or because people are being mean but because they are worried about me. You do realise now how isolated it has made me become."
It comes as national charity Scope has revealed that one in four people with a disability say they feel forgotten.
They are calling on Government to better support people with disabilities with an upcoming welfare report and National Disability Strategy.
In response to ITV News West Country, a Department for Health and Social Care spokesperson says:
"We understand this has been a particularly challenging time for disabled people and we remain committed to supporting them, their families and their carers.
"Across government we have published advice and guidance for disabled people as well as making £3.7 billion available to local authorities to help address pressures on local services, including adult social care.
"We want disabled people and those with learning disabilities to continue to be supported and the new National Disability Strategy will focus on the issues that disabled people say affect them the most."
Bristol Disability and Equality forum is starting a six month outreach project to support disabled people to build up the confidence to return to getting out of the house regularly.
The project will also loan people equipment and internet connection to access online support and communication with friends and family.