Nine out of 10 dementia units have been assessed as having aspects of poor care, the Care Quality Commission (CQC) has found.
The watchdog criticised hospitals and care home staff for having "a lack of understanding and knowledge of dementia care" as well as poor information sharing.
It also found that known risks to patients like falling over, urinary tract infections and malnutrition were not being managed properly.
Andrea Sutcliffe, the CQC's chief inspector of adult social care, said: "People living with dementia, their families and carers have every right to be treated with respect, dignity and compassion.
"Our review found some great care, delivered by committed, skilled and dedicated staff.
"But this is not the case everywhere or even within the same service meaning too many people are at risk of poor care. This has got to change."
The CQC plans to appoint a new national specialist adviser for dementia care and more training for inspectors to "understand what good dementia care looks like".
Health Secretary Jeremy Hunt said: "There can be no excuse, and no hiding place, for poor care within our NHS - we are focusing on improving the lives of dementia patients and their families as never before."