David Cameron has warned of the dangers of those who "quietly condone" Islamic State's extremist ideology.
Mr Cameron told a security conference in Slovakia that British Muslim communities should take more responsibility for countering the threat of radicalisation.
He also stressed the importance of tackling radicalisation at its source.
Mr Cameron said the cases highlight how young people from Britain are in danger of falling into violent extremism through online propaganda and encouragement from people who support Islamic State's extreme views on women and homosexuality.
The appeal of IS is stronger for young people "angry at the world" or "looking for an identity" if sections of society support sections of the group's extremist message, he said.
But some Muslim communities have reacted angrily to the Prime Minister's comments, accusing him of "pushing the blame back at the community".
Mr Cameron said police and intelligence agencies were "not responsible for people wanting to go".
He insisted families and communities must play a part in preventing would-be jihadis attempting to join IS in the first place.
If you're a troubled boy who is angry at the world or a girl looking for an identity, for something to believe in and there's something that is quietly condoned online or perhaps even in parts of your local community then it's less of a leap to go from a British teenager to an Isil fighter or an Isil wife than it would be for someone who hasn't been exposed to these things.
Analysis: ITV News Correspondent Debi Edward
Broadly speaking people have welcomed the Prime Minister's acknowledgement that more needs to be done about radicalisation, but there is some anger at him for pointing the finger of blame towards Muslim communities.
In Bradford they are all too aware of the threat of radicalisation and the threat that poses to our national security.
But they want to work together with the government to root the problem out.
There have been calls for greater trust, dialogue and partnership between the Muslim communities and the police authorities.
One Muslim leader said that if he had been told by counter terrorism officers, that the family of these three sisters were being watched, he might have been able to prevent them and their young children from travelling to Syria.