PM says change takes time, and those for whom the government is never right-wing enough can bring the government down through carelessness.