Grey marks on newly painted walls

A "professional" painter appears to have filled several large holes and cracks in our walls with, we think, decorators' caulk and painted over with matt emulsion. We now have unsightly grey marks in every room which we cannot seem to get rid of not matter what we try. Have tried digging out the stuff and refilling with Polyfilla but still no success. Any suggestions?