"It's a big galaxy and they can leave."

This. It's implied in the previous season that the Silents are on several planets, harassing several different species at different points in time.

I doubt they were all killed, or even most of them. They have the option of pulling up stakes and going off to find greener pastures, and probably did so.