"OMG its layzie!"
Anyway, what it comes down to is what type of game style are you playing and if you're actually trying.
I use silencers on occasion for...
SnD and WoGL (World Online Gaming League) and only if i'm trying to be sneaky. Personally, the m4a1's recoil is next to nothing anyway so the silencer doesn't make a huge difference as long as you tap your mouse and not try to full auto someone across the map.
From what I gathered from the OP, I would suggest going without one due to the fact that most maps aren't large enough to mess with the m4's accuracy at long range. Only ones I can see you might have problems on are snow valley and maybe waverider but that's if your opponent is across the map.