Imagine a "swarm" of UAS carrying high-value drugs across the US border. Some I suspect would fail.
Moving drugs is a business, and I'm not sure it would be profitable to move narcotics this way, and furthermore I'm not sure how a swarm would land in a consolidated location where the drugs could be received by the next link in the network without being detected? I can see the potential of criminals using UAVs for surveillance prior to moving their load in some cases, but so far spotters seem to be working. I suspect that UAVs will be used to fill a need, and if there isn't a need criminals won't adapt the technology. The same holds true for states and terrorists. States have the means to experiment and expand the realm of the possible. Criminals and terrorists will generally adapt existing technologies instead of developing new technologies.

I remain concerned about what creative terrorists will be able to do with this technology as it evolves, and while I'm thinking bigger than a hand grenade, one can't downplay the psychological effect of a handgrenade dropped on a target from a UAV in someplace like the UK, Canada, or the U.S..

However, to put things in perspective the non-lethal potential of UAVs to support social needs such as search and rescue, delivering supplies, fighting fires, supporting law enforcement, collecting various forms of data to support non-military planning, etc. is far greater than its potential negative impact.

This isn't much different than the bio-sciences in that regard, for the most part this science has greatly improved the lot of man, but there are elements of this science that could be used by state and non-state actors alike to threaten or harm.