Being concerned about border control and the prospect of criminals entering the UK is valid. We have enough criminals of our own, we don't need anybody else's.
But actually, the UK and Ireland are in complete control of our borders.
Freedom of movement doesn't affect border control. It's the Schengen Agreement that affects border control and the UK and Ireland have a permanent exemption from joining the Schengen Agreement.
And we can't be forced into it against our wishes.
You may think it's fair to say that the UK isn't very good at controlling its borders, and that may be a fair criticism. But being a member of the EU has no impact on that. Some criminals manage to get into the UK from the EU, but some criminals also manage to get into the UK from outside the EU. It's not freedom of movement that facilitates it, it's just international travel in general.
Security information sharing within the EU also helps to reduce international crime, so leaving the EU may actually make it more difficult to control our borders if the information sharing is hindered in any way.