I was running M1 a years ago on my VPS, but it was a pain in ass. Today I run M2 on specialized hosting (with Varnish, Redis, Elasticsearch, daily backups, ...) and my site is running pretty fast with 4k products and a lot of extensions. But the best thing is the "staging" feature, where you can clone your store and experiment with settings and extensions and your live store is untouched.
It cost me cca 60 €/m, but I don't need to mess with VPS myself.
And to your question. Maybe some sort of Cache Warmer will help you to speed up the first visit/page load.