The only difference is that you cannot verify the game files through Steam, and automatic updates are a nightmare (you would have to download a whole new repack).
Have you tried a highly compressed version of Serious Sam 4? Share your install times in the comments below (but keep comments about compression, not piracy links). serious sam 4 highly compressed