Some good points, and some points of contention.
One important thing that I had to come to grips with … don’t worry about getting a design “perfect” on the first try. You can’t always anticipate what the future will bring, so you can easily design a system to be extensible in a way that will never be needed, and miss something that will be.
Some refactorings may not be worth the effort, because that code may really never change. Usually, I just try to get a working solution, then refactor to make things cleaner. If you know something should probably be refactored, putting a comment to that effect in the code is a nice reminder.
I also personally believe readable code is generally WAY more important than performance-optimized code. Human cognition isn’t evolving at anywhere near the rate of computer performance. Plus, you generally are paying a LOT more for your human resources than computers!