There is a known issue in world project management which translates well into the field of HPC. I first heard of is as the “Trinity Problem” – in every endeavor you always aspire to reach a result that is good, fast and cheap but can only fully achieve two of these (at best).
The reason is not the futility of man, but the inevitability of math. Given the wishes of the recipient of the project’s result (i.e. the client) trade-offs will be made in order to satisfy the requirements. For the work to be done in the shortest time possible without settling on quality, you have to get help from the best people (or just more people than you have right now). Being the best (or more), comes with a price tag so things aren’t going to be cheap. Alternatively, you can relax the requirements, which may mean a less good* product, Or you can do with the personnel and requirements you have, and things will take as long as they have to.
In HPC (and cloud computing in general) we have the same problem, the more accurate (=good) we want our result to be the more computing time we would have to invest.
When handling a performance issue, we are effectively tackling the “fast” aspect of the trinity problem – this means we need to decide whether we want (or can) invest more money on bigger and faster computing hardware or can some corners be rounded as far as the accuracy of the results is concerned. Some calculations are unscalable – meaning using bigger machines will not help and some operations have no slack for us to work with, figure out what is right of you and your project, and act upon this decision as soon as possible, it’s software after all and you can always change your mind.
* For more on the subject of quality let me refer you to Zen and the Art of Motorcycle Maintenance