Why did I write this in the first place?

A startup founder and content creator in technology recently said that it is not necessary to hire professional or senior-level engineers or software engineers with computer science degrees because the salary demanded is too high with little difference in work delivered compared to junior engineers who relatively demand a much smaller salary. Of course, I agree with him for some part of that statement, however, I think it is not entirely true.

Introduction

In my personal experience in the IT consulting industry, several clients I met never cared about system optimization. It is simply because the optimization usually requires a significant increase in the development budget. In some cases, the budget increase due to the optimization requirements may be close to 100% of the initial budget. Thus, some clients with strict budgets just want the system to be developed without any optimization to be made just for the sake of saving money in creating their dream apps. To make the problem more complex, junior software engineers, especially those who do not have computer science degrees, usually also didn’t care about optimization. As long as their algorithm can solve the problem, the work is considered as done. Thus, a combination of small-budget clients with junior software engineers who never think about optimization always resulted in a slow-performance app.

Is Optimization Important?

In general, you already know the answer. Yes, it is very important. But how important is that? The answer depends on the system you want to build. A system that works with a small dataset may not require any optimization. However, if the system is dealing with large datasets with terabytes of data, optimization is much more important than developing the basic solution that works with small datasets.

A Simple Test

Let me give you a simple example. I have a simple task to calculate the minimal length of the sub-arrays in the given array, which equals or more than a given number. So, the input is the array and the number.

The algorithm in Picture 1 is the basic solution. It solves the problem but is not optimized.

The algorithm in Picture 2 is the optimized solution.

I will give 5000 items in the array as the input and let the algorithm find the minimum length of a sub-array in the given array which has a sum >= 9968933.

The result is as follows. The test is conducted with the same programming language (Python), and on the same machine. The unoptimized algorithm can solve the problem in 1 minute and 1 second for 5000 data points (see Picture 3). While the optimized algorithm can solve the same problem with not even close to 1 second (See Picture 4). It is clear that by only processing 5000 data points, the basic solution needs more than 1 minute to find the answer. What if the data point is billions of data? It could run forever, right? So now you can imagine how important optimization is.

Takeaway

For a startup company or small business owner, I understand that optimization will slow the development of the app and increase the budget. I agree that you may skip the optimization and save the money, but carefully consider it as a technical debt. As your business grows and your data scales quickly, it is necessary to take action for system optimization as soon as possible to ensure seamless user experiences. Most importantly, hire professional developers for your app as seasoned engineers already learn through experiences the nuances of complex problems.