Over the decades I’ve seen the praise and critique of the “Unix way”. I don’t remember reading about the aspect I’ll describe below. This might be a banality, and I haven’t invested much thought into it either. I’m just curious to hear your opinions.
At the core of the Unix way is that tools should do just one thing. Users very often need results that can only be achieved after doing several things. With only single-purpose tools at hand, users have to integrate them. In other words, the tool authors delegate integration to the users.
Here’s the idea I want to discuss. In programming languages, integrating various modules to achieve a goal is already a complex task because we often have to handle numerous failure modes and edge cases. In shell scripts this is much worse, since shells are more limited than programming languages. Writing a tool that does only one thing looks like avoiding the most complicated task. Which leads to a conclusion that the Unix way only works well in a rather small fraction of use cases; the majority of cases are better served by different approaches.
What do you think?
A data pipeline commonly has three phases: extract, transform, and load (ETL). The Unix philosophy of “do one thing well” can apply here by making each program handle only one ETL phase; for example, a program that only extracts data. This strategy works well whether orchestration is done with a shell script or with a Python-based de facto tool such as Apache Airflow. In contrast to mixing ETL within a single program, Airflow and shell scripts using GNU Parallel can handle retries, concurrency, logging, etc. Data pipeline engineering is not a rare case. Still, other kinds of applications, such as GUI, TUI, and web apps, are rarely built on shell scripts; however, shell scripts may work behind the scenes for many apps. Moreover, “do one thing well” still applies, but the unit may be a function, module or service instead.
Am I reading your message wrong? It appears to say that a single program cannot handle retries, concurrency, logging, etc.