Why engineers should treasure Python over Excels of the world

credit: @pakata
  • Sunken cost argument (different excuses but all cantered around why should I learn a new tool, when I have spent so much time learning the current ones)
  • My challenge/model is too complex for anything that Python can handle
  • General fear of scripting which goes hand to hand with the next argument
  • Anything you can do in python I can do in Excel (replace Excel with the software of your choice with Graphical User Interface (GUI))

Familiar Scenario

I have worked in my industry in different roles from an external consultant to the engineer responsible for delivering projects in operating companies and I have seen my share of useful but unreliable solutions developed on Excel and similar tools. The following story should sound familiar to you:

  • Senior Engineer/Manager (M): We have plenty of data to work with, we need x number of more engineers to be able to create and run all these models
  • Smart junior/grad engineer(E): Hey, it should be alright, I know advanced excel and VBA from my university days, I should be able to automate the model generation piece and we can do this within our team
  • M: great, show me how your tool works, let’s present it to the higher-up management, we are great
  • M and E decide to simplify their assumptions (sometimes to the point that they may not be valid, but the justification is we are constrained by the tool)
  • M leverages business continuity argument and complains to the IT that IT should supply a more stable version of Microsoft Excel/Environment
  • E suggests instead of running Microsoft Excel on one machine, why not running multiple instances on many, accepting 50% completion of macros over night as a success

Why YOU should treasure Python over Excels of the world

If you have not been convinced yet (which if I know my engineer friends and colleagues, the chances are highly likely that you have not), below I am making a case on why using Python to play with your data is a much more superior tool compared to Excel or any other Proprietary tool.

  • We should use more Open Source and less Proprietary software
  • Scripts are powerful. They can eat GUI for breakfast any day
  • There are future opportunities unforeseen today if you are not confined by your tools
  • Engineering work requires a lot of mental power and focus. You cannot get interrupted constantly and still produce quality work. Working with scripts rather than GUIs seems scary at the beginning. As you get more comfortable with writing scripts and navigate your way through things using text commands and scripts, you realise staying in a single environment minimises the distractions. This ability to focus more can lead to generating higher quality work. Using mouse and GUI while visually enticing would cause you to move from one screen to the next which means your eyes glance on various things on screen. They can distract you and keeping yourself focus drains your mental strength.
  • The second importance of the script vs GUI is the reproducibility of your work. There is a learning curve. It may make you slower than the rest at the start.As you get comfortable though, you will spend less time on the preparation and more on analysing and engineering while staying focus for longer. Imagine a simple case of copy/pasting a file. Compare automating or repeating a simple file copy/paste. The movement of the mouse, right clicking and selecting copy, going to the destination to paste the data, vs the following (bash not Python):

Final words

In this article I showed general scenarios on how engineers can end up with closed source solutions and run into scalability issue as the challenges are getting more complex and the datasets are getting larger. I offered my solution as adding Python to our toolbox.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Amin Noor

Amin Noor

Charted Reservoir Engineer, and Python Enthusiast, I love automation and efficiency gains