In a programmer’s mind

Keeping the gears running by Curious Expeditions

Many new business software projects are about automating some repetitive and boring tasks. Instead of having a big spreadsheet in which we all enter our time log that we share by sending it by email to get some reports at the end of the month, we create a centralized client/server application where everyone can enter whenever he wants his time log and automatically generate and send that report at the end of the month. Instead of manually copying our past time log from our old spreadsheets into our new system, we create a simple script that reads those spreadsheets and import that data in our new centralized application. Tasks that could have taken days are completed in minutes instead.

In many situations, I think we could push even more for automation. In the software development business, I feel like we are working on automating tasks for our clients but we are not thinking about automating our tasks enough. I guess it is different from places to places, but in each of my job experiences, I have found processes that could get an easy productivity boost simply by creating a simple script.

The reasons are numerous for not automating more. We do have time for it, we are not using the right tools, we do not know how to create the right tools, we fear changes, we do not know how to validate our automations, we are using applications or processes which do not have any automation entry points or we are plainly lazy and we do not want to learn how to automate things. I include myself in because I have been using each of those reasons at least once for not doing it.

If you do not have time to automate your tasks, you are just wasting your time. Just think about it. Automating is all about saving time. The exception would be a task that is quick to execute or rarely executed that would take an enormous about of effort to automate. I you let you be the judge on this one but do not pretend it always requires huge efforts to automate a process.

Using the wrong tool is a frequent reason. It is hard to overcome because in many cases, you just do not know that there are better tools, applications or ways to automate your task. One way is to try and stay up-to-date with the latest technologies, try to learn new stuff like a scripting language (Python is probably a good start). Automating a task that require you to use a GUI application where there are no alternatives like command line interface, a library interface or any public API is obvliously hard. Even then, there are tools which can help you out in those situations.

If you are like me, with a strong Windows background where the GUI is king, you might want to see how things are done in the unix/linux world where everything is a small command line program that is often used in a long tool chain to automate many tasks. If you are shy of installing a full-blown free linux operating system to toy with those tools, you can do like I do and install a linux-like environment for Windows.

To end this post, I will show you some examples and tools that I have been using recently to automate boring tasks.

Mass image manipulation

To automate the creation of thumbnails with a folder of a thousand pictures, I have been using ImageMagick and a command line similar to this one:

mogrify -format jpg -size 600x600 -auto-orient *.jpg

XML manipulation

To update large and complex xml configuration files, I have been using different XSLT templates with Saxon, a free and open source XSLT processor.

Data transformation, importation and exportation

I often had to import or export data from and to text files, spreadsheets, different databases, LDAP repositories or outlook/exchange. I used to create small C# command line programs but I have switch to Python recently for the productivity gain. My strategy is always the same. Find a component that reads the raw data of the input source, find another component that can write to the destination source and transform the data in the middle.