Jiby's toolbox

I’ve recently written a series of articles about Gherkin: its usage for requirements gathering, about its uses in BDD, and a lower-tech alternative to Cucumber, which I’m still experimenting with.

I wanted to showcase all of these ideas brought together, as well as show how comfy modern Python tooling can be. But more than writing code, I would need to explain my reasoning as we draft the program. So I picked up again a tool I always wanted to play with: Literate Programming.

Let’s explore how a file-format design exercise turned into reinventing the wheel, and let’s all appreciate how lucky we are that the protocols that the internet are built on have fantastic public documents called RFCs.

Designing a file format

I was idly thinking about designing a good binary file format for a side project of mine called qrxfil. The project aims at QR-code-based file export: split a file in many pieces (based on content limit of QR codes) before reassembling it on the other side.

In the previous post, I showed how Gherkin Scenarios provide a great framework for acceptance test definition. I also showed how the dreams of the BDD movement extend to automating Gherkin, using a tool called Cucumber, enforcing traceability of acceptance tests.

In this article, I want to explain my disilusionment towards Cucumber, and show how a low-tech alternative emerged that I believe covers most of the value for none of the effort.

In the previous post in this series on Gherkin, I showed the tools that Gherkin Features provide for requirements tracking, and mentioned ambitious goals for the Behaviour-driven Development (BDD) movement. In this article, I want to expand on these BDD ideals, show Gherkin Scenarios, and explain how the dream of traceability led to Cucumber, a tool for automating acceptance test execution, tracing them back to Gherkin Scenarios. We’ll set aside my personal feelings on these tools for a future post.

The most common mistake I see developers make is building the wrong thing: Not tracking exactly what is required from the start, and instead getting excited about the cool technical problems ahead. Fast forward a few days, weeks, months, showcasing the product near completion, suddenly the rift between what’s needed and what was built becomes obvious, in a frenzy of last minute changes to reach the original goal under pressure from deadlines.

Here’s how attempting to export images off a Word Document led to a quest for data deduplication and classification using the shell.

The images I wanted to export were MS Word diagrams drawn in Word, rather than PNG files¹. Because those doodle-shapes do not export to PNG well, I first copy-pasted them into Powerpoint to get the familiar “save as picture” context menu. But a couple of images were still deformed beyond recognition.

Like many developers, I keep track of my configuration files in version control, to make moving to new machines is simpler and share code with others. This process started out for my Emacs config files at first, but grew to consume other aspects: bash functions, git aliases… Since 2016, I have accumulated over 500 git commits.

Now the dotfiles are tracked, the next biggest pain is to install packages and folder structures and obscure commands required to set up some subsystems needs which config files. Wouldn’t it be nice to have an all-in-one setup?

I was trying to convert bank-issued CSV files that have a weird data layout, for automatic processing by the excellent ledger. This made me learn a new thing about awk. Follow me down the UNIXy goodness!

The data I was getting looks like this:

Transaction Date,Transaction Description,Debit Amount,Credit Amount,Balance
29/02/2018,Restaurant,14.5,,1234.2
12/03/2018,ATM withdrawal,50,,1184.2
23/04/2018,Deposited check,,100,1284.2

Note the last line having 100 in a different column: The problem is that, when receiving money, the transaction is logged in a separate entry (“Credit Amount”) than the (more common) case of debit, which goes to “Debit Amount” column. My importing tools required a single column, so the output must look like this:

In order to troubleshoot a bug in the very early history of a project using git, I needed a way to show the git diff from “nothing” to the first commit. Since git keeps a linked graph of commits, what I wanted was a diff from the commit before the first one. Searching the onlines dug up this answer on StackOverflow, showing that there’s a specific commit hash we can use to diff from “nothing” to the first commit:

I’m writing code to back up my user data off a website that lets me see all of my info (including querying by time, account etc) but doesn’t have export features (officially). I am certain there’s an API behind the site that I just have to make sense of.

Since the webapp is requesting data from the API when I click, we should be able to record the web traffic to explore the API. My first thought was to use tcpdump(8), a powerful network sniffer. But for HTTPS traffic, the TLS encryption makes the network-level capture useless. We can of course look into MITM HTTPS proxies. But it turns out it can be even easier than that, as we can tell Firefox to record this for us during normal website use. Let’s see how Firefox can help in reverse-engineering APIs!