Robots, automation, and AI will obliterate a lot of human jobs in the next 10–20 years. Very few people now argue with this prediction. We have been discussing this idea widely in the last few years, often using the example of autonomous driving systems and how they will replace human drivers (the most popular occupation in America today).

Most programmers think they won’t be at the forefront of this trend because software programming is a cognitively demanding activity that only a fraction of humans actually able to perform well. …

In this post, I review the recently published paper “Online capacity estimation of lithium-ion batteries with deep long short-term memory networks” (Li, Sengupta, Dechent, Howey, Annaswamy, Sauer) and compare the proposed Long-Short Term Memory neural network for estimating the remaining capacity (i. e., state-of-health) of Li-ion battery cells with already existing Kalman Filter and linear regression-based methods.

Summary: an LTSM-RNN model was trained to predict the cell remaining (degraded) capacity using as inputs just time-stamped voltage readings during charging cycles.

The promise of the paper is to make the estimation simpler. They achieve that: they use absolutely minimum inputs to…

I live in a city that is about to be hit hard by the coronavirus pandemic. People are shut in their flats and the health care system is already loaded over its maximum capacity.

It comes to me more acutely than ever before how ridiculously privileged I am as a programmer. My life almost hasn’t changed yet. I work and study exactly the same way I did before, remotely. …

Functional Data Engineering — a modern paradigm for batch data processing

Maxime Beauchemin summarizes hard-won lessons about building repeatable and reasonable data processing systems:

Functional programming brings clarity. When functions are “pure” — meaning they do not have side-effects — they can be written, tested, reasoned-about and debugged in isolation, without the need to understand external context or history of events surrounding its execution.

As ETL pipelines grow in complexity, and as data teams grow in numbers, using methodologies [of functional programming] that provide clarity isn’t a luxury, it’s a necessity.

Failure Modes and Continuous Resilience

Adrian Cockcroft adds a third component to the classic risk exposure formula in the context of engineering:

In addition to…

ScheduledThreadPoolExecutor is prone to unbounded clock drift

Recently, I’ve realized that ScheduledThreadPoolExecutor is prone to unbounded clock drift and therefore should not be used to schedule tasks at specific timestamps or specific rates in terms of UTC, Unix time, or system time (for example, once every hour) for longer than a few days, which is usually the case in the backend (unless you mandatory restart your server applications daily) as well as desktop software.

It is this question on StackOverflow which made me thinking about this problem, where somebody observed ~15 minutes of drift per day when using a ScheduledThreadPoolExecutor.

java.util.Timer freezes periodic tasks or piles them up when system time is shifted

There is also a dusty Timer class…

I’ve listened to Daniel Coyle’s The Culture Code after reading the excerpt which resonated with me greatly. Thanks to Subbu Allamaraju who pointed to this book in his post about Status Management.

The book has changed the way I think about work profoundly: even more than The Effective Executive by Peter Drucker and Principles: Life and Work by Ray Dalio, and comparably to Cal Newport’s Deep Work. I would say “The Culture Code” is a must-read for everyone who works in a team. These “Engineering Ideas” are my notes on the book. …

Simple Made Easy

One of those instant classic talks by Rich Hickey with plenty of powerful thoughts. First, a reminder:

Simplicity is a prerequisite for reliability. — Edsger Dijkstra

For me, this echoes the idea of “embracing operational simplicity” from Edmond Lau’s The Effective Engineer:

Simple solutions impose a lower operational burden because they’re easier to understand, maintain, and modify. At Instagram, simplicity was a key principle enabling the team to scale.

Hickey explains that simple means “one-folded”: one task, one role, one concept. It’s objective. Easy means “convenient, at hand, familiar”. It’s relative.

Modularity is not a silver bullet:

Modularity (partitioning and…

This year, I’m starting to record ideas that I encounter in my reading (listening, watching) about engineering and find novel, powerful, or just interesting. The scope is software, systems, reliability, and data engineering, as well as software operations and some peopleware.

Use of Formal Methods at Amazon Web Services

Amazon engineers Chris Newcombe, Tim Rath, Fan Zhang, Bogdan Munteanu, Marc Brooker, and Michael Deardeuff describe how TLA+ can be useful in the development of industrial systems. This idea may have sounded more uncommon 6 years ago, while now there is a whole book on the subject: Practical TLA+ by Hillel Wayne (see review).

A precise, testable, well commented…