Robocorp 3.1.4 and robocorp-log 3.1.5

Memory growth in output redaction

robocorp-log redacts sensitive values from the log output. Every value registered for hiding, whether explicitly through hide_from_output() or automatically because it was assigned to a variable with a sensitive-looking name such as password, is added to a single regular expression that is rebuilt each time a new value arrives.

Rebuilding that expression had a hidden cost (#489, fixed in #491). Each rebuild produced a new, slightly longer pattern, and CPython's regular expression cache kept the last several hundred of them alive. In a long-running process that registers a new sensitive value per processed record, the retained patterns grew without bound. One reported batch job reached roughly 13 GB of memory over seven hours before ending with:

Redaction itself was never at risk: every registered value was still hidden from the output. The defect only affected memory use and, before that, throughput.

The active pattern is now compiled outside the process-wide cache, so only the current version is kept. Measured with the reproduction from the issue, registering 2,400 values took about 34 MB of memory instead of roughly 800 MB, with redaction output unchanged.

Who should upgrade

You are most likely affected if all of the following apply:

• The automation runs for hours in one process, for example a batch loop over many records.
• A sensitive-looking variable name, or a pattern added with add_sensitive_variable_name_pattern(), matches a value that changes on every iteration.
• Memory use climbs steadily during the run, or the run slows down over time.

If you saw MemoryError raised from robocorp/log/_log_redacter.py, this release is the fix.

Upgrading

robocorp 3.1.4 pulls in the fixed robocorp-log automatically, so installing the meta package is enough:

If you depend on robocorp-log directly, upgrade it to 3.1.5.

Released packages

• package robocorp 3.1.4 (meta package)
• package robocorp-log 3.1.5