Fluent Bit Modify Nested JSON log with Lua script

Fluent Bit allows users to modify log data through a Modify filter with conditions. Find Part 1 of this blog for basic filters here: Fluent Bit Modify Log data with Modify Filter plugin Examples

Advanced Filter with Lua

This is not always enough for complex tasks, however, so FluentBit allows users to create custom Lua scripts for use in filters. This can manipulate or filter logs with more complexity like modifying nested JSON based on condition. Let's see some examples below.

In the following example, we have to change message which is inside another object status

{
   "status":{
   "code":200,
   "message":"true"
   }
}

We are not covering the basics of Lua here. If you want to learn the basics of Lua script, see Learn Lua Script .

Writing Lua script for the Filter Logic

Write the Lua script in a script file and add that to the fluentbit configuration

Step 1 : Create script file

Create an empty file and name testScript.lua on the same directory of fluentbit config file

Step 2 : Create a function inside the script file

The code below will replace status.message to "Success" if the status.code is 200

function replace_status(tag,timestamp,record)
  if status.code == 200 then
    status.message = "Success"
  end
  return 0,timestamp,record
end

Step 3 : Add the Lua script to filter condition

In the filter section, the function name and script file name are mentioned which are in the same location as the configuration file (add full path if it is on different path).

[INPUT]
    Name tcp
    Port 8888
[FILTER]
    Name lua
    Match *
    script  test.lua
    call    replace_status
[OUTPUT]
    Name stdout
    Match **

Sample Input

{
 "status":{
   "code":200,
   "message":"true"
  }
}

Sample Output

{
 "status":{
   "code":200,
   "message":"Success"
  }
}

Mutate data of Nested child based on condition

Let us check a bit more advanced function where we need to replace a list of child item, see the sample below

Sample Input

{
 "data":[{
   "item_staus":0,
   "message":"nil"
  },
{
   "item_staus":1,
   "message":"nil"
  },
{
   "item_staus":0,
   "message":"nil"
  }]
}

Replace the message to "True" if item_status is 1 and "False" if item_status is 0

function replace_status(tag, timestamp, record)
    data = record.data
    for itemCount = 1, #workouts do
        if workouts[itemCount].item_status == 1 then
            workouts[itemCount].message = "True"
        else
            workouts[nameCount].message = "False"     
        end
    end
    return 0, timestamp, record
end

Sample Output

{
 "data":[{
   "item_staus":0,
   "message":"False"
  },
{
   "item_staus":1,
   "message":"True"
  },
{
   "item_staus":0,
   "message":"False"
  }]
}

Advanced loops and Conditions in Lua script can be used to filter, mutate and enrich data while passing through fluent bit

TLDR;

• How to use filter in Fluent Bit to modify fields with Lua script

• Loop through child object and use conditions to replace

If you are not familiar with fluent bit, see related blogs to learn how to run these on fluent bit with and without docker.