New Component - Drop Columns Processor

Hi all, we have just published a new processor Drop Columns that is useful when you want to exclude some columns from your incoming sources (e.g. PIIs that you do not need).

To use this you can just add this processor into the after section. The bellow configuration excludes columns Location and Type from all tables that start with test prefix, e.g. test.csv:

{
  "definition": {
    "component": "kds-team.processor-drop-columns"
  },
  "parameters": {
    "table_mapping": {
      "test*": [
        "Location",
        "Type"
      ]
    }
  }
}

Note that the table name must match what the component output in the data/out/tables . The easiest way to see that is to run a debug job