Post

Generic Extractor templates

Hi all,

We need to download data from DataDog and Airtable and I was wondering if anyone can share "template" configuration for mentioned services for Generic Extractor.
This will help us speed up integration of such sources.

I would say it will be nice to have sort of marketplace for this.

Thanks
Ondra

9 replies

Liked by Leoš Gerlich and 3 others

Where am I?

In Keboola you can ask and answer questions and share your experience with others!

Matus Pavliscak

last year

For the Airtable, you should be off to a good start with this 😉

{
 "parameters": {
  "api": {
   "baseUrl": "https://api.airtable.com/v0/",
   "http": {
    "headers": {
     "Authorization": {
      "function": "concat",
      "args": [
       "Bearer ",
       {
        "attr": "#token"
       }
      ]
     }
    }
   },
   "pagination": {
    "method": "response.param",
    "responseParam": "offset",
    "queryParam": "offset"
   }
  },
  "config": {
   "id": "airtable",
   "debug": true,
   "#token": "API_TOKEN_GOES_HERE",
   "incrementalOutput": false,
   "jobs": [
    {
     "endpoint": "BASE_ID/TABLE_NAME",
     "dataField": "records"
    }
   ]
  }
 }
}

@Martin Fiser Maybe you could create another generic extractor template from this? 🙂

Ondřej Tichý

last year

Thanks Matus, we will try to implement it asap.
We owe you a beer ;-)

Matej Vidim

last year

Hi Matus, I just implemented your code and it works perfectly. Huge Thanks :)

Martin Fiser Keboola TEAM

last year

We are working on fully fledged Airtable component. Stay tuned.

Anyways, thanks for sharing the config here!

Matus Pavliscak

last year

Hi Ondrej, what are the API endpoints from the Datadog that you want to download?

Ondřej Tichý

last year

Hi Matus,

I do not know DataDog API well but basicaly we need to get data for one metric on aggregated level and breakdown by one dimension.

Matus Pavliscak

last year

Ondrej, It seems that you would need to use Datadog's Logs Analytics API, which requires POST call:

curl -X POST "https://api.datadoghq.com/api/v2/logs/analytics/aggregate" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_CLIENT_API_KEY}" \
...
...

I believe that the Generic extractor only supports GET calls, so I created a custom component in this case.

But I will let Keboola folks follow up on this if there is a way to set up a Generic extractor to send POST calls :) cc @Martin Fiser

Martin Fiser Keboola TEAM

last year

I think @Ondřej Tichý meant an extractor config. (we also have a Generic Writer now @Matus Pavliscak ).

At the moment I don't have a GenEx config on hand, though it looks like its a simple REST API with token auth.

Marcus Wong Keboola TEAM

last year

hey @Matus Pavliscak RE: method:

https://developers.keboola.com/extend/generic-extractor/configuration/config/jobs/#method

Generic Extractor templates

Where am I?

Related Posts

Share Post