There is a stellar Keboola Academy course for Generic Extractor - I would encourage everyone to start there:
However, I think this guide might be pretty useful for kickstarting new configuration:
Resources
Official documentation:
- Mentioned Keboola Academy for Generic Extractor https://academy.keboola.com/courses/generic-extractor
- For Users: https://help.keboola.com/extractors/other/generic/
- For Developers: https://developers.keboola.com/extend/generic-extractor/
- Refer to Fisa’s repository for various samples of Generic Extractor configuration
- Github Repo of Generic Extractor with various examples
Steps to follow
- Authentication type
- Base URL
- Basic endpoint(s)
- Pagination
- Is Mapping needed? (in most cases - yes, especially for incremental loads)
(Important) If you bounce on some serious issues and you cannot get a solution from your peers, stop wasting your time further and switch to a custom component (Python, PHP, etc.)
Getting Started
To configure your first Generic Extractor, follow the tutorial for ‘Basic Configuration’. Also, review the following sections: Generic Extractor > Generic Extractor Tutorial >
- REST HTTP API Introduction
- JSON Introduction
- Basic Configuration (same as ‘tutorial’ above)
- Pagination Tutorial
- Jobs Tutorial (special emphasis on ‘Child Jobs’)
- Mapping Tutorial (copy and save the example under ‘Review’ - very useful)
Note: It is important to make use of Column Mapping feature in order to avoid occurrence of duplicates during extraction.
Further configuration
To understand the specific parts of an existing configuration, check the Configuration Map (aka Parameter Map) and review following sections: Generic Extractor > Configuration >
Iterate through each underlined item of the Configuration Map and review details. Alternatively, go to this page containing useful links for various sections of the Generic Extractor.
Review various Authentication types
Note: Don’t consume yourself with advanced authentication types e.g. OAuth, in the beginning. Start with ‘URL Query, ‘Basic HTTP’, ‘Login’ and know where to come back to in case another authentication type is required.
Adopted from an internal article by Michal Hruska @Keboola