Hello everyone, we are constantly working on extending and improving our existing components. This time we'd like to get your opinion on the existing Generic Extractor which is one of the most powerful components in our collection. We are already considering some improvements such as easier mapping, predefined configuration templates to speedup the setup, etc.
I'd like to start a discussion here and ask you what is your experience with it and what functionalities are you missing or what could be improved?
Hi David, I might have one :-) Generic extractor and cursor/next page link returned in header
Yup!
Is there (any) way to support Oauth in Generic?
yes:
OAuth 2.0 Authentication | Keboola Connection Developers Knowledge Base
but from the definition a component has to be registered
That's what I meant, would there be a way to avoid the need for the registration...
You can skip the registration if you supply the credentials in the
authorizationsection of the config. That means that you have to go through the OAuth process yourself and it's some work. But that's sort of the point of OAuth :)The biggest pains for me are the iterations and mapping which looks like you know about already.
It would be great to have a "wizard" mode to help guide the "non data engineer" through the setup. If you're not familar with the generic extractor it is very difficult and time consuming to get up to speed.
Also would love to see more love for generic writer as I'm seeing more use cases for business automation.
There are some nice low-code API setup examples here from tray.io: https://tray.io/documentation/connectors/core/http-client/#overview
I like the authentication setup and loop logic
Will check it out, thanks!
btw. if you have a graphql based APi we also have a dedicated extractor and writer for those.
I would appreciate some kind of loop availability. Here is why: I have a case when I need to extract data from an API for a set of items (items serve as input to API parameters). Unfortunately, the API allows me to input to parameters only one item at a time (it does not accept an array of items). To my knowledge (and I believe I discussed it with your support team as well), I cannot create a simple loop that would loop through the set and extract the data.
Thanks! I suppose you would like have the input parameters dynamic right? Otherwise the Iterations are usable for this. What if you could add the parameters for iterations in an input table => 1 row =1 iteration, similarly to the current Generic Writer?
I don't remember why I have not tried the iteration or why it wasn't recommended to me by the support team. Probably because with the iteration I would have to write a really long extractor in order to iterate through the items. Having an input table as a parameter would be a good option.
I will throw the first one in the case ppl are shy :)
I would love the ability to pick a config from some sort of shared config library, so users do not have to reinvent wheels - there is a high chance someone has worked on that exact source already...