does anyone have experience with running Keboola Elasticsearch extractor on some decent amount of data?
For what use-cases and amounts of data you use this extractor?
Lets say we have 30 millions records per day spread in 4 indexes.
We need to get data to keboola to enrich other product data and this is some future estimate of data load.
Does this make sense or we should think about some pre-aggregation outside of KBC?
Thanks a lot for any thoughts
Ondřej Tichý 106 Senior BI specialist at integromat.com
Do you have questions for Ondřej Tichý?
Log in to ask Ondřej Tichý questions publicly or anonymously.
SQL coding question:
What is you opinion on using CTE vs. Create (temp) tables approach when writing complex transformations with lots of steps / or nested selects?
For a long time I am using create table when writing a complex SQL in order to split it to granular pieces. Wondering if CTE will be better option.
My reason is:
- in keboola crating temp tables in workspace do not matter much
- not see any performance issue
- can debug partial tables faster and more easier
Thanks for you ideas/opinions
Integromat is a fast growing Prague-based automation start-up (backed by a European tech ‘unicorn’ - Celonis). For our analytics team we need someone who will help us with mostly the backend side of data.
We are at the start of our journey to make Integromat truly data-driven (even if it sounds like a buzzword) and there are huge opportunities (as well as a few challenges) ahead, as we build an analytics capability to support the developments in the product and the business over the next 12 month and beyond.
- Basics are established but more advanced (fun?!) projects are ahead of us
- Data stack is built on no-devops techs like Keboola/Snowflake and we will need to add some more programmatic and devops parts
- We will be identifying the best architectural mix of new and existing technologies for our future needs
- We will be shifting to more secure dev-prod environment with help of automated tests
If you have any questions about Integromat or our data environment feel free to message me!
Link with the description:
I was wondering when Graph visualization of table dependencies in Storage component will be available in the new Transformation version (on Azure)?
We need to download data from DataDog and Airtable and I was wondering if anyone can share "template" configuration for mentioned services for Generic Extractor.
This will help us speed up integration of such sources.
I would say it will be nice to have sort of marketplace for this.