Hi all, some of you are already using the Google Apps script that runs in Google sheet and creates option for the user to trigger a Keboola Orchestration directly from the sheet. We have created and updated version of that which supports the new queue, e.g. Orchestrations v2 or Flows.Â
Here's a guide on how to deploy it in you sheet:
Navigate to Extensions -> Apps script
Insert the entire code into the Code.gs (replace the existing content)
Find the script attached on the post.
Save
Refresh or reopen the sheet.
New tab named Keboola will appear.
Hit reset settings.
Once you click Run orchestration you will be asked to provide url to your orchestration + Storage token (that will be saved for next run)
Now you can trigger your orchestration to collect the data directly from the sheet
Hi everyone, for all Czechs around you may find useful this new processor  we recently published. It allows you to process PDF account statements issued by Komercni Banka and store them in tabular form in Storage! It works best in combination with the IMAP Email Content datasource. Just schedule the statements to your email and set up the configuration.
Hi community! I am pleased to announce that we just added a new data source component that allows you to extract layout data from the FileMaker relational database via FileMaker Data API.Â
Hello community, I'd like to share with you a new Chrome extension useful for advanced users, who often trigger debug jobs on components to debug processors or just need to work with component configurations via API for whatever reason. It enables you to perform these tasks directly from the component configuration page.Â
Just click the button and the current token is displayed in the popup. No need to reset the token. All works only in browser, the token is not stored anywhere.
Run Debug job
Trigger debug job directly from the component page. Super useful for debugging of the processors
Enable processors
Enable processors UI element in any component configuration. All components support them in theory, but not all of them will provide you the UI. After clicking the button you'll be able to set up the processors in the UI. No API call needed.
Update configuration
Display the raw JSON of the currently opened configuration and update it directly from the component.
Run Tag
This one is useful for component developers. You can run alternate tag of the docker image with the currently opened configuration. This speeds up the debugging and development of the new components - no need to deploy new version to all users in order to test it in the platform.
The extension currently supports both old and new queue. It also supports orchestrations-v2. So give it a try and let me know if you have any suggestions for improvements.
Iâm happy to announce that we have just released a new updated version of the Generic Writer along with a new public documentation. This update brings several new things and also simplification of the configuration structure. The main changes are:
The structure is simplified and should be more understandable. Also it lays grounds for templatization of the component and row based configuration that will come in the future.
Default api properties options
Similarly to Generic Extractor specify default HTTP options to be sent with each request
Authentication methods
Setup Basic authentication method without need to replicate via functions
Unlocks the component for many more methods to be added in the future.
Retry config
Allows users to control retry strategy.Â
NOTE The update is fully backward compatible with old structure and previously created configurations will continue to work unaffected. To enable the new features for the old ones, please reach out to us via the support button and we can convert your current configuration into the new structure.
I am glad to announce the new addition to our component list, the Imap Email Contents and Attachments data source. As the name suggest, it allows you to get email content, e.g. body, header and additional metadata, and / or also email attachments.
The biggest advantage to other similar sources is that it uses IMAP protocol which means your emails will remain intact in your inbox and you can use rich queries to filter the emails (or attachments). You can also use it with any provider that supports IMAP (which is literally all of them), so you're not limited only to Gmail, Outlook, etc. Combine it with processors of your choice and you have a powerful integration tool to ingest data from sources that do not provide API but do scheduled exports to emails.Â
I would like to share with you some advanced tips on how to modify the behaviour of Event Triggered Orchestration. As you know, the event trigger is very useful when you need to trigger a downstream process only when there is some import on a specific table. Current orchestration trigger reacts on any import event, which in practice means that it may trigger even if there is no data coming in within that import event. A typical example is a Transformation that creates an empty table. If there is an output mapping for that table, the import event will be always generated.
It is very often that we need to modify this behaviour to trigger orchestration only when the monitored table is actually updated or even to react to some more complex condition. We can achieve this even using the current event trigger using a combination of a Python transformation and a âtriggerâ table that we will use to react on in our event trigger.
Example conditional trigger process:
The above process consists of following steps:
Extract data to Result Table
Run Transformation that takes Result Table and Modified Rows table on input. Filter Result Table data by last modified timestamp from previous run (Mofified Rows loopback) and overwrites the Modified Rows table with the result.
Python Transformations checks whether the Modified Rows contains any data and if it does it outputs row to sys_trigger table
Downstream Process is triggered.
(Alternatively you can skip the step 2 completely and just load data changed in last X into the Python transformation.)
Example Python Transformation
The key is to generate the import event only when a certain condition is met. This can be achieved using Python transformation with no output mapping but outputting tables using the table manifest instead. You do not need to understand the details about output manifests though. We provide a convenient Python library that abstracts this for you and which is by default present in all Python workspaces / transformations.
Example Python Code:
import csv
from datetime import datetime
from keboola.component import CommonInterface
ci = CommonInterface()
new_rows_table = ci.get_input_table_definition_by_name('modified_rows.csv')
with open(new_rows_table.full_path, 'r') as inp:
inp.readline()
lin = inp.readline()
has_data = len(lin)
if not has_data:
print("No new rows")
exit(0)
result_table_id = 'out.c-sys_trigger.trigger_systable'
out_table = ci.create_out_table_definition('data.csv', destination=result_table_id, incremental=True)
print(has_data)
now = datetime.now()
with open(out_table.full_path, 'w+', newline='') as out:
writer = csv.DictWriter(out, fieldnames=['timestamp', 'event'], lineterminator='\n', delimiter=',')
writer.writeheader()
writer.writerow({"timestamp": now.strftime("%H:%M:%S"), "event": "success"})
print('Triggering')
# finally write result manifest
ci.write_manifest(out_table)
Pleat note, that User and Password authentication method of our Tableau Trigger applicationwill be deprecated starting February 1, 2022. Any configurations using this method may start failing after this day. Please change the authentication method to PATs to avoid any outages.
Hello everyone, we are working on releasing a new Xero extractor, specifically the Accounting API. I would like to ask here whether you have any usecases for it and what would be the core functionality that you'd require from it. e.g. incremental loading, specific endpoints, different API etc.Â
So far we think of implementing following endpoints:
Accounts
BankTransactions
Contacts
Invoices
Overpayments
Prepayments
PurchaseOrders
CreditNotes
ManualJournals
Journals
If the above list is ok and you'd like to see the component in our catalog just give the post a like.