Hi all, some of you are already using the Google Apps script that runs in Google sheet and creates option for the user to trigger a Keboola Orchestration directly from the sheet. We have created and updated version of that which supports the new queue, e.g. Orchestrations v2 or Flows. 

Here's a guide on how to deploy it in you sheet:

1. Navigate to Extensions -> Apps script
2. Insert the entire code into the Code.gs (replace the existing content)
   1. Find the script attached on the post.
3. Save
4. Refresh or reopen the sheet.
5. New tab named Keboola will appear.
6. Hit reset settings.
7. Once you click Run orchestration you will be asked to provide url to your orchestration + Storage token (that will be saved for next run)
8. Now you can trigger your orchestration to collect the data directly from the sheet

Hello community, we have a new Zoho CRM extractor component ready for Beta testing. Is there anyone who would be willing to be part of the testing? 

Your help will be much appreciated and we'll be able to tune it to your liking.

Thank you!

Hi everyone, for all Czechs around you may find useful this new processor  we recently published. It allows you to process PDF account statements issued by Komercni Banka and store them in tabular form in Storage! It works best in combination with the IMAP Email Content datasource. Just schedule the statements to your email and set up the configuration.

Hi community! I am pleased to announce that we just added a new data source component that allows you to extract layout data from the FileMaker relational database via FileMaker Data API. 

Hello community, I'd like to share with you a new Chrome extension useful for advanced users, who often trigger debug jobs on components to debug processors or just need to work with component configurations via API for whatever reason. It enables you to perform these tasks directly from the component configuration page. 

The extension is available in the Chrome Web Store

What is it useful for?:

• Show current user's API token.
  • Just click the button and the current token is displayed in the popup. No need to reset the token. All works only in browser, the token is not stored anywhere.
• Run Debug job
  • Trigger debug job directly from the component page. Super useful for debugging of the processors
• Enable processors
  • Enable processors UI element in any component configuration. All components support them in theory, but not all of them will provide you the UI. After clicking the button you'll be able to set up the processors in the UI. No API call needed.
• Update configuration
  • Display the raw JSON of the currently opened configuration and update it directly from the component.
• Run Tag
  • This one is useful for component developers. You can run alternate tag of the docker image with the currently opened configuration. This speeds up the debugging and development of the new components - no need to deploy new version to all users in order to test it in the platform.

The extension currently supports both old and new queue. It also supports orchestrations-v2. So give it a try and let me know if you have any suggestions for improvements.

I’m happy to announce that we have just released a new updated version of the Generic Writer along with a new public documentation. This update brings several new things and also simplification of the configuration structure. The main changes are:

• Brand new documentation - https://developers.keboola.com/extend/generic-writer/configuration/
• Updated configuration structure
  • The structure is simplified and should be more understandable. Also it lays grounds for templatization of the component and row based configuration that will come in the future.
• Default api properties options
  • Similarly to Generic Extractor specify default HTTP options to be sent with each request
• Authentication methods
  • Setup Basic authentication method without need to replicate via functions
  • Unlocks the component for many more methods to be added in the future.
• Retry config
  • Allows users to control retry strategy. 

NOTE The update is fully backward compatible with old structure and previously created configurations will continue to work unaffected. To enable the new features for the old ones, please reach out to us via the support button and we can convert your current configuration into the new structure.

I am glad to announce the new addition to our component list, the Imap Email Contents and Attachments data source. As the name suggest, it allows you to get email content, e.g. body, header and additional metadata, and / or also email attachments.

The biggest advantage to other similar sources is that it uses IMAP protocol which means your emails will remain intact in your inbox and you can use rich queries to filter the emails (or attachments). You can also use it with any provider that supports IMAP (which is literally all of them), so you're not limited only to Gmail, Outlook, etc. Combine it with processors of your choice and you have a powerful integration tool to ingest data from sources that do not provide API but do scheduled exports to emails. 

I would like to share with you some advanced tips on how to modify the behaviour of Event Triggered Orchestration. As you know, the event trigger is very useful when you need to trigger a downstream process only when there is some import on a specific table. Current orchestration trigger reacts on any import event, which in practice means that it may trigger even if there is no data coming in within that import event. A typical example is a Transformation that creates an empty table. If there is an output mapping for that table, the import event will be always generated.

It is very often that we need to modify this behaviour to trigger orchestration only when the monitored table is actually updated or even to react to some more complex condition. We can achieve this even using the current event trigger using a combination of a Python transformation and a “trigger” table that we will use to react on in our event trigger.

Example conditional trigger process:

The above process consists of following steps:

1. Extract data to Result Table
2. Run Transformation that takes Result Table and Modified Rows table on input. Filter Result Table data by last modified timestamp from previous run (Mofified Rows loopback) and overwrites the Modified Rows table with the result.
3. Python Transformations checks whether the Modified Rows contains any data and if it does it outputs row to sys_trigger table
4. Downstream Process is triggered.

(Alternatively you can skip the step 2 completely and just load data changed in last X into the Python transformation.)

Example Python Transformation

The key is to generate the import event only when a certain condition is met. This can be achieved using Python transformation with no output mapping but outputting tables using the table manifest instead. You do not need to understand the details about output manifests though. We provide a convenient Python library that abstracts this for you and which is by default present in all Python workspaces / transformations.

Example Python Code:

import csv
from datetime import datetime

from keboola.component import CommonInterface

ci = CommonInterface()
new_rows_table = ci.get_input_table_definition_by_name('modified_rows.csv')

with open(new_rows_table.full_path, 'r') as inp:
    inp.readline()
    lin = inp.readline()
    has_data = len(lin)

if not has_data:
    print("No new rows")
    exit(0)

result_table_id = 'out.c-sys_trigger.trigger_systable'
out_table = ci.create_out_table_definition('data.csv', destination=result_table_id, incremental=True)
print(has_data)
now = datetime.now()
with open(out_table.full_path, 'w+', newline='') as out:
    writer = csv.DictWriter(out, fieldnames=['timestamp', 'event'], lineterminator='\n', delimiter=',')
    writer.writeheader()
    writer.writerow({"timestamp": now.strftime("%H:%M:%S"), "event": "success"})

print('Triggering')

# finally write result manifest 
ci.write_manifest(out_table)

Pleat note, that User and Password authentication method of our Tableau Trigger application will be deprecated starting February 1, 2022. Any configurations using this method may start failing after this day. Please change the authentication method to PATs to avoid any outages.

Please find more information in our status post.

Hello everyone, we are working on releasing a new Xero extractor, specifically the Accounting API. I would like to ask here whether you have any usecases for it and what would be the core functionality that you'd require from it. e.g. incremental loading, specific endpoints, different API etc. 

So far we think of implementing following endpoints:

• Accounts
• BankTransactions
• Contacts
• Invoices
• Overpayments
• Prepayments
• PurchaseOrders
• CreditNotes
• ManualJournals
• Journals

If the above list is ok and you'd like to see the component in our catalog just give the post a like.

Thank you!