I have a need to use keboola to check for the existence of a file in s3. Is there a good way to accomplish this?
I have a need to use keboola to check for the existence of a file in s3. Is there a good way to accomplish this?
Where am I?
In Keboola you can ask and answer questions and share your experience with others!
yes, or a python code that utilized Boto3 library to check that out.
Yes this will work. Unless it is a custom component, I would recommend the generic extractor approach since it allows you to encrypt your s3 tokens.
Cool use case! I would think generic extractor to call the ListObject API , do you agree @Martin Fiser , @David Esner or @Leo Chan ?
I think this would be a bit complicated to setup but doable. Authentication header and signature must be constructed via HMAC hash: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html It would need to use the unsigned payload option. The extractor supports sha256 https://developers.keboola.com/extend/generic-extractor/functions/#hash_hmac so it should be doable.
Or test that out in the python code (using BOTO which does plenty of stuff for you) and then bundle that as a component - yes, there is an overhead but not really that big and you would have it simpler to work on that actual code - plus it can enrich the community :)