#162 long running Kafka producer and client Python script
Description
EditContext:
This enterprise workflow engine will be used to run massive durable workflows, like long running data producers, ingestion tools, ETL pipelines, etc.
The goal is to build building blocks and tooling for that.
We need to build one workflow, with a couple of independent tasks and steps:
1- one producer, it will produces messages and will sit there indefinitely, until killed or send a signal to pause
2- three clients, which will listen to a consumer group.
5- a kafka health monitoring step. in next phases, this will be a durable cron to check health of kafka cluster
6- a long running task, which will send a signal to 3 clients and fetch their results and dump it into end of a file in /tmp/kafka_results.txt
now, since these are long running tools mostly, they should go with activity worker (excep the monitoring one)
the python run tool excepts artifact id, so we need to first upload the scripts to artifactory as zip and get their id.
notes:
- we have a kafka server available.
- read ~/client.py and ~/producer.py for the client and producer code.
Your task:
- create a folder in examples/etl/kafka_pipeline
- copy scripts there
- upload python scripts (client and producer) and get their artifact ids
- create a python dsl workflow, define a workflow to define the aforementioned workflow above.
remember this:
- since the tasks are long running, activity workers (and generally all workers) must have the ability to drain and accept signals, pause and continue.
- the signalling is important, as these are long running tasks
- think about using async deferal for some of the tasks, although we need to have signalling capability. these scripts must be able to response to each other.
ultrathink and make sure this will be an enterprise example.
all files related to the issue should be in that folder.
Any architecturally proven bug you should create a critical ticket and fix.
Always report back the progress as comments here in this issue #162.
run `hwe dsl-prompt` to know more about how to create workflows.
if you find any problem is prompt which is misleading, you must fix it in engine/cli/prompt.txt to make it better.
YOU must use activity workers, the servers won't need restart as you are just writing the workflow. server has the capability already.
everytime you change the code, you must uplaod it to artifactory and get new artifactory id
Comments
Loading comments...
Context
Loading context...
Audit History
View AllLoading audit history...