Skip to content

feat(backend): Support flow control (backpressure) visualization in backend#4270

Draft
yunyad wants to merge 12 commits intoapache:mainfrom
yunyad:yunyad-flow-control-vis
Draft

feat(backend): Support flow control (backpressure) visualization in backend#4270
yunyad wants to merge 12 commits intoapache:mainfrom
yunyad:yunyad-flow-control-vis

Conversation

@yunyad
Copy link
Copy Markdown
Contributor

@yunyad yunyad commented Mar 7, 2026

What changes were proposed in this PR?

This PR introduces backend support for flow control (backpressure) visualization in workflow execution.

During workflow execution, operators may experience backpressure when they receive data faster than they can process. In such cases, the operator signals its upstream operators to temporarily slow down or pause data transmission. This mechanism helps prevent buffer overflow and maintains stable execution.

This PR exposes the backpressure status of operators and edges from the backend so that the frontend can visualize the flow control behavior of the workflow.

The main changes include:

  • Adding backend support to collect and propagate flow control / backpressure status
  • Extending execution monitoring data structures to include flow control information
  • Merge per-worker physical channel usage into logical workflow-edge statistics so the frontend can render pressure on graph edges.
  • Keep the feature visualization-only: this PR does not change the existing flow-control algorithm, thresholds, scheduling behavior, or acknowledgement semantics. It only exposes and aggregates the runtime values already used by the backpressure mechanism.

In short, this PR makes the backend capable of reporting where backpressure is happening and how much queued data is currently contributing to flow control, so the frontend can visualize execution bottlenecks directly on workflow edges.

Any related issues, documentation, discussions?

This PR fixes Issue #4321.

How was this PR tested?

This PR was tested manually in a local development environment.

The following scenarios were verified:

  • Confirmed the backend successfully collects flow-control usage from workers during workflow execution.
  • Confirmed execution statistics events include edge-level usage data and the configured flow-control credit limit.
  • Verified the both Scala and Python worker reports more realistic queue usage values.
  • Ran workflows with intentionally slowed downstream operators to create backpressure and verified that the reported edge statistics increase under congestion and decrease as queued data is drained.

Was this PR authored or co-authored using generative AI tooling?

@yunyad yunyad self-assigned this Mar 7, 2026
@github-actions github-actions bot added engine python frontend Changes related to the frontend GUI labels Mar 7, 2026
@yunyad yunyad marked this pull request as draft March 7, 2026 21:32
@yunyad yunyad changed the title feat(backend): flow control visualization feat(backend): Support flow control (backpressure) visualization in backend and frontend Mar 8, 2026
@yunyad yunyad changed the title feat(backend): Support flow control (backpressure) visualization in backend and frontend feat(backend): [2/3] Support flow control (backpressure) visualization in backend Mar 15, 2026
@yunyad yunyad changed the title feat(backend): [2/3] Support flow control (backpressure) visualization in backend feat(backend): Support flow control (backpressure) visualization in backend Mar 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

engine frontend Changes related to the frontend GUI python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant