Skip to content

Enhance streaming API control for long-running transfers #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

let-def
Copy link

@let-def let-def commented Mar 14, 2025

This PR refactors the streaming API to grant users finer control over HTTP transfers, particularly for long-running cases. Key changes include:

  • Early access to response metadata by exposes HTTP response code and headers as soon as they’re received (previously only available after the entire body was transferred).
  • Progress monitoring and cancellation, by adding hooks to track transfer progress and interrupt streams programmatically.

Together, they allow clients to abort transfers early based on metadata (e.g., invalid status code).

Limitations:

  • Malformed headers trigger an immediate failure without giving access to the response code.
  • The on_respond callback may receive an early response, but if the transfer fails, the final return will be an error (e.g., because of network issues or when aborted by user).

Alternatively, revisiting the response type could better represent the different point of failures:

type response = {  
  code: int option; (* None if the request failed *)  
  headers: (header list, string) result; (* The headers or an error message if they failed to parse; we could also provide the raw headers, though this is probably a fringe case *)
  body: ('a, curl_error) result;  
}

However, this would break backward compatibility and is still insufficient for async scenarios, which control flow is actually closer to (response_code * (headers * body io) io) io, though it’s syntactically cumbersome.

While not perfect, this API is strictly more flexible than the current implementation, addressing core pain points for streaming use cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant