[🚀 Feature]: Implement high level BiDi network commands

[titusfortner]

Feature and motivation

At the Selenium Dev Summit we agreed on this API to be generally applied across the bindings; we'll keep this labeled beta while we make sure that it works for what is needed

We want the methods to be accessible from a network() method available directly from the Driver class (e.g., driver.network.addRequestHandler(), driver.network().addRequestHandler()). We can't do everything just like this in all the languages, because, for example, .NET uses events with a += and -= for adding and removing handler events so we don't went "add" and "remove" methods.

Implementations:

Method	Java	NodeJS	Python	Ruby	.NET
addRequestHandler()				#14751
removeRequestHandler()				#14751
clearRequestHandlers()				#14751
addResponseHandler()
removeResponseHandler()
clearResponseHandlers()
addAuthenticationHandler()	#14334	#14345		#14523
removeAuthenticationHandler()	#14334	#14345		#14523
clearAuthenticationHandlers()	#14334	#14345		#14523

Considerations:

If we can figure out how to get the "add" methods return an id that can be used by the "remove" methods, that would be a lot easier for or users. Might be too complicated to implement.

[titusfortner]

I've been thinking about this, and I think the add methods should return an id. This is going to be much easier for languages that want to pass in lambdas without storing a reference to the location os the object in memory.

This is how moz/addon/install endpoint works and the id gets passed into the payload to remove the addon.

I was looking for common wisdom here and it generally seems to match — https://softwareengineering.stackexchange.com/questions/314066/restful-api-should-i-be-returning-the-object-that-was-created-updated

[p0deje]

I've been thinking about this, and I think the add methods should return an id. This is going to be much easier for languages that want to pass in lambdas without storing a reference to the location os the object in memory.

Totally fine by me!

[pujagani]

Can someone help refresh my memory on the "addRequestHandler()"? - I vaguely remember us discussing the current C# implementation using the matcher and transformer https://github.com/SeleniumHQ/seleniumhq.github.io/blob/c6769ef064d8bc3ffb6343fba3c10ab4ddbad78c/examples/dotnet/SeleniumDocs/BiDi/CDP/NetworkTest.cs#L85. Is that what we want? How would that look like?

[pujagani]

@titusfortner @diemol @p0deje

[p0deje]

Can someone help refresh my memory on the "addRequestHandler()"? - I vaguely remember us discussing the current C# implementation using the matcher and transformer https://github.com/SeleniumHQ/seleniumhq.github.io/blob/c6769ef064d8bc3ffb6343fba3c10ab4ddbad78c/examples/dotnet/SeleniumDocs/BiDi/CDP/NetworkTest.cs#L85. Is that what we want? How would that look like?

I might be wrong, but IIRC the idea was to let addRequestHandler to accept two arguments:

• filter/matcher for request to be handled
• callback that receives an original request and return a handled request

[nvborisenko]

I've been thinking about this, and I think the add methods should return an id. This is going to be much easier for languages that want to pass in lambdas without storing a reference to the location os the object in memory.

I guess id can be any object. In .net I implemented it like:

var subscription = await context.Network.OnBeforeRequestSentAsync(Console.WriteLine);
await context.NavigateAsync("https://selenium.dev");
await subscription.UnsubscribeAsync();

Or network interception:

await using var intercept = await context.Network.AddInterceptedRequestAsync(async args =>
{
  await args.Request.Request.ContinueAsync(new() { Method = "POST" });
});

Subscription is a disposable object. The same for Intercept object. Generally saying it is object, not just magic ID.

[junjuping]

Can I use the INetwork-NetworkResponseReceived feature in Selenium. WebDriver version 4.24.0 in Firefox?

[bogishvili]

Hi I am trying to replace devtools with BiDi in my project. Can I get request and response bodies from BeforeRequestSent and ResponseDetails? any form like json or byte sequence or string would be great.

[pujagani]

@bogishvili I understand your use case here. However, we can support what is available in the WebDriver BiDi spec https://w3c.github.io/webdriver-bidi/#module-network. Currently, there is no way defined to get the request and response bodies. There is an issue to track this though w3c/webdriver-bidi#747. So this can be supported in the future once it is present in the spec and supported by the browsers.

[Vinodh7887]

is there any update when we should be able to extract Network Response body using BiDi?

[pujagani]

Please track the issue stated above w3c/webdriver-bidi#747.

[thulasipavankumar]

@pujagani
is there an issue that tracks the read request payload in Java.

The use case is -> for requests with graphql the mocking really depends on the request body rather than the request url since all the requests are sent to /graphql

[jkim2492]

Hi all! I'd opened a PR #14738 a few months ago for the Python implementations of some request handler methods. I would really appreciate your review/feedback on it!

[pujagani]

@pujagani is there an issue that tracks the read request payload in Java.

The use case is -> for requests with graphql the mocking really depends on the request body rather than the request url since all the requests are sent to /graphql

Thank you for this. Yes, I am working on adding those pieces so the user can get flexibility to match the request however they prefer. Since, it is part of this issue, it will not be tracked separately. But the PR will be linked to this.

[pujagani]

@thulasipavankumar, Please note that there is currently no way to get the request body with BiDi. So, filtering based on that is not possible since it is not part of the spec and hence not implemented by browsers. However, it is possible to filter by URL, headers and method.

[nvborisenko]

One more time for me please, this proposal is about high-level API, meaning this API should NOT expose low-level BiDi types. And meaning we should introduce abstraction layer?

[thulasipavankumar]

@thulasipavankumar, Please note that there is currently no way to get the request body with BiDi. So, filtering based on that is not possible since it is not part of the spec and hence not implemented by browsers. However, it is possible to filter by URL, headers and method.

@pujagani thanks for the reply.

In specific use case for testing graphql endpoints, unfortunately the headers,url and method are all same for every request

[shbenzer]

With the merge of #14592, Python is now at the same progress as Ruby