| View previous topic :: View next topic |
| Author |
Message |
Vieri
l33t
Joined: 18 Dec 2005
Posts: 901
|
 Posted: Mon Oct 28, 2024 7:50 pm Post subject: Perl: API client processing responses |
 |
|
Hi,
I need to write a Perl script that gets data from a server and processes all of it.
The following PoC code works OK when run without the sleep() call that simulates a call to a function that processes the data chunks received.
| Code: | my $app_ua_request = HTTP::Request->new('POST' => $app_url);
$app_ua_request->header(
'Content-Type' => 'application/x-www-form-urlencoded',
);
$app_ua_request->content($app_data);
my $app_ua_response = $app_ua->request($app_ua_request, sub {
my ($chunk, $app_ua_response, $protocol) = @_;
my $fromjson = from_json($chunk);
my $app_response = $fromjson->{'response'};
# process $app_response (simulate by sleeping)
# sleep(10);
});
if ($app_ua_response->is_success) {
print("replied all");
} else {
print("reply error: ".$app_ua_response->status_line);
} |
In a sample run, the "sub routine" is called exactly as many times as server replies are received.
However, if I enable the call to sleep() or the actual blocking function that processes the chunk data the "sub routine" is called just for the first 2 data chunks. The code execution then jumps to "is_success" and prints "replied all".
The Perl script is obviously receiving all of the server replies before each "sub routine" has a chance to finish.
Or maybe something's timing out in the meantime.
I'm using this to call the API service:
| Code: | my $app_ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 1 });
$app_ua->agent($UA_string);
$app_ua->env_proxy;
$app_ua->conn_cache(LWP::ConnCache->new());
$app_ua->timeout(120); |
The server takes more than 20 seconds but less than 120 to stop sending data.
What can I try?
If the code execution reaches "is_success", can I wait for all the "sub routines" to finish before exiting? How?
Regards,
Vieri |
|
| Back to top |
|
 |
turtles
Veteran
Joined: 31 Dec 2004
Posts: 1696
|
| Posted: Mon Oct 28, 2024 10:02 pm Post subject: |
|
|
Are you trying to process data as its being read/received from the server?
Seems like you need to have one function that makes the client request,
another that listens for response data from the server and stores it in a data structure.
and then a final function that does any processing to the stored data structure.
_________________
Donate to Gentoo |
|
| Back to top |
|
|
turtles
Veteran
Joined: 31 Dec 2004
Posts: 1696
|
| Posted: Mon Oct 28, 2024 10:58 pm Post subject: |
|
|
Here is a working example:
| Code: |
#!/usr/bin/env perl
use strict;
use warnings;
use diagnostics;
use LWP::UserAgent;
use HTTP::Request;
use JSON::MaybeXS qw(from_json);
use Data::Dumper;
my $app_url = "https://httpbin.org/anything/anything/";
my $app_data = "key1=value1&key2=value2";
my $app_ua_request = HTTP::Request->new( 'POST' => $app_url );
$app_ua_request->header( 'Content-Type' => 'application/x-www-form-urlencoded', );
$app_ua_request->content($app_data);
my $ua = LWP::UserAgent->new;
my $app_ua_response = $ua->request($app_ua_request);
if ( ($app_ua_response) and ( $app_ua_response->is_success ) ) {
# call a post process function
print $app_ua_response->decoded_content;
print "\n";
print "dump data structure response \n";
print Dumper $app_ua_response;
print "\n";
# post_process($app_ua_response);
}
elsif ($app_ua_response) {
print "dump data structure response no success\n";
print Dumper $app_ua_response;
#print "status_line: '$app_ua_response->status_line' \n";
}
else { die( "error" . $! ) }
sub post_process {
my $response = shift;
# do stuff ...
}
1;
|
_________________
Donate to Gentoo |
|
| Back to top |
|
|
Vieri
l33t
Joined: 18 Dec 2005
Posts: 901
|
| Posted: Tue Oct 29, 2024 10:49 am Post subject: |
|
|
Thanks for the reply and code.
As far as I can tell, your code is waiting for the full reply from the server before it actually processes the data.
In my case, the server is replying chunks of data, and I need to process each chunk as it arrives (I also need to wait for the process to finish before reading the next chunk).
So it can't be async.
That's why in my first post I gave the sleep(10) example as a replacement for the chunk data processing function.
Here's another code example using handlers:
| Code: | my $app_ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 1 });
$app_ua->agent($UA_string);
$app_ua->env_proxy;
$app_ua->conn_cache(LWP::ConnCache->new());
$app_ua->timeout(100);
$app_ua->add_handler(response_data => sub {
my ($response, $ua, $handler, $data) = @_;
print("response_data\n");
sleep(10);
return 1;
});
$app_ua->add_handler(response_done => sub {
my ($response, $ua, $handler) = @_;
print("response_done\n");
return 1;
}); |
The call to sleep(10) is there just to simulate a blocking function I need to call and wait for it's output.
If I run this code, the response_data subroutine is called each time the server sends a chunk.
However, there's a race condition in that the response_done subroutine is called BEFORE all of the data chunks had had the time to be processed by the response_data subroutine.
In other words, a lot of data chunks are discarded/ignored once "response_done" is reached.
That's because the response_data handler sub is slower (because of sleep() or other processing function) than the total time in takes for the server to fully reply and terminate.
Is there a way to "tell LWP" to "wait" for all response_data subs to finish even if response_done was reached? |
|
| Back to top |
|
|
turtles
Veteran
Joined: 31 Dec 2004
Posts: 1696
|
| Posted: Tue Oct 29, 2024 4:43 pm Post subject: |
|
|
| Vieri wrote: | Thanks for the reply and code.
As far as I can tell, your code is waiting for the full reply from the server before it actually processes the data.
|
Yeah I'd capture the full HTTP 1.1 reply then process the data.
_________________
Donate to Gentoo |
|
| Back to top |
|
|
Vieri
l33t
Joined: 18 Dec 2005
Posts: 901
|
| Posted: Tue Oct 29, 2024 6:56 pm Post subject: |
|
|
I'm unable to do what I want with LWP, so I "opened" a command to an external program such as curl and read its output.
Something like this:
| Code: | open my $cmd, '-|', "curl -q -s $url -d '$data'";
while ($chunk = <$cmd>) {
...
} |
I don't like this solution, but it's a workaround. |
|
| Back to top |
|
|
|
Portage & Programming
