| View previous topic :: View next topic |
| Author |
Message |
Vieri
l33t
Joined: 18 Dec 2005
Posts: 898
|
 Posted: Mon Oct 28, 2024 7:50 pm Post subject: Perl: API client processing responses |
 |
|
Hi,
I need to write a Perl script that gets data from a server and processes all of it.
The following PoC code works OK when run without the sleep() call that simulates a call to a function that processes the data chunks received.
| Code: | my $app_ua_request = HTTP::Request->new('POST' => $app_url);
$app_ua_request->header(
'Content-Type' => 'application/x-www-form-urlencoded',
);
$app_ua_request->content($app_data);
my $app_ua_response = $app_ua->request($app_ua_request, sub {
my ($chunk, $app_ua_response, $protocol) = @_;
my $fromjson = from_json($chunk);
my $app_response = $fromjson->{'response'};
# process $app_response (simulate by sleeping)
# sleep(10);
});
if ($app_ua_response->is_success) {
print("replied all");
} else {
print("reply error: ".$app_ua_response->status_line);
} |
In a sample run, the "sub routine" is called exactly as many times as server replies are received.
However, if I enable the call to sleep() or the actual blocking function that processes the chunk data the "sub routine" is called just for the first 2 data chunks. The code execution then jumps to "is_success" and prints "replied all".
The Perl script is obviously receiving all of the server replies before each "sub routine" has a chance to finish.
Or maybe something's timing out in the meantime.
I'm using this to call the API service:
| Code: | my $app_ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 1 });
$app_ua->agent($UA_string);
$app_ua->env_proxy;
$app_ua->conn_cache(LWP::ConnCache->new());
$app_ua->timeout(120); |
The server takes more than 20 seconds but less than 120 to stop sending data.
What can I try?
If the code execution reaches "is_success", can I wait for all the "sub routines" to finish before exiting? How?
Regards,
Vieri |
|
| Back to top |
|
 |
turtles
Veteran
Joined: 31 Dec 2004
Posts: 1688
|
| Posted: Mon Oct 28, 2024 10:02 pm Post subject: |
|
|
Are you trying to process data as its being read/received from the server?
Seems like you need to have one function that makes the client request,
another that listens for response data from the server and stores it in a data structure.
and then a final function that does any processing to the stored data structure.
_________________
Donate to Gentoo |
|
| Back to top |
|
|
turtles
Veteran
Joined: 31 Dec 2004
Posts: 1688
|
| Posted: Mon Oct 28, 2024 10:58 pm Post subject: |
|
|
Here is a working example:
| Code: |
#!/usr/bin/env perl
use strict;
use warnings;
use diagnostics;
use LWP::UserAgent;
use HTTP::Request;
use JSON::MaybeXS qw(from_json);
use Data::Dumper;
my $app_url = "https://httpbin.org/anything/anything/";
my $app_data = "key1=value1&key2=value2";
my $app_ua_request = HTTP::Request->new( 'POST' => $app_url );
$app_ua_request->header( 'Content-Type' => 'application/x-www-form-urlencoded', );
$app_ua_request->content($app_data);
my $ua = LWP::UserAgent->new;
my $app_ua_response = $ua->request($app_ua_request);
if ( ($app_ua_response) and ( $app_ua_response->is_success ) ) {
# call a post process function
print $app_ua_response->decoded_content;
print "\n";
print "dump data structure response \n";
print Dumper $app_ua_response;
print "\n";
# post_process($app_ua_response);
}
elsif ($app_ua_response) {
print "dump data structure response no success\n";
print Dumper $app_ua_response;
#print "status_line: '$app_ua_response->status_line' \n";
}
else { die( "error" . $! ) }
sub post_process {
my $response = shift;
# do stuff ...
}
1;
|
_________________
Donate to Gentoo |
|
| Back to top |
|
|
|
Portage & Programming
