CGI and Perl

Adapting the Code for General Purpose Use

The UserAgent module can prove useful in other examples as well. The code in Listing 9.2 that retrieved the stock quote can be turned into a general purpose URL retriever. This next example does this, adding to it the ability to send the request through a firewall. The code in Listing 9.4 should look quite familiar.
Output from the program
Figure 9.2. Output from the program.

Listing 9.4. General purpose URL retriever going through a firewall.

 require LWP::UserAgent;
 require HTTP::Request;
 $ua = new LWP::UserAgent;
 foreach $url (@ARGV) {
    $request = new HTTP::Request `GET', $url;
    $response = $ua->request($request);
    if ($response->is_success) {
    } else {

Listing 9.4 simply replaces the forever loop with a foreach loop where the iterator is a list of URLs to retrieve. You also may have noticed the line


This is how you can send a request through a firewall or proxy server. The mechanism used here is to define an environment variable called HTTP_PROXY. However, you could use a different approach, such as a hard-coded constant value, or the proxy server could be passed into the script as an argument. The functions handleResponse() and handleError() are left unimplemented. These are the functions that would turn this general-purpose URL retriever into something more useful such as our stock quote retriever or a Web spider, as you'll see next. That function can be specific to whatever might suit your requirements.

You'll see how this general URL retriever can be applied to useful functionality in the following examples. We will also explore some of the other powerful features that the LWP::UserAgent module provides.