11.6 How to manipulate URLs on-the-fly
One of the most powerful (and complex) customization features of LinkScan concerns the real-time manipulation of links during the course of the scan. This is typically used to control the testing of sites with complex dynamic content. The basic commands available are:
Sessionmatch expression
Substitute relative-path-expression expression
Substituteraw relative-path-expression expression
Substitutescript relative-path-expression expression
The Sessionmatch command is used to manipulate Session numbers. The Substitute command is used to perform transformations on resolved links. The Substituteraw is used to perform transformations on unresolved links (i.e. the raw contents of a tag or tag attribute). The Substitutescript is used to perform transformations of blocks of JavaScript code.
We shall consider a number of examples which may be adapted according to your specific needs.
Example 1
Consider a site that produces links such as:
http://www.example.com/page1.asp
http://www.example.com/page1.asp?Print
It is entirely possible that page1.asp has been designed in such a manner that it delivers the same basic content with minor variations in formatting depending upon the presence or absence of the Print query string. One might configure LinkScan with:
Substitute (.*\.asp)\?Print $1
Whenever LinkScan encounters a link matching the specified pattern it will make the substitution indicated before it tries to validate or follow that link. In this example, a link to:
http://www.example.com/page1.asp?Print
will immediately be transformed to:
http://www.example.com/page1.asp
Note, however, this is not the same as Excluding links which contain the Print query string; that would cause LinkScan to simply ignore the link. In this case, LinkScan will process the link but transform it on-the-fly during the scan.
Example 2
Next we will consider a significantly more complex scenario.
Sessionmatch .*&token=([^&]+)
Substitute (.*&token=)[^&]*(.*)$ $1!S$2
In this case, we use the special Sessionmatch command to capture and save the first value of the query parameter token that LinkScan sees. This is most likely some kind of session number assigned by the target server immediately following the submission of a login form. The Substitute command then instructs LinkScan to replace all subsequent values of token with the saved value (represented by the special parameter !S).
In this scenario, LinkScan ensures that the value of token can never change during the course of the scan from the originally assigned value.
Example 3
Next we'll consider a JSP site that produces URL's with the following structure:
http://www.example.com/content?A=123&B=456&C=789&D=XYZ
It may not be productive or efficient for LinkScan to scan all of the pages using every combination and permutation of values for the parameters A, B, C, D... etc.. We can control that by manipulating the individual name-value pairs during the scan. For example:
Substitute (content\.jsp\?.*)&B=[^&](.*) $1&B=456$2
Substitute (content\.jsp\?.*)&C=[^&](.*) $1$2
Taglimit content\.jsp\?.*&D= 20
The first command fixes the value of B=456. Whatever value the parameter B takes on during the scan, LinkScan will force the value back to 456. The second command deletes any references to the C parameter from every link that it finds. We have also included the third Taglimit command; this will cause LinkScan to completely ignore the twenty-first and subsequent links that include a D parameter. In other words, in this case, we only want to test a representative sample (20) of links that include a D parameter.
Example 4
For our next example, we shall consider a site that generates pages containing some links with the following structure:
http://www.example.com/cgi-bin/GenerateFrame?Referer=abc&Link=http://www.yahoo.com/
Rather than linking directly to Yahoo!, this page links to a script that generates a frameset that includes the referenced page. In a default configuration, LinkScan will happily follow the link, validating the frameset and the ultimate link to Yahoo!. However, it may not be productive to do that for potentially thousands of links. Furthermore, in the (extremely unlikely) event that the link to
http://www.yahoo.com/ was broken, the error would appear in one of the GenerateFrame documents and not the original referring document. In order to repair that link, one would have to backtrack through the frameset to locate the original source of the trouble.
Substitute cgi-bin/GenerateFrame.*&Link=([^&]+).* !U$1
This command will extract the value of the Link= parameter, and the special !U token instructs LinkScan that the string needs to be un-encoded. So the original link:
http://www.example.com/cgi-bin/GenerateFrame?Referer=abc&Link=http://www.yahoo.com/
is transformed on-the-fly to:
http%3A%2F%2Fwww.yahoo.com%2F
and then decoded to:
http://www.yahoo.com/
And this means LinkScan can validate the link to Yahoo! directly without checking the GenerateFrame script many, many times. Furthermore, any errors will be flagged against the original document (and not one or more steps removed).
Example 5
For our final example, we include for illustration the complete configuration for a real-world large and very complex dynamic site:
# Set the CGI limit to be very large
# Include all file types on the Map
Maxcgi = 10000
Mapinclude .*
# Force &A=B and insert it immediately after the '?'
Substitute (cgi-bin.*[&\?])A=[^&=]*&*(.*) $1$2
Substitute (cgi-bin.*\?)(.*) $1A=B&$2
# Discard null and undefined values
Substitute (cgi-bin.*)&B=(null|undefined)(.*) $1$3
Substitute (cgi-bin.*)&C=(null|undefined)(.*) $1$3
Substitute (cgi-bin.*)&D=(null|undefined)(.*) $1$3
Substitute (cgi-bin.*)&R=(null|undefined)(.*) $1$3
# For 'category', take the &C= if present, otherwise the &B=
Substitute (cgi-bin/bv/scripts/category.*\?A=B).*?(&C=[^&=]*).* $1$2
Substitute (cgi-bin/bv/scripts/category.*\?A=B).*?(&B=[^&=]*).* $1$2
# For 'content', take the &D= or &R= if present (call it &D=). Otherwise take the &B=
Substitute (cgi-bin/bv/scripts/content.*\?A=B).*?&[DR]=([^&=]*).* $1&D=$2
Substitute (cgi-bin/bv/scripts/content.*\?A=B).*?(&B=[^&=]*).* $1$2
# For 'frame', take the &D= or &R= if present (call it &D=). Otherwise take the &B=
Substitute (cgi-bin/bv/scripts/frame.*\?A=B).*?&[DR]=([^&=]*).* $1&D=$2
Substitute (cgi-bin/bv/scripts/frame.*\?A=B).*?(&B=[^&=]*).* $1$2
# For 'mailing...', take the &R=
Substitute (cgi-bin/bv/scripts/mailing.*\?A=B).*?(&R=[^&=]*).* $1$2
# For 'contact', take the &B=, &C= and &Comments
Substitute (cgi-bin/bv/scripts/contact.*\?A=B).*?(&B=[^&=]*).*?(&C=[^&=]*).*?(&Comments=[^&=]*).* $1$2$3$4
# Mark redirects to Error page as 404
# Mark documents containing 'Error Code:' as 404
Errordoc cgi-bin/bv/scripts/error.jsp
Errorbody Error\s+Code:[^\n<]*
# Hide some frequent arising errors
Noforms = 1
Exclude images/arrow.gif
Example 6
Next we will consider a reference to a JavaScript function:
<a href="javascript:MyFunction(4,5,6);">
The following Substitutescript command:
Substitutescript .*:MyFunction\((\d+),(\d+),(\d+)\) '/somepage.jsp?Par1=$1&Par2=$2&Par3=$3'
will transform the function call into the following link which will then be validated/processed by LinkScan.
/somepage.jsp?Par1=4&Par2=5&Par3=6
That wouldnt be worth the hassle.
