HTTP Server for i

Secure, powerful, and complete

Overview

Getting Started

Documentation

Support

iSeries Webserver Search Engine - Set up for Crawling Sites Requiring Authentication

This page contains the documentation that is specific to authentication requirements for the web crawler. Be sure to read Getting Started Running the Web Crawler.

Some sites cannot be entered without some sort of authentication, such as a Userid and Password, or a Certificate Authentication. The web crawler has the capacity to handle either case as long as you do the required set up.

For a site requiring a Userid and Password, you must create a Validation List Object, entering the URL, Userid, and Password. Then be sure to specify the Validation List Object when you start crawling. See How to create a validation list object .

For a site that requires Client Authentication, you will need to follow the steps listed below.

• Make sure base option 34 for 5722SS1 is installed in order to use the Digital Certificate Manager.

• Next get a VeriSign. level 1 certificate using Netscape Navigator:
  1. Go to VeriSign, Inc. at http://www.verisign.com/.
  2. Under Home & Home Office, click Secure E-Mail.
  3. In the right hand column click Try a Digital ID FREE for 60 Days.
  4. Under Class 1 Digital ID, click the Enroll Now button
  5. Enter a first name such as Web and a last name such as Crawler.
  6. Enter a valid e-mail address to receive information about your certificate.
  7. Use test for the Challenge Phrase, without quotes
  8. Make sure you select I'd like to test drive a 60-day trial Digital ID for free.
  9. Skip the the Billing Information fields
  10. Click Accept on the bottom when you are done
  11. In the Generate A Private Key window up click OK
  12. Type test in the Password Entry Dialog
  13. You should get an email within a half an hour or whatever is stated in the message.
  14. When you get the e-mail, follow the instructions there.
  15. Go to the web site specified in the email, https://digitalid.verisign.com/enrollment/nspickup.htm.
  16. Paste (or enter) your Digital ID PIN in the appropriate box and select the Submit button to install
  17. Accept the ID into Netscape
  18. Then click the security button on the toolbar in netscape
  19. In the left hand column of the window that popped up, under Certificates, click Yours.
  20. The certificate you just got from VeriSign should be listed there (ie. "webcrawler's Verisign, Inc. ID ...)
  21. Select it
  22. After you select it, click the Export button
  23. Give your password test twice.
  24. Save the certificate in you PC (ie. web_cert.p12) then FTP the certificate to your iSeries system, saving the certificate in a directory. Be sure to transfer the certificate in binary mode.
• Now set up your system for authentication
  1. On your browser go to the ADMIN Server - http://server:2001/.
  2. You must have already created your Server Certificate
  3. Click the Digital Certificate Manager link
  4. Click the Select a Certificate Store button
  5. Select *SYSTEM
  6. Click Continue.
  7. Enter the password test for your certificate.
  8. Now click Fast path in the left frame
  9. Click Work with Server and Client Certificates
  10. On the Work with Server and Client Certificates form click Import on the bottom.
  11. Select the file you just FTPed as your import file
  12. Then click Continue.
  13. Scroll to the bottom and click Import.
  14. In the Import file: field, input the path and filename of your certificate, for example, /directory/web_cert.p12).
  15. Click Continue.
  16. >Enter your password test.
  17. You should receive a successful message. The certificate has been imported.
  18. Now, in the left hand column under Fast Path, click Work with server applications.
  19. Select Webserver Search Engine radio button
  20. Then click the Work with Application button on the bottom.
  21. Click the Update Certificate Assignment button.
  22. Select the certificate you just imported, then click Assign New Certificate button on the bottom
  23. You must receive a successful message The certificate was assigned to the application.)
  24. You are now ready to crawl.
  25. Be sure to enter the following when you start the crawling session or when you create an options object:
      • Proxy server for HTTPS
      • Proxy port for HTTPS
  26. Return to Getting Started - Running the web crawler.

Note: To find out more details about the crawling activity, start a trace by entering mkdir '/CRLTRC' before you start the crawl. Once crawling has started, a file will be created in this directory called TRCHTTPCRL.xxxxx and will contain multiple statements about crawling activity. A new file with a different number will be created for each new session.