Crawler

Scope

Downloads the file corresponding to a given URL.

Interface

• It sends the following message to obtain an address :

  GET
  

  and in response it receives message :

  URL _urlid_ _urlstring_
  

  from the URL Server and crawls the URL.
• Sends to the store server messages of the form :
  

  STR _size_info_byte_ _info_
  |___________________||_____|
  readUTF()              readByte()
  

  where _info_ is a compressed packet containing the _urlid_, MIME type of the crawled file and the file.
• Reads a file describing the address of the servers it has to contact and maybe other parameters (delays for timeouts, etc.).