| This system provides the free (gratis) services, and, among other activities, explores the web links, which are used by the human beings. Thus, the robot of this system imitates the actions of the human beings, but, when it is practical, also respects the instructions within the (/robots.txt) files of the web sites. You may read about the reasons on the archive.org web site. |