The process of surfing the web, storing URLs and indexing keywords, links and text
A spider is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index The major search engines on the Web all have such a program, which is also known as a "crawler" or a "bot " Spiders are typically programmed to visit sites that have been submitted by their owners as new or updated