1. Web &
Working of Search Engine
Presented By:
Vinay Arora
Assistant Professor
CSED, Thapar University
2. Web Content
Web Content/Resource means content accessible/present on Internet.
Invisible Web
Visible Web
Visible Web – The Publicly Index able pages that have been picked up and
Indexed by conventional search engines, mainly consist of static HTML pages.
Invisible Web/Deep Web/Hidden Web - Information that cannot be Indexed/Seen
by the Crawlers or Spiders of conventional Search Engines.
Types of Invisible Web
Truly Invisible Web
Opaque Proprietary
Private
3. TYPES of Invisible Web & Reasons of being Invisible
Truly Invisible Web is not accessible for search engines mainly because of
technical reasons Dynamically generated pages, Pages with pdf, exe, swf format.
Proprietary Web Databases which are mainly fee based and are provided by
Information Providers. These Databases provide user with search facility however,
their contents are not searchable through the search engines.
Private Web Technically Indexable , but have purposely been excluded from
search engines using Password Protected Pages, Robot.txt, NoIndex META Tag.
Opaque Web Disconnected URL.
Size Of Invisible Web is approx.500 times larger than Visible Web.
4. Crawling & Indexing
A Search Engine operates, in the
Following order:
1. Web Crawling.
2. Indexing.
3. Searching.
7. Making Invisible Web Visible
Sitemap.xml - Sitemaps are an easy way for webmasters to inform search
engines about pages on their sites that are available for crawling. In its simplest
form, a Sitemap is an XML file that lists URLs for a site along with additional
metadata about each URL.
8. Making Invisible Web Visible
Making Entries into Robot.txt file for allowing the Robots to Crawl and Changing
META Content.
9. Making Invisible Web Visible
Providing links of the desired website from another Websites so that it can be
made accessible from other/different websites. And can be Crawled.
www.orkut.com
orkut www.gmail.com
Changing the Source Code of Web Crawlers – Making the crawlers efficient and
intelligent enough so that it can accept files with extension pdf, swf etc. and
list/Index the entries properly.
The content of Proprietary Web Databases are not searchable through the
search engines. They are assembled into Web pages as responses to queries
submitted through the “Query Interface” of an underlying database. Because
current search engines cannot effectively “Crawl” databases, such data is
believed to be “Invisible,” and thus remain largely “hidden” from users
14. User Form Interaction
For Form-based Search Interfaces when user is present for Input instead of
Crawler. Result will be obtained after Query execution as soon as User press
Submit button after filling the required fields present in the Form.
We want Response Page to be
listed in Search Engine.
We have to make this Visible.
15. Crawler Form Interaction & Steps for Hidden Web Crawler
Crawler at desired URL.
Form Analysis for Internal Form Representation.
Matching with the entries present in Task Specific Database.
Automatic FORM Processing and Submission.
Response Page from the Server.
Response Analysis of that Page.
Putting the results in the Repository.
16. References
The Deep Web: Surfacing Hidden Value. http://www.completeplanet.com/Tutorials/DeepWeb/.
Paper: Crawling the Hidden web Hector Garcia CSE Department Stanford University, USA
http://www.invisible-web.net
All About Invisible Web : Natalia Arroyo, Internet Lab, CINDOC – CSIC
Accessing the Deep Web: A Survey , Bin He, Mitesh Patel, Zhen Zhang, Kevin
Chen-Chuan Chang, Computer Science Department, University of Illinois at
Urbana-Champaign.
Towards a Model of User oriented Aspects of the Invisible Web, Yazdan
Mansourian, Department of Information Studies , The University of Sheffield
