Web archiving

This article includes a list of references, related reading or external links, but its sources remain unclear because it lacks inline citations. Please improve this article by introducing more precise citations where appropriate. (January 2010)

Web archiving is the process of collecting portions of the World Wide Web and ensuring the collection is preserved in an archive, such as an archive site, for future researchers, historians, and the public. Due to the massive size of the Web, web archivists typically employ web crawlers for automated collection. The largest web archiving organization based on a crawling approach is the Internet Archive which strives to maintain an archive of the entire Web. National libraries, national archives and various consortia of organizations are also involved in archiving culturally important Web content. Commercial web archiving software and services are also available to organizations who need to archive their own web content for corporate heritage, regulatory, or legal purposes.

Contents 1 Collecting the Web 2 Methods of collection 2.1 Remote harvesting 2.1.1 On-demand 2.2 Database archiving 2.3 Transactional archiving 3 Difficulties and limitations 3.1 Crawlers 3.2 General limitations 4 Aspects of Web curation 5 See also 6 References 7 External links

[edit] Collecting the Web

Web archivists generally archive all types of web content including HTML web pages, style sheets, JavaScript, images, and video. They also archive metadata about the collected resources such as access time, MIME type, and content length. This metadata is useful in establishing authenticity and provenance of the archived collection.

[edit] Methods of collection

[edit] Remote harvesting

The most common web archiving technique uses web crawlers to automate the process of collecting web pages. Web crawlers typically view web pages in the same manner that users with a browser see the Web, and therefore provide a comparatively simple method of remotely harvesting web content. Examples of web crawlers frequently used for web archiving include:

• Automated Internet Sessions in biterScripting
• Heritrix
• HTTrack
• Wget

[edit] On-demand

There are numerous services that may be used to archive web resources "on-demand", using web crawling techniques. Those offering archives for use as legal evidence comply with Federal laws and standards.

• WebCite, a free service specifically for scholarly authors, journal editors and publishers to permanently archive and retrieve cited Internet references ^[1]
• Archive-It, a subscription service which allows institutions to build, manage and search their own web archive.
• Hanzo Archives, offers commercial web archiving tools and services for corporate heritage, regulatory compliance, litigation-support, and electronic discovery.
• BackupURL.com, allows creation of "a copy of any website that you can share and view any time knowing it will last forever." ^[2]
• freezePAGE snapshots, a free/subscription service. To preserve snapshots, requires login every 30 days for unregistered users, 60 days for registered users. ^[3]
• Website-Archive.com, a subscription service. Captures screen-shots of pages, transactions and user journeys using "actual browsers". Screen-shots can be viewed online or downloaded in a monthly archive. Uses Cloud Testing technology.
• Iterasi, a subscription service to create web archives for the corporate, legal and government industries.
• PageFreezer, a subscription service to archive, browse and search websites in compliance with records management laws. Archives can be used as legal evidence.
• Compliance WatchDog, a subscription service that archives websites and allows users to browse the site as it appeared in the past. It also monitors sites for changes and alerts compliance personnel if a change is detected.
• Aleph Archives, offers web archiving services for regulatory compliance and eDiscovery aimed to corporate (Global 500 market), legal and government industries.

[edit] Database archiving

Database archiving refers to methods for archiving the underlying content of database-driven websites. It typically requires the extraction of the database content into a standard schema, often using XML. Once stored in that standard format, the archived content of multiple databases can then be made available using a single access system. This approach is exemplified by the DeepArc and Xinq tools developed by the Bibliothèque nationale de France and the National Library of Australia respectively. DeepArc enables the structure of a relational database to be mapped to an XML schema, and the content exported into an XML document. Xinq then allows that content to be delivered online. Although the original layout and behavior of the website cannot be preserved exactly, Xinq does allow the basic querying and retrieval functionality to be replicated.

[edit] Transactional archiving

Transactional archiving is an event-driven approach, which collects the actual transactions which take place between a web server and a web browser. It is primarily used as a means of preserving evidence of the content which was actually viewed on a particular website, on a given date. This may be particularly important for organizations which need to comply with legal or regulatory requirements for disclosing and retaining information.

A transactional archiving system typically operates by intercepting every HTTP request to, and response from, the web server, filtering each response to eliminate duplicate content, and permanently storing the responses as bitstreams. A transactional archiving system requires the installation of software on the web server, and cannot therefore be used to collect content from a remote website.

Examples of commercial transactional archiving software include:

• PageVault
• Vignette WebCapture

[edit] Difficulties and limitations

[edit] Crawlers

Web archives which rely on web crawling as their primary means of collecting the Web are influenced by the difficulties of web crawling:

• The robots exclusion protocol may request crawlers not access portions of a website. Some web archivists may ignore the request and crawl those portions anyway.
• Large portions of a web site may be hidden in the deep Web. For example, the results page behind a web form lies in the deep Web because a crawler cannot follow a link to the results page.
• Crawler traps (e.g., calendars) may cause a crawler to download an infinite number of pages, so crawlers are usually configured to limit the number of dynamic pages they crawl.

The Web is so large that crawling a significant portion of it takes a large amount of technical resources. The Web is changing so fast that portions of a website may change before a crawler has even finished crawling it.

[edit] General limitations

• Some web servers are configured to return different pages to web archiver requests than they would in response to regular browser requests. This is typically done to fool search engines into directing more user traffic to a website, and is also often done to to avoid accountability, or to provide enhanced content only to those browsers that can display it.

Not only must web archivists deal with the technical challenges of web archiving, they must also contend with intellectual property laws. Peter Lyman^[4] states that "although the Web is popularly regarded as a public domain resource, it is copyrighted; thus, archivists have no legal right to copy the Web". However national libraries in many countries do have a legal right to copy portions of the web under an extension of a legal deposit.

Some private non-profit web archives that are made publicly accessible like WebCite or the Internet Archive allow content owners to hide or remove archived content that they do not want the public to have access to. Other web archives are only accessible from certain locations or have regulated usage. WebCite cites a recent lawsuit against Google's caching, which Google won.^[5]

[edit] Aspects of Web curation

Web curation, like any digital curation, entails:

• Collecting verifiable Web assets
• Providing Web asset search and retrieval
• Certification of the trustworthiness and integrity of the collection content
• Semantic and ontological continuity and comparability of the collection content

Thus, besides the discussion on methods of collecting the Web, those of providing access, certification, and organizing must be included. There are a set of popular tools that addresses these curation steps:

A suite of tools for Web Curation by International Internet Preservation Consortium:

• Heritrix - official website - collecting Web asset
• NutchWAX - search Web archive collections
• Wayback (Open source Wayback Machine) - search and navigate Web archive collections using NutchWax
• Web Curator Tool - Selection and Management of Web Collection

Other open source tools for manipulating web archives:

• WARC Tools - for creating, reading, parsing and manipulating, web archives programmatically
• Search Tools - for indexing and searching full-text and metadata within web archives

[edit] See also

• Archive
• Archive site
• Digital preservation
• Heritrix
• Internet Archive
• Library of Congress Digital Library project
• National Digital Information Infrastructure and Preservation Program
• Pandora Archive
• Portuguese Web Archive
• Project MINERVA
• UK Web Archiving Consortium
• Web crawling
• WebCite
• Virtual artifact

[edit] References

[edit] External links

• International Internet Preservation Consortium (IIPC) - International consortium whose mission is to acquire, preserve, and make accessible knowledge and information from the Internet for future generations
• International Web Archiving Workshop (IWAW) - Annual workshop that focuses on web archiving
• The Library of Congress, Digital Collections and Programs
• National Library of Australia, Preserving Access to Digital Information (PADI)
• Library of Congress - Web Archiving
• Web archiving bibliography - Lengthy list of web-archiving resources
• Web archiving discussion list - Used for discussing the technical, legal, and organizational aspects of web archiving
• WebArchivist - Researchers that work with scholars, librarians, and archivists interested in preserving and analyzing Web resources
• Julien Masanès, Bibliothèque Nationale de France - Towards continuous web archiving
• Comparison of web archiving services
• SWAT - Snappy Web Archiving Tool. A proof-of-concept software that archives web pages by harvesting all files and taking screenshots of each page. All META data is saved in XML (METS, PREMIS, MODS and ADDML).
• The UK Government Web Archive at The National Archives - Archive of UK central government websites
• The UK Web Archive provided by The British Library - Archive of selected websites of UK cultural, social and historical significance - archived with permission from content owners