= DataSources et al =

The central parts in the architecture are currently !DataSource, !DataCrawler,
!DataAccessor and !DataObject. Together they are used to access the contents of
an information system, such as a file system or web site.

A !DataSource contains all information necessary to locate the information
items in a source. For example, a !FileSystemDataSource has a set of one or
more directories on a file system, a set of patterns that describe what files
to include or exclude, etc. For the rest it is completely passive.

A !DataCrawler is responsible for actually accessing the physical source and
reporting the individual information items as !DataObjects. Each !DataObject
contains all metadata provided by the data source, such as file names,
modification dates, etc., as well as the !InputStream that provides access to
physical resource (e.g. the file itself).

We have chosen to separate the functionalities offered by !DataSource and !DataCrawler as there
may be several alternative crawling strategies for a single !DataSource type.
Consider for example a generic !FileSystemCrawler that handles any kind of
file system accessible through java.io.File versus a !WindowsFileSystemCrawler
using OS-native functionality to get notified about file additions, deletions
and changes. Another possibility is various !DataCrawler implementations that
have different trade-offs in speed and accuracy.

Currently, A !DataSource also contains support for writing its configuration
to or initializing it from an XML file. We might consider putting this in a
separate utility class, because the best way to store such information is
often application dependent.

A !DataCrawler creates !DataObjects for the individual information items it
encounters in the physical data source. These !DataObjects are reported to
!DataCrawlerListeners registered at the !DataCrawler. An abstract base class
(!DataCrawlerBase) is provided that provides base functionality for
maintaining information about which files have been reported in the past,
allowing for incremental scanning.

In order to create a !DataObject for a single resource encountered by the
!DataCrawler, a !DataAccessor is used. This functionality is kept out of the
!DataCrawler implementations on purpose because there may be several crawlers
who can make good use of the same data accessing functionality. A good
example is the !FileSystemCrawler and !HypertextCrawler, which both make use of
the !FileDataAccessor. Although they arrive at the physical resource in
different ways (by traversing folder trees vs. following links from other
documents), they can use the same functionality to turn a java.io.File into a
!FileDataObject.

It should be clear now that a !DataCrawler is specific for the kind of
!DataSource it supports, whereas a !DataAccessor is specific for the url
scheme(s) it supports.

The !AccessData instance used in !DataCrawlerBase maintains the information
about which objects have been scanned before. This instance is passed to the
!DataAccessor as this is the best class to do this detection. For example,
this allows the !HttpDataAccessor to use HTTP-specific functionality to let
the webserver decide on whether the resource has changed since the last scan,
preventing an unchanged file from being transported to the crawling side in
the first place.

== HypertextCrawler ==

The !HypertextCrawler makes use of two external compoments: a mime type
identifier and a hypertext link extractor. The latter component is required
to know which resources are linked from a specific resource and should be
crawled next. This functionality is realized as a separate component/service
as there are many document types that support links (PDF might be a nice one
to support next). A specific link extractor is thus mimetype-specific.
However, in order to know which link extractor to use, one first needs to
know the mime type of the starting resource, which is handled by the first
component.