The !ApertureSimpleDataCrawler is responsible for a simple access to structured data sources. Instances of this interface would be classes like !FileDataSource, !ImapDataSource, !OutlookDataSource, ... {{{ public interface SimpleDataCrawler { /** * init the datasource passing variables like name, base path, passwords, server hostname, etc */ public void init(Map parameters); /** * open the passed data object so that it can be viewed / edited by the user. for a file, * this would mean that the operating system opens the file, for an address book entry * the address book application would have to start */ public void openObject(String uri); /** * get the root uri of this datasource */ public String getRootUri(); /** * get the detailed data of one object, including plaintext and metadata * this is costly. * This may (internally) make heavy reuse of Extractors */ public Map getDataOfObject(String uri); /** * List sub-folders, Iterator contains folder uris as Strings. * this may also return the uris of objects, if the objects can contain sub-objects. * (IMAP-attachments)-but this is bad as detection of sub-objects of emails is costly. * the first call of this method would be with the getRootUri() */ public Iterator listSubFolders(String uri); /** * List objects inside the passed folder, Iterator contains folder uris of objects as Strings. * the first call of this method would be with the getRootUri() */ public Iterator listSubObjects(String uri); /** * get a map of metadata about the passed object, * enough so that changes can be detected. * if one value in this map has changed compared to the previously returned map (in the last scan) * than getDataOfObject is called to get the current data. */ public Map getChangeDataOfObject(String uri); } }}} Remarks Chris: The concept of a hierarchy, including a root, is strongly present in this interface. I don't really like this as some data sources have no intrinsic hierarchy, e.g. graph- or table-oriented data sources, so that crawling such sources becomes awkward or even impossible. For example, what kind of folders would a !WebCrawler return? How would it be able to know all folders and all items in those folders a priori? In our code we prevent this problem by keeping the traversal of the hierarchy, graph, table or whatever kind of structure the data source has, internal in the crawler implementation. Information regarding the folder/row/whatever becomes just another part of the metadata of the returned !DataObjects. Another complaint is that change ''detection'' is apparently performed outside the crawler. I assume the idea here is that there will be a single piece of code to detect changes that will work with any kind of !SimpleDataCrawler? Else there wouldn't be a reason to have this getChangeDataOfObject method. In my opinion change detection will ideally be highly crawler dependent. You might be able to generalize it this way, but at the cost of not being able to do source-specific optimizations. Consider for example HTTP-specific improvements (the if-modified-since header that lets the webserver tell you whether a source has not been changed). Maybe we can also improve IMAP in a similar way, I don't know yet. "Crawler" is probably not a good term to use in the name of this interface as there is some other code using an instance of this interface that retrieves the folder and object uris and decides to retrieve them (i.e. the actual crawling). The Javadoc comments suggest that Extractors are applied internally in the crawler implementations. I think they should be applied somewhere outside the crawlers: it is not up to the crawlers to decide how the encountered objects are processed. For example, a wget-like utility using this framework will need all metadata obtained from the data source but will have no interest in the extracted text. What I do like is that folders become more prominent. This may seem to contradict what I said before. What I mean is that it is possible to retrieve information about folders itself. Using a simple extension (e.g., a getDataOfFolder method) it becomes possible to retrieve all metadata of a folder. This is something we have not considered before in our own architecture. Here's a new idea that in my opinion merges this idea with our own architecture. Create a super interface of !DataObject (Resource? - has a strong RDF association. Entity? - has other associations here at Aduna). !DataObject then gets a sibling named Folder. Crawlers do not only produce !DataObject instances, they produce instances of its supertype. This way, crawlers that crawl data sources with an intrinsic hierarchy can return Folder instances, which contain all metadata of the Folder, similar to how !DataObjects contain metadata of that object. Similarly, we can introduce other !DataObject siblings for capturing table- or graph-related metadata that is not specific to a single !DataObject. Crawler-using applications that have no interest in this information can simply ignore these events. Also, the crawler interface itself does not need to specify folder-/graph-/table-specific information. In our use case this also facilitates metadata indexing in because currently our !MetadataFetcher (the class transforming the information inside a !DataObject to RDF statements) interprets the document URIs and "reinvents" the folder hierarchy, modeling it as Resources with a partOf relation. This would then no longer be necessary, the Folder instance would already contain all necessary information. Leo: ok, the idea of having a kind of "Sub-Class" sounds good. I would make ApertureDataObject the parent of ApertureDataFolderObject, so that ApertureDataFolderObject has all properties of DataObject and more. Chris: I don't think so, as a DataObject also has properties that a Folder does not have. For example, a Folder has no InputStream, no byte size, etc. I think DataSources and Folders are really something different. They surely do share characteristics (they both have a URI and metadata) but this should be expressed by their super type. One should not inherit from the other.