User:Vieglais/SOC CoM

From Phyloinformatics
< User:Vieglais
Revision as of 15:20, 22 March 2009 by Vieglais (talk) (New page: === Catalog of Metadata === ;Rationale: There exists many different standards for the structure and content of metadata documents, and this already large number is growing. Furthermore, ...)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Catalog of Metadata

There exists many different standards for the structure and content of metadata documents, and this already large number is growing. Furthermore, since many of these metadata documents are not exactly interesting reading, there is a strong tendency for researchers and implementors to go out on their own and create a new standard rather than spend the time and effort necessary to discover, let alone digest these typically cumbersome, dense documents. This of course simply exacerbates the situation, as these fringe projects become successful and their corresponding metadata standard, adding to the already large pile of documents. By providing a simple catalog of existing metadata standards and a mechanism for searching them for keywords and/or phrases, it is hoped that the proliferation of metadata standards (at least for our target community) will be reduced because a) a list of standards will be readily available in the catalog, and b) simple search, browse and keywording/tagging mechanisms will assist interested parties in discovering standards of direct or related interest.
Several components need to be adapted or implemented for a successful outcome to be reached, though perhaps the most important is the identification of crucial terms to use when indexing the metadata documents (i.e. metadata about the metadata). Something as simple as Dublin Core () is expected to provide considerable assistance for discovering relevant documents within the system. The major components to be configured / adapted / implemented are:
  1. An information retrieval indexing service such as the Apache Lucene based SOLR system providing fuzzy search capabilities
  2. A set of tools that can parse various formats of metadata documents and generate index entries for the indexing service
  3. A web interface for searching and browsing the catalog, and a management interface for editing references to metadata documents.

Apart from the usual issues encountered when combining several libraries and applications developed for different purposes, it is expected that some interesting intellectual challenges will arise as well. One immediate issue is that there is no guarantee a consistent set of terms can be identified for the meta-metadata index, especially when considering the complexity of some scientific metadata standards such as EML (”) and FGDC ( Building an efficient content extraction framework that operates over several different types of content (e.g. HTML, plain text, MS Word Documents) with minimal reconfiguration requires both skill with low level document parsing techniques and higher level knowledge extraction tools, the later of which are likely to require some reconfiguration to operate effectively with scientific rather than general literature.

Involved Toolkits or Projects
Dave Vieglais (vieglais at ku edu)