Habitat

Project details

Released 4/29/2016

Habitat is a simple and yet powerful self-contained object storage management system. Based on Amazon Web Services, it is capable of virtually unlimited storage. Instead of a large centralized management system, Habitat can be used as a local repository for a single application or it can be shared and used with many clients.

Habitat is best used for situations where the client producers and consumers of the files do not require a file system protocol interface and can use http(s) to access the store.

GitHub repo

Key features

  • Upload/download via http(s) from any client that can issue HTTP POST requests
  • Upload/download via any tools that support writing/reading from S3. For example:
  • Future: upload via file interface where a file is stored by dropping in a temp directory
  • Future: checkout a collection of files into a local file system (setting user/group permissions as desired)
  • Immutable reference key for each stored file
  • Stores metadata about the object from a variety of sources:
    • Parsed from the file name (customized parsing via a regular expression)
    • Extracted from the file contents (via a custom plugin)
    • Exracted from a companion metadata file (uploaded either before or after the data file)
    • Extracted from the write action event itself
    • Extracted from the S3 metadata attribute on the object
  • Life cycle management to reduce to lower cost storage or to delete after defined time periods
  • Get file or file list based on metadata search
  • Search index using a discrete or shared Elastic Search instance
  • Future: Supports object versioning
  • Easily customized to tailor to unique requirements (config file driven, with custom plugins)
  • Supports saving a shadow copy of metadata into the S3 object metadata in addition to the Elastic Search index

Interested in NIBR Engineering?

At NIBR, you'll be at the forefront of technology — helping to shape it, develop it, and make it impactful. Partnering with scientists, our engineers create cutting-edge, state-of-the-art solutions that accelerate drug discovery and ultimately improve patients’ lives.

Learn more