ObjTables: open-source toolkit for quality controlling, reusing, and composing data with the ease of spreadsheets, the rigor of schemas & the power of object-oriented programming.
ObjTables builds on a community initiative to make supplementary spreadsheets more reusable. The long-term goal is to create an ecosystem of reusable data for comparative and integrative research. We invite the community to share feedback or get involved.
Contents
How does ObjTables make spreadsheets reusable?
Markup syntax for spreadsheets
ObjTables provides simple markup for indicating the class and attribute encoded into each worksheet and column.
Schemas for spreadsheets
ObjTables provides a simple format for describing the types of objects encoded in a spreadsheet, their attributes, and their possible relationships.
High-level software tools
ObjTables provides software for using schemas to parse, validate, compare, compose, and analyze annotated spreadsheets.
Example: Address book of CEOs
| !!!ObjTables objTablesVersion='1.0.0' date='2020-03-14 13:19:04' |
|---|
| !!ObjTables type='Data' class='Person' tableFormat='row' | |||
|---|---|---|---|
| !Name | !Company | !Email address | !Phone number |
| Mark Zuckerberg | zuck@fb.com | 650-543-4800 | |
| Reed Hastings | Netflix | reed.hastings@netflix.com | 408-540-3700 |
| !!ObjTables type='Schema' tableFormat='row' | ||||||
|---|---|---|---|---|---|---|
| !Name | !Type | !Parent | !Format | !Verbose name | !Verbose name plural | |
| Person | Class | row | Person | People | ||
| name | Attribute | Person | String(primary=True, unique=True) | Name | ||
| company | Attribute | Person | String | Company | ||
| email_address | Attribute | Person | Email address | |||
| phone_number | Attribute | Person | String | Phone number | ||
Getting started
Authors
- Use Excel and similar programs as graphical interfaces for complex datasets that consist of multiple distinct types of objects.
- Use complex data types (e.g., reaction equations) within tables.
- Use foreign keys, column groups, and grammars to encode relationships.
- Define schemas that describe the class represented by each table.
- Use schemas and the ObjTables software to debug, revision, and migrate tables.
Reviewers and editors
- Use schemas and the ObjTables software to verify that tables are free of errors, and that they can be re-used by other investigators.
Readers
- Use schemas and the ObjTables software to compare and merge spreadsheets.
- Use schemas and the ObjTables software to parse spreadsheets into object-oriented data structures for analysis with tools such as Python.
Features
Use Excel as a GUI for viewing and editing datasets
ObjTables enables users to leverage programs such as Microsoft Excel and LibreOffice Calc as graphical interfaces for viewing and editing datasets. ObjTables uses following features:
- Table of contents: Datasets can include a worksheet that describes the data represented by each worksheet.
- Formatted table titles: Each worksheet includes a formatted title bar that describes the data captured by the worksheet.
- Formatted attribute headings: Each worksheet includes formatted headings for each column.
- Inline help for attributes: ObjTables uses comments to embed help information about each attribute into it's heading.
- Select menus for enumerations and relationships: ObjTables provides dropdown menus for attributes that represent enumerations and *-to-one relationships.
- Instant validation: ObjTables can setup basic validation for attributes. Note, the ObjTables software provides more extensive validation.
- Hidden extra rows and columns: To help users focus on their data, ObjTables hides all empty rows and columns.
- Protection from unintentional editing: To help users avoid mistakes, ObjTables protects worksheets.
Create template spreadsheets for building datasets
To make it easy to build datasets, the ObjTables software can generate template spreadsheets for schemas with a table of contents, skeletons for the tables and columns, inline help, dropdown menus, and basic validation.
Iteratively build and revision datasets
The ObjTables software leverages Git to make it easy to build datasets iteratively, revision datasets, and track their provenance, including when each revision was made, who made it, and why it was made.
Iteratively build schemas and migrate datasets
To make it easy to build schemas iteratively, the ObjTables software can revision schemas, as well as migrate datasets between different versions of schemas (e.g., adding, removing, and renaming tables and columns).
Rigorously validate and quickly debug datasets
ObjTables makes it easy to validate and debug datasets at multiple levels:
- Attribute validation: Validations of individual attributes such as validating that molecular weights are non-negative.
- Object validation: Validations of entire objects such validating that reactions are element-balanced.
- Class validation: Validations of all of the instances of a class such as validating that each reaction has a unique id.
Merge and split datasets
To help users build large datasets, the ObjTables software can merge datasets by identifying and fusing common objects. ObjTables can also decompose datasets into smaller, more manageable pieces by cutting relationships.
Compare/difference datasets
To help users compare and review changes to datasets, the ObjTables software can determine if datasets are semantically equal and identify their differences.
Query and analyze datasets
The ObjTables Python package makes it easy to find objects in datasets and use Python to conduct complex analyses of datasets such as numerical simulations.
Pretty print datasets for publication
ObjTables can pretty print datasets with tables of contents, formatted table titles and column headings, and inline help.
Visualize schemas for datasets
To help users understand schemas, ObjTables can generate UML diagrams.
Components of the ObjTables toolkit
Format for schemas for spreadsheets
ObjTables schemas capture the format of each table, including the name and data type of each column, which cells represent relationships among the entries in the tables, and constraints on the value of each cell. ObjTables supports three modes of encoding relationships into cells in tables.
- Foreign keys: Relationships from one object to other objects can be captured by columns that represents the keys of the related objects.
- Embedded tables for *-to-one relationships: ObjTables can also encode instances of related classes into groups of columns.
- Embedded grammars for *-to-many relationships: Grammars can also be used to encode instances of related classes into a single column. Grammars can be defined in EBNF format .
Numerous data types
ObjTables provides numerous data types, including for mathematics, science, chemoinformatics, and genomics.
Markup syntax for spreadsheets
ObjTables provides syntax for declaring which cells represent each table, instance, and attribute; declaring which entries represent metadata such as the date that a table was updated; and declaring which entries represent comments.
High-level software tools for working with spreadsheets
ObjTables includes a web application, a REST API, a command-line program, and a Python package for working with datasets. These tools can be used to pretty print, validate, compare, revision, and migrate datasets.
Python package for additional flexibility
For more flexibility, the Python package can be used to implement custom data types, define custom validation, query, and analyze datasets.Software tools
Use cases
ObjTables was designed to help users work with complex data with the ease of spreadsheets and the rigor of schemas. ObjTables excels at cases where datasets need to be both human and machine-readable, such as supplementary materials of journal articles. ObjTables is also well-suited to emerging fields which need to quickly build new formats for new types of data.
Publishing re-usable supplementary spreadsheets
Although supplementary spreadsheets contain valuable data, supplementary spreadsheets are hard to reuse because they often contain errors and often capture data ad hoc.
ObjTables enables authors to create high-quality datasets that are both human- and machine-readable: (a) authors can use ObjTables to debug their data, (b) authors can use ObjTables to pretty print data with tables of contents and inline help, (c) authors can publish schemas for parsing their data, and (d) readers can use these schemas to parse and analyze published data with minimal effort.
Sharing re-usable data and models
Research often involves novel datasets and models that require new formats. Unfortunately, the substantial effort needed to reuse these custom formats is a frequent barrier to collaboration.
ObjTables makes it easier to share data and models with collaborators by (a) enabling researchers to clearly describe the structure of their data or model with a schema, (b) enabling researchers to capture metadata about their data or model, (c) providing researchers software tools for validating their data, and (d) enabling collaborators to use these schemas to quickly parse data from colleagues.
Building and analyzing complex datasets and models
Many fields aim to understand how behaviors emerge from complex networks. This often requires integrating diverse data. For example, systems biology aims to understand how cellular behavior emerges from genotype, often using genomics and other data. Spreadsheets are a popular tool for merging data because they are easy to use. However, spreadsheets only support a few data types, and spreadsheets have limited support for multi-dimensional data. In addition, it is difficult to debug spreadsheets.
ObjTables makes it easy to build, validate, and analyze complex datasets: (a) users can use spreadsheets to assemble diverse data, (b) users can quickly define schemas for their data, and (c) users can use these schemas to validate their data and parse it into object-oriented data structures for further analysis in languages such as Python. For example, we have used ObjTables to integrate data about the biochemistry of H1 human embryonic stem cells.
ObjTables also makes it easy to build datasets iteratively over time by helping users revision data with Git and migrate their data as they revise their schemas.
Defining formats for new types of data and models
New areas of science often require new types of data and new kinds of models. In turn, this often requires new formats to capture these data and models and new software for working with these formats. Creating these formats is often an obstacle for new domains that have limited resources. Furthermore, evolving these formats as new approaches emerge is challenging because this often requires updating the software tools and converting old files to the new format.
ObjTables addresses this issue by making it easy to define schemas for domain-specific data and providing software tools for parsing, manipulating, and validating data encoded in these schemas. For example, we have used ObjTables to create, WC-KB , a format for the experimental omics, biochemical, and physiological data needed to model cellular biochemistry. We have also used ObjTables to create, WC-Lang , a format for whole-cell models of all of the biochemical activity in a cell. Creating these formats required minimal code.
Examples, tutorials, documentation, and help
Extensive examples, interactive tutorials, and documentation for the ObjTables formats and software tools are available through the links below.
Getting started
The getting started page contains quick guides for (a) authors for creating reusable spreadsheets and (b) readers for reusing spreadsheets from other investigators.
Command-line program installation
Installation instructions for the command-line program are available at docs.karrlab.org. A Dockerfile for building an Ubuntu Linux image with ObjTables is available here .
Python package installation
Installation instructions for the Python package are available at docs.karrlab.org. A Dockerfile for building an Ubuntu Linux image with ObjTables is available here .
Tutorials for the Python package
A Jupyter notebook with interactive tutorials is available at sandbox.karrlab.org.
Docs for the schema & dataset formats
Documentation for the formats for schemas and the formats for datasets is available at objtables.org/docs.
Docs for the command-line program
Documentation for the command-line program is available inline by running obj-tables --help.
Docs for the Python package
An introduction to the Python package is available at objtables.org/docs. Detailed documentation is available at docs.karrlab.org.
