Gaiandb
A light-weight data-federation technology built on Apache Derby. Gaian accesses and transforms data securely "at the edge" and can discover other instances of itself to form a scalable data network.
Install / Use
/learn @gaiandb/GaiandbREADME
GaianDB
A light-weight data-federation technology built on Apache Derby. Gaian accesses and transforms data securely "at the edge" and can discover other instances of itself to form a scalable data network.
<HTML> <BODY> <h1 class="title">Readme.html taken from build/GAIANDB_V2.1.8_20160523.zip</h1> <br/> <hr><h2 class="nocount">Contents</h2> <ol class="nest"> <li class="nest"><a href="#contents8">What is the Gaian Database ?</a></li><br/> <li class="nest"><a href="#contents10">Key benefits - what does Gaian give you ?</a></li><br/> <li class="nest"><a href="#contents12">What's new in version 2</a></li><br/> <li class="nest"><a href="#contents160">Installation and First Steps</a> <ol class="nest"> <li class="nest"><a href="#contents162">Upgrading Gaian</a></li> <li class="nest"><a href="#contents165">Installation Prerequisites</a></li> <li class="nest"><a href="#contents337">Supported Operating Systems</a></li> <li class="nest"><a href="#contents168">First Steps</a></li> <li class="nest"><a href="#contents169">Verify the installation</a></li> <li class="nest"><a href="#contents52">Quick table federation examples (matching a Logical Table to a Physical One)</a> <ol class="nest"> <li class="nest"><a href="#contents53">Federate an RDBMS table</a></li> <li class="nest"><a href="#contents60">Federate an existing CSV File</a></li> <li class="nest"><a href="#contents201">Federate an existing Excel spreadsheet</a></li> </ol> </li> <li class="nest"><a href="WorkedExamples.html">Further worked examples</a></li> </ol> </li><br/> <li class="nest"><a href="#contents110">GaianDB Usage</a> <ol class="nest"> <li class="nest"><a href="#contents115">GaianDB node usage</a></li> <li class="nest"><a href="#contents120">Launching multiple GaianDB nodes</a></li> <li class="nest"><a href="#contents127">Launching a GaianDB node from a separate directory</a></li> <li class="nest"><a href="#contents130">How to specify a location for configuration and log files and the physical database folder</a></li> <li class="nest"><a href="#contents137">How to manage multiple GaianDB node executions with multiple working directories</a></li> <li class="nest"><a href="#contents138">How to initialise a GaianDB node with User defined Derby objects like Tables, views, procedures,functions</a></li> <li class="nest"><a href="#contents139">How to integrate GaianDB into a wrapping class or framework: startup, start detection, kill</a></li> </ol> </li><br/> <li class="nest"><a href="#contents339">Client Usage</a> <ol class="nest"> <li class="nest"><a href="#contents140">Command line processor options with 'queryDerby'</a></li> <li class="nest"><a href="#contents145">Issuing Distributed SQL Queries</a></li> <li class="nest"><a href="#contents341">Distributed Sub-queries or Pushing query processing to each node</a></li> <li class="nest"><a href="#contents217">Listing server warnings</a></li> <li class="nest"><a href="#contents344">Dashboard Graphical User Interface</a></li> </ol> </li><br/> <li class="nest"><a href="#contents175">Configuration</a> <ol class="nest"> <li class="nest"><a href="#contents343">Using the Configuration File</a> <ol class="nest"> <li class="nest"><a href="#contents180">Configuration file: Properties for Logical Tables, their Data Sources and associated Connections</a></li> <li class="nest"><a href="#contents209">Configuration file: Global System properties and identifiers</a></li> </ol> </li> <li class="nest"><a href="#contents177">Using the management API to apply configuration updates</a></li> <li class="nest"><a href="#contents67">Pluralized data sources</a></li> <li class="nest"><a href="#contents66">In-Memory tables and indexes on their columns</a></li> <li class="nest"><a href="#contents92">Constant Column definitions</a></li> <li class="nest"><a href="#contents64">Using the LISTQUERIES and CANCELQUERY procedures as well as the GDB_TIMEOUT and GDB_WID features</a></li> <li class="nest"><a href="#contents65">Connection maintenance configuration in intermittent/high latency/low bandwidth networks</a></li> <li class="nest"><a href="#contents215">Hard-wiring connections between GaianDB nodes (incl setting up gateways)</a></li> <li class="nest"><a href="#contents99">Text file federation configuration options (e.g: how to specify different record and field separators)</a></li> </ol> </li><br/> <li class="nest"><a href="#contents306">Security</a> <ol class="nest"> <li class="nest"><a href="#contents307">Communication encryption using SSL (Secure Sockets Layer)</a> <ol class="nest"> <li class="nest"><a href="#contents3071">Server Configuration for SSL</a></li> <li class="nest"><a href="#contents3072">Client Setup for SSL</a></li> </ol> </li> <li class="nest"><a href="#contents308">User authentication</a></li> <li class="nest"><a href="#contents309">User access restriction to chosen databases</a></li> <li class="nest"><a href="#contents310">Password scrambling</a></li> </ol> </li><br/> <li class="nest"><a href="#contents315">Advanced Features</a> <ol class="nest"> <li class="nest"><a href="#contents321">Special SQL Query Options (with_provenance, explain, maxDepth, ...)</a></li> <li class="nest"><a href="#contents234">Provenance Columns: Special columns identifying origin of data rows</a></li> <li class="nest"><a href="#contents270">Explain Queries: Getting and showing the route of a query</a></li> <li class="nest"><a href="#contents273">Specifying advanced table and column mappings</a></li> <li class="nest"><a href="#contents274">SQL Query as a Logical Table Data Source</a></li> <li class="nest"><a href="#contents275">Message Storer: Message Broker Integration (e.g. use with sensors)</a></li> <li class="nest"><a href="#contents401">Custom VTIs</a> <ol class="nest"> <li class="nest"><a href="#contents402">IBM Content Analytics Restful Interface (ICAREST)</a></li> <li class="nest"><a href="#contents403">Spatial Query</a></li> </ol> </li> </ol> </li><br/> <li class="nest"><a href="#contents325">Supported RDBMS Data sources</a></li><br/> <li class="nest"><a href="#contents330">Supported SQL Types for Logical Tables Definitions</a></li><br/> <li class="nest"><a href="#contents335">Supported Number of Logical Tables</a></li><br/> <li class="nest"><a href="#contents346">Error Messages</a></li><br/> <li class="nest"><a href="#contents359">FAQ & Troubleshooting</a></li><br/> <li class="nest"><a href="#contents121">Features</a></li><br/> <li class="nest"><a href="#contents400">Authors/Contacts</a></li> </ol> <hr><h2><a name="contents8">What is the Gaian Database ?</a></h2> <p>The Gaian Database - or GaianDB for short - is a lightweight (<4MB) dynamically distributed federated database (DDFD) engine based on Apache Derby 10.x. It provides a single centralized view over multiple, heterogeneous back-end data sources (e.g. RDBMS dbs, files, text indexes, spreadsheets, etc.) using an extensible logical table abstraction layer. However, its principal feature is its ability to automatically discover and federate other GaianDB instances (federating nodes) such that a whole network of these can be automatically formed.</p> <br/> <center> <TABLE class='image'> <TR class='image'> <TD class='image'><a Href="https://github.com/gaiandb/gaiandb/blob/master/doc/UserDocs/images/gaiandb3.gif"> <img src="https://github.com/gaiandb/gaiandb/blob/master/doc/UserDocs/images/gaiandb3.gif" width=500> </a></TD> <TD class='image'><a Href="https://github.com/gaiandb/gaiandb/blob/master/doc/UserDocs/images/gaiandb1.gif"> <img src="https://github.com/gaiandb/gaiandb/blob/master/doc/UserDocs/images/gaiandb1.gif"></a></TD> </TR> <TR class='image'> <TD class='image' align='center'>Fig. 1 - GaianDB high level architecture</TD> <TD class='image' align='center'>Fig. 2 - GaianDB connectivity model</TD> </TR> </TABLE> </center> <p>GaianDB advocates a flexible 'store locally, query anywhere' (SLQA) paradigm, which means the data remains where it is, but you have a centralized view over which you can query. Each data source should have its data managed autonomously and applications can choose which sections of it to make available to the GaianDB network. For this reason distributed insert/update/delete operations are not currently supported.</p> <p>Queries can be injected at any node and are routed effectively around the network using a 'biologically inspired' connectivity model that strives to minimise query time and maximise efficiency (by minimising network diameter and maximising connections to the fittest nodes).</p> <p>Architecturally, GaianDB sits <b>underneath</b> Derby, and is invoked by Derby to process any GaianDB-specific SQL. To query GaianDB you typically identify the 'GaianTable' class (which references a logical table, similar to a view definition), or the 'GaianQuery' class (which references an embedded subquery), either of which will retrieve federated data from every GaianDB node in a connected network. Other GaianDB SQL usage may be invocations of one of the stored procedures or functions that make up GaianDB's system management API. These can be used to define Logical Tables and their federated Data Sources, relational database connections, user-defined Gaian Connections (for nodes that couldn't be discovered), or any of GaianDB's general system properties individually.</p> <br /> <p>The distributed, federated and store-local nature of GaianDB makes it extremely scalable and suitable for a wide variety of deployments and scenarios. Also, GaianDB is 100% Java and so it can run in a multitude of places; anywhere from small devices, like mobile phones, to large enterprise systems; easily integrating between all of these.</p> <p>GaianDB has already been used in the context of complex text analytics applications, performing distributed semantic join queries, and drawn the attention of significant customers in t