Techniques are disclosed to safely and performantly store data in a distributed graph database. In various embodiments, a combination of a replication protocol for data redundancy with a chain-commit protocol is used to ensure a safe ordering of concurrent updates across servers. The resulting protocol allows a distributed graph database to simultaneously uphold ACID properties and a useful degree of scalability through concurrent processing of updates in the typical case, and serialization of updates only where data integrity would otherwise be at risk.

A method, apparatus, system, and computer program code for retrieving data records. A set of static configuration objects is provided, including: a set of resources that describe available data items, and a set of views that express a serialized transformation of resources objects into a response. In response to receiving a data request, a computer system generates a data fetch execution plan from the set of resources and the set of views. The data fetch execution plan is generated using an executor adapted to a particular data store and set of performance requirements. The computer system retrieves the data records according to the data fetch execution plan.

A big-data view integration platform generates integration guided user interfaces (GUIs). A first edge node ingests push-based and pull-based data from a plurality of platform services, which include legacy and non-legacy services having incompatible communication protocols. An event-based queue receives from the first edge node a plurality of queue events as indirect push-based data. A second set of queue events includes direct push-based data as received directly from a non-legacy platform service. A conformity component integrates the push-based data, the pull-based data, and the plurality of queue events into integration data having an enhanced integration format. A view integration component generates a plurality of data views from the integration data. A second edge node exposes the plurality of data views via an access services application programming interface (API). A new service execution component accesses the access services API to generate integration GUIs based on the data views.

Exemplary embodiments provide computer-implemented methods, mediums, and apparatuses configured to upload data stored in a data storage ecosystem to a cloud-based storage service. A database in the data storage ecosystem may store results sets from an analytical chemistry system. The results sets may be stored in a first model structure implemented by a library structure. An uploader may incorporate the library structure and may include logic to use the library structure to transform the results sets from first model structure into a second model structure suitable for use in a relational data store in the cloud-based storage system. By implementing the library structure in the uploader, the uploader can be decoupled from the data storage ecosystem. This allows the uploader to function without some of the overhead used by the data ecosystem, provide faster data uploads, and to automatically generate derived information for the results sets.

A database system supporting persistent queries, using an enhanced persistent query service and various data sources. On receiving a request to create a persistent query from a client software application, the persistent query service: creates a query virtual table; parses the persistent query; creates a plurality of intermediate virtual tables; establishes listeners for the query virtual table; creates a plurality of data source virtual tables; causes the plurality of data source virtual tables to retrieve initial data from data sources; and propagates data via intermediate virtual tables to the persistent query virtual table.

A query that is frequently processed to access an object storage is identified. Results from the query returned from the object storage is transformed into a relational database format as a materialized view. When the query is submitted a subsequent time, updated results are managed from the materialized view, other materialized views, and/or the object storage when needed.

Apparatus, methods, and computer-readable media facilitating efficiently scaling real-time indexing are disclosed herein. An example method includes generating a first plurality of source data objects based on source files having data received at the object storage system. The example method also includes generating one or more real-time manifest files based on the first plurality of source data objects. Additionally, the example method includes updating the index to include the one or more real-time manifest files. The example method also includes receiving a search query for at least one of the first plurality of source data objects and the second plurality of source data objects stored at the object storage system. Additionally, the example method includes generating a materialized view of a result set of the search query based on querying the index based on the search query, the manifest file, and the one or more real-time manifest files.

A central database system allows users to access and use data stored in a relational database. In order to ensure that the stored data is not detrimentally impacted and that the security of the stored data is maintained, the central database system generates a sanitized copy of the database. The central database system stores and accesses annotation files associated with data tables of the database and a schema identifying the structure of the database. Based on the schema, for each data table, the central database system validates the annotation file. A copy of the data table is created and sanitized corresponding to the annotation file. The sanitized copies of the data tables are used to generate a sanitized copy of the database. The sanitized copy of the database may then be accessed and used by users of the central database system without impacting the relational database.

Methods and systems are presented for collectively storing, managing, and analyzing data associated with different data sources. A data management system defines an enterprise data model schema based on different data model schemas associated with the different data sources. The data management system generates, for each data source, an enterprise data model instance based on the enterprise data model schema. Data is ingested from the different data sources, and then transformed and stored in a corresponding enterprise data model instance based on a mapping between a corresponding data model schema and the enterprise data model schema. Upon ingesting the data from the data sources, one or more consolidated data views are generated that combine at least portions of data from different enterprise data model instances. The data arranged according to the one or more consolidated data views is presented on a device and/or further analyzed to produce an analysis outcome.

Methods and systems are presented for collectively storing, managing, and analyzing data associated with different data sources. A data management system defines an enterprise data model schema based on different data model schemas associated with the different data sources. The data management system generates, for each data source, an enterprise data model instance based on the enterprise data model schema. Data is ingested from the different data sources, and then transformed and stored in a corresponding enterprise data model instance based on a mapping between a corresponding data model schema and the enterprise data model schema. Upon ingesting the data from the data sources, one or more consolidated data views are generated that combine at least portions of data from different enterprise data model instances. The data arranged according to the one or more consolidated data views is presented on a device and/or further analyzed to produce an analysis outcome.