A method for providing generating data extensions for a first data profile includes receiving a request for historical data, the data being associated with a first data profile, the first data profile being generated from data supplied by a data source system, creating a data model for obtaining the historical data based on data from at least one of the first data profile, a second data profile, or the data source system, executing the data model to obtain the historical data, and storing the historical data as a data extension of the first data profile. The first data profile and the data extension of the first data profile may be generated and stored separately, and the data extension of the first data profile is generated on the fly at runtime.

Systems and methods described herein involve a meta-graph management configured to link external data source to another external data mart through a data management platform which can involve managing characteristics of one or more tables of the data source and the data mart and a temporary table based on columns, managing characteristics of one or more Input data and Output data of data processing from the data source to the data mart based on columns; managing relationships of characteristics between data and data processing for the data source and the data mart based on the columns; managing one or more data flows between the data source and the data mart that include data, data processing, and relationships; and providing data, data processing, and relationships between the data source and the data mart for each data flow.

A method is provided for output validation of data processing systems, performed by one or more processors. The method comprises performing a data comparison between a first data table and a second data table to determine a data differentiating table, wherein the first data table is based on an output of a first data pipeline, and wherein the second data table is based on an output of a second data pipeline; performing a schema comparison between the first data table and the second data table to determine a schema differentiating table; generating a first output validation score based on the data differentiating table; generating a second output validation score based on the schema differentiating table; and generating a summary comprising both the first and second output validation scores.

A schema for a hierarchical data structure may include application specific extensions to the schema applied to a hierarchical data structure. Class may be added to the schema by individual applications granted access to a hierarchical data structure. When an access request for an object of the hierarchical data structure is received, the class may be identified in the schema and applied to process the access request to the object. Different classes may be added by different applications without disrupting the utilization of the schema for accessing the hierarchical data structure of other applications.

A system includes first and second subsystems and a third processor. The first subsystem includes a first memory and a first processor. The first memory stores data, which includes metadata associated with transmitted metadata fields. The first processor transmits the data to the second subsystem. The second subsystem includes a second memory and a second processor. The second memory stores expected metadata fields. The second processor receives the data. The third processor determines that the first subsystem transmitted the data to the second subsystem and that a mismatch exists between the transmitted and expected metadata fields. In response, the third processor prevents the second subsystem from executing an application configured to process the data using the expected metadata fields. The third processor resolves the mismatch by modifying the expected metadata fields such that they correspond to the transmitted metadata fields and allows the second subsystem to execute the application.

Techniques are disclosed that pertain to deploying immutable instances of a system. A computer system may maintain an active generation value that indicates an immutable instance of a database system that is permitted to write data to a database. The computer system may deploy a first immutable instance of the database system and update the active generation value to permit the first immutable instance to write data to the database. The computer system may receive a request to deploy a second immutable instance of the database system that includes an update not found in the first immutable instance. The computer system may deploy the second immutable instance and update the active generation value to cause the first immutable instance to cease writing data to the database and to permit the second immutable instance to write data to the database.

A system for querying a graph model and methods for making and using same. An initial vertex set can be received for one or more query blocks. The one or more query blocks can be executed to generate respective output vertex sets. The output vertex sets and the initial vertex set can be enabled to interconnect in a vertex-set-flow graph based on the query blocks. The vertex-set-flow graph can have a Directed Acyclic Graph shape. A selected query block can generate an output vertex set based on an input vertex set with or without traversal over an edge. A selected query block can calculate a runtime attribute. Edges and/or vertices of the graph model can be updated during querying. A selected block can call a graph query as a generic function. Functions for querying the graph model are powerful and can advantageously meet various graph query needs.

Database servers may maintain a database according to a database schema. A database change management system can include a profile service configured to collect database profile information. A forecast service can be configured to use SQL parsing to generate change objects and generate a forecast report indicative of a prediction of a failure or success of an implementation of the set of changes.

Disclosed is a method and system for joining datasets in a distributed computing environment. The system comprises a memory 206 and a processor 202. The processor 202 identifies a skewed dataset from two or more datasets to be joined. The processor 202 identifies a replication parameter from a configuration file. The processor 202 then assigns a randomly assigned machine number to each chunk of the skewed dataset owned by the nodes/machines involved in the join operation. The processor 202 forms copies of the non-skewed dataset equal to the replication parameter and adds the copy number to each sample of the copy of the non-skewed dataset formed. Further, the processor 202 merges each non-skewed dataset into the final copy of the non-skewed dataset, forming a single non skewed dataset. The processor 202 then repeats these steps for all the non-skewed datasets involved in the join operation resulting in generation of merged copies of all the non-skewed datasets and then performs the joining operation.

A code generator platform may receive source metadata and a target data model. The code generator platform may determine a parameter, of the target data model, that is associated with the attribute. The code generator platform may map, based on the attribute and the source metadata, the data to the parameter of the target data model. The code generator platform may generate, based on mapping the data to the parameter, data transformation code associated with the data and the target data model, wherein the data transformation code, when executed, generates target data that corresponds to the data according to the target data model. The code generator platform may perform an action associated with the data transformation code to permit the data transformation code to be executed in order to update a target database with the target data.