AUTOMATED INTEROPERATIONAL TRACKING IN COMPUTING SYSTEMS

Abstract

Techniques of automated interoperation tracking in computing systems are disclosed herein. One example technique includes tokenizing a first event log from a first software component and a second event log from the second software component by calculating frequencies of appearance corresponding to strings in the first and second event logs and selecting, as tokens, a first subset of the strings in the first event log and a second subset of the strings in the second event log individually having calculated frequencies of appearance above a preset frequency threshold. The example technique can also include generating an overall event log for a task executed by both the first and second software components by matching one of the strings in the first subset to another of the strings in the second subset.

Claims

1. A method of automated interoperations tracking performed by a server in a distributed computing system having a first software component configured to provide a first computing service and operatively coupled to a second software component configured to provide a second computing service to the first computing service, the method comprising: receiving, at the server, a first event log from the first software component and a second event log from the second software component, the first and second event components individually including data representing multiple entries each having multiple data fields related to a process executed at the first and/or second software components; and in response to receiving the first and second event logs, calculating first frequencies of appearance individually corresponding to first strings in the first event log from the first software component; calculating second frequencies of appearance individually corresponding to second strings in the second event log from the second software component; based on the first and second calculated frequencies of appearance, selecting a first subset of the first strings in the first event log having the calculated first frequencies of appearance in the first event log above a preset frequency threshold; selecting a second subset of the second strings in the second event log having the calculated second frequencies of appearance in the second event log above the preset frequency threshold; matching one of the first strings in the first subset to another of the second strings in the second subset; and using the matched one of the first strings in the first subset and the another of the second strings in the second subset as a token to create a new entry for a new event log by combining a first entry corresponding to the one of the first strings in the first subset to a second entry corresponding to the another of the second strings in the second subset.

2. The method of claim 1, further comprising: calculating the frequencies of appearance as respective percentages of appearance of the first and second strings in the first and second event logs, respectively; and selecting the first subset of the first strings in the first event log and the second subset of the second strings in the second event log includes selecting one or more of the first strings from the first or second event log having a calculated percentage of appearance above the frequency threshold.

3. The method of claim 1, further comprising: calculating the frequencies of appearance as respective percentages of appearance of the first and second strings in the first and second event logs, respectively; sorting the first and second strings in the first and second event logs according to the calculated percentages of appearance, respectively, into first and second lists; and selecting the first subset of the first strings in the first event log and the second subset of the second strings in the second event log includes selecting a preset number of the first or second strings from the first or second event log at top of the sorted first and second lists, respectively.

4. The method of claim 1 wherein the first subset of first strings in the first event log and the second subset of second strings in the second event log each include a globally unique identifier, a string without spaces, or at least a partial universal resource locator.

5. The method of claim 1, further comprising: removing one or more words or phrases of following grammatical types from the first and second event logs: article; coordinating conjunction; subordinating conjunction; correlative conjunction; conjunctive adverb; transition; or preposition; and removing a list of pre-defined words or phrases from the first and second event logs.

6. A method of automated interoperations tracking performed by a server in a distributed computing system having a first software component configured to provide a first computing service and operatively coupled to a second software component configured to provide a second computing service to the first computing service, the method comprising: receiving, at the server, a first event log from the first software component and a second event log from the second software component, the first and second event components individually including data representing multiple entries each having multiple data fields related to a process executed at the first and/or second software components; and in response to receiving the first and second event logs, tokenizing the first and second event logs including: calculating frequencies of appearance corresponding to multiple strings in the first and second event logs from the first and second software components, respectively; and selecting, as tokens, a first subset of the strings in the first event log and a second subset of the strings in the second event log individually having calculated frequencies of appearance above a preset frequency threshold; and generating an overall event log for a task executed by both the first and second software components by matching one of the strings in the first subset to another of the strings in the second subset.

7. The method of claim 6, further comprising: calculating the frequencies of appearance as respective percentages of appearance of the multiple strings in the first and second event logs; and selecting the first subset of the strings in the first event log and the second subset of the strings in the second event log includes selecting one or more of the multiple strings from the first or second event log having a calculated percentage of appearance above the frequency threshold.

8. The method of claim 6, further comprising: calculating the frequencies of appearance as respective percentages of appearance of the multiple strings in the first and second event logs; sorting the multiple strings in the first and second event logs according to the calculated percentages of appearance; and selecting the first subset of the strings in the first event log and the second subset of the strings in the second event log includes selecting a preset number of the multiple strings from the first or second event log at top of the sorted first or second event log according to the calculated percentages of appearance.

9. The method of claim 6 wherein the first subset of the strings in the first event log and the second subset of the strings in the second event log each include a globally unique identifier, a string without spaces, or at least a partial universal resource locator.

10. The method of claim 6 wherein tokenizing the first and second event logs also includes removing one or more words or phrases of following grammatical types from the first and second event logs before calculating the frequencies of appearance: article; coordinating conjunction; subordinating conjunction; correlative conjunction; conjunctive adverb; transition; or preposition.

11. The method of claim 6 wherein tokenizing the first and second event logs also includes removing a list of pre-defined words or phrases from the first and second event logs before calculating the frequencies of appearance.

12. The method of claim 6, further comprising: determining whether a count of strings in the first or second subset exceeds a threshold; and in response to determining that the number of strings in the first or second subset exceeds the threshold, decreasing a value of the frequency threshold such that the number of strings is below the threshold.

13. The method of claim 6 wherein: determining whether a count of strings in the first or second subset is below a threshold; and in response to determining that the number of strings in the first or second subset is below the threshold, increasing a value of the frequency threshold such that the number of strings is above the threshold.

14. The method of claim 6 wherein generating the overall event log for the task executed by both the first and second software components includes: matching one of the strings in the first subset to another of the strings in the second subset; and using the matched one of the strings in the first subset and the another of the strings in the second subset as one of the tokens to create a new entry for the overall event log by combining a first entry corresponding to the one of the strings in the first subset to a second entry corresponding to the another of the strings in the second subset.

15. A computing device, comprising: a processor; a memory operatively coupled to the processor, wherein the memory having instructions executable by the processor to cause the computing device to: receive a first event log from the first software component and a second event log from the second software component, the first and second event components individually including data representing multiple entries each having multiple data fields related to a process executed at the first and/or second software components; and upon receiving the first and second event logs, for each of the first or second event logs: calculate frequencies of appearance corresponding to multiple strings in the first and second event logs from the first and second software components, respectively; select a first subset of the strings in the first event log having the calculated frequencies of appearance in the first event log above a preset frequency threshold; select a second subset of the strings in the second event log having the calculated frequencies of appearance in the second event log above the preset frequency threshold; and generate an overall event log for a task executed by both the first and second software components by matching one of the strings in the first subset to another of the strings in the second subset.

16. The computing device of claim 15 wherein the first subset of the strings in the first event log and the second subset of the strings in the second event log each include a globally unique identifier, a string without spaces, or at least a partial universal resource locator.

17. The computing device of claim 15 wherein the memory includes additional instructions executable by the processor to cause the computing device to remove one or more words or phrases of following grammatical types from the first and second event logs before calculating the frequencies of appearance: article; coordinating conjunction; subordinating conjunction; correlative conjunction; conjunctive adverb; transition; or preposition.

18. The computing device of claim 15 wherein the memory includes additional instructions executable by the processor to cause the computing device to: determine whether a count of strings in the first or second subset exceeds a threshold; and in response to determining that the number of strings in the first or second subset exceeds the threshold, decrease a value of the frequency threshold such that the number of strings is below the threshold.

19. The computing device of claim 15 wherein the memory includes additional instructions executable by the processor to cause the computing device to: determine whether a count of strings in the first or second subset is below a threshold; and in response to determining that the number of strings in the first or second subset is below the threshold, increase a value of the frequency threshold such that the number of strings is above the threshold.

20. The computing device of claim 15 wherein to the overall event log for the task executed by both the first and second software components includes to: match one of the strings in the first subset to another of the strings in the second subset; and use the matched one of the strings in the first subset and the another of the strings in the second subset as one of the tokens to create a new entry for the overall event log by combining a first entry corresponding to the one of the strings in the first subset to a second entry corresponding to the another of the strings in the second subset.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIG. 1 is a schematic diagram illustrating a distributed computing system implementing automated interoperational tracking in accordance with embodiments of the disclosed technology.

[0021] FIG. 2 is a schematic diagram illustrating certain example hardware/software components of the distributed computing system of FIG. 1 implementing automated interoperational tracking in accordance with embodiments of the disclosed technology.

[0022] FIG. 3A is a schematic diagram illustrating examples of interoperational components in the distributed computing system of FIG. 1 individually configured to record a component trace in accordance with embodiments of the disclosed technology.

[0023] FIGS. 3B and 3C are schematic diagrams illustrating example component traces recorded by two of the interoperational components in FIG. 3A in accordance with embodiments of the disclosed technology.

[0024] FIG. 4 is a schematic diagram illustrating certain components of an example trace correlator configured for automated interoperational tracking.

[0025] FIGS. 5A and 5B are flowcharts illustrating example processes for automated interoperational tracking in accordance with embodiments of the disclosed technology.

[0026] FIG. 6 is a computing device suitable for certain components of the distributed computing system in FIG. 1.

DETAILED DESCRIPTION

[0027] Certain embodiments of systems, devices, components, modules, routines, data structures, and processes for automated interoperational tracking in a distributed computing system are described below. In the following description, specific details of components are included to provide a thorough understanding of certain embodiments of the disclosed technology. A person skilled in the relevant art will also understand that the technology can have additional embodiments. The technology can also be practiced without several of the details of the embodiments described below with reference to FIGS. 1-6. For example, instead of being implemented distributed computing systems, aspects of the disclosed technique disclosed herein can also be implemented in enterprise computing systems or other suitable types of computing systems.

[0028] As used herein, the term “distributed computing system” generally refers to an interconnected computer system having multiple network nodes that interconnect a plurality of servers or hosts to one another and/or to external networks (e.g., the Internet). The term “network node” generally refers to a physical network device. Example network nodes include routers, switches, hubs, bridges, load balancers, security gateways, or firewalls. A “host” generally refers to a physical computing device. In certain embodiments, a host can be configured to implement, for instance, one or more virtual machines, virtual switches, or other suitable virtualized components. For example, a host can include a server having a hypervisor configured to support one or more virtual machines, virtual switches, or other suitable types of virtual components. In other embodiments, a host can be configured to execute suitable applications directly on top of an operating system.

[0029] A computer network can be conceptually divided into an overlay network implemented over an underlay network in certain implementations. An “overlay network” generally refers to an abstracted network implemented over and operating on top of an underlay network. The underlay network can include multiple physical network nodes interconnected with one another. An overlay network can include one or more virtual networks. A “virtual network” generally refers to an abstraction of a portion of the underlay network in the overlay network. A virtual network can include one or more virtual end points referred to as “tenant sites” individually used by a user or “tenant” to access the virtual network and associated computing, storage, or other suitable resources. A tenant site can host one or more tenant end points (“TEPs”), for example, virtual machines. The virtual networks can interconnect multiple TEPs on different hosts. Virtual network nodes in the overlay network can be connected to one another by virtual links individually corresponding to one or more network routes along one or more physical network nodes in the underlay network. In other implementations, a computer network can only include the underlay network.

[0030] As used herein, the term “computing service” generally refers to one or more computing resources provided over a computer network such as the Internet. Example computing services include software as a service (“SaaS”), platform as a service (“PaaS”), and infrastructure as a service (“IaaS”). SaaS is a software distribution technique in which software applications are hosted by a cloud service provider in, for instance, datacenters, and accessed by users over a computer network. PaaS generally refers to delivery of operating systems and associated services over the computer network without requiring downloads or installation. IaaS generally refers to outsourcing equipment used to support storage, hardware, servers, network devices, or other components, all of which are made accessible over a computer network.

[0031] Also used herein, the term “resource provider” generally refers to a computing service that is configured to provide or make available one or more services or resources of a computing system. The resources can be deployed and managed through, for example, a “resource manager” accessible via a user portal. In certain embodiments, a resource provider can be configured to offer representational state transfer (“REST”) Application Programming Interfaces (“APIs”) for working with associated resources. For example, a resource provider can be configured to deploy a secret storage (e.g., a key vault) for storing keys and credentials. In other embodiments, resource providers can also provide computing resources (e.g., virtual machines), storage resources (e.g., network storage), network resources (e.g., virtual networks), database resources (e.g., database servers), or other suitable types of resources.

[0032] As used herein, an “event log” or a “trace” generally refers to a collection of data that records hardware/software operational occurrences and/or errors in computing systems. For example, an operating system or an application executing on a computing device can record various error messages, such as application stopped responding, memory fault, etc. In another example, firmware on a processor of the computing device can also record events and/or errors (e.g., internal memory error) encountered by the processor during operation. In further examples, multiple components of a distributed computing system can also individually record corresponding event logs. Examples of such components can include computing services and/or resources in the distributed computing system.

[0033] An event log can be organized in various suitable ways. For instance, an event log can include a file having common separated values each corresponding to a data field (e.g., a system ID, a timestamp, etc.). In another example, an event log can include a database table having columns of data fields and rows with values corresponding to the data fields in the columns. Each row can be referred to as an entry in the event log. In further examples, an event log can be organized as an array or can have other suitable structured or non-structured format.

[0034] Also used herein, a “frequency of appearance” generally refers to a percentage or ratio that a string, word, or phrase appears in a file over a total number of appearances of all strings, words, or phrases in the file, such as an event log. For instance, if a string, e.g., “65a7d61d-62c9” appears once in a file with two other strings each appearing twice in the file, then the string “65a7d61d-62c9” has a frequency of appearance of (1/(1+2*2))=20% or a ratio of 0.2. When calculating a frequency of appearance of the string, all strings in the file can be initially identified. The number of appearances of each of the strings can then be identified and used to derive a total appearance number. Then, the frequency of appearance of the string can be calculated by dividing the number of appearances of “65a7d61d-62c9” by the total number of appearances.

[0035] FIG. 1 is a schematic diagram illustrating a distributed computing system 100 implementing automated interoperational tracking in accordance with embodiments of the disclosed technology. As shown in FIG. 1, the distributed computing system 100 can include an underlay network 108 interconnecting a plurality of hosts 106, a plurality of client devices 102 associated with corresponding users 101, and a platform controller 125 operatively coupled to one another. The platform controller 125 can be a cluster controller, a fabric controller, a database controller, and/or other suitable types of controllers configured to monitor and manage resources and operations of the servers 106 and/or other components in the distributed computing system 100. Even though certain hardware components are shown in FIG. 1, in other embodiments, the distributed computing system 100 can also include additional and/or different components or arrangements. For example, in certain embodiments, the distributed computing system 100 can also include network storage devices, additional hosts, and/or other suitable components (not shown) in other suitable configurations.

[0036] As shown in FIG. 1, the underlay network 108 can include one or more network nodes 112 that interconnect the multiple hosts 106 and the client device 102 of the users 101. In certain embodiments, the hosts 106 can be organized into racks, action zones, groups, sets, or other suitable divisions. For example, in the illustrated embodiment, the hosts 106 are grouped into three host sets identified individually as first, second, and third host sets 107a-107c. Each of the host sets 107a-107c is operatively coupled to a corresponding network nodes 112a-112c, respectively, which are commonly referred to as “top-of-rack” network nodes or “TORs.” The TORs 112a-112c can then be operatively coupled to additional network nodes 112 to form a computer network in a hierarchical, flat, mesh, or other suitable types of topologies. The underlay network 108 can allow communications among hosts 106, the platform controller 125, and the users 101. In other embodiments, the multiple host sets 107a-107c may share a single network node 112 or can have other suitable arrangements.

[0037] The hosts 106 can individually be configured to provide computing, storage, and/or other suitable cloud or other suitable types of computing services to the users 101. For example, as described in more detail below with reference to FIG. 2, one of the hosts 106 can initiate and maintain one or more virtual machines 144 (shown in FIG. 2) or containers (not shown) upon requests from the users 101. The users 101 can then utilize the provided virtual machines 144 or containers to perform database, computation, communications, and/or other suitable tasks. In certain embodiments, one of the hosts 106 can provide virtual machines 144 for multiple users 101. For example, the host 106a can host three virtual machines 144 individually corresponding to each of the users 101. In other embodiments, multiple hosts 106 can host virtual machines 144 for the users 101.

[0038] The client devices 102 can each include a computing device that facilitates the users 101 to access computing services provided by the hosts 106 via the underlay network 108. In the illustrated embodiment, the client devices 102 individually include a desktop computer. In other embodiments, the client devices 102 can also include laptop computers, tablet computers, smartphones, or other suitable computing devices. Though three users 101 are shown in FIG. 1 for illustration purposes, in other embodiments, the distributed computing system 100 can facilitate any suitable numbers of users 101 to access cloud or other suitable types of computing services provided by the hosts 106 in the distributed computing system 100.

[0039] FIG. 2 is a schematic diagram illustrating certain example hardware/software components of the distributed computing system 100 in accordance with embodiments of the disclosed technology. In FIG. 2 and in other Figures herein, individual software components, objects, classes, modules, and routines may be a computer program, procedure, or process written as source code in C, C++, C#, Java, and/or other suitable programming languages. A component may include, without limitation, one or more modules, objects, classes, routines, properties, processes, threads, executables, libraries, or other components. Components may be in source or binary form. Components may include aspects of source code before compilation (e.g., classes, properties, procedures, routines), compiled binary units (e.g., libraries, executables), or artifacts instantiated and used at runtime (e.g., objects, processes, threads).

[0040] Components within a system may take different forms within the system. As one example, a system comprising a first component, a second component and a third component can, without limitation, encompass a system that has the first component being a property in source code, the second component being a binary compiled library, and the third component being a thread created at runtime. The computer program, procedure, or process may be compiled into object, intermediate, or machine code and presented for execution by one or more processors of a personal computer, a network server, a laptop computer, a smartphone, and/or other suitable computing devices.

[0041] Equally, components may include hardware circuitry. A person of ordinary skill in the art would recognize that hardware may be considered fossilized software, and software may be considered liquefied hardware. As just one example, software instructions in a component may be burned to a Programmable Logic Array circuit or may be designed as a hardware circuit with appropriate integrated circuits. Equally, hardware may be emulated by software. Various implementations of source, intermediate, and/or object code and associated data may be stored in a computer memory that includes read-only memory, random-access memory, magnetic disk storage media, optical storage media, flash memory devices, and/or other suitable computer readable storage media excluding propagated signals.

[0042] As shown in FIG. 2, the distributed computing system 100 can include hosts 106 (e.g., a source host 106a and a destination host 106′) interconnected by an overlay network 108′. The source host 106a and the destination hosts 106b and 106b′ can each include a processor 132, a memory 134, and a network interface card 136 operatively coupled to one another. In other embodiments, the source or destination host 106a and 106b can also include input/output devices configured to accept input from and provide output to an operator and/or an automated software controller (not shown), or other suitable types of hardware/software components.

[0043] The processor 132 can include a microprocessor, caches, and/or other suitable logic devices. The memory 134 can include volatile and/or nonvolatile media (e.g., ROM; RAM, magnetic disk storage media; optical storage media; flash memory devices, and/or other suitable storage media) and/or other types of computer-readable storage media configured to store data received from, as well as instructions for, the processor 132 (e.g., instructions for performing the methods discussed below with reference to FIGS. 5A-5D). Though only one processor 132 and one memory 134 are shown in the individual hosts 106 for illustration in FIG. 2, in other embodiments, the individual hosts 106 can include two, six, eight, or any other suitable number of processors 132 and/or memories 134.

[0044] The source host 106a and the destination host 106b can individually contain instructions in the memory 134 executable by the processors 132 to cause the individual processors 132 to provide a hypervisor 140 (identified individually as first and second hypervisors 140a and 140b) and an operating system 141 (identified individually as first and second operating systems 141a and 141b). Even though the hypervisor 140 and the operating system 141 are shown as separate components, in other embodiments, the hypervisor 140 can operate on top of the operating system 141 executing on the hosts 106 or a firmware component of the hosts 106.

[0045] The hypervisor 140 can be configured to generate, monitor, terminate, and/or otherwise manage one or more virtual machines 144 organized into tenant sites 142. For example, as shown in FIG. 2, the source host 106a can provide a first hypervisor 140a that manages first and second tenant sites 142a and 142b, respectively. The destination host 106b can provide a second hypervisor 140b that manages first and second tenant sites 142a′ and 142b′, respectively. The hypervisors 140 are individually shown in FIG. 2 as a software component. However, in other embodiments, the hypervisors 140 can be firmware and/or hardware components. The tenant sites 142 can each include multiple virtual machines 144 for a tenant (not shown). For example, the source host 106a and the destination host 106b can both host the tenant site 142a and 142a′ for a first tenant. The source host 106a and the destination host 106b can both host the tenant site 142b and 142b′ for a second tenant. Each virtual machine 144 can be executing a corresponding operating system, middleware, and/or applications.

[0046] Also shown in FIG. 2, the overlay network 108′ can include one or more virtual networks 146 that interconnect the tenant sites 142a and 142b across multiple hosts 106. For example, a first virtual network 142a interconnects the first tenant sites 142a and 142a′ at the source host 106a and the destination host 106b. A second virtual network 146b interconnects the second tenant sites 142b and 142b′ at the source host 106a and the destination host 106b. Even though a single virtual network 146 is shown as corresponding to one tenant site 142, in other embodiments, multiple virtual networks 146 (not shown) may be configured to correspond to a single tenant site 146.

[0047] The virtual machines 144 can be configured to execute one or more software applications 147 to provide suitable cloud or other suitable types of computing services to the users 101 (FIG. 1). For example, the source host 106a can execute a software application 147 that is configured to provide a computing service that monitors online trading and distribute price data to multiple users 101 subscribing to the computing service. The virtual machines 144 on the virtual networks 146 can also communicate with one another via the underlay network 108 (FIG. 1) even though the virtual machines 144 are located on different hosts 106.

[0048] As shown in FIG. 2, to facilitate communications with one another or with external devices, the individual hosts 106 can also include a network interface card (“NIC”) 136 for interfacing with a computer network (e.g., the underlay network 108 of FIG. 1). A NIC 136 can include a network adapter, a LAN adapter, a physical network interface, or other suitable hardware circuitry and/or firmware to enable communications between hosts 106 by transmitting/receiving data (e.g., as packets) via a network medium (e.g., fiber optic) according to Ethernet, Fibre Channel, Wi-Fi, or other suitable physical and/or data link layer standards. During operation, the NIC 136 can facilitate communications to/from suitable software components executing on the hosts 106. Example software components can include the virtual switches 141, the virtual machines 144, applications 147 executing on the virtual machines 144, the hypervisors 140, or other suitable types of components.

[0049] In operation, the software application 147, the virtual machine 144, and/or other suitable software components on the source host 106a can generate an outbound packet destined to, for instance, other software applications 147 at the destination host 106b. The NIC 136 at the source host 106a can forward the generated packet to the first TOR 112a, which in turn forwards the packet to the second TOR 112b via the overlay/underlay network 108 and 108′. The second TOR 112b can then forward the packet to the NIC 136 of the destination host 106b, which can then forward the packet to one of the software applications 147 executed at the destination host 106b.

[0050] The virtual machines 144 at the various hosts 106 can each execute suitable software applications 147 to provide a computing service to the users 101 and/or to other computing services in the distributed computing system 100. For example, FIG. 3A illustrates example interoperations of various computing services in the distributed computing system 100 of FIG. 1 in accordance with embodiments of the disclosed technology. As shown in FIG. 1, the distributed computing system 100 can include a resource manager 150 configured to receive a task request 160 from a user 101 and call various other computing services to execute the requested task. For example, the resource manager 150 can submit a task call 162 to a subscription service 152 configured to verify that the user 101 indeed has subscription to the requested task. Upon verification, the resource manager 150 can then submit additional task calls 162′ to, for instance, the compute resource provider 154 to allocate compute resources to execute the requested task and the storage resource provider 156 to allocate suitable storage space to store results of executing the requested task. In turn, the compute resource provider 154 can submit another task call 162″ to a network resource provider 158 to request network bandwidth or other suitable resources to facilitate execution of the requested task.

[0051] As shown in FIG. 3A, the various computing services can separately record a component trace 170 for events monitored during execution of the various computing services. However, such component traces 170 may not be readily combined to form an overall trace corresponding to the requested task from the user 101 due to a lack of tracking identifier shared among the computing services. In component traces 170, some entries are related to the requested task while others are related to other tasks executed by those computing services. Thus, without a tracking identifier, one may not readily discern which entry in one component trace corresponds to another entry in another component trace.

[0052] The recorded component traces 170 can also include different data fields (e.g., columns with different identifiers) configured to contain different data. For example, as shown in FIGS. 3B and 3C, a first component trace 170 recorded by the resource manager 150 can include a table having data fields of “Status code,” “Service Request ID,” and “Time Stamp” which a second component trace 170′ recorded by the subscription service 152 can have another table having data fields of “Status code,” “Service Request ID,” “Outgoing Request,” and “Time Stamp.” As such, without a tracking ID to identify associated entries, different entries in the component traces 170 and 170′ may not be easily correlated to the requested task. Thus, when an issue is detected during execution of a request task (e.g., instantiation of a VM), a technician or administrator may have difficulty determining which computing service(s) in the distributed computing system 100 that caused the problem.

[0053] One solution to the foregoing difficulty is to perform correlation of entries from different component traces 170 manually. However, given that a recorded component trace 170 from one data source can be hundreds of even thousands of gigabytes in thirty minutes, such manual correlation can be labor intensive, impractical, or even impossible at times. Another solution is to instrument the various components (e.g., the resource manager, the subscription manager, the compute resource provider, the storage resource provider, and the network resource provider) in the distributed computing system 100 to pass on a tracking ID corresponding to the requested task when recording the component traces 170. However, instrumenting a software application 147 (FIG. 2) that provides a corresponding computing service can be costly and error prone because code changes to the various components can be complicated and time consuming.

[0054] As shown in FIG. 4, several embodiments of the disclosed technology can address some aspects of the foregoing difficulties by deploying a trace correlator 180 that is configured to automate interoperation tracking without instrumenting distinct data sources in the distributed computing system 100. As shown in FIG. 4, the trace correlator 180 can include a remover 182, an extractor 184, and a linker 186 operatively coupled to one another. Though particular components of the trace correlator 180 is shown in FIG. 4 for illustration purposes, in other embodiments, the trace correlator 180 can also include input/output, calculation, or other suitable types of components.

[0055] In certain implementations, the trace correlator 180 can be configured to tokenize individual component traces 170 from distinct data sources (e.g., the computing services shown in FIG. 3A) to identify strings or “tokens” that appear less frequently in the component traces 170 than others based on a frequency threshold (e.g., 1%, 3%, 5%, etc.) or other suitable criteria. Examples of a token can include a globally unique identifier (“GUID”), a string without spaces (e.g., “65a7d61d-62c9” shown in FIG. 3B), or at least a partial universal resource locator (“URL”). In other examples, a token can also include a combination of at least two of the foregoing or other suitable types of data. Example operations of extracting such tokens are described in more detail below.

[0056] As shown in FIG. 4, the remover 182 can be configured to perform various operations to exclude non- or less-unique words and/or phrases from the component traces 170 to generate a filtered list 172. For example, the remover 182 can be configured to initially remove the following categories of words and/or phrases from the component traces: [0057] articles, e.g., “a,” “the,” “an,” etc.; [0058] coordinating conjunctions, e.g., “for,” “and,” “nor,” “but,” “or,” “yet,” “so,” etc.; [0059] subordinating Conjunctions, e.g., “though,” “before,” “until,” “that,” etc. [0060] correlative Conjunctions, e.g., “as,” “if . . . then,” “also,” etc. [0061] conjunctive Adverbs, e.g., “also,” “however,” “still,” then,” etc.
In other examples, the remover 182 can be configured to remove transitions for similarity (e.g., “too”), contradiction (e.g., “instead”), cause (e.g., “due to”), examples (e.g., “like”), effect (e.g., “as a result”), conclusion (e.g., “in conclusion”), time (e.g., “during”), space (e.g., “here”), or other suitable functions. In further examples, prepositions such as “on,” “at,” “to,” “from,” or other suitable types of non-unique words can also be removed. In yet further examples, the remover 182 can be configured to remove a list of non-unique words such as “failed”, “exception”, “deployment,” “error,” etc. from a pre-defined list established by an administrator or user 101.

[0062] Upon removing the non- or less-unique words and/or phrases from the component traces 170, the remover 182 can be configured to transmit the filtered list 172 to the extractor 184 for further processing. In one embodiment, the extractor 184 can be configured to sort the remaining words and/or phrases in the filtered list 172 according to a frequency of appearance in the corresponding component traces 170. Based on the sorted component traces 170, the extractor 184 can be configured to extract a subset of sufficiently unique words and/or phrases from the individual component traces 170 as tokens 174 based on a frequency threshold 192. In one example, words and/or phrases appearing less than a frequency threshold (e.g., 1%, 3%, 5%, etc.) in the component traces 170 can be selected. In other examples, top three, five, ten, twenty, or one hundred words and/or phrases with the lowest frequencies of threshold may be selected. In further examples, the sufficiently unique words and/or phrases can be selected according to a length of characters, a type of characters, or in other suitable ways.

[0063] In certain embodiments, the frequency threshold 192 can be adjusted based on numbers of items in the tokens 174 of extracted subset of words and/or phrases. For example, when the tokens 174 of extracted subset of words and/or phrases exceeds a high threshold (e.g., ten, fifty, one hundred, etc.), the frequency threshold 192 can be decreased to the high threshold is not exceeded. On the other hand, when the tokens 174 of the subset of words and/or phrases is below a low threshold (e.g., one, two, five, etc.), the frequency threshold 192 can be increased. In other embodiments, the frequency threshold 192 can be adjusted based on correlation results of entries in the component traces 170 described below or based on other suitable criteria.

[0064] Upon extracting the tokens 174, the extractor 184 can be configured to provide the tokens 174 to the linker 186 to correlate entries in the component traces 170. Upon receiving the tokens 174, the linker 186 can be configured to use the tokens 174 of extracted subset of sufficiently unique words and/or phrases from the component traces to correlate entries in individual component traces 170. The inventors have recognized that different components in the distributed computing system 100 may use the same sufficiently unique strings in different data fields of the recorded component traces 170. For instance, as shown in FIGS. 3B and 3C, the resource manager 150 may record the first component trace 170 identifying the requested task using a “Service Request ID” of, e.g., “65a7d61d-62c9.” However, when the resource manager 150 calls the other computing services (e.g., the subscription service 152), the subscription service 152 may not record events indexed to the same “Service Request ID.” Instead, the subscription service 152 may record events associated with the requested task using, e.g., a “Http Outgoing Request ID” with the same sufficiently unique string in the second component trace 170′ while the “Service Request ID” contains a null value in the second component trace 170′. Such mismatch in data fields can occur when, for instance, the resource manager 150 calls the subscription service 152 for the requested task. The resource manager 150 then periodically polls the subscription service 152 regarding completion of the requested task with an identifier or token (e.g., “65a7d61d-62c9”) known to both the resource manger 150 and the subscription service 152. Alternatively, the subscription service 152 can notify the resource manager 150 regarding completion of the requested task using the same identifier upon completion.

[0065] Though the “Http Outgoing Request” in the second component trace 170′ and the “Service Request ID” in the first component trace 170 can be different data fields, the string “65a7d61d-62c9” shared by both the first and second component traces 170 and 170′ can be used as a token 174 to correlate entries from the first and second component traces 170 and 170′. For example, a first entry 171 corresponding to “65a7d61d-62c9” in the first component trace 170 can be correlated to a second entry 171′ corresponding to the same value in the second component trace 170′ as a call from the resource manager 150 to the subscription service 152 even though “65a7d61d-62c9” appears in different data fields of the first and second component traces 170 and 170′. As such, events from both the resource manager 150 and the subscription service 152 can be correlated to form an interoperation trace 176 (FIG. 4) of the requested task. In certain embodiments, a single token 174, e.g., “65a7d61d-62c9” can be used to correlate the entries in the first and second component traces 170 and 170′. In other embodiments, a combination of multiple strings shared between the first and second component traces can be used as a composite token to correlate entries in the first and second component traces. As shown in FIG. 4, the interoperational trace 176 can include a first entry 177 from the first component trace 170 in FIG. 3A and a second entry 177′ from the second trace 170′ in FIG. 3B. The interoperational trace 176 can also include a new data filed “Component ID” to identify which component trace 170 an entry 177 in the interoperational trace 176 comes from.

[0066] In certain examples, the same sufficiently unique string, e.g., “65a7d61d-62c9” can be used as a token 174 to correlate entries in multiple pairs of component traces 170 when, for example, the same token 174 is used by another computing service (e.g., the compute resource provider 154 in FIG. 3A) called by the subscription service 152 when executing the request from the resource manager 150. In other examples, different sufficiently unique strings can be used to correlate different pairs of entries in multiple component traces 170. For example, “65a7d61d-62c9” can be used to correlate the first and second entries in the first and second component traces 170 and 170′ recorded by the resource manager 150 and the subscription service 152, respectively. Another sufficiently unique string, e.g., “1a-9009b5598” in a “Responsible body” of the second component trace 170′ can be used to correlate to a third entry (not shown) in a third trace (not shown) having a “Service ID” having the same string and recorded by, for instance, the compute resource provider 154. As such, by recursively performing the foregoing tokenization and correlation, the trace correlator 180 can be configured to build the interoperational trace 176 corresponding to the requested task by the user 101.

[0067] Several embodiments of the disclosed technology can be efficient and effective in generating an interoperational trace 176 of executing a requested task from a user 101 in a complex computing system such as the distributed computing system 100, without instrumenting each hardware/software components. By identifying sufficiently unique strings in the component traces 170, the trace correlator 180 can be configured to use such strings as tokens 174 to correlate entries from different component traces 170 to form the interoperational trace of the requested task without code changes to implement a tracking ID in the various hardware/software components providing the computing services. Such correlation can be performed based on historical event logs or may be performed near real time based on real-time event logs. Thus, the interoperational trace 176 corresponding to the requested task can be generated to aid debugging, troubleshooting, or other suitable purpose without impacting the code base or operations of the various hardware/software components in the distributed computing system 100.

[0068] FIGS. 5A and 5B are flowcharts illustrating example processes for automated interoperational tracking in accordance with embodiments of the disclosed technology. Though aspects of the processes are described below in the context of the distributed computing system 100, in other embodiments, the processes may also be implemented in computing systems with additional and/or different hardware/software components.

[0069] As shown in FIG. 5A, a process 200 can include receiving component traces from distinct data sources at stage 202. In one example, the distinct data sources can be different computing services in the distributed computing system 100. In other examples, the distinct data sources can also include hardware/software components external to the distributed computing system 100 or include other suitable types of data sources. The process 200 can also include tokenizing the received component traces at stage 208. Example operations of tokenizing the component traces are described below with reference to FIG. 5B. The process 200 can then include correlating entries in different received component traces using tokens identified in the received component traces at stage 206. Example operations of correlating the entries are described above with reference to FIG. 4. The process 200 can then include generating an interoperational trace with the correlated entries at stage 210.

[0070] As shown in FIG. 5B, example operations of tokenizing a component trace can include removing non- or less unique words and/or phrases at stage 212, such as those described above with reference to FIG. 4. The operations can then include sorting the remaining words and/or phrases in the component trace at stage 212. The operations can then include a decision stage 214 to determine whether each of the remaining words and/or phrases has a frequency of appearance greater than a frequency threshold. In response to determining that a remaining word and/or phrase has a frequency of appearance greater than the frequency threshold, the operations can proceed to adding the word and/or phrase to a subset of the remaining words and/or phrases as tokens at stage 216. Otherwise, the word and/or phrase can be eliminated from the subset as a token at stage 218.

[0071] FIG. 6 is a computing device 300 suitable for certain components of the distributed computing system 100 in FIG. 1. For example, the computing device 300 can be suitable for the hosts 106, the client devices 102, or the platform controller 125 of FIG. 1. In a very basic configuration 302, the computing device 300 can include one or more processors 304 and a system memory 306. A memory bus 308 can be used for communicating between processor 304 and system memory 306.

[0072] Depending on the desired configuration, the processor 304 can be of any type including but not limited to a microprocessor (μP), a microcontroller (μC), a digital signal processor (DSP), or any combination thereof. The processor 304 can include one more level of caching, such as a level-one cache 310 and a level-two cache 312, a processor core 314, and registers 316. An example processor core 314 can include an arithmetic logic unit (ALU), a floating-point unit (FPU), a digital signal processing core (DSP Core), or any combination thereof. An example memory controller 318 can also be used with processor 304, or in some implementations memory controller 318 can be an internal part of processor 304.

[0073] Depending on the desired configuration, the system memory 306 can be of any type including but not limited to volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.) or any combination thereof. The system memory 306 can include an operating system 320, one or more applications 322, and program data 324. This described basic configuration 302 is illustrated in FIG. 6 by those components within the inner dashed line.

[0074] The computing device 300 can have additional features or functionality, and additional interfaces to facilitate communications between basic configuration 302 and any other devices and interfaces. For example, a bus/interface controller 330 can be used to facilitate communications between the basic configuration 302 and one or more data storage devices 332 via a storage interface bus 334. The data storage devices 332 can be removable storage devices 336, non-removable storage devices 338, or a combination thereof. Examples of removable storage and non-removable storage devices include magnetic disk devices such as flexible disk drives and hard-disk drives (HDD), optical disk drives such as compact disk (CD) drives or digital versatile disk (DVD) drives, solid state drives (SSD), and tape drives to name a few. Example computer storage media can include volatile and nonvolatile, removable, and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. The term “computer readable storage media” or “computer readable storage device” excludes propagated signals and communication media.

[0075] The system memory 306, removable storage devices 336, and non-removable storage devices 338 are examples of computer readable storage media. Computer readable storage media include, but not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other media which can be used to store the desired information, and which can be accessed by computing device 300. Any such computer readable storage media can be a part of computing device 300. The term “computer readable storage medium” excludes propagated signals and communication media.

[0076] The computing device 300 can also include an interface bus 340 for facilitating communication from various interface devices (e.g., output devices 342, peripheral interfaces 344, and communication devices 346) to the basic configuration 302 via bus/interface controller 330. Example output devices 342 include a graphics processing unit 348 and an audio processing unit 350, which can be configured to communicate to various external devices such as a display or speakers via one or more NV ports 352. Example peripheral interfaces 344 include a serial interface controller 354 or a parallel interface controller 356, which can be configured to communicate with external devices such as input devices (e.g., keyboard, mouse, pen, voice input device, touch input device, etc.) or other peripheral devices (e.g., printer, scanner, etc.) via one or more I/O ports 358. An example communication device 346 includes a network controller 360, which can be arranged to facilitate communications with one or more other computing devices 362 over a network communication link via one or more communication ports 364.

[0077] The network communication link can be one example of a communication media. Communication media can typically be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and can include any information delivery media. A “modulated data signal” can be a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), microwave, infrared (IR) and other wireless media. The term computer readable media as used herein can include both storage media and communication media.

[0078] The computing device 300 can be implemented as a portion of a small-form factor portable (or mobile) electronic device such as a cell phone, a personal data assistant (PDA), a personal media player device, a wireless web-watch device, a personal headset device, an application specific device, or a hybrid device that include any of the above functions. The computing device 300 can also be implemented as a personal computer including both laptop computer and non-laptop computer configurations.

[0079] From the foregoing, it will be appreciated that specific embodiments of the disclosure have been described herein for purposes of illustration, but that various modifications may be made without deviating from the disclosure. In addition, many of the elements of one embodiment may be combined with other embodiments in addition to or in lieu of the elements of the other embodiments. Accordingly, the technology is not limited except as by the appended claims.