METHOD AND SYSTEM FOR ADAPTIVE CORNER DETECTION USING DYNAMIC VISION SENSORS

Abstract

A method for adaptive corner detection is provided. The method includes: obtaining one or more event data from a dynamic vision sensor; capturing and organizing a plurality of recorded events of the event data into a 2D array; transforming the 2D array into one or a plurality of Ordered Surface (OS) matrices by populating one or a plurality of empty Image Matrices with the recorded events based on their coordinates and assigning order values; and applying a corner detector to the OS matrices. A system for adaptive corner detection is also provided.

Claims

1. A method for adaptive corner detection in machine vision, comprising: obtaining one or more event data from a dynamic vision sensor; capturing and arranging, by a machine vision processor, a plurality of recorded events of the event data into a 2D array; transforming, by the machine vision processor the 2D array into one or a plurality of Ordered Surface matrices by populating one or a plurality of empty Image Matrices with the recorded events based on their coordinates and assigning order values; applying, by the machine vision processor, a corner detector to the Order Surface matrices; and outputting a corner detection result from the corner detector.

2. The method of claim 1, wherein the recorded events are arranged in global temporal ordering in the 2D array.

3. The method of claim 1, wherein the recorded events are arranged in a plurality of rows in the 2D array, and each of the rows has a series of the recorded events arranged in global temporal ordering; and wherein the numbers of the recorded events in the rows are the same, and the rows are arranged in global temporal ordering.

4. The method of claim 3, wherein the maximum number of the recorded events in every row ranges from 25 to 100.

5. The method of claim 3, wherein each Ordered Surface matrix is derived from all the rows of the recorded events in the 2D array.

6. The method of claim 1, wherein the step of applying the corner detector to one of the Ordered Surface matrices includes: extracting a patch of elements in the Ordered Surface matrix; employing a Harris detector to the patch to generate a Harris score; and comparing the Harris score generated by the Harris detector with a predefined threshold.

7. The method of claim 6 further comprising: performing sort normalization to the patch of elements before employing the Harris detector.

8. The method of claim 6, wherein the patch has M rows and N columns, and the M ranges from 7 to 11, and the N ranges from 7 to 11.

9. The method of claim 1 further comprising: applying a spatial-temporal correlation filter to the event data before organizing the recorded events into the 2D array.

10. A system for adaptive corner detection in machine vision, comprising: a dynamic vision sensor; and a machine vision processor electrically connected to the dynamic vision sensor; wherein the machine vision processor is configured to: obtain one or more event data from the dynamic vision sensor; capture and arrange a plurality of recorded events of the event data into a 2D array; transform the 2D array into one or a plurality of Ordered Surface matrices by populating one or a plurality of empty Image Matrices with the recorded events based on their coordinates and assigns order values; and apply a corner detector to the Order Surface matrices to output a corner detection result.

11. The system of claim 10, wherein the recorded events are arranged in global temporal ordering in the 2D array.

12. The system of claim 10, wherein the recorded events are arranged in a plurality of rows in the 2D array, and each of the rows has a series of the recorded events arranged in global temporal ordering, and the numbers of the recorded events in the rows are the same.

13. The system of claim 12, wherein the maximum number of the recorded events in every row ranges from 25 to 100.

14. The system of claim 12, wherein each Ordered Surface matrix is derived from all of the row of the recorded events in the 2D array.

15. The system of claim 10, wherein the processing device extracts a patch of elements in the Ordered Surface matrix; wherein the processing device employs a Harris detector to the patch to generate a Harris score; and wherein the processing device compares the Harris score generated by the Harris detector with a predefined threshold.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Embodiments of the invention are described in more details hereinafter with reference to the drawings, in which:

[0020] FIG. 1 depicts a schematic view of a 2D array in accordance with an embodiment of the present invention;

[0021] FIG. 2 depicts a schematic view of a 2D array and the corresponded OS in accordance with another embodiment of the present invention;

[0022] FIG. 3 depicts a schematic view of a system for adapting corner detection in accordance with another embodiment of the present invention;

[0023] FIG. 4 depicts a flow chart of the procedures of a method or a system for adapting corner detection in accordance with some embodiments of the present invention;

[0024] FIG. 5 depicts a flow chart of the procedures of a method or a system for adapting corner detection in accordance with another embodiment of the present invention;

[0025] FIG. 6 depicts a schematic view of a 2D array in accordance with another embodiment of the present invention;

[0026] FIG. 7 depicts a flow chart of the procedures of a method or a system for adapting corner detection in accordance with another embodiment of the present invention; and

[0027] FIG. 8 depicts a flow chart of the procedures of a method or a system for adapting corner detection in accordance with still another embodiment of the present invention.

DETAILED DESCRIPTION

[0028] In the following description, methods and systems of adaptive corner detection and the likes are set forth as preferred examples. It will be apparent to those skilled in the art that modifications, including additions and/or substitutions may be made without departing from the scope and spirit of the invention. Specific details may be omitted so as not to obscure the invention; however, the disclosure is written to enable one skilled in the art to practice the teachings herein without undue experimentation.

[0029] In accordance with a first aspect of the present invention, a method for adaptive corner detection using dynamic vision sensors is provided. The method presents a corner detection pipeline that can deployed on edge devices. Features of the method includes: a way to save events without using too much memory while keeping their time order; an efficient method to turn events into a fixed-sized matrix for corner detection; a way to quickly identify corners in events without losing accuracy.

[0030] The method includes the following procedures: updating a 2D array with temporal ordering; constructing Ordered Surface (OS) matrices which introduce relative spatial ordering; and perform corner detection.

[0031] However, the method is not limited to these procedures. The following will provide a well-illustrated detailed description of this method.

[0032] The steps of the method of this embodiment include obtaining an event data from a dynamic vision sensor (DVS).

[0033] The DVS of this embodiment is a type of sensor used in computer vision and robotics to detect changes in visual information in a highly efficient and asynchronous manner. The DVS may include an event camera. In some other embodiments, the DVS may include neuromorphic cameras, silicon retinas, or asynchronous cameras.

[0034] An event camera is a specialized type of camera that fundamentally differs from traditional cameras. Instead of capturing images at a fixed rate, event cameras detect changes in brightness or intensity at the pixel level and only transmit information when a significant change occurs. This unique feature allows them to operate with exceptionally low latency and high temporal resolution, making them well-suited for capturing fast and dynamic events in real-time, such as rapid motion or sudden changes in lighting.

[0035] The DVS of the embodiment provides event data. The event data includes a plurality of recorded events, and each of the recorded events contains four key parameters. The recorded events generated by an event camera encapsulate crucial information about dynamic changes in the visual environment. The four key parameters include: pixel coordinates (x, y), timestamp (t), and polarity (p).

[0036] The pixel coordinates pinpoint the location in the camera's field of view where the event occurred, offering spatial specificity. The timestamp provides the precise moment when the event took place, allowing for accurate temporal resolution. Importantly, the polarity indicates whether the change in brightness was an increase or decrease, conveying the directionality of the observed alteration in luminance.

[0037] These parameters collectively contribute to the unique advantages of event cameras. The asynchronous nature of event data transmission, driven by these parameters, enables them to capture fast and subtle changes in real-time with minimal latency. This distinctive approach to sensing visual information makes DVS of this embodiment particularly valuable in applications requiring quick and precise responsiveness, such as robotics, autonomous systems, and computer vision tasks where traditional frame-based cameras may fall short.

[0038] In this embodiment, after the event data is obtained, the method includes capturing and organizing the recorded events of the event data into the 2D array.

[0039] In this step, at least some of the recorded events in series is captured. In this embodiment, the recorded events are captured in real time, and the recorded events are organized according to its timestamp. To be specific, the pixel coordinates, i.e. x, y, are the main information of elements that is saved in the 2D array, and each of the elements is corresponded to one of the recorded events, while the arrangement is according to the timestamp of each recorded events.

[0040] In other words, in the 2D array of this embodiment, each element corresponds to a recorded event and contains essential pixel coordinates, denoted as x and y. The arrangement of the elements is based on the timestamps of the recorded events, providing a chronological order of the events.

[0041] The method accordance with this embodiment constructs a 2D array, initialized with 0's as its initial entry values. Subsequently, upon acquiring data from the event camera, the temporal sequence of event records is systematically populated into the corresponding 2D array entries. This process ensures that the 2D array encapsulates the temporal evolution of the recorded events, providing a structured representation of the data within the context of data processing.

[0042] In other words, in this embodiment, after multiple recorded events are captured, the recorded events are arranged in a plurality of rows. During organizing the recorded events, after all the entries of a row entries are filled with the recorded events, subsequent recorded events are arranged to fill in the entries of the next row, so on and so forth. Entries of rows not filled by recorded events remain to contain the 0. As the plurality of rows is built, the 2D array of information corresponding to the recorded events is formed.

[0043] In the 2D array of this embodiment, the arrangement of the recorded events contains the temporal information. Each of the recorded events has the coordinate parameters indicating a location, specifically the pixel coordinates on the frame. Thus, the 2D array contains both spatial and temporal information.

[0044] FIG. 1 is a schematic drawing of a 2D array of this embodiment. In this embodiment, each element in the 2D array exclusively includes coordinates (x, y). For example, the element A(1) contains (x.sub.1, y.sub.1), and the element A(C) contains (x.sub.C, y.sub.C), resulting in a small and compact array size, which is CxD to store the events. The C indicates the number of columns, and the D indicates the depth of the array. The memory cells, i.e., elements, in the 2D array are now storing coordinates instead of timestamp of the incoming events, and the design is particularly advantageous for subsequent corner detection procedure. In another embodiment, the elements may be Address Event Representation (AER) events and contain other information from the recorded events of the event data form the DVS.

[0045] In this embodiment, after the 2D array is organized, the method includes transforming the 2D array into one OS matrix by populating one empty Image Matrices (IM) with the recorded events based on their coordinates and assigning order values.

[0046] In the transformation process from the structured 2D array to the OS, we first capture and organize recorded events into the 2D array, where each element represents a specific event and contains pixel coordinates (x, y). Instead of storing timestamps, we employ the order values, which are indices indicating the temporal order of the events. This eliminates the need for large timestamp storage, resulting in a more memory-efficient representation.

[0047] Next, the OS is constructed by populating an empty IM with the pixel coordinates of the recorded events and assigning order values to each element. The IM effectively organizes events into a spatial structure, restoring the spatial ordering that might be lost in the 2D array representation. The assignment of order values maintains the temporal ordering of events, creating a coherent and efficient representation for subsequent corner detection.

[0048] This approach not only reduces the memory footprint compared to traditional timestamp-based representations like Time Surfaces but also provides a structured OS matrix that is conductive to effective corner detection methods. The resulting OS matrix, organized both spatially and temporally, serves as a foundation for further image processing and analysis in various applications.

[0049] Referring to FIG. 2 for the following description. An exemplary and simplify 2D array 10 is presented. The 2D array 10 has 4 rows, and each row has 4 elements. In this embodiment, the elements in a row is arranged from left to right, and the rows are arranged from top to bottom. According to the order of these elements, the order values are assigned respectively from 1 to 16, and each order value is fill in the position in the IM according to each coordinate of the elements, so as to form the OS matrix 11.

[0050] Through this transformation, the spatial information is reconstructed in the OS, and the subsequent procedure of the corner detection can proceed. In other words, by employing the 2D array and the OS matrix, we minimize the event data from the DVS. This reduction in data enhances the efficiency of the subsequent corner detection process. Essentially, these steps optimize data storage without compromising spatial and temporal information essential for corner detection. As a result, the detector operates swiftly and efficiently.

[0051] After the transformation of the OS matrix, the method of this embodiment proceeds with applying a corner detector to the OS matrix. Due to the reduction in data, the efficiency of the performance of the corner detector is improved. Therefore, the method for adaptive corner detection of this embodiment can provide an effective solution for corner detection. This tailored approach not only enhances the efficiency of corner detection but is particularly well-suited for resource-constrained environments, such as IoT. Consequently, the detector operates swiftly and efficiently, making it an ideal solution for edge IoT applications.

[0052] In another embodiment of the present invention, a system for the adapting corner detection is provided. The system comprises a DVS and a processing device, which is electrically connected to the DVS.

[0053] The processing device of this embodiment acquires event data from the DVS and executes subsequent steps for corner detection. Specifically, the device captures and organizes multiple recorded events within the event data into a structured 2D array. Subsequently, it transforms this 2D array into an OS matrix by populating an IM with recorded events based on their coordinates, and assigns order values. Finally, the processing device applies a corner detector to the resulting OS matrix.

[0054] Similar to the methodology outlined in the previous embodiment, the adaptive corner detection system in this embodiment excels in efficiently executing corner detection with low system requirement. This characteristic makes the system a suitable choice for deployment in edge IoT applications.

[0055] Referring to FIG. 3 for the following description. The system 100 for adaptive corner detection has the DVS 110 and the processing device 120. The processing device 120 obtains the event data 101 from the DVS 110, and the processing device 120 organize the recorded events of the event data 101 into the 2D array 102, and the processing device 120 transforms the 2D array 102 into the OS matrix 103, and the processing device 120 applies the corner detector to the OS matrix and generate a detection score 104, and the processing device 120 can output an indicator 105 by comparing the detection score 104 with a predefined threshold. The indicator 105 can indicate whether a corner is detected in the event data 101 or not, and the indicator 105 can be transmit to a output device such as display, light signal, or proceed to a subsequent procedure for robotics or autonomous control of a vehicle.

[0056] In this embodiment, the processing device 120 can include an edge processing unit. The edge processing unit is optimized for power efficiency and can handle real-time processing. In some embodiments, the edge processing unit includes: ARM based microcontrollers (MCU) from ST Micro (STM32F412, STM32F746, STM32H743), NXP (LPC1751FBD80) and Microchip (PIC32C series). By leveraging adaptive corner detection procedures of this embodiment, our system excels in optimizing corner detection efficiency across variety of edge processing units. This makes it particularly well-suited for machine vision applications and other automated moving surveillance devices integrated into vehicles, showcasing its versatility and effectiveness in dynamic environments.

[0057] FIG. 4 shows a flowchart of the procedure of the method and system for adaptive corner detection of some embodiments of the present invention. In some embodiments, in the process of organizing recorded events into a 2D array (S12), the events are structured and arranged according to a global temporal order within the 2D array.

[0058] In these embodiments, global temporal ordering ensures that recorded events in the 2D array maintain a coherent temporal sequence, which is crucial in corner detection. The 2D array contains clear and precise chronological order of recorded events for corner detection.

[0059] Also, the global temporal ordering facilitates efficient processing of recorded events in the 2D array. This can lead to streamlined methods for corner detection, resulting in improved overall system efficiency.

[0060] In these embodiments, the global temporal ordering helps in accurately capturing and representing the temporal evolution of visual information. This precision is beneficial for corner detection, contributing to the ability to detect corners with high accuracy.

[0061] In accordance with various embodiments of the present invention, the recorded events are arranged in a plurality of rows in the 2D array. Each of the rows has a series of the recorded events arranged in global temporal ordering, and the numbers of the recorded events in the rows are the same.

[0062] The rows of these embodiments are arranged in global temporal ordering. To be specific, the timestamps of the elements (i.e., events) arranged in one of the rows are smaller than all the timestamps of the elements (i.e., events) arranged in the following rows, and the timestamps of the elements (i.e., events) in the second row are larger than all the timestamps of the elements (i.e., events) in the first row.

[0063] In these embodiments, the rows with consistent number of elements simplify the data processing pipeline. This uniformity allows for standardized procedures and computations when working with each row, streamlining the computational process steps of the system.

[0064] Also, in these embodiments, uniform row lengths create opportunities for parallelization in data processing. By having rows with the same number of elements, parallel processing techniques can be more easily applied with GPU and multi-core processors, potentially leading to significant speedups in the overall event data processing. This benefit is particularly relevant in applications where real-time or high-throughput processing is essential.

[0065] In an embodiment of the present invention, the maximum number of the recorded events in every row is 100. However, the invention is not limited to the number. In some other embodiments, the maximum number of the recorded events in every row can be 25, 50, 100, or any values from 25 to 100. By utilizing the 2D array having these configurations, the method or the system can change the frequency at which the system performs corner detection, by varying C. The 2D array is also customizable, by being able to shrink down as event rate of the scene decreases.

[0066] In this embodiment, the recorded events are captured and organized continuously, which is forming a one-dimensional sequence in multiple rows, and no repetition of recorded events are present in these rows. However, the present invention is not limited to this manner. Referring to FIG. 1 for the following description. In some other embodiments of the present invention, the recorded events are organized in batches. For example, the number of recorded events in every batch is C, and the number of the batches in this 2D array is D. In these embodiments, overlapping stride is applied on the rows of the recorded events in the 2D array. In other words, some recorded events at the end of one batch are included at the beginning of the following batch, allowing for a more continuous and overlapping analysis of the event history.

[0067] In these embodiments, applying overlapping stride on the rows of the recorded events within the 2D array introduces a crucial feature that optimizes the system's ability to capture dynamic and evolving patterns in the visual data. The use of overlapping stride enables the system or method to explore and analyze temporal relationships between recorded events more comprehensively. Instead of processing each row in isolation, the overlapping stride ensure that adjacent rows share a portion of their recorded events, allowing for a seamless integration of temporal information across the 2D array.

[0068] The main advantage of applying overlapping stride in the method or system of these embodiments lies in its enhancement of temporal context. By incorporating events from neighboring rows in the processing pipeline, the method or system gain a more nuanced understanding of the temporal evolution of visual information. This is particularly beneficial in scenarios where recorded events occur rapidly or in quick succession, ensuring that the system does not miss crucial temporal correlations. The overlapping stride effectively create a temporal buffer, enabling the system or method to capture and interpret recorded events within a broader time window.

[0069] Furthermore, in these embodiments, the use of overlapping stride aligns with the principle of adaptability in dynamic environments. It provides the system or method with the flexibility to adapt varying speed or rates of change in the visual input. Whether events are occurring rapidly or at a slower pace, the overlapping stride enables the system or method to maintain responsiveness and accuracy in capturing temporal dynamics. The feature enhances the system's robustness in real-world applications, making it well-suited for corner detection in dynamic and unpredictable environments.

[0070] FIG. 5 shows a flowchart of the procedure of the method and system for adaptive corner detection of some other embodiments of the present invention. In these embodiments, the method or system applies a Spatial-Temporal Correlation Filter (STCF) to the event data (S22) before organizes the recorded events into the 2D array.

[0071] In these embodiments, applying the STCF to recorded events before organizing them into a 2D array introduces a powerful feature that enhances the system's ability to discern meaningful patterns in the event data. This filtering mechanism operates by considering both the spatial and temporal dimensions of the recorded events, capturing the correlation between neighboring events in a coherent manner.

[0072] The method or system of these embodiments apply STCF after obtaining the event data from the DVS. The primary advantage of this STCF lies in its capability to prioritize relevant information, discarding noise or inconsequential events. By considering the correlation between events in both space and time, the filter refines the event data, emphasizing sequences of events that are likely to represent meaningful patterns or visual features. This not only enhances the precision of subsequent processing steps but also contributes to the overall efficiency of the system by focusing computational resources on pertinent information.

[0073] Moreover, the step S22 of applying the STCF aligns with the principles of neuromorphic vision, mimicking the brain's ability to recognize patterns in a holistic and context-aware manner. This feature ensures that the system or method of these embodiments not only captures individual events but also interprets their collective significance, facilitating more nuanced and contextually relevant analyses. In essence, the STCF acts as a sophisticated preprocessing step, imbuing the system or the method with the capability to discern meaningful patterns from the stream of recorded events, thereby enhancing the system's overall performance in corner detection or any other related event-based processing.

[0074] In some embodiments of the present invention, the method and the system processing the method include usage of row pointer and column pointer. Referring to FIG. 6 for the following description. A row pointer 12 and a column pointer 13 are included in the 2D array. The row pointer and the column pointer keep track of the insertion location in the 2D array.

[0075] In these embodiments, the row pointer 12 and the column pointer 13 are essential elements used to navigate and access specific elements within the 2D array where recorded events are organized. The rows of the 2D array are logical rows implemented by the column pointer that moves in a cyclical fashion across the physical rows in the memory.

[0076] The row pointer of these embodiments is a variable or indicator that keep track of the current row being processed or accessed within the 2D array. As the system iterates through the rows, the row pointer is updated to point to the next row of interest. This allows the system to sequentially process events along the temporal dimension.

[0077] On the other hand, the column pointer of these embodiments operates within a specific row. It indicates the current position or element within the row that the system or method is examining or processing. As recorded events are organized into columns, the column pointer assists in navigating through the spatial dimension of the 2D array.

[0078] Both pointers of these embodiments are crucial for maintaining the system's awareness of its position within the data structure, enabling it to efficiently process and analyze the recorded events. They play a key role in the sequential analysis of recorded events or when specific recorded events need to be accessed and manipulated within the 2D array.

[0079] In other words, the row pointer and column pointer are like the navigation tools that keep our system precisely tuned into the vast landscape of recorded events in the architecture of our 2D array. The row pointer takes charge of our temporal journey, diligently indicating the specific row under scrutiny as we traverse through the chronological sequence of events. It's the guide that ensures we're always on track, never losing our place in the dynamic flow of temporal data. Meanwhile, the column pointer is our spatial maestro, orchestrating the movement within a given row. It directs our attention to the exact recorded event within that row, allowing us to meticulously navigate the spatial dimension of our array. Together, these pointers provide the system or the method with the agility to explore both time and space, ensuring an efficient and coherent analysis of the recorded event data.

[0080] Referring to FIG. 6 for the following description. The number of the rows of the 2D array of these embodiments is (D+1), which means an additional row is included in this 2D array. The additional row is used to allow insertion of recorded events while subsequent procedures are performed. It allows the dynamic insertion of events while concurrently performing the step of applying the corner detector on the OS matrix.

[0081] The additional row of these embodiments is likely implemented as a buffer or a placeholder row where new events can be temporarily stored before being incorporated into the OS. This approach is particularly advantageous when dealing with asynchronous event data from a DVS where events occur at irregular intervals.

[0082] For instance, the step of applying the corner detector to the OS matrix in these embodiments includes employing a Harris detector. As new events are detected by the DVS, they can be inserted into the additional row without interrupting the ongoing Harris detector operation on the rest of the OS matrix. This ensures a continuous flow of data processing without the need to pause or reset the analysis when new events occur.

[0083] Also, the ability to insert recorded events into the separate additional row while concurrently performing Harris detector operation on the OS matrix helps minimize latency. The method and the system using the method can adapt real-time changes in the visual input, capturing and evaluating recorded events in near real-time without significant delays.

[0084] Furthermore, the additional row of these embodiments provides a mechanism for efficient memory management. New events can be selectively incorporated into the OS matrix when deemed necessary, preventing unnecessary expansion or reallocation of memory, which is crucial in resource-constrained environment.

[0085] Referring to FIG. 2 for the following description. In some embodiments, the OS matrix 11 is derived from all the rows of the recorded events in the 2D array 10, so there will be enough temporal and spatial information to make meaningful corner detection. Transforming the full 2D array into a signal OS matrix offers several advantages in data processing.

[0086] One key benefit of the transforming of these embodiments is the consolidation of spatial-temporal information into a unified structure, simplifying subsequent analyses. This transformation streamlines the computations, which can be applied more efficiently to a singular OS rather than navigating through the complexities of a 2D array. Furthermore, the transition to a single OS matrix enhances data locality, potentially optimizing cache utilization and reducing memory access times. This consolidation also aids in parallel processing, allowing for concurrent operations on a cohesive dataset. Overall, in these embodiments, the transformation to a single OS enhances computational efficiency, reduces memory overhead, and fosters a more cohesive and accessible representation of the underlying spatial-temporal information.

[0087] In some embodiments, the transformation procedure may be represented by the following execution steps: [0088] Require: 2D Array, rowPtr (row pointer), OS [0089] SET T, OS to 1, 0 [0090] // rowPtr (row pointer): Variable indicating the current row being processed in the 2D array. [0091] For r from ((rowPtr+2) mod (D+1)) to rowPtr do [0092] For c from 0 to C do [0093] x, yarray[r,c] [0094] OS[x,y]T [0095] TT+1 [0096] End For [0097] End For

[0098] In these embodiments, the OS only require D write per event, which require much less writing times than luvHarris. While the method in accordance with these embodiments utilizes a logical pointer (i.e., row pointer or column pointer) that denotes the current entry in the process, the logical pointer's movement across the physical rows of the memory is facilitated by a cyclic operation determined by the modulus (mod) operation in the steps above, ensuring that the logical pointer cycles through the rows of the memory in a cyclical manner.

[0099] In another embodiment, the OS matrix is derived from some of the adjacent rows of the recorded events in the 2D array. For example, an OS matrix derived from three adjacent rows in the 2D array, and a plurality of the OS matrices is transformed from one 2D array.

[0100] In this embodiment, the method or system including this transformation is optimized for DVS that has higher sensitivity or sensing rate.

[0101] In still another embodiment, the OS matrix is derived from one of the rows of the recorded events in the 2D array. Therefore, a plurality of the OS matrices are transformed from one 2D array, and every OS matrix is corresponded to latest row of the recorded events.

[0102] In this embodiment, the method or system including this transformation is optimized for DVS that has high throughput in every second, and the adaptive corner detection is provided with high sensitivity.

[0103] Referring to FIG. 7 for the following description. In another embodiment, the method includes extracting a patch of elements in the Ordered Surface matrix S34; employing a Harris detector to the patch to generate a Harris score S35; and comparing the Harris score generated by the Harris detector with a predefined threshold S36.

[0104] In this embodiment, a patch surrounding the event of interest is extracted from the OS for the Harris detector or Harris calculation, so as to take advantage of the sparsity of events. The steps S34, S35, S36 optimized the process for efficient feature detection. The steps define a systematic approach to selecting a region or patch from the OS centered around a particular event, allowing the Harris detector to focus on specific spatial locations of interest within the OS.

[0105] One notable feature of this patch extraction process is its parameter tunability, enabling the algorithm to dynamically adjust the size and configuration of the patch. This flexibility ensures that the Harris detector can effectively capture and analyze features at different scales, contributing to the algorithm's robustness in identifying corners and relevant patterns in the visual data.

[0106] Additionally, the step S34 often incorporates considerations for neighboring events and their contributions to the patch. This ensures that the Harris detector leverages spatial-temporal correlations present in the OS matrix, enhancing its ability to discern meaningful features. The adaptive and parameterizable nature of the patch extraction process plays a pivotal role in the overall success of the Harris detector, making it well-suited for diverse applications in data processing and computer vision.

[0107] In various embodiments, the patch has M rows and N columns, and the M ranges from 5 to 11, and the N ranges from 5 to 11. Therefore, the method or system of the embodiments can capture the salient information stored within the OS while consuming minimal memory when using the Harris detector. Moreover, the patch sizes play a crucial role in corner detection, such as using the Harris detector, particularly for images with resolutions such as 240180 or 360260. In certain embodiments, it is beneficial to choose alternate patch sizes based on varied resolution of the event camera. In other words, the method and system in these embodiments offer flexibility in choosing the patch size. This adaptability allows the method and system to be employed with various types of event cameras, ensuring effective corner detection across different scenarios.

[0108] In certain embodiments, the values of M and N may be the same, and the method utilizes square patches. Therefore, the output of the Harris detector can be determined by referring to the existing dataset, so as to improve the efficiency. For example, the M and N can both be 5, 7, 9, 11, or any proper odd number, depending on the resolution of the event camera. As such, the methods in accordance with the embodiments of the present invention are flexible and applicable in systems with different event camera resolution.

[0109] In some embodiments, a sort normalization may be applied to the OS matrix before the corner detection is performed. Referring to FIG. 8 for the following description. The method comprises performing sort normalization to the patch of elements S45 before employing the Harris detector.

[0110] The sort normalization of these embodiments modified the range of value of the temporal information in the OS matrix. To be specific, the step bypasses approaches involving costly division operations. In this procedure S45, the non-zero values within the patch are collected and arranged in descending order within a queue. Subsequently, these values undergo sort normalization, where the normalized pixel value is computed as 255 minus the index. Here, the index denotes the pixel's position within the sorted queue. This approach not only optimizes computational efficiency but also aligns with theoretical concepts like rank order coding, reflecting a sophisticated strategy reminiscent of visual cortex encoding. The result is a streamlined and adaptable normalization method that significantly enhances the system's or method's overall performance in visual data processing applications.

[0111] In some embodiments, the local patch construction and Harris calculation may be represented by the following execution steps below: [0112] Require: x, y, OS, k (Patch Half Size) [0113] For r from k to k do [0114] For c from k to k do [0115] px, py+c+k, r+k [0116] Patch [px, py]OS [x+c, y+r] [0117] End For [0118] End For [0119] Sort SortQueue in descending order [0120] SortValue255 [0121] For value in SortQueue do [0122] Patch ValueSortValue [0123] SortValueSortValue1 [0124] End For [0125] ScoreHarrisDetector (Patch) [0126] If ScoreThreshold then [0127] CornerFlagTrue [0128] Else [0129] CornerFlagFalse [0130] End If.

[0131] The method and the system of the embodiments of the present invention provide an event-based corner detection for edge device, which uses minimized memory. Despite the smaller memory footprint and energy usage, the method and system achieve accuracy comparable/better than luvHarris.

[0132] In various embodiments, in-memory computing (IMC) techniques are utilized to accelerate processing, and the adaptive corner detection may be executed by a machine vision processor implemented using specially configured computing devices, computer processors, or electronic circuitries including but not limited to application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), microcontrollers, and other programmable logic devices configured or programmed according to the teachings of the present disclosure. Machine instructions running in the computing devices, computer processors, or programmable logic devices can readily be prepared by practitioners skilled in the software or electronic art based on the teachings of the present disclosure.

[0133] All or portions of the methods in accordance to the embodiments may be executed in one or more computing devices including server computers, personal computers, laptop computers, mobile computing devices such as smartphones and tablet computers.

[0134] The embodiments may include computer storage media, transient and non-transient memory devices having machine instructions stored therein, which can be used to program or configure the computing devices, computer processors, or electronic circuitries to perform any of the processes of the present invention. The storage media, transient and non-transient memory devices can include, but are not limited to, floppy disks, optical discs, Blu-ray Disc, DVD, CD-ROMs, and magneto-optical disks, ROMs, RAMs, flash memory devices, or any type of media or devices suitable for storing instructions, codes, and/or data.

[0135] Each of the functional units and modules in accordance with various embodiments also may be implemented in distributed computing environments and/or Cloud computing environments, wherein the whole or portions of machine instructions are executed in distributed fashion by one or more processing devices interconnected by a communication network, such as an intranet, Wide Area Network (WAN), Local Area Network (LAN), the Internet, and other forms of data transmission medium.

[0136] The foregoing description of the present invention has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations will be apparent to the practitioner skilled in the art.

[0137] The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to understand the invention for various embodiments and with various modifications that are suited to the particular use contemplated.