CAMERA SYSTEM INCLUDING A PROXIMITY SENSOR AND RELATED METHODS
20220030156 ยท 2022-01-27
Inventors
Cpc classification
H04N23/66
ELECTRICITY
H04N23/11
ELECTRICITY
H04N23/10
ELECTRICITY
H04N25/702
ELECTRICITY
H04N25/00
ELECTRICITY
H04N23/74
ELECTRICITY
International classification
Abstract
A camera for capturing an image comprising: an image sensor configured to generate image sensor data in response to received light; a processing resource configured to process the image sensor data to obtain image data and communication data, wherein obtaining the communication data comprises performing a demodulation process in respect of at least part of the image sensor data, wherein the processing resource is further configured to transmit the communication data and the image data to at least one further processing resource.
Claims
1. A proximity sensor configured to generate proximity data representative of a nearby object using optical signals, the proximity sensor comprising: an emitter configured to produce an outgoing optical signal comprising at least one of a proximity signal or a communication data signal; a receiver configured to receive an incoming optical signal comprising at least one of an optical proximity signal or an optical wireless communication modulation signal modulated in accordance with an optical wireless communication scheme, wherein the optical wireless communication modulation signal is transmitted by a remote device; and a processing resource configured to perform at least one of a) or b) a) a demodulation process in respect of at least part of the incoming optical signal to obtain incoming optical wireless communication data b) a modulation process to encode outgoing optical wireless communication data on at least part of the outgoing optical signal.
2. The proximity sensor of claim 1, wherein the processing resource is further configured to transmit the incoming optical wireless communication data to at least one further processing resource and/or to receive the outgoing optical wireless communication data from at least one further processing resource.
3. The proximity sensor of claim 1, wherein the emitter, receiver, and processing resource are integrated on a single chip.
4. The proximity sensor of claim 1, further comprising: driving circuitry configured to provide a modulation signal to the emitter, wherein the modulation signal is based on the outgoing optical wireless communication data received from one further processor.
5. The proximity sensor of claim 4, wherein the driving circuitry comprises: a digital to analog convertor configured to provide a modulation signal at a frequency characteristic of an optical light communication signal.
6. The proximity sensor of claim 1, wherein the receiver comprises: a receiver sensor configured to generate a communication signal in response to received light, and conditioning circuitry configured to collect the communication signal and condition the communication signal into a form suitable for the processing resource.
7. The proximity sensor of claim 6, wherein the receiver sensor comprises: a photodiode configured to receive the proximity signal, and a further photodiode configured to receive the optical wireless communication modulation signal.
8. The proximity sensor of claim 1, further comprising: an enclosure, wherein the optical proximity signal is sensed through a first enclosure entrance from a first direction, and wherein the optical wireless communication modulation signal is sensed through a second enclosure entrance from a second direction that is different from the first direction.
9. The proximity sensor of claim 1, further comprising: an enclosure, wherein the optical proximity signal is emitted through a first enclosure exit in a first direction, and wherein the optical wireless communication modulation signal is emitted through a second enclosure exit in a second direction that is different from the first direction.
10. The proximity sensor of claim 1, wherein the proximity sensor is further configured to perform ambient light sensing.
11. The proximity sensor of claim 1, wherein the processing resource is configured to transmit and/or receive optical wireless communication data and proximity data in series over a first interface.
12. The proximity sensor of claim 1, wherein the processing resource is configured to transmit and/or receive optical wireless communication data and proximity data in parallel over a first and second interface.
13. The proximity sensor of claim 11, wherein the first interface comprises a high speed interface.
14. The proximity sensor of claim 11, wherein the first interface comprises a camera serial interface (CSI).
15. A method of receiving communication data comprising: receiving a light signal by using the proximity sensor of claim 1, where the light signal comprises an optical wireless communication signal modulated in accordance with an optical communication protocol; processing and demodulating at least part of the light signal that was received to obtain the incoming optical wireless communication data; and transmitting the incoming optical wireless communication data to at least one further processing resource.
16. A method of transmitting or receiving communication data using a proximity sensor configured to generate proximity data representative of a nearby object using optical signals, wherein the proximity sensor comprises: an emitter configured to produce an outgoing optical signal comprising at least one of a proximity signal or a communication data signal; a receiver configured to receive an incoming optical signal comprising at least one of an optical proximity signal or an optical wireless communication modulation signal modulated in accordance with an optical wireless communication scheme, wherein the optical wireless communication modulation signal is transmitted by a remote device; and a processing resource configured to perform at least one of a) or b) a) a demodulation process in respect of at least part of the incoming optical signal to obtain incoming optical wireless communication data b) a modulation process to encode outgoing optical wireless communication data on at least part of the outgoing optical signal; the method comprising: using an emitter of the proximity sensor to emit light comprising the outgoing optical wireless communication data; using a receiver of the proximity sensor to receive light comprising the incoming optical wireless communication data; generating a signal by the receiver in response to the light received; and processing the signal to extract optical wireless communication data.
17. A camera system comprising a camera for capturing an image, the camera comprising: an image sensor configured to generate image sensor data in response to received light; a camera processing resource configured to process the image sensor data to obtain image data and incoming optical wireless communication data, wherein the incoming optical wireless communication data are modulated in accordance with an optical wireless communication modulation scheme, wherein obtaining the incoming optical wireless communication data comprises performing a demodulation process in respect of at least part of the image sensor data, and wherein the camera processing resource is further configured to transmit the incoming optical wireless communication data and the image data to at least one further processing resource; and a proximity sensor configured to generate data representative of a nearby object using optical signals, wherein the proximity sensor comprises: an emitter configured to produce an outgoing optical signal; a receiver configured to receive an incoming optical signal comprising an incoming optical proximity signal and an incoming optical wireless communication signal modulated in accordance with an optical wireless communication scheme; and a proximity processing resource configured to perform at least one of a) and b): a) a modulation process to encode outgoing optical wireless communication data on at least part of the outgoing optical signal, b) a demodulation process in respect of at least part of the incoming optical signal to obtain incoming communication data.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0069] Various aspects of the invention will now be described by way of example only, and with reference to the accompanying drawings, of which:
[0070]
[0071]
[0072]
[0073]
[0074]
[0075]
[0076]
[0077]
[0078]
[0079]
[0080]
DETAILED DESCRIPTION OF THE DRAWINGS
[0081]
[0082] The image sensor 10 is configured to receive and process light containing image information and communication information. The image sensor 10 is configured to operate in two parallel channels: an image channel and a communication channel. The image channel is concerned with sensing image signals and outputting image data. The communication channel is concerned with sensing communication signals and outputting communication data. The input to both channels is the image sensor array 12. The image sensor array 12 is a grid of sensor elements or pixels for receiving and sensing incident light. The sensor elements may be either image sensors elements for capturing image information or communication sensor elements for sensing communication signals. The sensor elements comprise photodiodes or other optical sensing elements. The image sensor elements receive the input light to the image channel and the communication sensor elements receive the input light to the communication channel. Different suitable arrangement patterns of sensor elements are described with reference to
[0083] Light reflected or transmitted from an object is suitable for capturing an image of the object, as the intensity of the light received contains image information about the object and this information may be used to construct an image of the object. Colour information is obtained by having individual image sensor elements in the image sensor array 12 that respond to specific frequencies of light. Each individual image sensor element converts light into an electrical signal the magnitude of which is proportional to the intensity of the light sensed by the sensor element. Therefore, at each image sensor element location, the intensity of light of a specific frequency is measured. By collecting intensities from all image sensor elements, raw image information including information about the intensity of different colours over the image sensor array 12 is gathered. The image sensor array 12 can contain active pixels, for example it could be a complementary metal-oxide-semiconductor (CMOS) sensor, where each sensor element contains a photodetector, for example a photodiode, and an active amplifier. Another example of a suitable image sensor array 12 is a hybrid charge-coupled-device (CCD) and CMOS sensor. A hybrid image sensor array has CCD image sensor elements for image capture and CMOS sensor elements for communication signal capture.
[0084] Together with collecting image information, the intended use of the image sensor 10 is to receive and process light that also includes one or more components corresponding to a wireless optical communication signal sent from a wireless optical communication transmitter. The image sensor array 12 has dedicated communication sensor elements for capturing communication information. The communication sensor elements receive light for input into the communication channel.
[0085] The analogue front end 14 is coupled to the image sensor array 12 and has two parts, each corresponding to one of the two channels. For the image channel, the analogue front end 14 has dedicated circuitry for collecting and conditioning the sensed image signals and for the communication channel, the analogue front end 14 has dedicated circuitry for collecting and conditioning the sensed communication signals. The analogue front end 14 contains analogue to digital convertors for converting the sensed electronic signals into digital signals for the processing resource 16. Due to the nature of the incoming optical and hence electronic signals, the properties of the analogue to digital convertors may differ between the two parts.
[0086] An analogue to digital convertor converts an input analogue signal into an output digital signal. Two properties of analogue to digital convertors relevant to the design and manufacture of the different parts of the analogue front end 14 are: resolution and size. The resolution of the convertor is the number of discrete levels the convertor can produce over the range of the input analogue signal. For image electronic signals, the analogue to digital convertors are chosen such that the digital resolution of the image meets a pre-determined image quality. On the other hand, the analogue to digital convertors for communication should be suitable to respond and sample a higher frequency of electronic signals stemming from received wireless optical communication signals. Values of the frequency of electronic signals may be 100 MHz and above. For example, to deliver bandwidth of 40 MBps, a digital to analogue convertor is usually configured to sample at at least 160 MHz.
[0087] The size of an analogue to digital convertor is a further physical consideration when manufacturing circuits. Of particular importance here is the physical area spanned by each analogue to digital convertor. For capturing an image, each image sensor element of the image sensor array 12 has a corresponding analogue to digital convertor. Alternatively, the image sensor elements of the image sensor array 12 are grouped into subgroups each having a dedicated analogue to digital convertor. In either case, thousands of analogue to digital convertors need to be implemented to capture an image and thus the physical limits in size of the chip and size of the circuit board limit the choice of suitable analogue to digital convertors to ones that have a small area. On the other hand, the communication analogue to digital convertors are fewer in number and can therefore have a larger area.
[0088] The processing resource 16 has processes digital signals relating to a captured image and processes digital signals relating to a wireless optical communication signal. The processing resource 16 may be two separate processors: a first processor dedicated to processing image signals and a second processor dedicated to processing communication signals.
[0089] In use, light containing both information about the image being captured and an incoming wireless optical communication signal is incident on the image sensor array 12. As described above, the image sensor 10 operates in two channels: the image channel and the communication channel. These channels operate simultaneously and in parallel. Firstly, the image channel has input sensed by the image sensor elements of the image sensor array 12 that convert light into electronic signals such that the electronic signals contain raw image information. The conversion is carried out by a photodiode. The electronic signals are then provided to the analogue front end 14. The circuitry of the analogue front end 14 collects the electronic image signals and conditions these signals into a suitable form for the image processor of the processing resource 16. Conditioning includes one or more filtering steps applied to the electronic signal. These steps may be a low pass filter and a high pass filter. A value for a low pass filter according to an embodiment is 25 MHz, e.g. the frequency value below which signals are passed may be 25 MHz. A value for a high pass filter according to an embodiment is 300 KHz, e.g. the frequency value below which signals are passed may be 300 KHz. Conditioning includes converting the analogue electronic image signals into digital image signals using at least one dedicated analogue to digital convertor. The image processor of the processing resource 16 is then configured to process the digital image signals to produce image data suitable for constructing a digital representation of the captured image. The image data output of the processing resource 16 is then provided to the data interface 18 to be sent to a further processor.
[0090] Turning to the communication channel, the image sensor 10 captures incoming communication signals and communication data is produced. The communication sensor elements of the image sensor array 12 convert incident light into electronic signals such that the electronic signals contain raw communication information. The electronic signals are provided to the dedicated communication circuitry of the analogue front end 14 which collects and conditions the electronic communication signals into a suitable form for the processing resource 16. Conditioning can include the optional steps of amplifying the weak electrical signal induced in the sensor element and equalisation of received signals. Conditioning includes converting the analogue electronic communication signals into digital communication signals using at least one dedicated analogue to digital convertor. The digital signal is then passed to the communication part of the processing resource 16 which demodulates the digital signal to extract communication data that are encoded in the wireless optical communication signal. Any suitable modulation schemes may be used, for example non on-off keying modulation schemes are used in some embodiments, and the demodulation is a demodulation from the non on-off keying modulation scheme. Other non-complex modulation schemes, for example modulation schemes that do not include or that are not based on real and imaginary parts, may be implemented in some other embodiments.
[0091] The communication data output from the processing resource 16 is then provided to the data interface 18. The data interface 18 sends communication data output from the communication channel, together with the image data from the image channel, to the host processor 20. The interface and operation of the interface is described in more detail with reference to
[0092]
[0093] The image sensor array 12 is represented by a rectangular grid with a length of 12 sensor elements and a height of 8 sensor elements. The image sensor array 12 is made up of two sensor element arrays: a colour sensor array 22 and a communication sensor array 24. The colour sensor array 22 has four component image sensor elements: a blue sensor element 26, a first green sensor element 28, a second green sensor element 30 and a red sensor element 32. Together these four component image sensor elements, arranged is a square of two sensor elements by two sensor elements define a unit cell 34, where the unit cell 34 is the smallest, repeating pattern of the colour sensor array 22. Starting from the left hand side, the first row of the unit cell 34 is the blue sensor element 26 followed by the first green sensor element 28. Again, starting from the left hand side, the second row of unit cell 34, below the first row, has the second green sensor element 30 followed by the red sensor element 32. The unit cell 34 has the first green sensor element 28 and the second green sensor element 30 in opposite diagonal positions from each other. The human eye is more sensitive to green colour than red or blue. By providing two green sensor elements for every one red and one blue sensor element, the colour sensor array 22 provides an enhanced perceptual signal to noise ratio for a human eye. The colour sensor array 22 can be drawn by repeating the unit cell 34. The unit cell 34 shown in
[0094] The pattern of the communication sensor array 24 shown in image sensor array 12 of
[0095] Positioning the communication sensor array 24 at the periphery of the image sensor array 12 allows for communication signals to be received while keeping the same perceptual signal to noise ratio for imaging due to maintaining the number of green sensor elements in the colour sensor array 22. However, compared to an image sensor array 12 with no communication sensor elements, the overall number of image sensor elements is reduced leading to a reduction in number of sensor elements available for an image. Advantageously, no optical artefacts are introduced by this arrangement.
[0096]
[0097] The pattern 38 of
[0098] The image sensor array represented by pattern 38 has an equal number of green, red, blue and communication sensor elements. This sensor element arrangement is more suited to applications where pixel resolution is more important than human perceptual signal to noise ratio, for example in military imaging applications.
[0099]
[0100] Arbitration is a process of allocating access to shared resources. There is no need for arbitration in parallel mode. Instead, sensor configuration information is shared between the host processor 20 and the image sensor 10 over a separate control bus during a configuration phase. The implementation of the separate control bus is described with reference to
[0101]
[0102]
[0103] In operation, the host processor 20 provides a communication data signal to the flash processing resource 74 which encodes the communication data signal onto a drive signal. The drive signal is provided to the flash analogue front end 76 to produce a drive current for driving the LEDs of the flash unit 72. A suitable LED has an operating range in the region 0 to 20 MHz. The flash unit 72, once driven, produces an outgoing wireless communication signal. The flash unit 72 can operate either in an image mode or in a communication mode.
[0104] A standard flash unit 72 may contain more than one LED. An alternative to the above arrangement is to provide more than one different driving circuit in the flash analogue front end 76 for the more than one LEDs of the flash unit 72. In this arrangement, the flash unit 72 has a first group of image LEDs and a second group of communication LEDs and the image LEDs are driven by image part of the processing resource 16 and the flash LEDs are driven by the flash processing resource 74. This mode of operation, allows the flash unit 72 to produce a flash and a wireless optical communication signal simultaneously.
[0105]
[0106] The proximity sensor 78 is configured to generate proximity data representative of a nearby object or to make an ambient light measurement. Proximity sensors are commonly found placed next to camera sensors on mobile phone devices and tablets. Proximity data may be used for a variety of different purposes. The first data interface 94 is a dedicated high throughput interface, for example a camera serial interface (CSI-2/3). The second data interface 96 is a low throughput interface, for example a configuration interface that is used for both control and read-out. The second data interface 96 may be, for example, an inter-integrated circuit (I2C) or serial peripheral interface (SPI).
[0107] The proximity sensor operates to perform proximity or ambient light sensing together with wireless optical communication operation. The host processor 80 sends a signal to the proximity signal module 90 via the second data interface 96. The proximity signal module 90 drives the emitter 84 to generate an outgoing optical signal, for example an infrared signal. The outgoing optical signal strikes a nearby object and is then sensed by and the receiver 86 which senses the incoming optical signal. The incoming optical signal is then processed by the proximity signal module 90 to obtain proximity data. The proximity data is then sent to the host processor 20 via the second data interface 96. In a similar fashion, the proximity sensor 78 is able to perform an ambient light measurement by sensing an incoming optical signal at the receiver 86 which converts the incoming ambient light into an electrical signal. The sensed electrical signal is then processed by the ambient light signal module 92 to obtain ambient light data which may be sent to the host processor 80 via the second data interface 96.
[0108] In addition to the above functionality the proximity sensor 78 is configured to emit wireless optical light communication signals. Communication data to be transmitted is sent from the host processor 80 to the communication module 88 via the first data interface 94. The communication module 88 receives digital data signal via the first data interface 94 and has a processor to modulate this communication data onto a drive current and driving circuitry to provide the drive current to the emitter 84. The emitter 84 then produces an outgoing modulated wireless optical communication signal that carries the communication data. The signal is then received by the access point 82.
[0109] Optionally, the communication module 88 also has a receiver processor and an analogue front end for processing received light. In operation, light containing a wireless optical communication signal is sensed by the receiver 86 and the receiver 86 converts the incident light into an electric signal. The analogue front end of the communication module 88 collects and conditions this signal, with the optional step of amplifying it. The analogue front end also include analogue to digital convertors to produce a digital signal from the electric signal. The analogue to digital convertors are capable of sampling at the frequency of the wireless optical communication signals. These digital signals are then processed by the receiver processor. Processing the digital signals involves demodulating them to extract encoded communication data. The extracted communication data is then sent to the host processor 80 via the first data interface 94.
[0110] Suitable modes of operation of the first data interface 94 and the second data interface 96 are described in relation to
[0111]
[0112]
[0113] As described above, the proximity sensor can produce an outgoing optical proximity signal and an outgoing wireless optical communication signal. The proximity sensor can receive an incoming optical proximity signal and an incoming wireless optical communication signal. The holes of the enclosure 120 are positioned relative to emitter and receiver of the proximity sensor as follows. The first transmission hole 126 is positioned such that the outgoing optical proximity signal can be emitted through it. The first receiving hole 128 is positioned such that the incoming optical proximity signal can be received through it.
[0114] Concerning the optical communication signals
[0115] Alternatively, the proximity sensor may be configured to emit the outgoing proximity signal through the first transmission hole 126 and the outgoing wireless optical communication signal through the second transmission hole 130. In this case, the proximity sensor is configured such that a first LED produces the outgoing proximity signal and a second LED produces the outgoing wireless optical communication signal. Likewise, the proximity sensor may be configured to receive the incoming proximity signal through the first receiving hole 128 and the incoming wireless optical communication signal through the second receiving hole 132. In this case, the proximity sensor is configured such that a first photodiode receives the incoming proximity signal and a second photodiode receives the incoming wireless optical communication signal.
[0116]
[0117] A configuration bus 146 connects the image sensor 138, the proximity sensor 140 and the host processor 136. The configuration bus 146 provides an interface for configuration data to be transmitted from the host processor 136 to the module 134. The configuration bus is a low-bandwidth communication bus. The configuration bus 146 also provides a data interface between the image sensor 138 and the proximity sensor 140 to allow for communication between the two sensors to adjust various image capture parameters, for example determining an optimal zoom configuration. The configuration bus may be an inter-integrated circuit (I2C) or a serial peripheral interface (SPI).
[0118] A first data bus 148 connects the image sensor 138 and the host processor 136. The first data bus 148 carries communication data and image data transmitted from the image sensor 138. The first data bus 148 is a high throughput interface, for example a camera serial interface (CSI-2/3). Compared to a known camera sensor and proximity sensor module no physical modification to the first data bus 148 is required. However, unlike a known camera sensor and proximity sensor module, the first data bus 148 carries both types of data. The configuration bus 146 can send configuration data including permission data between the image sensor 138 and the host processor 136.
[0119] A second data bus 150 connects the proximity sensor 140 and the host processor 136. In addition to modifications to sensor hardware, the second data bus 150 is added to a known camera sensor and proximity sensor module to carry communication data from the host processor 136 to the proximity sensor 140 to be transmitted as part of a wireless optical communication signal. The second data bus 150 is a high throughput interface, for example a camera serial interface (CSI-2/3). Using a camera serial interface (CSI-2/3) provides uniformity between first data bus 148 and the second data bus 150.
[0120] A skilled person will appreciate that variations of the disclosed arrangements are possible without departing from the invention. Accordingly, the above description of the specific embodiment is made by way of example only and not for the purposes of limitation. It will be clear to the skilled person that in alternative embodiments straightforward modifications may be made to each of the features described.