Estimation of harmonic frequencies for hearing implant sound coding using active contour models

Abstract

A signal processing arrangement generates electrical stimulation signals to electrode contacts in an implanted cochlear implant array. An input sound signal is processed to generate band pass signals that each represent an associated band of audio frequencies. A spectrogram representative of frequency spectrum present in the input sound signal is generated. A characteristic envelope signal is produced for each band pass signal based on its amplitude. An active contour model is applied to estimate dominant frequencies present in the spectrogram, and the estimate is used to generate stimulation timing signals for the input sound signal. The electrode stimulation signals are produced for each electrode contact based on the envelope signals and the stimulation timing signals.

Claims

1. A method for generating electrode stimulation signals for electrode contacts in an implanted cochlear implant electrode array, the method comprising: processing an input sound signal to generate a plurality of band pass signals, each band pass signal representing an associated band of audio frequencies and having a characteristic amplitude; generating a spectrogram representative of frequency spectrum present in the input sound signal; extracting a characteristic envelope signal for each band pass signal based on its amplitude; applying an active contour model to estimate dominant frequencies present in the spectrogram; using the estimate of dominant frequencies to generate stimulation timing signals for the input sound signal; and producing the electrode stimulation signals for each electrode contact based on the envelope signals and the stimulation timing signals stimulating the auditory nerve tissue with the electrode contacts using the electrode stimulation signals.

2. The method according to claim 1, wherein the spectrogram is generated using a short time Fourier transformation (STFT).

3. The method according to claim 1, wherein the electrode stimulation signals include channel-specific sampling sequences (CSSS).

4. The method according to claim 1, wherein using the estimate of dominant frequencies includes smoothing the spectrogram.

5. The method according to claim 1, wherein the estimate of dominant frequencies includes a determination of one or more harmonic frequencies present in the spectrogram.

6. The method according to claim 1, wherein the method is iteratively repeated over a period of time intervals.

7. A system for generating electrode stimulation signals of a cochlear implant to electrode contacts in an implantable cochlear implant electrode array, the system comprising: an implantable electrode array having a plurality of electrode contacts; a preprocessor filter bank configured to process an input sound signal to generate a plurality of band pass signals, each band pass signal representing an associated band of audio frequencies and having a characteristic amplitude; a spectrogram module configured to generate a spectrogram representative of frequency spectrum present in the input sound signal; an envelope detector configured to extract a characteristic envelope signal for each band pass signal based on its amplitude; an active contour model module configured to: i. apply an active contour model to the spectrogram to estimate dominant frequencies present in the spectrogram, and ii. using the estimate of dominant frequencies to generate stimulation timing signals for the input sound signal; and a pulse generator configured to produce and apply the electrode stimulation signals to each electrode contact so as to stimulate the auditory nerve tissue, the electrode stimulation signals based on the envelope signals and the stimulation timing signals.

8. The system according to claim 7, wherein the spectrogram module is configured to use a short time Fourier transformation (STFT) to generate the spectrogram.

9. The system according to claim 7, wherein the active contour model module is configured to use Channel-Specific Sampling Sequences (CSSS) to generate the stimulation timing signals.

10. The system according to claim 7, wherein the active contour model module is configured to include in the estimate of dominant frequencies a smoothing of the spectrogram.

11. The system according to claim 7, wherein the active contour model module is configured to include in the estimate of dominant frequencies a determination of one or more harmonic frequencies present in the spectrogram.

12. The system according to claim 7, wherein the system is configured to iteratively repeat the processing of the input sound signal over a period of time intervals.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The file of this patent contains at least one photograph. Copies of this patent with photograph will be provided by the Office upon request and payment of the necessary fee.

(2) FIG. 1 shows a section view of a human ear with a typical cochlear implant system designed to deliver electrical stimulation to the inner ear.

(3) FIG. 2 shows a sample spectrogram for a sample of clean speech including estimated instantaneous frequencies for Channels 1 and 3 based on zero-crossings.

(4) FIG. 3 shows various functional blocks in a signal processing arrangement for a hearing implant according to an embodiment of the present invention.

(5) FIG. 4 shows various logical steps in developing electrode stimulation signals according to an embodiment of the present invention.

(6) FIG. 5 shows an example of a short time period of an audio speech signal from a microphone.

(7) FIG. 6 shows an acoustic microphone signal decomposed by band-pass filtering by a bank of filters into a set of band pass signals.

(8) FIG. 7 shows a spectrogram of a clean speech input sound signal with estimated harmonics.

(9) FIG. 8A-8B shows the principle of an active contour model.

(10) FIG. 9 shows another spectrogram of a clean speech input sound signal with estimated harmonics.

(11) FIG. 10 shows a spectrogram of a clean speech input sound signal with modified estimated harmonics according to an embodiment of the present invention.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

(12) The bandwidths of the band pass filters in a typical cochlear implant signal processor are quite large compared to the auditory filters in normal hearing, and there is likely to be more than one frequency harmonic in each electrode channel. This can cause a poor estimation of the instantaneous frequency of the dominant harmonic in a given channel.

(13) Aubert, Gilles, et al. Image segmentation using active contours: Calculus of variations or shape gradients?. SIAM Journal on Applied Mathematics 63.6 (2003): 2128-2154 (incorporated herein by reference in its entirety) describes active contour models to segment parts of image frames out of a video.

(14) Fruehauf, Florian, et al. Experiments and algorithms to detect snow avalanche victims using airborne ground-penetrating radar. Geoscience and Remote Sensing, IEEE Transactions on 47.7 (2009): 2240-2251.(incorporated herein by reference in its entirety) describes using an active contour model to segment the snow layer out of radar data to automatically detect avalanche victims by flying with a helicopter over an avalanche and using a radar antenna mounted on the helicopter that receives the radar data. The amount of data is very large and the evaluation must be available in real time, but still the snow layer can be extracted out of the radar data.

(15) Embodiments of the present invention are based on applying an active contour model to a spectrogram of the input sound signal, and using that to estimate the course of dominant frequencies such as the dominant harmonics. This estimation is then independent of the cochlear implant filter bank. Such embodiments may provide improved speech intelligibility and perception of music and pitch in hearing implant systems.

(16) FIG. 3 shows various functional blocks in a signal processing arrangement for a hearing implant and FIG. 4 is a flow chart showing various logical steps in producing electrode stimulation signals to electrode contacts in an implanted cochlear implant array according to an embodiment of the present invention. A pseudo code example of such a method can be set forth as:

(17) TABLE-US-00001 Input Signal Preprocessing: BandPassFilter (input_sound, band_pass_signals) Spectrogram: Spectrogram (input_sound, spectrogram) Envelope Extraction: BandPassEnvelope (band_pass_signals, band_pass_envelopes) Active Contour Model: DominantFrequencies (spectrogram, dom_freqs) StimulationTiming (dom_freqs, stim_timing) Pulse Generation: PulseGenerate (band_pass_envelopes, stim_timing, out_pulses)
The details of such an arrangement are set forth in the following discussion.

(18) In the arrangement shown in FIG. 3, the initial input sound signal is produced by one or more sensing microphones, which may be omnidirectional and/or directional. Combined Preprocessor and Filter Bank 301 pre-processes this input sound signal u, step 401, with a bank of multiple parallel band pass filters, each of which is associated with a specific band of audio frequencies; for example, using a filter bank with 12 digital Butterworth band pass filters of 6th order, Infinite Impulse Response (IIR) type, so that the input sound signal is filtered into some K band pass signals, U.sub.1 to U.sub.K where each signal corresponds to the band of frequencies for one of the band pass filters. Each output of the sufficiently narrow CIS band pass filters for a voiced speech input signal may roughly be regarded as a sinusoid at the center frequency of the band pass filter which is modulated by the envelope signal. This is also due to the quality factor (Q3) of the filters. In case of a voiced speech segment, this envelope is approximately periodic, and the repetition rate is equal to the pitch frequency. Alternatively and without limitation, the Preprocessor Filter Bank 301 may be implemented based on use of a fast Fourier transform (FFT) or a short-time Fourier transform (STFT). Based on the tonotopic organization of the cochlea, each electrode contact in the scala tympani typically is associated with a specific band pass filter of the combined Preprocessor and Filter Bank 301. The combined Preprocessor and Filter Bank 301 also may perform other initial signal processing functions such as for example automatic gain control (AGC) and/or noise reduction and/or wind noise reduction and/or beamforming and other well-known signal enhancement functions. It is understood that instead of the combined Preprocessor and Filter Bank 301, the Preprocessor and Filter Bank may be separate. In this embodiment the Preprocessor may process input sound signal u and the preprocessed output signal is subsequently used for input and processing by Filter Bank and Spectrogram Module 302.

(19) FIG. 5 shows an example of a short time period of an input sound signal u from a sensing microphone, and FIG. 6 shows the microphone signal decomposed by band-pass filtering by a bank of filters. An example of pseudocode for an infinite impulse response (IIR) filter bank based on a direct form II transposed structure is given by Fontaine et al., Brian Hears: Online Auditory Processing Using Vectorization Over Channels, Frontiers in Neuroinformatics, 2011; incorporated herein by reference in its entirety.

(20) The band pass signals U.sub.1 to U.sub.K (which can also be thought of as electrode channels) are output to an Envelope Detector 303, which extracts characteristic envelope signals outputs X.sub.1, . . . , X.sub.K, step 403, that represent the channel-specific band pass envelopes. The envelope extraction can be represented by X.sub.k=LP(|U.sub.k|), where |.| denotes the absolute value and LP (.) is a low-pass filter; for example, using 12 rectifiers and 12 digital Butterworth low pass filters of 2nd order, IIR-type. Alternatively, the Envelope Detector 303 may extract the Hilbert envelope, if the band pass signals U.sub.1, . . . , U.sub.K are generated by orthogonal filters.

(21) A Spectrogram Module 302 generates a spectrogram S representative frequency spectrum present in the input sound signal u, step 402; for example by using a short time Fourier transformation (STFT). FIG. 7 shows a spectrogram S(f, t) for a clean speech input sound signal u, where the first and second dimension refers to the frequency f and time t, respectively, and where the estimated harmonic frequencies are shown by the black-white-black lines. The k.sup.th harmonic is given through a function

(22) $h_k:\{\begin{array}{ccc}T& .fwdarw.& F\\ t& .fwdarw.& h_k\left(t\right)\end{array},$
which assigns each time tT to a frequency h.sub.k(t)F. A timing signal Y.sub.k(t) can be obtained by

(23) $Y_k\left(t\right)=\{\begin{array}{cc}1& t=t_k\left[n+1\right]=t_k\left[n\right]+\frac{1}{h_k\left(t_k\left[n\right]\right)}\\ 0& \mathrm{otherwise}\end{array}.$
Then the time differences t.sub.k[n+1]t.sub.k[n] of the ones in Y.sub.k correlate with the estimated frequency of the k.sup.th harmonic.

(24) The spectrogram S then is the input signal for an Active Contour Model Module 304, which applies an active contour model to the spectrogram S, step 404. This may be generally as based on the use of active contour models as described in the prior art as to image processing, embodiments of the present invention represent the first use of such active contour model-based image processing techniques to the processing of input sound signals for a hearing implant. The Active Contour Model Module 304 then uses the estimate of the dominant frequencies present in the spectrogram S to generate stimulation timing signals, step 405.

(25) The extracted signal envelopes X.sub.1, . . . , X.sub.K from the Envelope Detector 303, and the stimulation timing signals Y.sub.1, . . . , Y.sub.K from the Active Contour Model Module 304 are input signals to a Pulse Generator 305 that produces the electrode stimulation signals Z for the electrode contacts in the implanted electrode array of the Implant 306, step 406. The Pulse Generator 305 applies a patient-specific mapping functionfor example, using instantaneous nonlinear compression of the envelope signal (map law)That is adapted to the needs of the individual cochlear implant user during fitting of the implant in order to achieve natural loudness growth. The Pulse Generator 305 may apply logarithmic function with a form-factor C as a loudness mapping function, which typically is identical across all the band pass analysis channels. In different systems, different specific loudness mapping functions other than a logarithmic function may be used, with just one identical function is applied to all channels or one individual function for each channel to produce the electrode stimulation signals. The electrode stimulation signals typically are a set of symmetrical biphasic current pulses.

(26) Returning to describe in greater detail the operation of the Active Contour Model Module 304, the spectrogram S can be treated as a continuous mapping from FTR.sub.+.sup.2.fwdarw.R.sub.+, where R.sub.+ denotes the positive real numbers. First, the Active Contour Model Module 304 smooths the spectrogram S for robustness reasons: .sub.1(f, t)=S(f, t)+.Math..sub.fS(f,t), where .sub.f is the Laplace operator corresponding to the frequency and >0. This smoothing corresponds to solving a one dimensional heat equation with initial condition S up to time . is a parameter and determines the smoothing. The larger is chosen, the stronger the smoothing will be.

(27) Then the Active Contour Model Module 304 can detect the first harmonic present in the spectrograms S, given p.sub.1,t:ff+.sub.1(f, t) be a potential for fixed t and >0. FIG. 8A shows a plot of p.sub.1,t where the peaks of at 200 and 400 Hz correspond to the first two harmonics, and the Active Contour Model Module 304 finds the first local minima of p.sub.1,t, which is indicated in the plot by the rolling ball. The first found local minimum is denoted by h.sub.1(t). Then the Active Contour Model Module 304 finds the second harmonic. Since the second harmonic has a higher frequency than h.sub.1(t), then:

(28) ${}_2\left(f,t\right)=\{\begin{array}{ccc}0& \mathrm{for}& f{h}_1\left(t\right)+\\ {}_1\left(f,t\right)& \mathrm{for}& f>h_1\left(t\right)+\end{array}.$
FIG. 8B shows the potential p.sub.2,t where .sub.1 is replaced by .sub.2. By choosing an appropriate , the first harmonic is neglected in .sub.2 and the second harmonic is given by the first local minimum h.sub.2(t) of p.sub.2,t. The Active Contour Model Module 304 then repeats the calculation for k=3,4, . . . , to estimate the higher harmonics, getting h.sub.k by replacing h.sub.1, p.sub.2,t, .sub.1 and .sub.2 through h.sub.k1, p.sub.k,t, .sub.k1 and .sub.k, respectively.

(29) The foregoing uses only the information for a single moment in time, so that a single false estimation can occur and the estimated harmonics then will not be smooth, and during speech pauses, the course of the estimation will become erratic. To avoid that, the calculations can be iteratively repeated over a period of time intervals. Consider the function:

(30) $H_k\left(h\right)={}_{t_1}^{t_2}{p}_{k,t}\left(h\left(t\right)\right)\mathrm{dt}+\frac{}{2}{}_{t_1}^{t_2}{h^{}\left(t\right)}^2\mathrm{dt},$
where h(t) denotes the derivative with respect to time. The k.sup.th harmonic can be chosen as the local minimizer h.sub.k of H.sub.k. The first term of H.sub.k forces h.sub.k to be in the first local minimum, while the second term makes h.sub.k smooth. The parameter controls the influence of the two terms. The calculation of the minimizer can be done, for example, by a steepest descent method where the corresponding Euler Lagrange equation must be solved:

(31) $\frac{H_k}{h}-\frac{}{t}\frac{H_k}{h^{}}=-+\frac{{}_k\left(h\left(t\right),t\right)}{f}-h^{}\left(t\right)=0.$
This is done iteratively by

(32) $h_{k,i+1}\left(t\right)=h_{k,i}\left(t\right)+d.Math.\left(-\frac{{}_k\left(h_{k,i}\left(t\right),t\right)}{f}+h_{k,i}^{}\left(t\right)\right)$
with step size d.

(33) Based on those considerations, the following implementation is yielded in a discrete setting where the spectrogram S and .sub.k are given as a matrix: S: {f.sub.1, . . . , f.sub.M}{t.sub.1, . . . , t.sub.N}.fwdarw.R.sub.+: 1. Define a rounding operator: f=argmin.sub.{f.sub.1.sub., . . . , f.sub.M.sub.}|ff.sub.k|. 2. Calculate: .sub.1(f.sub.m, t.sub.n)=S(f.sub.m, t.sub.n)+.Math.(S(f.sub.m1, t.sub.n)2S(f.sub.m, t.sub.n)+S(f.sub.m +1, t.sub.n)) for n=1, . . . , N. Set k=1, h.sub.k,1(t.sub.n)=f.sub.1 and g.sub.1(t.sub.n)=0 for n=1, . . . , N, and choose an appropriate stopping threshold parameter g>0. 3. Set i=1 and ={1, . . . , N}. 4. Calculate for n=1, . . . , N. a. If n:

(34) $i.h_{k,i}^{}\left(t_n\right)=h_{k,i}\left(t_{n-1}\right)-2h_{k,i}\left(t_n\right)+h_{k,i}\left(t_{n+1}\right).\mathrm{ii}.\begin{array}{c}{g}_{i+1}\left(t_n\right)=g_i\left(t_n\right)+\frac{{}_k\left(h_{k,i}\left(t_n\right),t_n\right)}{f}\\ =g_i\left(t_n\right)+\frac{{}_k\left(.Math.h_{k,i}\left(t_n\right).Math.,t_n\right)-{}_k\left(.Math.h_{k,1}\left(t_n\right).Math.,t_n\right)}{h_{k,i}\left(t_n\right)-h_{k,1}\left(t_n\right)}\end{array}$$\mathrm{iii}.h_{k,i+1}\left(t_n\right)=h_{k,i}\left(t_n\right)+d.Math.\left(-g_{i+1}\left(t_n\right)+h_{k,i}^{}\left(t_n\right)\right).$ b. Otherwise, h.sub.k,i+1(t.sub.n)=h.sub.k,i(t.sub.n). 5. Check a stopping criteria for n. a. If g.sub.i+1(t.sub.n)>g, set =\{n}. b. If h.sub.k,i+1(t.sub.n)>f.sub.M or h.sub.k,i+1(t.sub.n)<f.sub.1, set =\{n} and h.sub.k,i+1(t.sub.n)=h.sub.k,i(t.sub.n). c. If = go to step 7. 6. Set i=i+1 and go to step 4. 7. Set the k.th harmonic h.sub.k=h.sub.k,i+1. If k is equal the number of wanted harmonics, then stop. 8. Initialization for the next harmonic: For all (f, t){f.sub.1, . . . , f.sub.M}{t.sub.1, . . . , t.sub.N} let

(35) ${}_{k+1}\left(f,t\right)=\{\begin{array}{cc}0& \mathrm{for}f{h}_k\left(t\right)+\\ {}_k\left(f,t\right)& \mathrm{for}f>h_k\left(t\right)+\end{array},g_1\left(t\right)=0\mathrm{and}{h}_{k+1,1}\left(t\right)=h_k\left(t\right)+.\text{.Math.}$
Go to step 3.

(36) FIG. 9 shows an example from such an implementation where the spectrogram S has a frequency resolution from 78-966 Hz with a step size of 10Hz. The time resolution is given by 0 to 2.3 seconds with step size 51 msec. The estimation of the first three harmonics is good during the speech phases, but the calculated values are too small since the algorithm is designed to stop at the first local minimum of the potentials p.sub.k,t. To avoid such stopping in the local minima, the term

(37) $\frac{{}_k\left(.Math.h_{k,i}\left(t_n\right).Math.,t_n\right)-{}_k\left(.Math.h_{k,1}\left(t_n\right).Math.,t_n\right)}{h_{k,i}\left(t_n\right)-h_{k,1}\left(t_n\right)}$
can be exchanged by

(38) 0${.Math.}_{\underset{f{h}_{k,i}\left(t_n\right)}{f\left[f_1,\mathrm{.Math.},f_M\right]}}{}_k\left(f,t_n\right)$
in 4.a.ii. of the above implementation. This yields for h.sub.k,i(t.sub.n)=f.sub.i that g.sub.i(t.sub.n) is the cumulative sum of .sub.k(.Math., t.sub.n). Thus, the stopping criteria 5.a. above is reached at a high value of .sub.k(.Math., t.sub.n). Furthermore, a factor also can be introduced into the calculation of .sub.k in step 8 since the harmonic amplitudes generally decrease. Thus

(39) ${}_{k+1}\left(f,t\right)=\{\begin{array}{cc}0& \mathrm{for}f{h}_k\left(t\right)+\\ .Math.{}_k\left(f,t\right)& \mathrm{for}f>h_k\left(t\right)+\end{array}\mathrm{with}{}_{k}1.$
In FIG. 7, the result of this modified implementation is shown where it can be seen that the first six harmonics are well estimated.

(40) Considering an actual in a hearing implant speech processor, the spectrogram S can be divided into segments in time: S.sub.l,L: {f.sub.1, . . . , f.sub.M}{t.sub.l+1, . . . , t.sub.L+1}.fwdarw.R.sub.+ with S.sub.l,L(.Math., t.sub.j)=S(.Math., t.sub.j+1) for j=1, . . . , L. Then the modified implementation just discussed can be applied to each S.sub.l,L and achieve the harmonics h.sub.k.sup.l for l=0, . . . , NL. Setting h.sub.k(t.sub.j)=h.sub.k.sup.0(t.sub.j) for j=0, . . . , L (initialization) and for the following segments, using only the last value: h.sub.k(t.sub.L+l)=h.sub.k.sup.l(t.sub.L+l) for l=1, . . . , NL. The resultant harmonics are shown in FIG. 10 where the estimations are comparable to the results in FIG. 7.

(41) Various refinements or modifications alone or in combination are possible in different specific embodiments, including: Other methods can be used to get the spectrogram S; for example, applying an STFT on the band pass signals U.sub.1, . . . , U.sub.K. The calculation of the stimulation timing signals Y.sub.k can be changed. The approach of the active contour model can be changed; for example, using another smoothing term .sub.t.sub.1.sup.t.sup.2 h(t).sup.2dt or potential function. The calculation of .sub.k also can be modified. The calculation of the envelope signals can be modified to be a function of the spectrogram: (X.sub.1, . . . , X.sub.K)=F(S). The calculation of the envelope signals can be modified to be a function of the spectrogram and the estimated harmonics and/or the timing signals. The input signal of the generation of the spectrogram S can be modified; for example, though Preprocessor that for example may pre-process the input sound signal with automatic gain control and/or noise reduction functions. Harmonics have the property that f.sub.k=k.Math.f.sub.0 with fundamental frequency f.sub.0. This information of harmonics can be introduced into the active contour model, for example by adding the additional term

(42) $\frac{{}_2}{2}{}_{t_1}^{t_2}\left(\underset{j=1}{\overset{k}{.Math.}}{\left(h\left(t\right)-\frac{k}{j}{h}_{j-1}\left(t\right)\right)}^2\right)$
into the functional H.sub.k. The information of the estimated harmonics can also be used in noise reduction and/or a classification algorithm to improve the signal-processing in these modules.

(43) Applying an active contour model to estimate dominant frequencies present in a spectrogram of the input sound signal can further lead to the development of new coding strategies concepts where the actual harmonics determine the starting points of a CSSS and/or the psychoacoustic phenomenon of the phantom fundamental can be exploited. The course of the dominant frequencies that is determined could also be useful in a scene classification algorithm, and the acquired classification could then be used to control further signal processing. For example a stationary noise reduction (NR) could be turned off when listening to music, or a beamformer could be turned on in a conversation with loud surrounding background noise. The knowledge of the dominant frequencies can be used in a NR as information for a voice-activity detector, which might be able to distinguish between speech and other sounds based on the harmonics present in speech.

(44) Embodiments of the invention may be implemented in part in any conventional computer programming language. For example, preferred embodiments may be implemented in a procedural programming language (e.g., C) or an object oriented programming language (e.g., C++, Python). Alternative embodiments of the invention may be implemented as pre-programmed hardware elements, other related components, or as a combination of hardware and software components.

(45) Embodiments can be implemented in part as a computer program product for use with a computer system. Such implementation may include a series of computer instructions fixed either on a tangible medium, such as a computer readable medium (e.g., a diskette, CD-ROM, ROM, or fixed disk) or transmittable to a computer system, via a modem or other interface device, such as a communications adapter connected to a network over a medium. The medium may be either a tangible medium (e.g., optical or analog communications lines) or a medium implemented with wireless techniques (e.g., microwave, infrared or other transmission techniques). The series of computer instructions embodies all or part of the functionality previously described herein with respect to the system. Those skilled in the art should appreciate that such computer instructions can be written in a number of programming languages for use with many computer architectures or operating systems. Furthermore, such instructions may be stored in any memory device, such as semiconductor, magnetic, optical or other memory devices, and may be transmitted using any communications technology, such as optical, infrared, microwave, or other transmission technologies. It is expected that such a computer program product may be distributed as a removable medium with accompanying printed or electronic documentation (e.g., shrink wrapped software), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the network (e.g., the Internet or World Wide Web). Of course, some embodiments of the invention may be implemented as a combination of both software (e.g., a computer program product) and hardware. Still other embodiments of the invention are implemented as entirely hardware, or entirely software (e.g., a computer program product).

(46) Although various exemplary embodiments of the invention have been disclosed, it should be apparent to those skilled in the art that various changes and modifications can be made which will achieve some of the advantages of the invention without departing from the true scope of the invention.