Deconvolution with varying ILS

[RichardJ]

Does anyone know how to do deconvolution when the ILS varies across the spectrum? See the attached file for an example (this is measured data, but the natural peak width is so small the only significant broadening is from the instrument). I don't know how to do this, and don't even know what I should be looking for. Does deconvolution like this have a standard name, such as "local deconvolution"? It's hard to search for something when you don't even know what it's called!

Richard

[AlvaroDíaz]

You can see convol and deconvol functions in mcad 11 and a little program to emulate this at

http://collab.mathsoft.com/read?124389,8e#124403

Convolution came from integrals, so, the concept of 'local' convolutions (or deconvolutions) could be strange. But if you have some parameter with variations in your range study probably you can (a) divide in small intervals where the things came constants, or (b) let the parameter free and hope that in some stage you can assign values or range values to this parameters.

Deconvolutions are only the entire quotient of two polynomials. That because convolution is the product of two polys, and (again) because within Parseval theorem and the develops in series you reduce the calculus of convolutions to this product.

Hope this helps.

Regards. Alvaro.

[RichardJ]

On 9/27/2009 8:20:56 PM, adiaz wrote:
>You can see convol and
>deconvol functions in mcad 11
>and a little program to
>emulate this at
>
>http://collab.mathsoft.com/rea
>d?124389,8e#124403

If it's just a standard deconvolution it's easy to do it using FFTs. In principle, anyway. In practice deconvolution is not always so easy.

>Convolution came from
>integrals, so, the concept of
>'local' convolutions (or
>deconvolutions) could be
>strange.

That is a point. This situation is not that uncommon though, so I'm sure there must be algorithms to address it.

>But if you have some
>parameter with variations in
>your range study

I am assuming that I can come up with some function that describes the ILS that has a parameter that varies over the spectral range.

>probably you
>can (a) divide in small
>intervals where the things
>came constants,

I had thought of this approach, but I can't end up with any discontinuities in the data, so I would have to do a separate deconvolution for every data point (in other words, for data point i the deconvolution parameter takes on the value p[i). That would be rather slow, but possible.

>or (b) let the
>parameter free and hope that
>in some stage you can assign
>values or range values to this
>parameters.

I'm not sure what you mean. What I'm looking for is an algorithm that would be faster than doing a separate deconvolution for every data point. Or confirmation that no such algorithm exists.

Richard

[AlvaroDíaz]

On 9/27/2009 9:27:11 PM, rijackson wrote:
>I'm not sure what you mean. What I'm looking for is an algorithm that would be faster than doing a separate deconvolution for every data point. Or confirmation that no such algorithm exists.

If you don't know the kernel of the convolution, I think that the name of your problem it's a "blind deconvolution".

The only one algo that I know (this is, I understand somethings, because there are a lot of new algorithms in this matter) is:

http://en.wikipedia.org/wiki/Richardson%E2%80%93Lucy_deconvolution

This assume a Poisson distributed values, but doesn't work with other free parameters (this is the reason that I say that you hope assign unknows). The generalization of this algo have a lot of variants, and I can't focusse never in a practical use of some general formulism (obviously, "I").

Regards. Alvaro.

Regards. Alvaro.

[RichardJ]

On 9/27/2009 10:00:17 PM, adiaz wrote:
>On 9/27/2009 9:27:11 PM, rijackson
>wrote:
>>I'm not sure what you mean. What I'm looking for is an algorithm that would be faster than doing a separate deconvolution for every data point. Or confirmation that no such algorithm exists.
>
>If you don't know the kernel of the
>convolution, I think that the name of
>your problem it's a "blind
>deconvolution".

Assume I know the kernel

>The only one algo that I know (this is,
>I understand somethings, because there
>are a lot of new algorithms in this
>matter) is:
>
>http://en.wikipedia.org/wiki/Richardson%
>E2%80%93Lucy_deconvolution

Yes, I could implement a 2D version of that. My problem, however, is that my p[ij is not a single, constant psf (well, what I have is actually an instrument line shape, not a psf, but I'm relating what I have to the article). It varies, depending on i. I could adapt this algorithm to accommodate a varying ILS, but I'm not sure that's any better than just doing N FFT based deconvolutions for N points. The ILS has a limited extent, so I would not need to FFT the entire data vector each time, just a small subset of it, so maybe it will not actually be that slow.

Richard

[RichardJ]

On 9/28/2009 11:21:59 AM, jmG wrote:
>It could be we are not talking
>about the same thing.

We are talking about the same thing.

>Deconvolution works fine in
>Mathcad, lots of them were
>done in the past,

This is not a simple case, and I am not aware of one like this being done in the past.

Richard

[TheodoreM.Bones]

Type several buttons marked in green as shown. Then the program runs OK.

[RichardJ]

ILS column 1 is the ILS convolved with a series of infinitely narrow peaks. I posted it to show how the ILS varies. If you look at the peaks they are symmetric at one end, and badly skewed at the other. The skew is what I need to get rid of in other data. The standard, simple, approach of FFT, divide by the FT of the ILS, multiply by the FT of a more desirable ILS, then IFFT, will not work because the ILS is not constant across the data vector.

Richard

[LouP]

If the impulse response of your deconvolving kernel is position (time) dependent, then I believe that you have described what is effectively a time varying system (except your domain is wavelength). If the support of the impulse response is small compared to the overall domain (1661 in your example), then why not try the brute force integral/summation approach to the convolution with wavelength as a param of the kernel, as approp?

I suspect that the convolution including 2-D impulse response( with the wavelength dependence) of the general time-varying convolution expression may be solvable with a 2D DFT setup, but I don't have time to look at it right now. There may be a general 2D formulation which covers a 1-D, time varying system. If you don't like the brute force approach, you can explore how to setup this method. I may have some time later in the week.

ILS? I found 27 acronyms, and ruled out Insurance Linked Security and Israeli Shekel; my bet is on Intracavity Laser Spectroscopy (http://www.abbreviations.com/ILS).

Lou

[RichardJ]

On 9/28/2009 11:13:42 AM, lpoulo wrote:
>If the impulse response of
>your deconvolving kernel is
>position (time) dependent,
>then I believe that you have
>described what is effectively
>a time varying system (except
>your domain is wavelength).

Yes, that's exactly the situation.

>If
>the support of the impulse
>response is small compared to
>the overall domain (1661 in
>your example), then why not
>try the brute force
>integral/summation approach to
>the convolution with
>wavelength as a param of the
>kernel, as approp?

It's not the convolution I need, but the deconvolution. Although that of course is just the convolution with 1/kernel. So maybe the brue force approach would work.

>I suspect that the convolution
>including 2-D impulse
>response( with the wavelength
>dependence) of the general
>time-varying convolution
>expression may be solvable
>with a 2D DFT setup, but I
>don't have time to look at it
>right now. There may be a
>general 2D formulation which
>covers a 1-D, time varying
>system. If you don't like the
>brute force approach, you can
>explore how to setup this
>method. I may have some time
>later in the week.

Don't spend too much time on it. I can think of more than one approach to solving this. They are approaches I made up though, so before spending time trying them out I thought I would post here in case someone was familiar with the problem and could point me to a tried as tested algorithm. If not, I'll try the brute force approach and/or the multiple FFT approach. Either one should work in theory,

Sorry. Instrument Line Shape. Such a common acronym in what I do I didn't even think to define it. It's basically the impulse response, except the impulse is on a wavelength axis rather than a time axis.

Richard

[LouP]

The deconvolution kernel will be essentially a high pass filter(HPF) that will emphasize the noise, with some parameters that vary with wavlength. From a practical viewpoint, it may be that making the tradoff between resultant noise and the upper freq. limit of the HPF (which also limits the resultant deconvolved line width) may be the messiest part since this is a wavelength dependent tradeoff. Keep us posted (pun intended) on the results.

Lou

[ptc-1368288]

>The deconvolution kernel will be essentially a high pass filter(HPF) that will emphasize the noise,...<<br> _________________________

Quite right, the reconstruct is only noise !
noise on an unknown carrier.

jmG

[ptc-1368288]

On 9/28/2009 2:56:44 PM, jmG wrote:
>>The deconvolution kernel will be essentially a high pass filter(HPF) that will emphasize the noise,...<<br> >_________________________
>
>Quite right, the reconstruct
>is only noise !
>noise on an unknown carrier.
>
>jmG
__________________________
PSD comes out nicely from icfft of the deconvolved.

jmG

[RichardJ]

On 9/28/2009 1:37:46 PM, lpoulo wrote:
>The deconvolution kernel will
>be essentially a high pass
>filter(HPF) that will
>emphasize the noise, with some
>parameters that vary with
>wavlength. From a practical
>viewpoint, it may be that
>making the tradoff between
>resultant noise and the upper
>freq. limit of the HPF (which
>also limits the resultant
>deconvolved line width) may be
>the messiest part since this
>is a wavelength dependent
>tradeoff.

That usually is the messiest part, but in this case it probably won't be. If you look at the spectrum I posted it's a series of lines that represent the ILS (some are doublets, so ignore those). At one end the lines are symmetric, at the other they are seriously skewed. In principle, the only difference should be the skew. All I want to do is remove the skew, not try to get resolution enhancement. So if the ILS is a function ILS(p(lambda)) with a parameter, p, that represents the skew, I want to deconvolve with ILS(p(lambda)) and then convolve with ILS(0).

The data point density is not high enough to do any significant resolution enhancement anyway. If you want to do resolution enhancement then the data needs to have been measured with a data point density that is high enough for the target resolution, not just for the measured resolution.

Richard

[PhilipOakley]

Given the large (ish) number of data points, I'd suggest something along the lines of one of the wavelet decompositions.

The reason being that you get 'local support' (while fft is global). From this you should then be able to start doing a local deconvolution.

The trick with wavelets is to use one of the Dual Tree approaches which allow a pseudo magnitude & phase approach which is reasonable constant locally, but avoid the globality issue you mentioned (that the Instrument response varies with wave number).

The other option/theory would be to transform the data to a domain where the ILS is constant, deconvolve in that domain, then invert back to the wavenumber domain (think FTIR..) .

Philip Oakley

[RichardJ]

On 9/29/2009 6:07:56 AM, philipoakley wrote:

>The other option/theory would
>be to transform the data to a
>domain where the ILS is
>constant,

In this case I can't imagine what that domain would be though. And if I did manage to convert to such a domain, I would need to convolve with an ILS that was not constant before transforming back to the wavelength domain (where the ILS would then be constant).

> deconvolve in that
>domain, then invert back to
>the wavenumber domain (think
>FTIR..) .

I think FTIR just about every day, but I don't see the connection with this problem. The ILS of an FTIR is guaranteed to be constant if the axis is in some unit of frequency (such as wavenumbers).

Richard

[ptc-1368288]

Not a case of data reduction. The deconvolution I posted works great, but deconvolution is only "approximate" and is not in theory possible as it involves a "divide by 0" by the nature of the mathematical operation. The other point is that we don't know how Richard did the convolution and it seems not containing enough figures [3 does not seem enough]. Fourier would probably want the all machine precision and a very low digitization noise from the ADC. A far as the project is concerned, the data set is purely out of the blue with no scope and no end purpose.

Maybe proprietary ?
Does ILS means ILS (Instrument Landing System] ?

jmG

[RichardJ]

On 9/29/2009 11:35:57 AM, jmG wrote:
>Not a case of data reduction.
>The deconvolution I posted
>works great,

Sorry, but I have no idea what you posted is even trying to do.

> but deconvolution
>is only "approximate" and is
>not in theory possible as it
>involves a "divide by 0" by
>the nature of the mathematical
>operation. The other point is
>that we don't know how Richard
>did the convolution

I didn't do the convolution. I told you, it's real data.

> and it
>seems not containing enough
>figures [3 does not seem
>enough]. Fourier would
>probably want the all machine
>precision and a very low
>digitization noise from the
>ADC. A far as the project is
>concerned, the data set is
>purely out of the blue with no
>scope and no end purpose.

I explained the scope and the end purpose.

>Maybe proprietary ?

The reason I want to do this is proprietary, yes.

>Does ILS means ILS (Instrument
>Landing System] ?

Instrument Line Shape.

Richard

[IRstuff]

This has a very surreal and existential feel to it. I'll bet someone could write a book...

TTFN,
Eden

[ptc-1368288]

Richard,

I understand "proprietary", that the data set is given to you, that it may be incorrectly collected and/or under-digitized, and that on some of these accounts, Mathcad 11.2a crashes dramatically ! at least my Mathcad. Now, to read more, I have to read Philip's links as bed time reading, hopefully to avail. The work sheet I just posted above proves w/o doubt that my Mathcad 11.2a supports Convoluion/deConvolution, that was my QED.
What I have done and posted, I have deleted, but part of it can be found in Theodore reply above this thread.

Cheers Richard !

Jean

[PhilipOakley]

http://www-imk.fzk.de/asf/ame/ClosedProjects/assfts/P_II_1_Bouchard_JP.pdf ?


Philip Oakley

[PhilipOakley]

Also found http://trs-new.jpl.nasa.gov/dspace/bitstream/2014/18028/1/99-1484.pdf

Haven't had chance to digest but they 'suggest' options.

Further http://www.eusipco2009.org/Tutorials-g.asp#Tutorial 4
may be of interest (I have the papers)


Philip Oakley

[RichardJ]

On 9/29/2009 12:59:20 PM, philipoakley wrote:
>Also found
>http://trs-new.jpl.nasa.gov/ds
>pace/bitstream/2014/18028/1/99
>-1484.pdf
>
>Haven't had chance to digest
>but they 'suggest' options.

That does make the good point that if the FT spectrometer has an array detector then while the ILS is constant over all wavenumbers in the spectrum from each pixel, it can be different for each pixel. I don't see where that gets me though. I don't have a hyperspectral data set, I just have a single spectrum, with an ILS that is wavelength dependent.

Richard

[ptc-1368288]

Thanks Philip !!! for the links.

ILS is like they all say "ILS"

Instrument Landing System,
Instrument Line Shape
... what else ?
So, if you have a good signal, good methods then good project. In this collab: a secret garden with a snake in the garden... worth to follow as it will come down.

Jean

[RichardJ]

On 9/29/2009 12:36:41 PM, philipoakley wrote:
>http://www-imk.fzk.de/asf/ame/
>ClosedProjects/assfts/P_II_1_B
>ouchard_JP.pdf ?

Well, it shows asymmetry in the ILS, but that asymmetry is constant with wavenumber. I could see how to take what they have done and make it non-constant though. If you introduced chromatic aberration so that the beam divergence was wavenumber dependent, and then the ILS would be wavenumber dependent. That would leave them with the same problem I have, but based on what's in the poster they would be no closer to a solution.

Richard

[LouP]

On 9/28/2009 5:54:58 PM, rijackson wrote:
That
>usually is the messiest part, but in
>this case it probably won't be. If you
>look at the spectrum I posted it's a
>series of lines that represent the ILS
>(some are doublets, so ignore those). At
>one end the lines are symmetric, at the
>other they are seriously skewed. In
>principle, the only difference should be
>the skew. All I want to do is remove the
>skew, not try to get resolution
>enhancement. So if the ILS is a function
>ILS(p(lambda)) with a parameter, p, that
>represents the skew, I want to
>deconvolve with ILS(p(lambda)) and then
>convolve with ILS(0).

The data point
>density is not high enough to do any
>significant resolution enhancement
>anyway. If you want to do resolution
>enhancement then the data needs to have
>been measured with a data point density
>that is high enough for the target
>resolution, not just for the measured
>resolution.

Is this something ad hoc, need only for this particular analysis, or, as I suspect, are you looking for a more general technique that you can use of other data sets?

Do you need to get the ILS params from the data itself, or do you have a separate, independent model or test data that can be used for the corrections? Are the isolated peaks usable to extract params?

I would guess that this is a fairly common problem, so I'm surprised that there are no standard tools to deal with it. Unfortunately, this will likely gnaw at me and I'll work on it(slowly) in spite of your admonition.

Lou

[RichardJ]

On 9/29/2009 2:51:57 PM, lpoulo wrote:

>Is this something ad hoc, need only for
>this particular analysis, or, as I
>suspect, are you looking for a more
>general technique that you can use of
>other data sets?

It would be applied to several thousand spectra, yes.

>Do you need to get the ILS params from
>the data itself, or do you have a
>separate, independent model or test data
>that can be used for the corrections?

What I posted is the separate test data 🙂
The natural peak width in what I posted is, to all intents and purposes, zero. So the data represents the ILS at different wavelengths.

The actual data that the deconvolution will be applied to has broader peaks than this. My first step would be to deconvolve this data though, so that the skew is removed and the peaks all look like the ones around 950.

>Are the isolated peaks usable to extract
>params?

Yes, that was my intent when I acquired this data.

I expect the symmetric part of the ILS to be very close to Gaussian (so a Gaussian should fit the single peaks around 950, but I haven't tried it yet). The asymmetry is going to be a guessing game, but I would try to model it with some sort of asymmetric Gaussian (even though that is a contradiction in terms!)

>I would guess that this is a fairly
>common problem, so I'm surprised that
>there are no standard tools to deal with
>it.

That was exactly my thinking. I didn't know of such a tool, and the fact that I had never heard of one surprised a little bit. But I figured that if I searched the web I would find something. When I couldn't find something I was more surprised, but figured that I must just be searching for the wrong terms, because I didn't know what it was called. So then I posted the question here.

>Unfortunately, this will likely gnaw
>at me and I'll work on it(slowly) in
>spite of your admonition.

If it makes you feel any better, it's bugging me too. And now my client has found an alternative solution (using a more expensive instrument that doesn't have the problem in the first place), so I actually also have no better reason to work on it than curiosity! I'll spend some time on it over the next few days I expect, and will post anything useful.

Richard

[ptc-1368288]

What I posted is the separate test data...
My first step would be to deconvolve this data though ...
If it makes you feel any better, it's bugging me too. And now my client has found an alternative solution (using a more expensive instrument that doesn't have the problem in the first place), so I actually also have no better reason to work on it than curiosity! I'll spend some time on it over the next few days I expect, and will post anything useful. [Richard]
__________________________

As far as the data set from the client, Mathcad can deconvolve. The more expensive instrument is simply embedding the maths required to do. About the traceability ? maybe not needed if a calibration/check procedure is supplied c/w with all necessary primary standards.

Lot of projects started on square wheel, ending as great Mathcad tools. At this point, your project is for the general interest of the collab community and as it stands added to your original work sheet. From what I read quick [Philip's links], vaguely understandable for the unfamiliar to ILS.

jmG