What is preventing me from achieving better accuracy for this delay?

[Nate Duong]

I have 2 PERFECT data of the transmitter and receiver. From 2 data, I can calculate the delay estimation:

Fs = 8e6; % sample rate
...
for i = 1:2
[cc_correlation,lag] = xcorr(signal2(i), signal1);
[cc_maximum, cc_time] = max(abs(cc_correlation));
cc_estimation = abs(length(signal1) - cc_time);
delay(i) = cc_estimation/Fs;
end

Then I have the matrix of delays are 11 microseconds and 13.875 microseconds.

The expectation in nanosecond from this function because from the sampling rate, I can see the period time T=1/Fs=125ns. Therefore, the delay should be in nanosecond, not microsecond as I had.

When I call the MATLAB function above:

[cc_maximum, cc_time] = max(abs(cc_correlation));

It returns the values which are called cc_maximum, and another value cc_time. It is sample data.

What did I do wrong for this algorithm?

My professor also said:"you don’t have function, you have sample version of the function, the xcorr is a waveform of continuous function, they have a maximum in the current of time. When you work with a sample, the waveform you have entire function tell the value of the function you have discrete time"

and I still do not understand what his mean?

I hope someone can help me out.

Thank you.

 

[FactChecker]

Some thoughts:

• How do you know that the first data point of the transmitter and receiver signals were recorded at exactly the same time? You are counting on nanosecond clock synchronization. Are you sure you have that?
• I recommend simplifying this by only looking at one receiver signal till everything makes sense. Get rid of the loops and indices.
• If the first data of transmitter and receiver are at exactly the same time, then you probably only need the first maximum cross correlation index, not an array of them.
• Can you generate a test signal to transmit that will maximize the correlated times and minimize the others?
• I don't understand what you are doing in the cc_estimator calculation. What does the length of the signal have to do with anything?

 

[Nate Duong]

Some thoughts:

• How do you know that the first data point of the transmitter and receiver signals were recorded at exactly the same time? You are counting on nanosecond clock synchronization. Are you sure you have that?
• I recommend simplifying this by only looking at one receiver signal till everything makes sense. Get rid of the loops and indices.
• If the first data of transmitter and receiver are at exactly the same time, then you probably only need the first maximum cross correlation index, not an array of them.
• Can you generate a test signal to transmit that will maximize the correlated times and minimize the others?
• I don't understand what you are doing in the cc_estimator calculation. What does the length of the signal have to do with anything?

@FactChecker: thank you the reply, I will review first four suggestions from you. and the 5th one I am trying to explain to you:

I have 2 signal, 1 for the TX signal and 1 for the RX signal. The TX signal is sent from the transmitter have the waveform lTXdata.jpg and sent repeatly in 100 seconds, therefore, I have the second image TXdata_vs_RXdata.jpg , every second the TX sends to RX, we will have delay, so we have 100 times of delay.

So, I need to cut RX data file out into 100 parts and calculate delay each of them. I am not sure i did right or wrong but my professor mentioned about 100 delays, there is only way to get 100 delays.

and the delay changed as this plot show.

 

[f95toli]

I haven't looked at this in any detail.
However, what you wrote still does not explain what you are doing. Note that the "units" of the CC_estimator calculations make no sense. You are subtracting cc_time (which presumably is in seconds) from the number of elements of the arrays that contains signal1
Hence, unless you've set it up in some very unusual way (so that each element is always equal to a specific number of ns) I can't see how this could be correct.

 

[Nate Duong]

I haven't looked at this in any detail.
However, what you wrote still does not explain what you are doing. Note that the "units" of the CC_estimator calculations make no sense. You are subtracting cc_time (which presumably is in seconds) from the number of elements of the arrays that contains signal1
Hence, unless you've set it up in some very unusual way (so that each element is always equal to a specific number of ns) I can't see how this could be correct.

So, Could you give me some suggestions how to fix it to make more sense?
Let me explain what I am trying to do?
I have a transmitted waveform and send to the RX device, such as TXdata.jpg and TXdata_vs_RXdata.jpg so I want to calculate the time delay with expectation is few nanoseconds. I also looked up https://www.mathworks.com/help/signal/ref/xcorr.html , and it is a good example. I applied the similar way from it. Now you said I did wrong and no make sense. That is also confuse me.

 

[FactChecker]

It would be more clear if you renamed cc_time to cc_max_index. It is an array of the indices where the maximum absolute cross correlation is achieved.
CORRECTION: Rename cc_time to cc_max_lag_frames. It is the number of frames of lag that give the highest cross correlation.
I think that the absolute value should be removed. You are looking for the maximum positive cross correlation. Negative cross correlations should not be confused with the positive cross correlations.

 

[FactChecker]

My general advice is to make the main part of the algorithm as simple and bare-bones as possible. Don't look at multiple seconds of data. Start with a data sample that is small enough that you can print/display/plot intermediate results to see step-by-step if your algorithm is doing what you expect.

Once the basic algorithm is working, you can quickly make it as complicated as you want.

 

[Nate Duong]

My general advice is to make the main part of the algorithm as simple and bare-bones as possible. Don't look at multiple seconds of data. Start with a data sample that is small enough that you can print/display/plot intermediate results to see step-by-step if your algorithm is doing what you expect.

Once the basic algorithm is working, you can quickly make it as complicated as you want.

Yes, I did. I already tried with 1 second of data, and still give me the same delay in microsecond, I know the algorithm which I posted, was wrong and I am still trying to figure out the way to get in nanosecond. According to this link which I am following https://www.mathworks.com/help/signal/ref/xcorr.html#bub68m7 and I see nothing wrong with the algorithm:
TX.dat : https://www.dropbox.com/s/zr04pff9skeh8cn/TX.dat?dl=0
RX.dat: https://www.dropbox.com/s/pj9bguc0bu7wtba/RX.dat?dl=0
%% initial values:
nsamps = inf;
nstart = 0;
Fs = 8e6; % sample rate
F_0 = 520e6; % center freq
c = 3e8; % speed of light

%% input data
file_tx = 'TX.dat';
file_rx = 'RX.dat';
x_tx = readcplx(file_tx, nsamps,nstart);
x_rx = readcplx(file_rx, nsamps,nstart);
s2 = x_rx;
s1 = x_tx;
[acor,lag] = xcorr(s2,s1);
[~,I] = max(abs(acor));
lagDiff = lag(I);

timeDiff = lagDiff / Fs;

%% read .dat data
function x = readcplx(filename,nsamps,nstart);
fid = fopen(filename);
fseek(fid,4 * nstart,'bof');
y = fread(fid,[2,inf],'short');
fclose(fid);
x = complex(y(1,:),y(2,;
end

 

[Nate Duong]

My general advice is to make the main part of the algorithm as simple and bare-bones as possible. Don't look at multiple seconds of data. Start with a data sample that is small enough that you can print/display/plot intermediate results to see step-by-step if your algorithm is doing what you expect.

Once the basic algorithm is working, you can quickly make it as complicated as you want.

I just discussed with the professor, he also mentioned the expectation for the delay will be 3ns or better (less than 1m).

I do not have continue function, I just have function of the sample time (sample waveform different to the continuous of time waveform)

I am taking sample number of delay, that way never going to be more accurate then half of 125ns (about 20m)

I combined data taken in different time and I am treating the result I taken the same.

He also mentioned about the solution on the curve of auto-correlation which is attached below.

Do you have any ideal?

 

[FactChecker]

You are working with time series, which are discrete samples of signals at fixed time steps. That is a very common situation and it is what the xcorr is designed to work with. The fact that your signal is not continuous will actually help you if you can control the transmitted signal to make the cross correlation clearly best at one lag and clearly worse for the others.
You should limit the lag amounts using the maxlag parameter of xcorr (see https://www.mathworks.com/help/signal/ref/xcorr.html#inputarg_maxlag )

Notice the correction in post #6:Rename cc_time to cc_max_lag_frames. It is the number of frames of lag that give the highest cross correlation.

 

[Nate Duong]

You are working with time series, which are discrete samples of signals at fixed time steps. That is a very common situation and it is what the xcorr is designed to work with. The fact that your signal is not continuous will actually help you if you can control the transmitted signal to make the cross correlation clearly best at one lag and clearly worse for the others.
You should limit the lag amounts using the maxlag parameter of xcorr (see https://www.mathworks.com/help/signal/ref/xcorr.html#inputarg_maxlag )

Notice the correction in post #6:Rename cc_time to cc_max_lag_frames. It is the number of frames of lag that give the highest cross correlation.

@FactChecker: very helpful, even I am still trying to understand the meaning of MAXLAG.

but 1 thing I also thinking of, that is: the professor is asking to find the interpolated maximum of xcorr rather than max sample location. So, the continuous signal is difference from computed sample peak to actual peak is less than 1 sample (mean: at the point after correlating, we have 1 point, and that point will be divided by sample rate Fs, so 1/8e6 ) delay has to be less than 125e-9ns

By actual peak I mean the peak of curve at top, so it requires interpolation between two samples on either side. one way is to use MATLAB interp of xcorr result using very high interpolation rate then find new max and subtract from sample based peak.

SO, do you have any ideal to find the interpolation between 2 samples?

 

[FactChecker]

I think that trying to interpolate the delay is getting way ahead of things. You are saying that the calculated answers are wrong by orders of magnitude. I don't know the details of your transmitter, receiver, and clock synchronization, but I think that you are being over optimistic about how much accuracy you have in the timing.

 

[Nate Duong]

I think that trying to interpolate the delay is getting way ahead of things. You are saying that the calculated answers are wrong by orders of magnitude. I don't know the details of your transmitter, receiver, and clock synchronization, but I think that you are being over optimistic about how much accuracy you have in the timing.

@FactChecker: Sorry for lacking of knowledge, but I thought the calculations were wrong. you also helped me know more the way to debug the problem. I appreciated for it.

Consider I have this question "how much accuracy you have in the timing?" as you mentioned. Fix me if I state: "an interpolation rate of 125 will take you to 1 ns accuracy and I do that on signal1 and signal2 instead of doing it on xcorr result and find delay from that as before."

So, I have to interpolate rate of 125 first on signal1 and signal2, then I have 2 new signals let say s1 and s2. I can cross-correlation s1 and s2 to get the delay, that delay will give me in nanosecond.

Did I say right? please fix me!

Thank you.

 

[FactChecker]

Oh! I may have made a wrong assumption from the beginning. I assumed you are transmitting a digital message at a certain rate to a receiver and comparing the data in the message to determine the delay. Now I think that you may be sending an analog continuous signal and locally sampling the signals at both ends. So the sampled data at the transmitter and receiver will not be the same values. Is that right?

 

[Nate Duong]

Oh! I may have made a wrong assumption from the beginning. I assumed you are transmitting a digital message at a certain rate to a receiver and comparing the data in the message to determine the delay. Now I think that you may be sending an analog continuous signal and locally sampling the signals at both ends. So the sampled data at the transmitter and receiver will not be the same values. Is that right?

they are not the same value, because of noise added due to signal bouncing in the lab, the amplitude of the TX data bigger than RX data 30 times. original length of TX data is about 208000, but the length of RX data is 800000, therefore after the Transmitted real TX data, and the rest of time in 1s transmitted only noise.

 

[f95toli]

What is the following line supposed to do?
[cc_correlation,lag] = xcorr(signal2(i), signal1);

Since signal2 is (presumably) an array and i is an index that can be either 1 or 2, you are calculating the cross correlation between a single number (which I believe Matlab will zero-pad so that you still get an array the same length as signal2 as an answer) and an array.

Is this really what you want to do? I can't see how this would give you anything sensible.

 

[Nate Duong]

What is the following line supposed to do?
[cc_correlation,lag] = xcorr(signal2(i), signal1);

Since signal2 is (presumably) an array and i is an index that can be either 1 or 2, you are calculating the cross correlation between a single number (which I believe Matlab will zero-pad so that you still get an array the same length as signal2 as an answer) and an array.

Is this really what you want to do? I can't see how this would give you anything sensible.

@f95toli: index i means, I calculate delay in multiple RX signal from the receiver (not only one), I want to see the variation of the delay.

 

[FactChecker]

For the type of accuracy you desire, I wonder about how your clocks on the transmitter and the receiver are set. On the transmitter side you have a clock that is giving you the time of the recorded transmitted data. And you have the same thing on the receiver side. How are the two clocks set to their time? Are they GPS accuracy (15 - 40 ns) or are they synchronized by an interrupt (? interrupt clock response time)?

 

[Nate Duong]

For the type of accuracy you desire, I wonder about how your clocks on the transmitter and the receiver are set. On the transmitter side you have a clock that is giving you the time of the recorded transmitted data. And you have the same thing on the receiver side. How are the two clocks set to their time? Are they GPS accuracy (15 - 40 ns) or are they synchronized by an interrupt (? interrupt clock response time)?

@FactChecker: yes, I am using the rubidium clock the time synchronized by interrupt (http://www.thinksrs.com/products/PRS10.htm) and 1 set on the TX and another on set on the RX. the accuracy of it is 5e-11s

 

[f95toli]

@f95toli: index i means, I calculate delay in multiple RX signal from the receiver (not only one), I want to see the variation of the delay.

No, at least not if what you have written is what you actually type into Matlab,
How are the two signals stored? If you have 100 Rx signals each of some length n and these are stored in a matrix that is 100xn in size then signal1(i,:) would be an index to a signal 1.
signal1(i) where i=1 will just return whatever number happens to be first in the array.
It looks to me like you have the signals stored as two arrays and you just want the cross correlation between them; in this case you don't need an index at all (or a for-loop).

Do you understand the examples for xcorr in the Matlab documentation?

 

[Nate Duong]

@FactChecker: yes, I am using the rubidium clock the time synchronized by interrupt (http://www.thinksrs.com/products/PRS10.htm) and 1 set on the TX and another on set on the RX.

No, at least not if what you have written is what you actually type into Matlab,
How are the two signals stored? If you have 100 Rx signals each of some length n and these are stored in a matrix that is 100xn in size then signal1(i,:) would be an index to a signal 1.
signal1(i) where i=1 will just return whatever number happens to be first in the array.
It looks to me like you have the signals stored as two arrays and you just want the cross correlation between them; in this case you don't need an index at all (or a for-loop).

Do you understand the examples for xcorr in the Matlab documentation?

Yes, I do. I did wrong. I fixed it.

 

[FactChecker]

they are not the same value, because of noise added due to signal bouncing in the lab, the amplitude of the TX data bigger than RX data 30 times. original length of TX data is about 208000, but the length of RX data is 800000, therefore after the Transmitted real TX data, and the rest of time in 1s transmitted only noise.

So these are analog signals rather than digital messages containing data?

 

[Nate Duong]

So these are analog signals rather than digital messages containing data?

So these are analog signals rather than digital messages containing data?

I am not sure, but what if they are analog signal? and what if they are digital signal?

I also have an image which show the relationship of them.

 

[Nate Duong]

So these are analog signals rather than digital messages containing data?

Did I do right? make sense to you? please fix me!

https://www.dropbox.com/s/le4ppw32l4sp5ll/RX1.dat?...

https://www.dropbox.com/s/zr04pff9skeh8cn/TX.dat?d...

clear all;
close all;
format long;
%% initial values:
nsamps = inf;
nstart = 0;
Fs = 8e6; % sample rate 8MHz
F_0 = 520e6; % center freq 520MHz
c = 3e8; % speed of light
%% input data
file_tx = 'TX.dat';
file_rx = 'RX1.dat';
x_tx = readcplx(file_tx, nsamps,nstart);
x_rx = readcplx(file_rx, nsamps,nstart);
%% add interpolation
% x1 = 1:length(x_tx);
% xq1 = 1:1/6:length(x_tx);
% x2 = 1:length(x_rx);
% xq2 = 1:1/6:length(x_rx);
% x_tx= interp1(x1, x_tx, xq1);
% x_rx= interp1(x2, x_rx, xq2);
%%

s1 = x_tx;
s2 = x_rx;
[acor,lag] = xcorr(s2,s1);
[~,I] = max(abs(acor));
lagDiff = lag(I);
timeDiff = lagDiff/Fs;
%%
function x = readcplx(filename,nsamps,nstart);
fid = fopen(filename);
fseek(fid,4 * nstart,'bof');
y = fread(fid,[2,inf],'short');
fclose(fid);
x = complex(y(1,:),y(2 , : ) ) ;
end
%%

 

[FactChecker]

I don't know. You may need to get help at your place from a computer/network knowledgeable person.
Here are some comments:

• In your system diagram, I don't see an interrupt between the transmitter clock and the receiver clock to force them to the exact same time. I only see a GPS time source feeding one clock. The GPS clock accuracy would need to be researched.
• You need to find out what is being sent by the transmitter -- a time-tagged message or an analog signal. I am not familiar with transmitter/receivers and the Ettis B210 board. Maybe someone else can help
• I can't read a data file of unknown format that you put in dropbox.

I'm afraid that I can not help you any better on this. Maybe someone who is more familiar with this subject can help.