Can't get tutorial to work with new data

In summary, the conversation is about a backtracking tutorial and code that runs well for the user. The code involves loading data, converting it to a table, and using only 15 of the 30 DJIA component stocks. The code also includes local functions for equal-weighted portfolio allocation, mean-variance portfolio allocation, inverse-variance portfolio allocation, robust portfolio allocation, and Markowitz portfolio allocation. The user is trying to run the code with different data and the code has been shortened for this purpose.
  • #1
member 428835
Hi PF!

I'm going through a backtracking tutorial here. That code runs well for me, and is below:
Matlab: 
%% LOAD DATA

% Read a table of daily adjusted close prices for 2006 DJIA stocks.
T = readtable('dowPortfolio.xlsx');

% For readability, use only 15 of the 30 DJI component stocks.
assetSymbols = ["AA","CAT","DIS","GM","HPQ","JNJ","MCD","MMM","MO","MRK","MSFT","PFE","PG","T","XOM"];

% Prune the table to hold only the dates and selected stocks.
timeColumn = "Dates";
T = T(:,[timeColumn assetSymbols]);

% Convert to the table to a timetable.
pricesTT = table2timetable(T,'RowTimes','Dates');

% View the structure of the prices timetable.
head(pricesTT)

% View the size of the asset price data set.
numSample = size(pricesTT.Variables, 1);
numAssets = size(pricesTT.Variables, 2);
table(numSample, numAssets)

% USE FIRST 40 DAYS
% This example uses the first 40 days of the data set (about 2 months) to initialize the strategies
% The backtest is then run over the remaining data (about 10 months).
warmupPeriod = 40;

% No current weights (100% cash position).
current_weights = zeros(1,numAssets);

% Warm-up partition of data set timetable.
warmupTT = pricesTT(1:warmupPeriod,:);

%%
% Compute the initial portfolio weights for each strategy.
equalWeight_initial     = equalWeightFcn(current_weights,warmupTT);
maxSharpeRatio_initial  = maxSharpeRatioFcn(current_weights,warmupTT);
inverseVariance_initial = inverseVarianceFcn(current_weights,warmupTT);
markowitz_initial       = markowitzFcn(current_weights,warmupTT);
robustOptim_initial     = robustOptimFcn(current_weights,warmupTT);

%% LOCAL FUNCTIONS

function new_weights = equalWeightFcn(current_weights, pricesTT)
% Equal-weighted portfolio allocation

nAssets = size(pricesTT, 2);
new_weights = ones(1,nAssets);
new_weights = new_weights / sum(new_weights);

end

function new_weights = maxSharpeRatioFcn(current_weights, pricesTT)
% Mean-variance portfolio allocation

nAssets = size(pricesTT, 2);
assetReturns = tick2ret(pricesTT);  % tick2ret GIVES PERCENT RETURN AS DECIMAL FROM ONE DAY TO THE NEXT
% Max 25% into a single asset (including cash)
p = Portfolio('NumAssets',nAssets,...
    'LowerBound',0,'UpperBound',0.1,...
    'LowerBudget',1,'UpperBudget',1); % Portfolio IS DICTIONARY WITH PREDEFINED KEYS
p = estimateAssetMoments(p, assetReturns{:,:}); % estimates mean & covariance of assetReturns from data for a Portfolio object
new_weights = estimateMaxSharpeRatio(p); % outputs sharp ratio

end

function new_weights = inverseVarianceFcn(current_weights, pricesTT) 
% Inverse-variance portfolio allocation
% Risk averse by weighting prop to inverse variance, or inverse uncertainty

assetReturns = tick2ret(pricesTT); % tick2ret GIVES PERCENT RETURN AS DECIMAL FROM ONE DAY TO THE NEXT
assetCov = cov(assetReturns{:,:}); % cov calculates covariance, measures amount two quantities move together or inversed
new_weights = 1 ./ diag(assetCov);
new_weights = new_weights / sum(new_weights);

end

function new_weights = robustOptimFcn(current_weights, pricesTT) 
% Robust portfolio allocation
% maximizing return and minimizing risk with fixed risk-aversion
% coefficient (lambda)

nAssets = size(pricesTT, 2);
assetReturns = tick2ret(pricesTT); % tick2ret GIVES PERCENT RETURN AS DECIMAL FROM ONE DAY TO THE NEXT

Q = cov(table2array(assetReturns));
SIGMAx = diag(diag(Q)); % make matrix of zeros with diag the cov_ii

% Robust aversion coefficient
k = 1.1;

% Robust aversion coefficient
lambda = 0.05;

rPortfolio = mean(table2array(assetReturns))'; % mean of assetReturns

% Create the optimization problem
pRobust = optimproblem('Description','Robust Portfolio');

% Define the variables
% xRobust - x  allocation vector
xRobust = optimvar('x',nAssets,1,'Type','continuous','LowerBound',0.0,'UpperBound',0.1);
zRobust = optimvar('z','LowerBound',0);

% Define the budget constraint
pRobust.Constraints.budget = sum(xRobust) == 1;

% Define the robust constraint
pRobust.Constraints.robust = xRobust'*SIGMAx*xRobust - zRobust*zRobust <=0;
pRobust.Objective = -rPortfolio'*xRobust + k*zRobust + lambda*xRobust'*Q*xRobust;
x0.x = zeros(nAssets,1);
x0.z = 0;
opt = optimoptions('fmincon','Display','off');
[solRobust,~,~] = solve(pRobust,x0,'Options',opt);
new_weights = solRobust.x;

end

function new_weights = markowitzFcn(current_weights, pricesTT) 
% Robust portfolio allocation

nAssets = size(pricesTT, 2);
assetReturns = tick2ret(pricesTT);

Q = cov(table2array(assetReturns));

% Risk aversion coefficient
lambda = 0.05;

rPortfolio = mean(table2array(assetReturns))';

% Create the optimization problem
pMrkwtz = optimproblem('Description','Markowitz Mean Variance Portfolio ');

% Define the variables
% xRobust - x  allocation vector
xMrkwtz = optimvar('x',nAssets,1,'Type','continuous','LowerBound',0.0,'UpperBound',0.1);

% Define the budget constraint
pMrkwtz.Constraints.budget = sum(xMrkwtz) == 1;

% Define the Markowitz objective
pMrkwtz.Objective = -rPortfolio'*xMrkwtz + lambda*xMrkwtz'*Q*xMrkwtz;
x0.x = zeros(nAssets,1);

opt = optimoptions('quadprog','Display','off');
[solMrkwtz,~,~] = solve(pMrkwtz,x0,'Options',opt);
new_weights = solMrkwtz.x;

end

function [buy, sell] = variableTransactionCosts(deltaPositions)
I've shortened it substantially, where I only go until we get initial portfolio weights. Now I try to run this code with different data, so the code slightly changes to this:
Matlab: 
load('T.csv')

% Convert to the table to a timetable.
pricesTT = table2timetable(T,'RowTimes','Dates');

% View the structure of the prices timetable.
head(pricesTT)

% View the size of the asset price data set.
numSample = size(pricesTT.Variables, 1);
numAssets = size(pricesTT.Variables, 2);
table(numSample, numAssets)

% USE FIRST 40 DAYS
% This example uses the first 40 days of the data set (about 2 months) to initialize the strategies
% The backtest is then run over the remaining data (about 10 months).
warmupPeriod = 40;

% No current weights (100% cash position).
current_weights = zeros(1,numAssets);

% Warm-up partition of data set timetable.
warmupTT = pricesTT(1:warmupPeriod,:);

%%
% Compute the initial portfolio weights for each strategy.
equalWeight_initial     = equalWeightFcn(current_weights,warmupTT);
maxSharpeRatio_initial  = maxSharpeRatioFcn(current_weights,warmupTT);
inverseVariance_initial = inverseVarianceFcn(current_weights,warmupTT);
markowitz_initial       = markowitzFcn(current_weights,warmupTT);
robustOptim_initial     = robustOptimFcn(current_weights,warmupTT);

%% LOCAL FUNCTIONS

function new_weights = equalWeightFcn(current_weights, pricesTT)
% Equal-weighted portfolio allocation

nAssets = size(pricesTT, 2);
new_weights = ones(1,nAssets);
new_weights = new_weights / sum(new_weights);

end

function new_weights = maxSharpeRatioFcn(current_weights, pricesTT)
% Mean-variance portfolio allocation

nAssets = size(pricesTT, 2);
assetReturns = tick2ret(pricesTT);  % tick2ret GIVES PERCENT RETURN AS DECIMAL FROM ONE DAY TO THE NEXT
% Max 25% into a single asset (including cash)
p = Portfolio('NumAssets',nAssets,...
    'LowerBound',0,'UpperBound',0.1,...
    'LowerBudget',1,'UpperBudget',1); % Portfolio IS DICTIONARY WITH PREDEFINED KEYS
p = estimateAssetMoments(p, assetReturns{:,:}); % estimates mean & covariance of assetReturns from data for a Portfolio object
new_weights = estimateMaxSharpeRatio(p); % outputs sharp ratio

end

function new_weights = inverseVarianceFcn(current_weights, pricesTT) 
% Inverse-variance portfolio allocation
% Risk averse by weighting prop to inverse variance, or inverse uncertainty

assetReturns = tick2ret(pricesTT); % tick2ret GIVES PERCENT RETURN AS DECIMAL FROM ONE DAY TO THE NEXT
assetCov = cov(assetReturns{:,:}); % cov calculates covariance, measures amount two quantities move together or inversed
new_weights = 1 ./ diag(assetCov);
new_weights = new_weights / sum(new_weights);

end

function new_weights = robustOptimFcn(current_weights, pricesTT) 
% Robust portfolio allocation
% maximizing return and minimizing risk with fixed risk-aversion
% coefficient (lambda)

nAssets = size(pricesTT, 2);
assetReturns = tick2ret(pricesTT); % tick2ret GIVES PERCENT RETURN AS DECIMAL FROM ONE DAY TO THE NEXT

Q = cov(table2array(assetReturns));
SIGMAx = diag(diag(Q)); % make matrix of zeros with diag the cov_ii

% Robust aversion coefficient
k = 1.1;

% Robust aversion coefficient
lambda = 0.05;

rPortfolio = mean(table2array(assetReturns))'; % mean of assetReturns

% Create the optimization problem
pRobust = optimproblem('Description','Robust Portfolio');

% Define the variables
% xRobust - x  allocation vector
xRobust = optimvar('x',nAssets,1,'Type','continuous','LowerBound',0.0,'UpperBound',0.1);
zRobust = optimvar('z','LowerBound',0);

% Define the budget constraint
pRobust.Constraints.budget = sum(xRobust) == 1;

% Define the robust constraint
pRobust.Constraints.robust = xRobust'*SIGMAx*xRobust - zRobust*zRobust <=0;
pRobust.Objective = -rPortfolio'*xRobust + k*zRobust + lambda*xRobust'*Q*xRobust;
x0.x = zeros(nAssets,1);
x0.z = 0;
opt = optimoptions('fmincon','Display','off');
[solRobust,~,~] = solve(pRobust,x0,'Options',opt);
new_weights = solRobust.x;

end

function new_weights = markowitzFcn(current_weights, pricesTT) 
% Robust portfolio allocation

nAssets = size(pricesTT, 2);
assetReturns = tick2ret(pricesTT);

Q = cov(table2array(assetReturns));

% Risk aversion coefficient
lambda = 0.05;

rPortfolio = mean(table2array(assetReturns))';

% Create the optimization problem
pMrkwtz = optimproblem('Description','Markowitz Mean Variance Portfolio ');

% Define the variables
% xRobust - x  allocation vector
xMrkwtz = optimvar('x',nAssets,1,'Type','continuous','LowerBound',0.0,'UpperBound',0.1);

% Define the budget constraint
pMrkwtz.Constraints.budget = sum(xMrkwtz) == 1;

% Define the Markowitz objective
pMrkwtz.Objective = -rPortfolio'*xMrkwtz + lambda*xMrkwtz'*Q*xMrkwtz;
x0.x = zeros(nAssets,1);

opt = optimoptions('quadprog','Display','off');
[solMrkwtz,~,~] = solve(pMrkwtz,x0,'Options',opt);
new_weights = solMrkwtz.x;

end

function [buy, sell] = variableTransactionCosts(deltaPositions)
but my code is choking at line 28. I've read the error message, compared my table T to the tutorial's, but can't seem to figure out the issue. Any help is greatly appreciated.
 

Attachments

  • T.csv
    15.3 KB · Views: 117
Physics news on Phys.org
  • #2
joshmccraney said:
I've read the error message,
Which is ... ?
 
  • Like
Likes berkeman
  • #3
Mark44 said:
Which is ... ?
Sorry, it says "
Error using mv_optim_transform
Portfolio set appears to be either empty or unbounded. Check constraints.

Error in Portfolio/estimateFrontierLimits>frontierLimitsContinuous (line 108)
[A, b, f0, f, H, g, d] = mv_optim_transform(obj);

Error in Portfolio/estimateFrontierLimits (line 75)
pwgt = frontierLimitsContinuous(obj, minsolution, maxsolution);

Error in Portfolio/estimateMaxSharpeRatio (line 71)
[pwgtAtMaxRet, pbuyAtMaxRet, psellAtMaxRet] = estimateFrontierLimits(obj, 'max');

Error in stocks>maxSharpeRatioFcn (line 180)
new_weights = estimateMaxSharpeRatio(p); % outputs sharp ratio

Error in stocks (line 81)
maxSharpeRatio_initial = maxSharpeRatioFcn(current_weights,warmupTT);"

For me, the MATLAB file is called stocks.m and line 81 refers to line 28 in post 1 (my code was a little bulky since I was importing and modifying data, where I just saved it for ease of reproducability above).

Line 28 calls the function maxSharpeRatioFcn, which is defined on line 44. Within that function call, the error is line 54, the estimateMaxSharpeRatio(p) function. When I googled the error it appears my portfolio p is not bounded, as when I execute estimateBounds(p) I get NAN for all seven values, where the tutorial gets 0. Any idea?
 
  • #4
Caveat: I don't have MatLab and have never worked directly with it, at least not in the past 30 years or so..
joshmccraney said:
Line 28 calls the function maxSharpeRatioFcn, which is defined on line 44. Within that function call, the error is line 54, the estimateMaxSharpeRatio(p) function. When I googled the error it appears my portfolio p is not bounded, as when I execute estimateBounds(p) I get NAN for all seven values, where the tutorial gets 0. Any idea?
I don't see estimateBounds(p) anywhere in your program, so can't comment on why it is producing NAN for whatever values it's working on.

Your program differs from the tutorial quite a lot at the start. The tutorial calls readtable() to read in an Excel spreadsheet, while your modified version calls load() to load a CSV file. The tutorial also has an array of DJI stock symbols.

If you're not using a debugger, you should start doing so. Here's a link to the mathworks documentation on how to view the value of a variable - https://www.mathworks.com/help/matlab/matlab_prog/examine-values.html.
 
  • #5
Mark44 said:
Caveat: I don't have MatLab and have never worked directly with it, at least not in the past 30 years or so..

I don't see estimateBounds(p) anywhere in your program, so can't comment on why it is producing NAN for whatever values it's working on.

Your program differs from the tutorial quite a lot at the start. The tutorial calls readtable() to read in an Excel spreadsheet, while your modified version calls load() to load a CSV file. The tutorial also has an array of DJI stock symbols.

If you're not using a debugger, you should start doing so. Here's a link to the mathworks documentation on how to view the value of a variable - https://www.mathworks.com/help/matlab/matlab_prog/examine-values.html.
Yea, the beginning is different, but that's because the data set I'm using is different (the tutorial's is built in). Once loaded, don't the two tables T have identical form and structure?

You don't see estimateBounds(p) because this was where google led me in order to debug. However, after receiving the NANs outputs, no further suggestions were given, so I'm kind of stuck.

I've used the debugger, but it doesn't look like there's a bug anywhere.
 
  • #6
joshmccraney said:
Once loaded, don't the two tables T have identical form and structure?
No idea. Where I'd start with debugging is seeing what the table looks like after it's been loaded.
 
  • #7
Mark44 said:
No idea. Where I'd start with debugging is seeing what the table looks like after it's been loaded.
My thoughts exactly. Looks identical, so I'm kinda of lost.
 
  • #8
joshmccraney said:
My thoughts exactly. Looks identical, so I'm kinda of lost.
Single step through the tutorial program and write down what it does to the variables. Do the same with your program. There has to be a difference between the two programs, otherwise your version wouldn't be throwing an error message.
 
  • #9
Mark44 said:
Single step through the tutorial program and write down what it does to the variables. Do the same with your program. There has to be a difference between the two programs, otherwise your version wouldn't be throwing an error message.
Well, this is very annoying, but evidently estimateMaxSharpeRatio only works with 10 or more assets (documentation doesn't say anything about this). I know this because when I trim the demo number from 15 to 7,8,9 I get an error but 10+ seems to be fine...very annoying.
 

1. Why is my tutorial not working with new data?

There could be several reasons why your tutorial is not working with new data. It could be due to differences in the format or structure of your new data compared to the data used in the tutorial. It could also be a compatibility issue with the software or tools used in the tutorial. Additionally, errors in your code or incorrect input data could also be the cause.

2. How can I troubleshoot the issue with the tutorial not working?

Start by carefully reviewing the tutorial instructions and making sure you have followed all the steps correctly. Check for any error messages or warnings that may provide clues about the problem. You can also try reaching out to the tutorial creator or looking for online forums or resources for troubleshooting tips.

3. Is there a specific software or tool that I need to use for the tutorial?

Most tutorials will specify the software, tools, or libraries that are required to complete the tutorial successfully. Make sure you have all the necessary programs and versions installed. If you are using different software or tools, it may not be compatible with the tutorial.

4. Can I modify the tutorial to work with my data?

Yes, you can modify the tutorial to work with your data, but it may require some adjustments to the code or steps in the tutorial. Make sure you understand the tutorial instructions thoroughly before making changes. You may also need to consult additional resources or seek help from the tutorial creator or online community.

5. What should I do if I still can't get the tutorial to work with my data?

If you have exhausted all troubleshooting options and still can't get the tutorial to work with your data, consider seeking help from the tutorial creator or an expert in the field. You can also try finding alternative tutorials or resources that may be better suited for your specific data and research goals.

Similar threads

MATLAB Frisbee flight in Matlab, can't run (Input argument x is undefined)
    • MATLAB, Maple, Mathematica, LaTeX
Replies
5
Views
5K

Hot Threads

Recent Insights

Back
Top