Can't get tutorial to work with new data

  • Context: MATLAB 
  • Thread starter Thread starter member 428835
  • Start date Start date
  • Tags Tags
    Data Tutorial Work
Click For Summary

Discussion Overview

The discussion revolves around issues encountered while adapting a backtracking tutorial for portfolio optimization to work with new data. Participants are examining the code and its functionality, specifically focusing on the initialization of portfolio weights and the handling of different data formats.

Discussion Character

  • Technical explanation
  • Exploratory
  • Homework-related

Main Points Raised

  • One participant shares a code snippet that successfully initializes portfolio weights using a specific dataset and outlines the functions used for various portfolio allocation strategies.
  • The same participant attempts to modify the code to work with a different dataset loaded from a CSV file, indicating changes made to the data handling process.
  • There is mention of the need to ensure that the new data is structured similarly to the original data to avoid errors in the portfolio optimization functions.
  • Some functions, such as tick2ret and cov, are highlighted for their roles in calculating returns and covariance, which are crucial for the optimization process.
  • Participants express uncertainty about whether the modifications will yield the same results as the original tutorial due to potential differences in data structure.

Areas of Agreement / Disagreement

Participants have not reached a consensus on whether the modified code will function correctly with the new data. There are indications of uncertainty regarding the compatibility of the new dataset with the existing functions.

Contextual Notes

Limitations include potential differences in data structure between the original dataset and the new CSV file, which may affect the performance of the portfolio optimization functions. The discussion does not resolve these issues.

member 428835
Hi PF!

I'm going through a backtracking tutorial here. That code runs well for me, and is below:
Matlab: 
%% LOAD DATA

% Read a table of daily adjusted close prices for 2006 DJIA stocks.
T = readtable('dowPortfolio.xlsx');

% For readability, use only 15 of the 30 DJI component stocks.
assetSymbols = ["AA","CAT","DIS","GM","HPQ","JNJ","MCD","MMM","MO","MRK","MSFT","PFE","PG","T","XOM"];

% Prune the table to hold only the dates and selected stocks.
timeColumn = "Dates";
T = T(:,[timeColumn assetSymbols]);

% Convert to the table to a timetable.
pricesTT = table2timetable(T,'RowTimes','Dates');

% View the structure of the prices timetable.
head(pricesTT)

% View the size of the asset price data set.
numSample = size(pricesTT.Variables, 1);
numAssets = size(pricesTT.Variables, 2);
table(numSample, numAssets)

% USE FIRST 40 DAYS
% This example uses the first 40 days of the data set (about 2 months) to initialize the strategies
% The backtest is then run over the remaining data (about 10 months).
warmupPeriod = 40;

% No current weights (100% cash position).
current_weights = zeros(1,numAssets);

% Warm-up partition of data set timetable.
warmupTT = pricesTT(1:warmupPeriod,:);

%%
% Compute the initial portfolio weights for each strategy.
equalWeight_initial     = equalWeightFcn(current_weights,warmupTT);
maxSharpeRatio_initial  = maxSharpeRatioFcn(current_weights,warmupTT);
inverseVariance_initial = inverseVarianceFcn(current_weights,warmupTT);
markowitz_initial       = markowitzFcn(current_weights,warmupTT);
robustOptim_initial     = robustOptimFcn(current_weights,warmupTT);

%% LOCAL FUNCTIONS

function new_weights = equalWeightFcn(current_weights, pricesTT)
% Equal-weighted portfolio allocation

nAssets = size(pricesTT, 2);
new_weights = ones(1,nAssets);
new_weights = new_weights / sum(new_weights);

end

function new_weights = maxSharpeRatioFcn(current_weights, pricesTT)
% Mean-variance portfolio allocation

nAssets = size(pricesTT, 2);
assetReturns = tick2ret(pricesTT);  % tick2ret GIVES PERCENT RETURN AS DECIMAL FROM ONE DAY TO THE NEXT
% Max 25% into a single asset (including cash)
p = Portfolio('NumAssets',nAssets,...
    'LowerBound',0,'UpperBound',0.1,...
    'LowerBudget',1,'UpperBudget',1); % Portfolio IS DICTIONARY WITH PREDEFINED KEYS
p = estimateAssetMoments(p, assetReturns{:,:}); % estimates mean & covariance of assetReturns from data for a Portfolio object
new_weights = estimateMaxSharpeRatio(p); % outputs sharp ratio

end

function new_weights = inverseVarianceFcn(current_weights, pricesTT) 
% Inverse-variance portfolio allocation
% Risk averse by weighting prop to inverse variance, or inverse uncertainty

assetReturns = tick2ret(pricesTT); % tick2ret GIVES PERCENT RETURN AS DECIMAL FROM ONE DAY TO THE NEXT
assetCov = cov(assetReturns{:,:}); % cov calculates covariance, measures amount two quantities move together or inversed
new_weights = 1 ./ diag(assetCov);
new_weights = new_weights / sum(new_weights);

end

function new_weights = robustOptimFcn(current_weights, pricesTT) 
% Robust portfolio allocation
% maximizing return and minimizing risk with fixed risk-aversion
% coefficient (lambda)

nAssets = size(pricesTT, 2);
assetReturns = tick2ret(pricesTT); % tick2ret GIVES PERCENT RETURN AS DECIMAL FROM ONE DAY TO THE NEXT

Q = cov(table2array(assetReturns));
SIGMAx = diag(diag(Q)); % make matrix of zeros with diag the cov_ii

% Robust aversion coefficient
k = 1.1;

% Robust aversion coefficient
lambda = 0.05;

rPortfolio = mean(table2array(assetReturns))'; % mean of assetReturns

% Create the optimization problem
pRobust = optimproblem('Description','Robust Portfolio');

% Define the variables
% xRobust - x  allocation vector
xRobust = optimvar('x',nAssets,1,'Type','continuous','LowerBound',0.0,'UpperBound',0.1);
zRobust = optimvar('z','LowerBound',0);

% Define the budget constraint
pRobust.Constraints.budget = sum(xRobust) == 1;

% Define the robust constraint
pRobust.Constraints.robust = xRobust'*SIGMAx*xRobust - zRobust*zRobust <=0;
pRobust.Objective = -rPortfolio'*xRobust + k*zRobust + lambda*xRobust'*Q*xRobust;
x0.x = zeros(nAssets,1);
x0.z = 0;
opt = optimoptions('fmincon','Display','off');
[solRobust,~,~] = solve(pRobust,x0,'Options',opt);
new_weights = solRobust.x;

end

function new_weights = markowitzFcn(current_weights, pricesTT) 
% Robust portfolio allocation

nAssets = size(pricesTT, 2);
assetReturns = tick2ret(pricesTT);

Q = cov(table2array(assetReturns));

% Risk aversion coefficient
lambda = 0.05;

rPortfolio = mean(table2array(assetReturns))';

% Create the optimization problem
pMrkwtz = optimproblem('Description','Markowitz Mean Variance Portfolio ');

% Define the variables
% xRobust - x  allocation vector
xMrkwtz = optimvar('x',nAssets,1,'Type','continuous','LowerBound',0.0,'UpperBound',0.1);

% Define the budget constraint
pMrkwtz.Constraints.budget = sum(xMrkwtz) == 1;

% Define the Markowitz objective
pMrkwtz.Objective = -rPortfolio'*xMrkwtz + lambda*xMrkwtz'*Q*xMrkwtz;
x0.x = zeros(nAssets,1);

opt = optimoptions('quadprog','Display','off');
[solMrkwtz,~,~] = solve(pMrkwtz,x0,'Options',opt);
new_weights = solMrkwtz.x;

end

function [buy, sell] = variableTransactionCosts(deltaPositions)
I've shortened it substantially, where I only go until we get initial portfolio weights. Now I try to run this code with different data, so the code slightly changes to this:
Matlab: 
load('T.csv')

% Convert to the table to a timetable.
pricesTT = table2timetable(T,'RowTimes','Dates');

% View the structure of the prices timetable.
head(pricesTT)

% View the size of the asset price data set.
numSample = size(pricesTT.Variables, 1);
numAssets = size(pricesTT.Variables, 2);
table(numSample, numAssets)

% USE FIRST 40 DAYS
% This example uses the first 40 days of the data set (about 2 months) to initialize the strategies
% The backtest is then run over the remaining data (about 10 months).
warmupPeriod = 40;

% No current weights (100% cash position).
current_weights = zeros(1,numAssets);

% Warm-up partition of data set timetable.
warmupTT = pricesTT(1:warmupPeriod,:);

%%
% Compute the initial portfolio weights for each strategy.
equalWeight_initial     = equalWeightFcn(current_weights,warmupTT);
maxSharpeRatio_initial  = maxSharpeRatioFcn(current_weights,warmupTT);
inverseVariance_initial = inverseVarianceFcn(current_weights,warmupTT);
markowitz_initial       = markowitzFcn(current_weights,warmupTT);
robustOptim_initial     = robustOptimFcn(current_weights,warmupTT);

%% LOCAL FUNCTIONS

function new_weights = equalWeightFcn(current_weights, pricesTT)
% Equal-weighted portfolio allocation

nAssets = size(pricesTT, 2);
new_weights = ones(1,nAssets);
new_weights = new_weights / sum(new_weights);

end

function new_weights = maxSharpeRatioFcn(current_weights, pricesTT)
% Mean-variance portfolio allocation

nAssets = size(pricesTT, 2);
assetReturns = tick2ret(pricesTT);  % tick2ret GIVES PERCENT RETURN AS DECIMAL FROM ONE DAY TO THE NEXT
% Max 25% into a single asset (including cash)
p = Portfolio('NumAssets',nAssets,...
    'LowerBound',0,'UpperBound',0.1,...
    'LowerBudget',1,'UpperBudget',1); % Portfolio IS DICTIONARY WITH PREDEFINED KEYS
p = estimateAssetMoments(p, assetReturns{:,:}); % estimates mean & covariance of assetReturns from data for a Portfolio object
new_weights = estimateMaxSharpeRatio(p); % outputs sharp ratio

end

function new_weights = inverseVarianceFcn(current_weights, pricesTT) 
% Inverse-variance portfolio allocation
% Risk averse by weighting prop to inverse variance, or inverse uncertainty

assetReturns = tick2ret(pricesTT); % tick2ret GIVES PERCENT RETURN AS DECIMAL FROM ONE DAY TO THE NEXT
assetCov = cov(assetReturns{:,:}); % cov calculates covariance, measures amount two quantities move together or inversed
new_weights = 1 ./ diag(assetCov);
new_weights = new_weights / sum(new_weights);

end

function new_weights = robustOptimFcn(current_weights, pricesTT) 
% Robust portfolio allocation
% maximizing return and minimizing risk with fixed risk-aversion
% coefficient (lambda)

nAssets = size(pricesTT, 2);
assetReturns = tick2ret(pricesTT); % tick2ret GIVES PERCENT RETURN AS DECIMAL FROM ONE DAY TO THE NEXT

Q = cov(table2array(assetReturns));
SIGMAx = diag(diag(Q)); % make matrix of zeros with diag the cov_ii

% Robust aversion coefficient
k = 1.1;

% Robust aversion coefficient
lambda = 0.05;

rPortfolio = mean(table2array(assetReturns))'; % mean of assetReturns

% Create the optimization problem
pRobust = optimproblem('Description','Robust Portfolio');

% Define the variables
% xRobust - x  allocation vector
xRobust = optimvar('x',nAssets,1,'Type','continuous','LowerBound',0.0,'UpperBound',0.1);
zRobust = optimvar('z','LowerBound',0);

% Define the budget constraint
pRobust.Constraints.budget = sum(xRobust) == 1;

% Define the robust constraint
pRobust.Constraints.robust = xRobust'*SIGMAx*xRobust - zRobust*zRobust <=0;
pRobust.Objective = -rPortfolio'*xRobust + k*zRobust + lambda*xRobust'*Q*xRobust;
x0.x = zeros(nAssets,1);
x0.z = 0;
opt = optimoptions('fmincon','Display','off');
[solRobust,~,~] = solve(pRobust,x0,'Options',opt);
new_weights = solRobust.x;

end

function new_weights = markowitzFcn(current_weights, pricesTT) 
% Robust portfolio allocation

nAssets = size(pricesTT, 2);
assetReturns = tick2ret(pricesTT);

Q = cov(table2array(assetReturns));

% Risk aversion coefficient
lambda = 0.05;

rPortfolio = mean(table2array(assetReturns))';

% Create the optimization problem
pMrkwtz = optimproblem('Description','Markowitz Mean Variance Portfolio ');

% Define the variables
% xRobust - x  allocation vector
xMrkwtz = optimvar('x',nAssets,1,'Type','continuous','LowerBound',0.0,'UpperBound',0.1);

% Define the budget constraint
pMrkwtz.Constraints.budget = sum(xMrkwtz) == 1;

% Define the Markowitz objective
pMrkwtz.Objective = -rPortfolio'*xMrkwtz + lambda*xMrkwtz'*Q*xMrkwtz;
x0.x = zeros(nAssets,1);

opt = optimoptions('quadprog','Display','off');
[solMrkwtz,~,~] = solve(pMrkwtz,x0,'Options',opt);
new_weights = solMrkwtz.x;

end

function [buy, sell] = variableTransactionCosts(deltaPositions)
but my code is choking at line 28. I've read the error message, compared my table T to the tutorial's, but can't seem to figure out the issue. Any help is greatly appreciated.
 

Attachments

  • T.csv
    T.csv
    15.3 KB · Views: 216
Physics news on Phys.org
joshmccraney said:
I've read the error message,
Which is ... ?
 
  • Like
Likes   Reactions: berkeman
Mark44 said:
Which is ... ?
Sorry, it says "
Error using mv_optim_transform
Portfolio set appears to be either empty or unbounded. Check constraints.

Error in Portfolio/estimateFrontierLimits>frontierLimitsContinuous (line 108)
[A, b, f0, f, H, g, d] = mv_optim_transform(obj);

Error in Portfolio/estimateFrontierLimits (line 75)
pwgt = frontierLimitsContinuous(obj, minsolution, maxsolution);

Error in Portfolio/estimateMaxSharpeRatio (line 71)
[pwgtAtMaxRet, pbuyAtMaxRet, psellAtMaxRet] = estimateFrontierLimits(obj, 'max');

Error in stocks>maxSharpeRatioFcn (line 180)
new_weights = estimateMaxSharpeRatio(p); % outputs sharp ratio

Error in stocks (line 81)
maxSharpeRatio_initial = maxSharpeRatioFcn(current_weights,warmupTT);"

For me, the MATLAB file is called stocks.m and line 81 refers to line 28 in post 1 (my code was a little bulky since I was importing and modifying data, where I just saved it for ease of reproducability above).

Line 28 calls the function maxSharpeRatioFcn, which is defined on line 44. Within that function call, the error is line 54, the estimateMaxSharpeRatio(p) function. When I googled the error it appears my portfolio p is not bounded, as when I execute estimateBounds(p) I get NAN for all seven values, where the tutorial gets 0. Any idea?
 
Caveat: I don't have MatLab and have never worked directly with it, at least not in the past 30 years or so..
joshmccraney said:
Line 28 calls the function maxSharpeRatioFcn, which is defined on line 44. Within that function call, the error is line 54, the estimateMaxSharpeRatio(p) function. When I googled the error it appears my portfolio p is not bounded, as when I execute estimateBounds(p) I get NAN for all seven values, where the tutorial gets 0. Any idea?
I don't see estimateBounds(p) anywhere in your program, so can't comment on why it is producing NAN for whatever values it's working on.

Your program differs from the tutorial quite a lot at the start. The tutorial calls readtable() to read in an Excel spreadsheet, while your modified version calls load() to load a CSV file. The tutorial also has an array of DJI stock symbols.

If you're not using a debugger, you should start doing so. Here's a link to the mathworks documentation on how to view the value of a variable - https://www.mathworks.com/help/matlab/matlab_prog/examine-values.html.
 
Mark44 said:
Caveat: I don't have MatLab and have never worked directly with it, at least not in the past 30 years or so..

I don't see estimateBounds(p) anywhere in your program, so can't comment on why it is producing NAN for whatever values it's working on.

Your program differs from the tutorial quite a lot at the start. The tutorial calls readtable() to read in an Excel spreadsheet, while your modified version calls load() to load a CSV file. The tutorial also has an array of DJI stock symbols.

If you're not using a debugger, you should start doing so. Here's a link to the mathworks documentation on how to view the value of a variable - https://www.mathworks.com/help/matlab/matlab_prog/examine-values.html.
Yea, the beginning is different, but that's because the data set I'm using is different (the tutorial's is built in). Once loaded, don't the two tables T have identical form and structure?

You don't see estimateBounds(p) because this was where google led me in order to debug. However, after receiving the NANs outputs, no further suggestions were given, so I'm kind of stuck.

I've used the debugger, but it doesn't look like there's a bug anywhere.
 
joshmccraney said:
Once loaded, don't the two tables T have identical form and structure?
No idea. Where I'd start with debugging is seeing what the table looks like after it's been loaded.
 
Mark44 said:
No idea. Where I'd start with debugging is seeing what the table looks like after it's been loaded.
My thoughts exactly. Looks identical, so I'm kinda of lost.
 
joshmccraney said:
My thoughts exactly. Looks identical, so I'm kinda of lost.
Single step through the tutorial program and write down what it does to the variables. Do the same with your program. There has to be a difference between the two programs, otherwise your version wouldn't be throwing an error message.
 
Mark44 said:
Single step through the tutorial program and write down what it does to the variables. Do the same with your program. There has to be a difference between the two programs, otherwise your version wouldn't be throwing an error message.
Well, this is very annoying, but evidently estimateMaxSharpeRatio only works with 10 or more assets (documentation doesn't say anything about this). I know this because when I trim the demo number from 15 to 7,8,9 I get an error but 10+ seems to be fine...very annoying.