Can't get tutorial to work with new data

  • Context: MATLAB 
  • Thread starter Thread starter member 428835
  • Start date Start date
  • Tags Tags
    Data Tutorial Work
Click For Summary
SUMMARY

The forum discussion centers on troubleshooting a MATLAB script for portfolio optimization, specifically using the maxSharpeRatioFcn function. The user encounters an error indicating that the portfolio appears to be either empty or unbounded, which is confirmed by executing estimateBounds(p) that returns NaN values. The issue arises when attempting to estimate the maximum Sharpe ratio using the Portfolio object, which is not properly configured due to missing constraints or data inconsistencies. The user seeks assistance in resolving this error to successfully run the backtracking tutorial.

PREREQUISITES
  • Familiarity with MATLAB programming and syntax.
  • Understanding of financial portfolio optimization concepts.
  • Knowledge of MATLAB's Portfolio object and its methods.
  • Experience with data manipulation in MATLAB, particularly with tables and timetables.
NEXT STEPS
  • Investigate the configuration of the Portfolio object in MATLAB, ensuring all constraints are properly set.
  • Learn about the use of the estimateBounds function to diagnose portfolio issues in MATLAB.
  • Explore the implications of data integrity when importing datasets into MATLAB for financial analysis.
  • Review the documentation for the estimateMaxSharpeRatio function to understand its requirements and limitations.
USEFUL FOR

Data analysts, financial engineers, and MATLAB users involved in portfolio optimization and seeking to troubleshoot common errors in financial modeling scripts.

member 428835
Hi PF!

I'm going through a backtracking tutorial here. That code runs well for me, and is below:
Matlab: 
%% LOAD DATA

% Read a table of daily adjusted close prices for 2006 DJIA stocks.
T = readtable('dowPortfolio.xlsx');

% For readability, use only 15 of the 30 DJI component stocks.
assetSymbols = ["AA","CAT","DIS","GM","HPQ","JNJ","MCD","MMM","MO","MRK","MSFT","PFE","PG","T","XOM"];

% Prune the table to hold only the dates and selected stocks.
timeColumn = "Dates";
T = T(:,[timeColumn assetSymbols]);

% Convert to the table to a timetable.
pricesTT = table2timetable(T,'RowTimes','Dates');

% View the structure of the prices timetable.
head(pricesTT)

% View the size of the asset price data set.
numSample = size(pricesTT.Variables, 1);
numAssets = size(pricesTT.Variables, 2);
table(numSample, numAssets)

% USE FIRST 40 DAYS
% This example uses the first 40 days of the data set (about 2 months) to initialize the strategies
% The backtest is then run over the remaining data (about 10 months).
warmupPeriod = 40;

% No current weights (100% cash position).
current_weights = zeros(1,numAssets);

% Warm-up partition of data set timetable.
warmupTT = pricesTT(1:warmupPeriod,:);

%%
% Compute the initial portfolio weights for each strategy.
equalWeight_initial     = equalWeightFcn(current_weights,warmupTT);
maxSharpeRatio_initial  = maxSharpeRatioFcn(current_weights,warmupTT);
inverseVariance_initial = inverseVarianceFcn(current_weights,warmupTT);
markowitz_initial       = markowitzFcn(current_weights,warmupTT);
robustOptim_initial     = robustOptimFcn(current_weights,warmupTT);

%% LOCAL FUNCTIONS

function new_weights = equalWeightFcn(current_weights, pricesTT)
% Equal-weighted portfolio allocation

nAssets = size(pricesTT, 2);
new_weights = ones(1,nAssets);
new_weights = new_weights / sum(new_weights);

end

function new_weights = maxSharpeRatioFcn(current_weights, pricesTT)
% Mean-variance portfolio allocation

nAssets = size(pricesTT, 2);
assetReturns = tick2ret(pricesTT);  % tick2ret GIVES PERCENT RETURN AS DECIMAL FROM ONE DAY TO THE NEXT
% Max 25% into a single asset (including cash)
p = Portfolio('NumAssets',nAssets,...
    'LowerBound',0,'UpperBound',0.1,...
    'LowerBudget',1,'UpperBudget',1); % Portfolio IS DICTIONARY WITH PREDEFINED KEYS
p = estimateAssetMoments(p, assetReturns{:,:}); % estimates mean & covariance of assetReturns from data for a Portfolio object
new_weights = estimateMaxSharpeRatio(p); % outputs sharp ratio

end

function new_weights = inverseVarianceFcn(current_weights, pricesTT) 
% Inverse-variance portfolio allocation
% Risk averse by weighting prop to inverse variance, or inverse uncertainty

assetReturns = tick2ret(pricesTT); % tick2ret GIVES PERCENT RETURN AS DECIMAL FROM ONE DAY TO THE NEXT
assetCov = cov(assetReturns{:,:}); % cov calculates covariance, measures amount two quantities move together or inversed
new_weights = 1 ./ diag(assetCov);
new_weights = new_weights / sum(new_weights);

end

function new_weights = robustOptimFcn(current_weights, pricesTT) 
% Robust portfolio allocation
% maximizing return and minimizing risk with fixed risk-aversion
% coefficient (lambda)

nAssets = size(pricesTT, 2);
assetReturns = tick2ret(pricesTT); % tick2ret GIVES PERCENT RETURN AS DECIMAL FROM ONE DAY TO THE NEXT

Q = cov(table2array(assetReturns));
SIGMAx = diag(diag(Q)); % make matrix of zeros with diag the cov_ii

% Robust aversion coefficient
k = 1.1;

% Robust aversion coefficient
lambda = 0.05;

rPortfolio = mean(table2array(assetReturns))'; % mean of assetReturns

% Create the optimization problem
pRobust = optimproblem('Description','Robust Portfolio');

% Define the variables
% xRobust - x  allocation vector
xRobust = optimvar('x',nAssets,1,'Type','continuous','LowerBound',0.0,'UpperBound',0.1);
zRobust = optimvar('z','LowerBound',0);

% Define the budget constraint
pRobust.Constraints.budget = sum(xRobust) == 1;

% Define the robust constraint
pRobust.Constraints.robust = xRobust'*SIGMAx*xRobust - zRobust*zRobust <=0;
pRobust.Objective = -rPortfolio'*xRobust + k*zRobust + lambda*xRobust'*Q*xRobust;
x0.x = zeros(nAssets,1);
x0.z = 0;
opt = optimoptions('fmincon','Display','off');
[solRobust,~,~] = solve(pRobust,x0,'Options',opt);
new_weights = solRobust.x;

end

function new_weights = markowitzFcn(current_weights, pricesTT) 
% Robust portfolio allocation

nAssets = size(pricesTT, 2);
assetReturns = tick2ret(pricesTT);

Q = cov(table2array(assetReturns));

% Risk aversion coefficient
lambda = 0.05;

rPortfolio = mean(table2array(assetReturns))';

% Create the optimization problem
pMrkwtz = optimproblem('Description','Markowitz Mean Variance Portfolio ');

% Define the variables
% xRobust - x  allocation vector
xMrkwtz = optimvar('x',nAssets,1,'Type','continuous','LowerBound',0.0,'UpperBound',0.1);

% Define the budget constraint
pMrkwtz.Constraints.budget = sum(xMrkwtz) == 1;

% Define the Markowitz objective
pMrkwtz.Objective = -rPortfolio'*xMrkwtz + lambda*xMrkwtz'*Q*xMrkwtz;
x0.x = zeros(nAssets,1);

opt = optimoptions('quadprog','Display','off');
[solMrkwtz,~,~] = solve(pMrkwtz,x0,'Options',opt);
new_weights = solMrkwtz.x;

end

function [buy, sell] = variableTransactionCosts(deltaPositions)
I've shortened it substantially, where I only go until we get initial portfolio weights. Now I try to run this code with different data, so the code slightly changes to this:
Matlab: 
load('T.csv')

% Convert to the table to a timetable.
pricesTT = table2timetable(T,'RowTimes','Dates');

% View the structure of the prices timetable.
head(pricesTT)

% View the size of the asset price data set.
numSample = size(pricesTT.Variables, 1);
numAssets = size(pricesTT.Variables, 2);
table(numSample, numAssets)

% USE FIRST 40 DAYS
% This example uses the first 40 days of the data set (about 2 months) to initialize the strategies
% The backtest is then run over the remaining data (about 10 months).
warmupPeriod = 40;

% No current weights (100% cash position).
current_weights = zeros(1,numAssets);

% Warm-up partition of data set timetable.
warmupTT = pricesTT(1:warmupPeriod,:);

%%
% Compute the initial portfolio weights for each strategy.
equalWeight_initial     = equalWeightFcn(current_weights,warmupTT);
maxSharpeRatio_initial  = maxSharpeRatioFcn(current_weights,warmupTT);
inverseVariance_initial = inverseVarianceFcn(current_weights,warmupTT);
markowitz_initial       = markowitzFcn(current_weights,warmupTT);
robustOptim_initial     = robustOptimFcn(current_weights,warmupTT);

%% LOCAL FUNCTIONS

function new_weights = equalWeightFcn(current_weights, pricesTT)
% Equal-weighted portfolio allocation

nAssets = size(pricesTT, 2);
new_weights = ones(1,nAssets);
new_weights = new_weights / sum(new_weights);

end

function new_weights = maxSharpeRatioFcn(current_weights, pricesTT)
% Mean-variance portfolio allocation

nAssets = size(pricesTT, 2);
assetReturns = tick2ret(pricesTT);  % tick2ret GIVES PERCENT RETURN AS DECIMAL FROM ONE DAY TO THE NEXT
% Max 25% into a single asset (including cash)
p = Portfolio('NumAssets',nAssets,...
    'LowerBound',0,'UpperBound',0.1,...
    'LowerBudget',1,'UpperBudget',1); % Portfolio IS DICTIONARY WITH PREDEFINED KEYS
p = estimateAssetMoments(p, assetReturns{:,:}); % estimates mean & covariance of assetReturns from data for a Portfolio object
new_weights = estimateMaxSharpeRatio(p); % outputs sharp ratio

end

function new_weights = inverseVarianceFcn(current_weights, pricesTT) 
% Inverse-variance portfolio allocation
% Risk averse by weighting prop to inverse variance, or inverse uncertainty

assetReturns = tick2ret(pricesTT); % tick2ret GIVES PERCENT RETURN AS DECIMAL FROM ONE DAY TO THE NEXT
assetCov = cov(assetReturns{:,:}); % cov calculates covariance, measures amount two quantities move together or inversed
new_weights = 1 ./ diag(assetCov);
new_weights = new_weights / sum(new_weights);

end

function new_weights = robustOptimFcn(current_weights, pricesTT) 
% Robust portfolio allocation
% maximizing return and minimizing risk with fixed risk-aversion
% coefficient (lambda)

nAssets = size(pricesTT, 2);
assetReturns = tick2ret(pricesTT); % tick2ret GIVES PERCENT RETURN AS DECIMAL FROM ONE DAY TO THE NEXT

Q = cov(table2array(assetReturns));
SIGMAx = diag(diag(Q)); % make matrix of zeros with diag the cov_ii

% Robust aversion coefficient
k = 1.1;

% Robust aversion coefficient
lambda = 0.05;

rPortfolio = mean(table2array(assetReturns))'; % mean of assetReturns

% Create the optimization problem
pRobust = optimproblem('Description','Robust Portfolio');

% Define the variables
% xRobust - x  allocation vector
xRobust = optimvar('x',nAssets,1,'Type','continuous','LowerBound',0.0,'UpperBound',0.1);
zRobust = optimvar('z','LowerBound',0);

% Define the budget constraint
pRobust.Constraints.budget = sum(xRobust) == 1;

% Define the robust constraint
pRobust.Constraints.robust = xRobust'*SIGMAx*xRobust - zRobust*zRobust <=0;
pRobust.Objective = -rPortfolio'*xRobust + k*zRobust + lambda*xRobust'*Q*xRobust;
x0.x = zeros(nAssets,1);
x0.z = 0;
opt = optimoptions('fmincon','Display','off');
[solRobust,~,~] = solve(pRobust,x0,'Options',opt);
new_weights = solRobust.x;

end

function new_weights = markowitzFcn(current_weights, pricesTT) 
% Robust portfolio allocation

nAssets = size(pricesTT, 2);
assetReturns = tick2ret(pricesTT);

Q = cov(table2array(assetReturns));

% Risk aversion coefficient
lambda = 0.05;

rPortfolio = mean(table2array(assetReturns))';

% Create the optimization problem
pMrkwtz = optimproblem('Description','Markowitz Mean Variance Portfolio ');

% Define the variables
% xRobust - x  allocation vector
xMrkwtz = optimvar('x',nAssets,1,'Type','continuous','LowerBound',0.0,'UpperBound',0.1);

% Define the budget constraint
pMrkwtz.Constraints.budget = sum(xMrkwtz) == 1;

% Define the Markowitz objective
pMrkwtz.Objective = -rPortfolio'*xMrkwtz + lambda*xMrkwtz'*Q*xMrkwtz;
x0.x = zeros(nAssets,1);

opt = optimoptions('quadprog','Display','off');
[solMrkwtz,~,~] = solve(pMrkwtz,x0,'Options',opt);
new_weights = solMrkwtz.x;

end

function [buy, sell] = variableTransactionCosts(deltaPositions)
but my code is choking at line 28. I've read the error message, compared my table T to the tutorial's, but can't seem to figure out the issue. Any help is greatly appreciated.
 

Attachments

  • T.csv
    T.csv
    15.3 KB · Views: 212
Physics news on Phys.org
joshmccraney said:
I've read the error message,
Which is ... ?
 
  • Like
Likes   Reactions: berkeman
Mark44 said:
Which is ... ?
Sorry, it says "
Error using mv_optim_transform
Portfolio set appears to be either empty or unbounded. Check constraints.

Error in Portfolio/estimateFrontierLimits>frontierLimitsContinuous (line 108)
[A, b, f0, f, H, g, d] = mv_optim_transform(obj);

Error in Portfolio/estimateFrontierLimits (line 75)
pwgt = frontierLimitsContinuous(obj, minsolution, maxsolution);

Error in Portfolio/estimateMaxSharpeRatio (line 71)
[pwgtAtMaxRet, pbuyAtMaxRet, psellAtMaxRet] = estimateFrontierLimits(obj, 'max');

Error in stocks>maxSharpeRatioFcn (line 180)
new_weights = estimateMaxSharpeRatio(p); % outputs sharp ratio

Error in stocks (line 81)
maxSharpeRatio_initial = maxSharpeRatioFcn(current_weights,warmupTT);"

For me, the MATLAB file is called stocks.m and line 81 refers to line 28 in post 1 (my code was a little bulky since I was importing and modifying data, where I just saved it for ease of reproducability above).

Line 28 calls the function maxSharpeRatioFcn, which is defined on line 44. Within that function call, the error is line 54, the estimateMaxSharpeRatio(p) function. When I googled the error it appears my portfolio p is not bounded, as when I execute estimateBounds(p) I get NAN for all seven values, where the tutorial gets 0. Any idea?
 
Caveat: I don't have MatLab and have never worked directly with it, at least not in the past 30 years or so..
joshmccraney said:
Line 28 calls the function maxSharpeRatioFcn, which is defined on line 44. Within that function call, the error is line 54, the estimateMaxSharpeRatio(p) function. When I googled the error it appears my portfolio p is not bounded, as when I execute estimateBounds(p) I get NAN for all seven values, where the tutorial gets 0. Any idea?
I don't see estimateBounds(p) anywhere in your program, so can't comment on why it is producing NAN for whatever values it's working on.

Your program differs from the tutorial quite a lot at the start. The tutorial calls readtable() to read in an Excel spreadsheet, while your modified version calls load() to load a CSV file. The tutorial also has an array of DJI stock symbols.

If you're not using a debugger, you should start doing so. Here's a link to the mathworks documentation on how to view the value of a variable - https://www.mathworks.com/help/matlab/matlab_prog/examine-values.html.
 
Mark44 said:
Caveat: I don't have MatLab and have never worked directly with it, at least not in the past 30 years or so..

I don't see estimateBounds(p) anywhere in your program, so can't comment on why it is producing NAN for whatever values it's working on.

Your program differs from the tutorial quite a lot at the start. The tutorial calls readtable() to read in an Excel spreadsheet, while your modified version calls load() to load a CSV file. The tutorial also has an array of DJI stock symbols.

If you're not using a debugger, you should start doing so. Here's a link to the mathworks documentation on how to view the value of a variable - https://www.mathworks.com/help/matlab/matlab_prog/examine-values.html.
Yea, the beginning is different, but that's because the data set I'm using is different (the tutorial's is built in). Once loaded, don't the two tables T have identical form and structure?

You don't see estimateBounds(p) because this was where google led me in order to debug. However, after receiving the NANs outputs, no further suggestions were given, so I'm kind of stuck.

I've used the debugger, but it doesn't look like there's a bug anywhere.
 
joshmccraney said:
Once loaded, don't the two tables T have identical form and structure?
No idea. Where I'd start with debugging is seeing what the table looks like after it's been loaded.
 
Mark44 said:
No idea. Where I'd start with debugging is seeing what the table looks like after it's been loaded.
My thoughts exactly. Looks identical, so I'm kinda of lost.
 
joshmccraney said:
My thoughts exactly. Looks identical, so I'm kinda of lost.
Single step through the tutorial program and write down what it does to the variables. Do the same with your program. There has to be a difference between the two programs, otherwise your version wouldn't be throwing an error message.
 
Mark44 said:
Single step through the tutorial program and write down what it does to the variables. Do the same with your program. There has to be a difference between the two programs, otherwise your version wouldn't be throwing an error message.
Well, this is very annoying, but evidently estimateMaxSharpeRatio only works with 10 or more assets (documentation doesn't say anything about this). I know this because when I trim the demo number from 15 to 7,8,9 I get an error but 10+ seems to be fine...very annoying.