Portfolio optimization bug fixes #3675

montezdesousa · 2022-12-01T15:23:45Z

Description

Updated portfolio optimization menu scripts
Redirected params menu save to /OpenBBUserData/portfolio/optimization and added to load paths
Fix user warning when tries to plot --heat with just one value in category
Adds autocomplete with available portfolios for show, rpf and plot
Increase tolerance to avoid frequent "MaxUtil is returning a $ instead of %"
Fixed bug with boolean parameters being incorrectly parsed in SDK
Fixes issues in the comments of Fix portfolio optimization dates parameters bug #3619
Fixes [Bug] load from portfolio/po is missing its examples files #3640
Fixes [Bug] Portfolio/PO - autocomplete goes weird after loading a parameters file, then terminal crashes. #3613
Fixes [Bug] Portfolio/po - maxdecorr -p max - Terminal Crashes #3594
Fixes [IMPROVE] Change decimals format to percentage in parameters Excel template #3622

How has this been tested?

Terminal:
Run the portfolio optimization scripts in openbb_terminal/miscellaneous/scripts/portfolio
Try portfolio/po/params commands load and save

SDK:

from openbb_terminal.sdk import openbb

d = {
            "SECTOR": {
                "AAPL": "INFORMATION TECHNOLOGY",
                "MSFT": "INFORMATION TECHNOLOGY",
                "AMZN": "CONSUMER DISCRETIONARY",
            },
            "CURRENCY": {
                "AAPL": "USD",
                "MSFT": "USD",
                "AMZN": "USD",
            },
            "CURRENT_INVESTED_AMOUNT": {
                "AAPL": "100000.0",
                "MSFT": "200000.0",
                "AMZN": "300000.0",
            },
        }

p = openbb.portfolio.po.load(symbols_categories=d)
openbb.portfolio.po.maxsharpe(portfolio_engine=p)
openbb.portfolio.po.minrisk(portfolio_engine=p)
openbb.portfolio.po.maxutil(portfolio_engine=p)
openbb.portfolio.po.maxret(portfolio_engine=p)
openbb.portfolio.po.maxdiv(portfolio_engine=p)
openbb.portfolio.po.maxdecorr(portfolio_engine=p)
openbb.portfolio.po.blacklitterman(portfolio_engine=p)
openbb.portfolio.po.ef(portfolio_engine=p)
openbb.portfolio.po.ef_chart(portfolio_engine=p)
openbb.portfolio.po.riskparity(portfolio_engine=p)
openbb.portfolio.po.relriskparity(portfolio_engine=p)
openbb.portfolio.po.hrp(portfolio_engine=p)
openbb.portfolio.po.herc(portfolio_engine=p)
openbb.portfolio.po.nco(portfolio_engine=p)
openbb.portfolio.po.equal(portfolio_engine=p)
openbb.portfolio.po.mktcap(portfolio_engine=p)
openbb.portfolio.po.dividend(portfolio_engine=p)
openbb.portfolio.po.property(portfolio_engine=p, prop="forwardPE")
openbb.portfolio.po.plot(p, category="SECTOR", chart_type="pie")
openbb.portfolio.po.plot(p, category="SECTOR", chart_type="hist")
openbb.portfolio.po.plot(p, category="SECTOR", chart_type="dd")
openbb.portfolio.po.plot(p, category="SECTOR", chart_type="rc")
openbb.portfolio.po.plot(p, category="SECTOR", chart_type="heat")
openbb.portfolio.po.plot(p, category="CURRENCY", chart_type="heat")

montezdesousa · 2022-12-01T18:42:41Z

@deeleeramone can you test this to see if the bugs you found are fixed pls?

deeleeramone · 2022-12-01T23:09:37Z

Will take a look at this shortly, thanks!

deeleeramone · 2022-12-02T04:11:32Z

Amazing, @montezdesousa!! We're cooking with gas now! The ability to modify parameters in the terminal makes a much smoother user-experience, and now because you don't have leave the terminal, fine adjustments can be incorporated into routines!

Haven't yet tested with the SDK layer, but in the Terminal it is not crashing anymore and the argparse bug is gone. Error handling is also much more graceful.

I ran into a few minor things. First, a typo here in the output print, MaxUtil is returning a $ instead of %.

Secondly, I can't seem to make the window large enough to be able to have the title and axes labels on screen with plot --heat. In the image below, my screen is maxed out and I'm not able to get a usable image from it.

Zooming out doesn't help either.

If the syntax is incorrect, or using it wrong, plot --heat will print a big screen of nested dictionaries belonging to the portfolio file.

Thirdly, /po/params load does not see the OpenBBUserData folder.

Then a few comments, and I realize these fall outside of the scope of this PR; so, do with them as you will.

If I am remembering correctly, there was a previous effort to clean up all arguments, terminal-wide, so that - are assigned to only single letters; and arguments which can't reasonably be abbreviated as one letter should convert to being only -- with the full argument name. I'm pretty sure there was a specific reason for this, argparse-related, but it was a little while back.

Using the S&P 500 example, optimizing will show lots of 0.00 % allocations, but intuitively I know they are not actually zeros. There are +/- 0 % allocations, and you are unable to distinguish what this means. The user is also not able to verify if rounded numbers are being used for downstream calculations, which would create compounding errors. 0 is a much different outcome than 0.004.

I personally see no benefit to limiting the number of decimal places displayed, especially when the value is < 1. Displaying 1,000,000 is the same number of characters as 0.0000001. If the number is > 1, then a reasonable number of decimals might be 2 - 6, with less decimals as the number grows - i.e., 10, 100, 1000, 100000. For example, stocks and options are quoted on exchange on a sub-penny level; many crypto assets have a ridiculous number of zeros before the first integer.

Lastly, there must be a way to export the weightings of optimized portfolios for use in the main portfolio menu and elsewhere, or for recall. In the other direction, categories are already known by the main portfolio engine after a transactions file has been loaded. These two menus need to work together instead of independently.

montezdesousa · 2022-12-02T09:13:03Z

Amazing, @montezdesousa!! We're cooking with gas now! The ability to modify parameters in the terminal makes a much smoother user-experience, and now because you don't have leave the terminal, fine adjustments can be incorporated into routines!

Haven't yet tested with the SDK layer, but in the Terminal it is not crashing anymore and the argparse bug is gone. Error handling is also much more graceful.

I ran into a few minor things. First, a typo here in the output print, MaxUtil is returning a $ instead of %.

Secondly, I can't seem to make the window large enough to be able to have the title and axes labels on screen with plot --heat. In the image below, my screen is maxed out and I'm not able to get a usable image from it.

Zooming out doesn't help either.

If the syntax is incorrect, or using it wrong, plot --heat will print a big screen of nested dictionaries belonging to the portfolio file.

Thirdly, /po/params load does not see the OpenBBUserData folder.

Then a few comments, and I realize these fall outside of the scope of this PR; so, do with them as you will.

If I am remembering correctly, there was a previous effort to clean up all arguments, terminal-wide, so that - are assigned to only single letters; and arguments which can't reasonably be abbreviated as one letter should convert to being only -- with the full argument name. I'm pretty sure there was a specific reason for this, argparse-related, but it was a little while back.

Using the S&P 500 example, optimizing will show lots of 0.00 % allocations, but intuitively I know they are not actually zeros. There are +/- 0 % allocations, and you are unable to distinguish what this means. The user is also not able to verify if rounded numbers are being used for downstream calculations, which would create compounding errors. 0 is a much different outcome than 0.004.

I personally see no benefit to limiting the number of decimal places displayed, especially when the value is < 1. Displaying 1,000,000 is the same number of characters as 0.0000001. If the number is > 1, then a reasonable number of decimals might be 2 - 6, with less decimals as the number grows - i.e., 10, 100, 1000, 100000. For example, stocks and options are quoted on exchange on a sub-penny level; many crypto assets have a ridiculous number of zeros before the first integer.

Lastly, there must be a way to export the weightings of optimized portfolios for use in the main portfolio menu and elsewhere, or for recall. In the other direction, categories are already known by the main portfolio engine after a transactions file has been loaded. These two menus need to work together instead of independently.

Thanks for your feedback! Taking a look

montezdesousa · 2022-12-02T10:37:01Z

Can you describe the steps to reproduce this pls?

Edit: No need. Oddly, we are assuming that the values are in $ terms if the weights don't add up to 100% (with slight tolerance). I increase the tolerance to avoid this too often, but the logic itself does not make much sense to me. Are you aware of optimization methods in the menu that deal with actual $ instead of %?

If not, I'd include this issue in the decimals display issue I quoted below so that we address it later.

montezdesousa · 2022-12-02T11:33:54Z

Regarding these points below, can you open specific issues so they don't get lost here in the comments? Already opened the arguments with more than one letter when '-' #3687

Quote:
Using the S&P 500 example, optimizing will show lots of 0.00 % allocations, but intuitively I know they are not actually zeros. There are +/- 0 % allocations, and you are unable to distinguish what this means. The user is also not able to verify if rounded numbers are being used for downstream calculations, which would create compounding errors. 0 is a much different outcome than 0.004.

Screenshot 2022-12-01 at 6 32 31 PM

I personally see no benefit to limiting the number of decimal places displayed, especially when the value is < 1. Displaying 1,000,000 is the same number of characters as 0.0000001. If the number is > 1, then a reasonable number of decimals might be 2 - 6, with less decimals as the number grows - i.e., 10, 100, 1000, 100000. For example, stocks and options are quoted on exchange on a sub-penny level; many crypto assets have a ridiculous number of zeros before the first integer.

Lastly, there must be a way to export the weightings of optimized portfolios for use in the main portfolio menu and elsewhere, or for recall. In the other direction, categories are already known by the main portfolio engine after a transactions file has been loaded. These two menus need to work together instead of independently.

montezdesousa · 2022-12-02T13:32:33Z

Can you open a separate issue about this plot window thing? In my end it works fine when I maximize the window, not sure what's happening there tbh

deeleeramone · 2022-12-02T18:46:37Z

I don't think there are any models returning $ amounts. Will add to the decimal issue.

montezdesousa added 30 commits November 28, 2022 18:40

remove hcp

0aa9283

Merge branch 'main' of github.com:OpenBB-finance/OpenBBTerminal

923405b

add prams dict to statics

71eeab0

little controller bug

b4e8d8e

fix view bug

b4ee549

avoid crash if one stock

f1a7396

check params on setup

0afeb4f

Merge branch 'main' into po_toolkit

ed0890b

create parameter class

fa9d72c

check and convert parameters using class

a58d4c2

change params name to avoid confusion

136156d

create a parameter statics dict

d00a31e

change some funcs names

0822f73

remove unused imports

8cd9af0

remove default dict

92404cf

optional type

9133754

add multi choices

b70e8c1

cast only int and float

d28adcd

fix completer

cb64c6e

fix bugs with parameter validation

d13fd92

fix bugs with parameter validation

32d95b0

add excel formatting

6ecdd2c

sdk needs mapping as well, controller takes care of this in terminal

19391ca

small formating

f38aab6

add some safe guard try except

bd11405

controller bugs

a510800

oops

2caf49a

change export path of parameters to portfolio folder

be2151f

add more commands to scripts

5f62c91

Merge branch 'main' into po_toolkit

316858f

montezdesousa added 3 commits December 1, 2022 18:29

black

0f22687

flake8

0b641d2

forgot this

410d5c6

pylint

2732174

montezdesousa marked this pull request as ready for review December 1, 2022 19:01

montezdesousa added the do not merge Label to prevent pull request merge label Dec 1, 2022

montezdesousa added 3 commits December 1, 2022 20:20

change defaults

13b7a42

fix ef default

36a4581

fix bl

87f36ae

montezdesousa removed the do not merge Label to prevent pull request merge label Dec 1, 2022

montezdesousa added 2 commits December 1, 2022 22:03

sync sdk defaults

64eea79

sync last defaults

e3b0e9b

Merge branch 'main' into po_toolkit

349f266

fix plot heat and add more choices to controller autocomplete

838d0e9

montezdesousa added 2 commits December 2, 2022 11:56

patch weights

d638e2e

fix wrong bool parsing

9209003

montezdesousa requested a review from jmaslek December 2, 2022 14:01

deeleeramone mentioned this pull request Dec 2, 2022

[Bug] Portfolio/PO - MaxUtil returns value as $ when weights do not add up to 100% #3689

Closed

Merge branch 'main' into po_toolkit

65cc48d

jmaslek approved these changes Dec 2, 2022

View reviewed changes

montezdesousa merged commit faca7ab into main Dec 2, 2022

montezdesousa deleted the po_toolkit branch December 2, 2022 23:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Portfolio optimization bug fixes #3675

Portfolio optimization bug fixes #3675

montezdesousa commented Dec 1, 2022 •

edited

Loading

montezdesousa commented Dec 1, 2022

deeleeramone commented Dec 1, 2022

deeleeramone commented Dec 2, 2022

montezdesousa commented Dec 2, 2022

montezdesousa commented Dec 2, 2022 •

edited

Loading

montezdesousa commented Dec 2, 2022 •

edited

Loading

montezdesousa commented Dec 2, 2022 •

edited

Loading

deeleeramone commented Dec 2, 2022

Portfolio optimization bug fixes #3675

Portfolio optimization bug fixes #3675

Conversation

montezdesousa commented Dec 1, 2022 • edited Loading

Description

How has this been tested?

montezdesousa commented Dec 1, 2022

deeleeramone commented Dec 1, 2022

deeleeramone commented Dec 2, 2022

montezdesousa commented Dec 2, 2022

montezdesousa commented Dec 2, 2022 • edited Loading

montezdesousa commented Dec 2, 2022 • edited Loading

montezdesousa commented Dec 2, 2022 • edited Loading

deeleeramone commented Dec 2, 2022

montezdesousa commented Dec 1, 2022 •

edited

Loading

montezdesousa commented Dec 2, 2022 •

edited

Loading

montezdesousa commented Dec 2, 2022 •

edited

Loading

montezdesousa commented Dec 2, 2022 •

edited

Loading