Rate limit with project.directory.get_directory_list()

[anthony-chaudhary]

Solution

1. Set init_default_directory = False when initting Project()
2. use .get() and .new() methods

project = Project(
	project_string_id = "abc",
	client_id = "abc",
	client_secret = "abc",
	init_default_directory = False
)

try:
    directory = project.directory.get(name= "string_directory_name")
except: 
    directory = project.directory.new(name= "string_directory_name")

(this is instead of manually getting list and checking matches)

Context

existing_dirs = project.directory.get_directory_list()
will result in n number of calls for every dataset in the project

because eventually it calls init_files()
https://github.com/diffgram/python-sdk/blob/main/sdk/diffgram/core/directory.py#L76

in most cases this is not an issue but if a project has something like 300 datasets then every call is 300 x (number of calls in init_files)
so at least 300.

Future

1. A "best practice" example of doing the above instead of iterating over list?
2. Something like adding init false flag in the list concept too, e.g. so depending on use case if person doesn't need files

More generally we may need to reflect on "lazy loading" / expectations around one SDK call being "about" one API call