Dear fairy: first desires

[mischaschaub]

Hi, you made more than my day. This is such a lovely tool that you created. Some desires that would make my life as a user a bit easier. I plan to add points to this list during the coming days.

1. it would be great to be able so output a higher resolution - would 1920 x 1080 be feasible, if I run it on a A100 ?
2. I would love to be able to adapt the number of frames per prompt, and be able to influence other features within a prompt. So I would love to see a public prompt-part (for the text line) and an insider prompt-part to steer the visual qualities, like the number of steps, colorization etc).
3. I just love the painterly qualities of the output - all kinds of variables to influence this would be great.
4. I am starting to think of a longer movie project with your tool, based on many prompt groups, and would love to insert chapter title between prompt groups, like this was done in silent movies (of course I could do this in a video editor, but it would be nice as an integrated feature).

Keep up the good work!
Mischa

[pschaldenbrand]

Hi @mischaschaub Thanks so much for these suggestions, and I'm glad you enjoy the tool :)

1. The post processing step starts to break down with resolutions above around 720p. It "works" with arbitrarily large resolutions, but the quality is not nearly as good. If you need higher resolutions, I would suggest using an AI upscaler for the time being, and I will try to think of ways to produce higher resolutions with good quality.
2. This is a great suggestion. I will work on updating the code to support the ability to specify how many frames each prompt gets. As a work around, right now, you could duplicate prompts if you want to give them more frames.
3. I'm glad you like the painterly effects. I enjoy it too. Unfortunately the method is not super flexible in the sense that there are no parameters to tune to try to get different appearances/effects... But I'd really like to add this feature too, so I will try to come up with a way to support it.
4. This is very exciting!! I'd like to support in any way I can. I can try to work on this as a feature. If you would, could you also reach out to me via email? I'd like to hear more about your movie idea: pschalde@andrew.cmu.edu

[mischaschaub]

Hi, thanks for your kind answer. So let's go back to our list and extend this a bit:

1. Will try different upscalers. If you could output a 16:9 proportioned picture as an option this would be VERY helpful for my movie project.
2. For creators the concept of txt2vid is utterly fascinating, but less so for viewers. If a viewer sees two persons quarrelling in a movie, he/she does not want to read that there are two quarrelling persons, but what they are quarrelling about. Of course sometimes it may be great fun, to read what you see, but not always. Your prompt structure should allow such choice. If you could offer some choice to select another font or letter size this might be nice. Another suggestion: I would love to render in a first run just one test pix per prompt, and I would love to be able to finetune its variables to my desires, as possible with https://github.com/amotile/stable-diffusion-studio , before I start rendering out all pix. Such a first overview would be helpful.
3. As a special desire about looks: all kinds of black and white would be great! Long live Ilford HP5 Plus...
4. I just started with a naive movie called "what's wrong with the world?" – and now I see that I plan to have a voice-over rap of my sad world view as the basic structure, along which I would like to position the keyframes of the prompts. So I hope to have some driving energy from the sound to pull the movie through its time.
5. Sometimes it might be useful to be able to choose a jump-cutting prompt between two text prompts, otherwise everything becomes an endless stream of transforming continuity, what quickly may become boring.
6. I am a bit worried that your basic simplified creation-process might lose its beauty by all such complicated desires – I could imagine as well that you should go down another path, by just radicalizing your basic idea of o a movie based on some sentences, but then the colab-interface should offer to create a movie title and some kind of finishing signature, so that the user could publish the result directly. So for creating a useful movie in one app it would be helpful to get some support for structuring the whole material, for instance by introducing a prompt for a text intro to a chapter or act, just as it was useful in silent movies.
7. Will write you a direct mail with pleasure, thanks for your interest! mischa.schaub@virtualvalley.ch

[pschaldenbrand]

I'll try to implement these points where possible @mischaschaub , but many of these user interface/application suggestions go beyond the scope of this research project. I will never be able to create a video editor that compares to something a large company has produced, so I see this work as being something that could generate content to use with the polished video editing systems rather than being an all-in-one software product. But again, thank you for your great suggestions!