The final project is a 3-5 minute short film, using different AI applications. They put all their AI skills together to assemble something complex, working with text prompts and reference images to create video sequences. The main Gen AI applications that we use are ChatGPT, Midjourney and RUNWAY Gen3 Alpha.
Ulrich Gaulke is an award-winning documentary filmmaker who has taught his art across the world, from Bosnia to Bolivia. Last year he took a position as a senior lecturer at the Media and Journalism Center at the University of Hong Kong. One of the courses he teaches is Generative AI for Media Applications, for which he has been awarded the Social Science Outstanding Teaching Award. In this interview with our science editor Dr Pavel Toropov, Ulrich Gaulke talks about this new course, the role of AI in filmmaking, and why AI cannot yet replace a human storyteller.
❓How did this course – Generative AI for Media Applications, come about?
💬I studied computer science – a long time ago! And I am interested in technical things – like AI. I had an idea – how I can develop a new, pioneering class where students are storytellers, but they develop story-telling skills by using all the newest applications of Gen AI.
❓What do students do in your course?
💬First we do introduction to AI – how large language models work, how a diffusion model that creates images works… basic understanding of neural networks. Students learn what happens with the data inside an AI model and how the model creates a proper outcome.
Then, the students build, and feed, their own AI model with data – pictures. The students take pictures of Hong Kong with mobile phones to feed their own diffusion AI model. Based on this AI model we can create more Hong Kong-related stuff.
These are the beginnings. Then, step by step, we go through Gen AI applications: text to text, text to image, text to video, text to animation, text to speech, and text to music. After this the students can create stories and video sequences, and can start working on the final project.
The final project is a 3-5 minute short film, using different AI applications. They put all their AI skills together to assemble something complex, working with text prompts and reference images to create video sequences. The main Gen AI applications that we use are ChatGPT, Midjourney and RUNWAY Gen3 Alpha.
But first, the students must create a proper story, then divide it in different parts – like a story board in a fiction film. Each chapter, location, characters, everything must be visualised. I let students start with the final project only when I agree with the story. If their story has something poetic, revealing, touching, then I let them create characters, set ups, visuals.
We also have weekly assignments – to create something. We also discuss the ethical aspects of AI and technical background. I wanted to include people working with AI and combine their experience with my perspective of a storyteller. So, I have guest speakers from the computer science department.
I also invite guest speakers who are leading figures in AI and media, for example Professor Sylvia Rothe, Chair of Artificial Intelligence from Munich University of Television and Film in Germany.
We established a pioneering course, nobody else does this kind of work.
❓How popular is this course?
💬It was booked out immediately. But more students want to attend! Journalism and media students are a priority, and so we established a summer class in June. It has the same content, but the course is open to students from other departments and faculties.
❓You teach this course at the Media and Journalism Center at HKU. Is the course for filmmakers or journalists?
💬The course is open to both journalism students and filmmaker students, but our work is focused on the creation of works of fiction, not journalistic work.
Journalism is fact-based. If you are a journalist, you have to be very responsible in your use of generative AI. An AI model is not a research tool like Google. We teach that with AI, you cannot trust the outcome – an AI model does what it wants, creates its own patterns. The result may look very detailed, but it needs to be checked.
❓So, what is the main use of AI in filmmaking?
💬Filmmakers can use it creatively as a visualisation tool – for visualising something that is not possible to shoot, for example something that happened in the past, something that you have no video materials for.
Animation has traditionally been used for this purpose, in combination with powerful, real stories. For example (the animated film) FLEE, about a family from Afghanistan escaping to Europe, was nominated for three Oscars. It is based on a real story but is fully animated.
❓Do you use AI in your own work as a filmmaker?
💬I am now using it for historical re-enactments in my latest documentary about five 100-years-old ladies. They are talking about their past, but they only have still pictures from the time when they were very young. So, I can create video sequences based on these still images.
❓Storytelling is key to making films. Do you think this will be done by AI at some point?
💬Yes and No. It depends on what your expectations are. ChatGPT can help you write a story, but the main constellation – the plot, the characters, this must come from you! If you let AI create everything by itself, then you will see that it is just mimicking something that already exists. Stories are what humans use to communicate with each other. A good story must include something unique, surprising, that has something to do with your own life.
AI is based on patterns, these patterns come from the learning material, and the learning material is based on what has already been made… A script for a TV soap opera is based on very simple elements, and writers write the same stuff every year, so that is something that AI can do.
❓A little while ago, when text to video applications came out, there was talk that AI will now make films for us, eliminating the need for actors, directors…. This does not seem to be happening.
💬A lot of people are giving up on that idea.
❓Why?
💬The expectations are too high. Try giving AI a simple story to do. For example, a teacher is angry at a pupil, a little girl. Try to keep consistency with both characters, try to bring them into a serious conversation – for example, the angry teacher tells the pupil that there is something wrong with her homework. It is a very simple story but try to create it with AI – and it becomes very complicated!
Try to find a video on YouTube, one that can make you forget that it was created by AI. It is always more than obvious that a video was created by AI – a character disappears, another appears randomly, there are many random actions, there are aliens…
❓So, you see AI as a tool to create visuals for a creatively written story, one done by a human?
💬If you let AI do something on its own, then it goes weird, random. AI-created work is totally different from our idea of creativity. It´s more like a dream.
If you want to use AI application as a tool to create something that is based on our understanding of storytelling, of creating characters, of emotional expression, then it is very hard. Consistency is the problem – the movement, and the facial expressions of characters are not consistent.
It is very hard to develop a character using an AI model. You can create a realistic photo using Midjourney – for example of an old guy who is looking sad. The AI model will create an image of an old guy looking sad, but is this the old guy that you want to use in your story? Or is he completely different?
❓Does AI allow you to fine-tune these discrepancies?
💬Gen AI can do very impressive things, but it is not like applications such as Photoshop or After Effects, where you have direct control of the outcome by changing the parameters.
With Gen AI, if the result is something that you cannot use for your work – then try to change something, and – it is impossible! You can instead create something else, something new, but it can differ, again, from what you expect. You can become more and more frustrated, because you do not have direct control of the outcome.
What you can do with AI is write another prompt. But you cannot be sure that the AI model will give you exactly what you want.
❓Where would the skill of operating AI tools be? We know what a good photoshop operator can do, what is the AI equivalent?
💬The equivalent is an AI operator who is very experienced in writing prompts. The communication with an AI model needs AI communication skills and this means prompting, prompt design: how can I design the prompt to make the AI model to fulfil my expectations?
❓Do you teach prompt design in your course?
💬I try, in each lecture, to talk about prompt design. But, the more complex the outcome needs to be, the more skills you need to write a prompt.
There are prompt design tools: you can write something that is not really good as a prompt, and this tool can turn your idea into a proper prompt. I teach that too.
The students must keep in mind that it takes a lot of work to design prompts. So, they must be prepared – have a proper story ready, and only then spend time to create, using AI, the right visuals for that story.
❓What is the plan for the future of this course?
💬The AI applications keep improving. Last year we were mostly focused on managing the challenges and the difficulties of all the AI applications. This year we can expect more from them. Runway, the video application, is more advanced, so now we can do more with storytelling.
I want to make sure that the students develop better storytelling skills. Then, they can use the AI applications that are now more advanced, which means that we can create more sophisticated visuals.
👏Thank you, Ulrich!