A language model trained with to create editable watercolor images

Overview

Reinforcement learning can significantly enhance creative tasks but requires carefully defined reward functions to maintain a balance between flexibility and precision; overly strict models risk stagnation, while lenient ones may produce inconsistent outcomes. The approach used a judge model to improve watercolour selection among hand-rated paintings and employed an optimized prompt evolution method that underscores the importance of concise guidelines, while also addressing the challenge of model hallucinations stemming from prolonged reference documents.

Key points

Reinforcement Learning in Creative Tasks

• This project explores reinforcement learning applications in creative tasks, where rewards are subjective.
• Defining the reward function is crucial to balance flexibility and precision in design tasks.
• An overly strict model may stagnate, while too much leniency can lead to inconsistency in outcomes.

Model Evaluation and Training

• The pool includes 581 hand-rated paintings from 1664 generations.
• A judge model selects the superior watercolour from two reference paintings to update the model.

Prompt Optimization

• The optimisation process utilized GEPA for prompt evolution based on a scoring function over 200 iterations.
• A concise and clear allowlist improves output quality more effectively than the initial specifications.

Challenges in Model Output

• Long reference documentation causes models to create incorrect information regarding APIs.
• Models tend to hallucinate when exposed to extensive reference materials in prompts.