411. Procedures For Text Generation Projects

▮ Framework

I am currently involved in several text-generation model development for non-creative tasks.
So for this post, I’d like to share what I’ve learned on how to proceed text-generation projects.

If you can specifically define all the elements in each step, there might be a high chance that your project is feasible.

Before going into the details, I want to note that the procedures I’m about to share are mainly for non-creative text generation tasks. If your project is facing tasks that require creativity, such as generating novels, this article may not be for you.

▮ 1. Break Things Down

The first step of the framework is to break the text that you want to generate into sections. It may be better to optimize section-wise instead of generating the whole thing at once.

Breaking Things Down

The reasons for this are the following.

  • Debugging a model that generates everything at once can be hard
    • when you change your prompt, some parts may improve but others may not
  • Can be difficult to partially control the output
  • Can be difficult to evaluate

▮ 2. Pick The Right Algorithm

After you’ve divided the text into sections, the next step is to pick the right algorithm for each part.

Algorithm Selection

You don’t necessarily want/have to use generative models for everything due to the risk of hallucinations and high costs.

The algorithms you can use may differ depending on what type of text you want to generate for each section. Here are some examples.

  • Copy and Paste the whole text from the References
  • Copy and Paste different text depending on context using conditioning statements such as “if statements”.
    • Ex: 
    if is_text_related_to_alcohol():
    print("Only 20 years or older")
else:
    print("everyone can participate")
  • Use generative models to extract words/vocabularies from reference and generate the text in a fill-in-the-blank format.
    • Ex: 
    generating_text = "Hi, My name is {extracted_word_by_gpt}"
  • Use generative models to partially generate the section
  • Use generative models to fully generate the section

After you’ve decided which algorithm solution you’ll use for each section, make sure you have your client’s confirmation on your proposal.

▮ 3. Unify Section Volume

After assigning algorithms for each section, make sure that the expected amount of information included in each section is unified.

Unify Volume

Let’s say you are generating a section called “Procedures” and are expecting to have information A, B, and C included. 1 sentence each.

You want to make sure that the label data created by humans for this section also includes information A, B, and C with the same amount of volume.

When creating label data, you don’t want some people only including information A and B, while others only include information C with 5 sentences. When this doesn’t match, it gets extremely hard to evaluate the performance.

If your client already has labeled data, check each data whether the information included and the volume is consistent. If not, align your and your client’s expected
information.

3. Set Qualitative Criteria

There are many “quantitative” metrics proposed by researchers, such as BLEU, AQuA, etc., but in most cases, these metrics do not consider the specific problem your client is facing. This means it is hard to use these metrics to directly evaluate the task you want to accomplish.

Due to the above reasons, it is important to set a “qualitative” metric to evaluate the performance. These metrics should be considered by the people who are going to use this LLM. It is crucial to reflect the user’s needs to this metric.

Related Posts