Designing a Data Source Discovery App - Part 17: Migrating LLM Prompts To Jinja2...

Last time I spoke about using SQL to find records that don't exist in another table. Today, I am going to focus on how we are moving away from inefficient use of hard-coded LLM (Large Language Model) prompts in the KDS Discovery App prototype. After evaluating several alternatives, we found that Jinja2 templates are a good choice. It allows us to store prompts in the database rather than embedding them directly in code, creating a more flexible and maintainable system for prompt management.

Enter Jinja2: A Powerful Templating Engine

Jinja2 is a modern and designer-friendly templating engine for Python, created by the same developers who wrote Flask—the web framework that powers the KDS Discovery App. This synergy makes Jinja2 a natural fit for our technology stack. It's fast, widely used, and comes packed with powerful capabilities including:

• Template inheritance
• Variable substitution using {{ variable }} syntax
• Control structures (if/else statements, loops)
• Expression evaluation
• Filters to modify variables during rendering

At its core, Jinja2 lets us create templates with placeholders that get filled in at runtime. This separation between template structure and the actual data makes our code more organized and maintainable.

Implementation Approach

The approach involves storing prompt templates in the database and using Jinja2 to render them at runtime. Here's how we implemented this change:

1. Created a dedicated table (admin.prompt_setting) to store our templates
2. Modified our scripts to:
   • Fetch the appropriate template from the database
   • Use Jinja2 to render the template with the specific variables for each execution
   • Send the rendered prompt to the LLM(Large Language Model) for processing

A Real-World Test Case: The generate_test_answer.py Script

We started by refactoring the generate_test_answer.py script, which uses GPT-4 to generate test answers for testing the interview process. Here's what changed:

Before: Hardcoded Prompt

Previously, the script contained a hardcoded function that constructed the prompt string:

def generate_sample_answer_prompt(industry_description, role_, question_text, question_id):
    prompt = (
        f"Based on the following industry_description and role, please generate a sample answer to the question. Reference what tool is in use when known.\n\n"
        f"role_: {role_}\n"
        f"industry_description: {industry_description}\n"
        f"question_text: {question_text}\n\n"
        f"Respond in JSON format as follows:\n"
        f"{{\n"
        f"  \"question_id\": \"{question_id}\",\n"
        f"  \"sample_answer\": \"\"\n"
        f"}}"
    )
    return prompt

After: Database-Stored Template with Jinja2

Now, we've moved the prompt template to the database and added a function to fetch and render it:

# Fetch prompt template from the database
def fetch_prompt_template():
    conn = connect_to_db()
    cur = conn.cursor()
    query = """
    SELECT prompt_text 
    FROM admin.prompt_setting 
    WHERE name_ = 'generate_sample_answer' 
    ORDER BY create_date DESC 
    LIMIT 1;
    """
    
    cur.execute(query)
    result = cur.fetchone()
    cur.close()
    conn.close()
    
    if not result:
        raise ValueError("Prompt template 'generate_sample_answer' not found in the database.")
    
    return result[0]

# Generate a prompt using Jinja2 template
def generate_sample_answer_prompt(template_text, industry_description, role_, question_text, question_id):
    template = Template(template_text)
    rendered_prompt = template.render(
        role_=role_,
        industry_description=industry_description,
        question_text=question_text,
        question_id=question_id
    )
    return rendered_prompt

The database-stored template looks like this:

Based on the following industry_description and role, please generate a sample answer to the question. Reference what tool is in use when known.

role_: {{ role_ }}
industry_description: {{ industry_description }}
question_text: {{ question_text }}

Respond in JSON format as follows:
{
  "question_id": "{{ question_id }}",
  "sample_answer": ""
}

Benefits of This Approach

This architectural shift offers many advantages:

1. Separation of Concerns: Prompt content is now separated from code logic, making both easier to maintain.
2. Non-Technical Updates: Business analysts and prompt engineers can now modify prompts without needing developer assistance or code changes.
3. Version Control: Each prompt version is stored in the database with timestamps and author information, enabling better tracking of changes.
4. A/B Testing: We can now easily test different prompt variations by adding multiple entries in the database with different names or using a versioning scheme.
5. Centralized Management: All prompts are stored in one place, making it easier to audit, update, and maintain them.
6. Reduced Deployment Risk: Prompt changes no longer require code deployments, reducing the risk of application disruption.
7. Dynamic Selection: Applications can now select different prompts based on context, user role, or other factors.
8. Parameterization: Templates make it clear exactly which parameters are needed and how they're used within prompts.

Managing Line Feeds in Templates

One tricky issue we faced was handling line feeds in SQL when inserting templates into the database. PostgreSQL can be sensitive to how multi-line strings are stored, especially when using certain string quotation methods.

We solved this by using PostgreSQL's escaped string syntax with newline characters (\n):

INSERT INTO admin.prompt_setting (name_, prompt_text, response_count, create_date, created_by, source_)
VALUES (
  'generate_sample_answer',
  E'Based on the following...\n\nrole_: {{ role_ }}\nindustry_description: {{ industry_description }}\n...',
  20,
  CURRENT_DATE,
  'automated script',
  'generate_test_answer.py conversion'
);

Interestingly, we discovered that while line feeds enhance human readability, they're not strictly necessary for either Jinja2 template processing or for GPT-4's understanding of the prompt. But we decided to keep them anyway for better organization and clarity of our prompts.

Final Thoughts

By transitioning from hardcoded prompts to database-stored Jinja2 templates, our KDS Discovery App has gained significant flexibility and maintainability. This architectural improvement represents our commitment to building systems that can evolve rapidly with changing requirements while maintaining robustness and clarity.

Furthermore, the successful refactoring of our generate_test_answer.py script serves as a proof of concept for this approach, which we plan on adopting in the entire app.

And as mentioned in the past, please don't forget that kDS LLC is actively seeking organizations interested in adopting a beta version of the kDS Data Source Discovery App. If your organization is interested in exploring how our app can streamline your data source discovery process, please let us know. We’d love to collaborate. Tell us in the comments below or email us at info@keeshinds.com

As always, thanks for stopping by!