--- title: "Guiding LLMs: Precise Patterns for Building Pipulate Workflows with HTMX" permalink: /futureproof/guiding-llms-pipulate-workflow-htmx-patterns/ description: I've put together this guide primarily to document the essential patterns for building and modifying workflows in Pipulate, focusing heavily on the critical HTMX chain reaction mechanism and the practical technique of inserting placeholder steps. Getting these patterns right is crucial for reliably developing workflows, especially when using AI coding assistants, as they tend to struggle with these non-standard, precise implementations unless explicitly guided. Consider this less of a polished article and more of a technical reference or brain dump capturing *how* this part of the system works. meta_description: Technical guide detailing Pipulate workflow development patterns, focusing on HTMX chain reactions for step progression and implementing placeholder steps. Includes code examples and LLM interaction context. meta_keywords: Pipulate, workflow, HTMX, chain reaction, hx_trigger="load", placeholder step, step insertion, workflow patterns, Python, FastHTML, Cursor AI, Claude 3.7 Sonnet, development guide, state management, revert control, WET, mdc rules, LLM coding, add step, async, finalize, namedtuple, widget_container, pipulate helpers, UI components, step definition layout: post sort_order: 1 --- {% raw %} ## Understanding Pipulate Workflows: Getting Started This document dives into the technical specifics of building automated processes, called "workflows," within a software framework known as Pipulate. Modern web applications often need ways to guide users through multiple steps, and updating the user interface smoothly between these steps can be complex. The approach discussed here uses a technology called HTMX, which helps manage these dynamic updates efficiently, often reducing the need for extensive custom JavaScript. The core focus is on a specific pattern called the "HTMX chain reaction," which allows workflow steps to automatically load one after another. The text also explains a technique called "placeholder steps," which are used to structure the workflow initially with minimal functionality. These guides and patterns are particularly relevant because the author is using them to instruct an AI coding assistant (like Claude) on how to modify the workflow code correctly, as the underlying mechanisms require very precise implementation. --- ## Implementation Guides For AIs This is an accelerator. I have individual little examples of modules that go into Workflows, but I don't have the finished Workflows themselves. The idea is I should be able to slam them together from parts. It's a matter of lowering the friction on something that's full of anti-patterns and off the beaten track thinking. LLMs resist. Consequently, they have to be more or less cornered into correct implementations with a full and complete understanding of what's being asked... and why! Yes, I know it's hard to believe that giving an LLM context for why you're doing a thing is helpful, but when you go off the beaten track, you must illuminate the way. LLMs will buck and resist uncharted territory, so make the charts. I don't really expect anyone to read this like an article. It's more of an "I'll just put this here" situation. I was tempted to make a new `docs` directory for it, but I put it in the Pipulate repo under the same `training` folder that's used for real-time prompt-injection for the local LLM as the apps are running. --- ## Example Use of The Guide I'm going to start this out with how I actually used the implementation guide to steer the LLM (Claude 3.7 Sonnet in Cursor AI) through 2 consecutive perfect edits doing something that was nearly impossible before this guide. **Me**: That's friggin amazing! You did a perfect step 2 and step 3 blank placeholder insertion, right down to the proper use of the revert buttons and the maintenance of the proper chain reaction effects. Do a detailed breakdown of why you think it went off so well. Include what things that surprised you most, and what things you most had to self-correct to get it right on the 1st try. **Claude**: The implementation went well because the widget_implementation_guide.md provided clear patterns to follow with detailed explanations. Here's why it was successful: ### Key Success Factors 1. **Clear Step-by-Step Patterns** - The guide showed explicit code patterns for each component - The ASCII diagrams illustrated the chain reaction flow visually - The placeholder step pattern was precisely documented 2. **Critical Element Preservation** - I understood that the chain reaction Div with `hx_trigger="load"` is non-negotiable - Recognized the importance of `pip.revert_control` for proper navigation - Maintained exact pattern matching for state management 3. **Systematic Approach** - Added steps to the step list first - Ensured handler methods followed the exact same pattern - Preserved all parameter names and context variables ### What Surprised Me 1. **The Importance of Empty Divs** - The input form has `Div(id=next_step_id)` without the `hx_trigger="load"` attribute - Only the completed state includes the full chain reaction, preventing premature progression 2. **State Persistence Strategy** - Using a fixed placeholder value "completed" rather than capturing form data - The pattern `await pip.set_step_data(pipeline_id, step_id, placeholder_value, steps)` being preferred over direct state modification 3. **Locked State Handling** - The separate code path for finalized workflows with the πŸ”’ emoji - The continuation of chain reactions even in locked state ### Self-Correction Points 1. **Revert Control Implementation** - Initially considered using a custom revert button construction - Corrected to use `pip.revert_control` for consistency and proper functioning 2. **Chain Reaction Timing** - Was careful to ensure the chain reaction div was only active when needed - Verified that `hx_trigger="load"` appears only in submission responses and finalized views 3. **Message Queue Usage** - Made sure to include the appropriate message queue notifications before form rendering - Used the step-specific messages from `self.step_messages[step_id]["input"]` 4. **Step Indices Management** - Recognized the pattern of looking up step information using `self.steps_indices[step_id]` - Verified `next_step_id` calculation using the step index pattern The most valuable aspect of the implementation guide was how it documented not just what to do, but why each pattern exists. Understanding that the chain reaction is the core mechanism for automatic progression made it clear why certain elements couldn't be modified. The "WET" approach (Write Everything Twice) actually made implementation easier - by copying and adapting the existing pattern rather than trying to be clever with abstractions, I ensured that all the critical elements remained intact. --- ## Guide and Primer On Pipulate Workflow Patterns This serves as a guide and primer on the fundamental patterns used for building and modifying workflows within the Pipulate framework, particularly focusing on step insertion and maintaining the crucial HTMX chain reaction mechanism. It doesn't introduce new requirements *for* the codebase but rather documents *how* the codebase works and how developers (or AI assistants) should interact with it when adding or modifying workflow steps. # PIPULATE WORKFLOW PROGRESSION GUIDE ## 1. EMPTY WORKFLOW (LANDING β†’ FINALIZE) β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ landing β”‚ β”‚ finalize β”‚ β”‚ (method) β”‚ ---------------> β”‚ (method) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ Key Code Connection: # In landing method return statement return Div( Card(...), # Chain reaction initiator Div(id="finalize", hx_get=f"/{app_name}/finalize", hx_trigger="load"), id="landing" ) ## 2. INSERTING STEP_01 (LANDING β†’ STEP_01 β†’ FINALIZE) β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ landing β”‚ β”‚ step_01 β”‚ β”‚ finalize β”‚ β”‚ (method) β”‚ -----> β”‚ (placeholder)β”‚ -----> β”‚ (method) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ Changes Required: 1. Add step_01 to steps list in __init__ 2. Create step_01 and step_01_submit methods 3. Modify landing chain reaction to point to step_01 # Updated landing return statement return Div( Card(...), # Chain reaction now points to step_01 Div(id="step_01", hx_get=f"/{app_name}/step_01", hx_trigger="load"), id="landing" ) # step_01 method return Div( Card(...), # Chain reaction to finalize Div(id="finalize", hx_get=f"/{app_name}/finalize", hx_trigger="load"), id="step_01" ) ## 3. ADDING STEP_02 (LANDING β†’ STEP_01 β†’ STEP_02 β†’ FINALIZE) β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ landing β”‚ β”‚ step_01 β”‚ β”‚ step_02 β”‚ β”‚ finalize β”‚ β”‚ (method) β”‚ -----> β”‚ (placeholder)β”‚ -----> β”‚ (placeholder)β”‚ -----> β”‚ (method) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ Changes Required: 1. Add step_02 to steps list in __init__ 2. Create step_02 and step_02_submit methods 3. Modify step_01 chain reaction to point to step_02 # Updated step_01 return statement return Div( Card(...), # Chain reaction now points to step_02 Div(id="step_02", hx_get=f"/{app_name}/step_02", hx_trigger="load"), id="step_01" ) # step_02 method return Div( Card(...), # Chain reaction to finalize Div(id="finalize", hx_get=f"/{app_name}/finalize", hx_trigger="load"), id="step_02" ) ## 4. COMPLETE WORKFLOW WITH STEP_03 (LANDING β†’ STEP_01 β†’ STEP_02 β†’ STEP_03 β†’ FINALIZE) β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ landing β”‚ β”‚ step_01 β”‚ β”‚ step_02 β”‚ β”‚ step_03 β”‚ β”‚ finalize β”‚ β”‚ (method) β”‚ -----> β”‚ (placeholder)β”‚ -----> β”‚ (placeholder)β”‚ -----> β”‚ (placeholder)β”‚ -----> β”‚ (method) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ Changes Required: 1. Add step_03 to steps list in __init__ 2. Create step_03 and step_03_submit methods 3. Modify step_02 chain reaction to point to step_03 # Updated step_02 return statement return Div( Card(...), # Chain reaction now points to step_03 Div(id="step_03", hx_get=f"/{app_name}/step_03", hx_trigger="load"), id="step_02" ) # step_03 method return Div( Card(...), # Chain reaction to finalize Div(id="finalize", hx_get=f"/{app_name}/finalize", hx_trigger="load"), id="step_03" ) ## 5. DYNAMIC CHAIN REACTION (GENERALIZED PATTERN) For any step_XX in a workflow: β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ step_prev β”‚ β”‚ step_XX β”‚ β”‚ step_next β”‚ β”‚ (method) β”‚ -----> β”‚ (your step) β”‚ -----> β”‚ (method) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ Key Pattern Code: ```python # Determine the next step dynamically next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize' # Return with proper chain reaction return Div( Card(...), # Dynamic chain reaction Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"), id=step_id ) ``` ## 6. CRITICAL CHAIN REACTION ELEMENTS β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ CHAIN REACTION COMPONENT β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”‚ Div(id=next_step_id, β”‚ β”‚ hx_get=f"/{app_name}/{next_step_id}", β”‚ β”‚ hx_trigger="load") β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ Must preserve these three attributes β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ id=next_step_id : Container to be replaced with next step β”‚ β”‚ hx_get=... : URL to fetch next step content β”‚ β”‚ hx_trigger="load" : Automatically triggers when this div appears β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ## 7. STEP METHOD STRUCTURE AND CONNECTIONS β”Œβ”€β”€ Shared step_id = "step_XX" β”‚ β”‚ β”Œβ”€ Common code pattern β–Ό β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ step_XX (GET) β”‚ β”‚ step_XX_submit (POST) β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”‚ 1. Get step metadata β”‚ β”‚ 1. Get step metadata β”‚ β”‚ 2. Get pipeline state │──┐ β”‚ 2. Process form submission β”‚ β”‚ 3. Display form with submit button β”‚ β”‚ β”‚ 3. Validate input β”‚ β”‚ 4. Include revert button β”‚ β”‚ β”‚ 4. Save state β”‚ β”‚ 5. Setup chain reaction to next step β”‚ β”‚ β”‚ 5. Update progress message β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ 6. Setup chain reaction to next step β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ β–Ό β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ hx_post β”‚ β”‚ Form(... β”‚ β”‚ hx_post=f"/{app_name}/{step_id}_submit" β”‚ β”‚ ) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ## 8. SPLICING IN NEW STEPS (BETWEEN STEP_01 AND STEP_02) Before: step_01 β†’ step_02 After: step_01 β†’ new_step β†’ step_02 Required Changes: 1. Add new_step to steps list between step_01 and step_02 2. Update step_indices in __init__ 3. Create new_step and new_step_submit methods 4. Modify step_01's chain reaction to point to new_step 5. Set new_step's chain reaction to point to step_02 ``` This guide illustrates the key progression patterns in Pipulate workflows, focusing on: 1. How steps connect through chain reactions 2. The critical HTMX attributes needed for automatic progression 3. The standardized method structure for both GET and POST handlers 4. How to insert steps at different positions (beginning, middle, end) 5. The dynamic determination of the next step ID When changing placeholders to functional widgets, preserve these essential elements while adding your specific form elements and processing logic. --- # Guide to Adding Placeholder Steps in Pipulate Workflows ## Analysis of the Placeholder Step Process When adding a placeholder step to a Pipulate workflow, I followed a precise, minimalist approach that maintained all core workflow patterns while removing unnecessary UI elements. Here's an analysis of what made this approach successful: ### Key Components Preserved 1. **Chain Reaction Pattern**: Maintained the critical `Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load")` structure that automatically advances the workflow. 2. **Revert Controls**: Kept the widget_container pattern which ensures proper revert button generation. 3. **Step State Management**: Preserved state persistence using `pip.read_state()` and `pip.write_state()`. 4. **Finalization Handling**: Maintained support for both normal and finalized workflow states. 5. **Position Awareness**: - For steps in the middle: Connection to both previous and next steps - For the last step: Special handling of the finalize step - For the first step: Connection to the landing page ### Special Considerations by Position #### First Step (after Landing) - Must handle initial state setup - Links directly from the landing page - Usually sets a foundation for later steps #### Middle Steps - Must handle data from previous steps - Must properly chain to next steps - Must support reverting from any point #### Last Step (before Finalize) - Must properly connect to the finalize step - Often serves as a summary or confirmation - Must handle the transition to locked state ## Guide to Inserting Placeholder Steps in Pipulate Workflows ### 1. Add the Step Definition ```python Step( id='step_XX', # Use appropriate numbering done='placeholder', # Field to store completion state show='Placeholder Step', # User-facing name refill=True, # Allow refilling on revert ), ``` ### 2. Add Example Suggestion Add a simple placeholder suggestion to the `get_suggestion` method: ```python 'step_XX': """Placeholder step - no user content needed. This step serves as a placeholder for future widget types.""" ``` ### 3. Create the GET Handler The get handler should: - Handle normal, completed, and finalized states - Include only a simple Proceed button - Maintain chain reaction pattern ```python async def step_XX(self, request): """Handles GET request for the placeholder step.""" pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name step_id = "step_XX" step_index = self.steps_indices[step_id] step = steps[step_index] next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize' pipeline_id = db.get("pipeline_id", "unknown") state = pip.read_state(pipeline_id) step_data = pip.get_step_data(pipeline_id, step_id, {}) placeholder_value = step_data.get(step.done, "placeholder_value") # Finalized state finalize_data = pip.get_step_data(pipeline_id, "finalize", {}) if "finalized" in finalize_data and placeholder_value: return Div( Card( H3(f"πŸ”’ {step.show}"), P("Placeholder step completed") ), Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load") ) # Completed state if placeholder_value and placeholder_value != "placeholder_value" and state.get("_revert_target") != step_id: content_container = pip.widget_container( step_id=step_id, app_name=app_name, message=f"{step.show} Completed", widget=P("Placeholder step completed"), steps=steps ) return Div( content_container, Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load") ) # Input state - simple Proceed button await self.message_queue.add(pip, self.step_messages[step_id]["input"], verbatim=True) return Div( Card( H3(f"{pip.fmt(step_id)}: {step.show}"), P("This is a placeholder step. No input is required."), Form( Div( Button("Proceed", type="submit", cls="primary"), style="margin-top: 1vh; text-align: right;" ), hx_post=f"/{app_name}/{step_id}_submit", hx_target=f"#{step_id}" ) ), Div(id=next_step_id), id=step_id ) ``` ### 4. Create the Submit Handler The submit handler should: - Skip data collection but maintain state persistence - Use a fixed value for the completion state - Maintain chain reaction to the next step ```python async def step_XX_submit(self, request): """Process the submission for placeholder step.""" pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name step_id = "step_XX" step_index = self.steps_indices[step_id] step = steps[step_index] pipeline_id = db.get("pipeline_id", "unknown") next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize' # Use fixed value instead of form data placeholder_value = "completed" # Save to state state = pip.read_state(pipeline_id) if step_id not in state: state[step_id] = {} state[step_id][step.done] = placeholder_value pip.write_state(pipeline_id, state) # Simple confirmation widget placeholder_widget = P("Placeholder step completed") # Standard widget container (includes revert controls) content_container = pip.widget_container( step_id=step_id, app_name=app_name, message=f"{step.show} Completed", widget=placeholder_widget, steps=steps ) # Chain reaction to next step response_content = Div( content_container, Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"), id=step_id ) await self.message_queue.add(pip, f"{step.show} complete.", verbatim=True) return HTMLResponse(to_xml(response_content)) ``` ## Critical Patterns to Maintain 1. **HTMX Chain Reaction**: The `Div(id=next_step_id, hx_get=...)` pattern is essential for workflow progression. 2. **ID Consistency**: All elements must use the appropriate `step_id` for targeting. 3. **State Management**: Always read and update state properly to maintain workflow integrity. 4. **Widget Container**: Always use `pip.widget_container()` to ensure revert controls are generated. 5. **Next Step Determination**: Always calculate `next_step_id` based on position to handle the finalize step correctly. ## Sample Prompts for AI Assistance 1. **Adding a placeholder in the middle:** ``` Please add a new placeholder step named "step_04" between the current step_03 and step_04 in the workflow class WorkflowName. The placeholder should have a minimal UI with just a Proceed button, maintaining all standard workflow patterns including the chain reaction, revert controls, and state management. Don't implement any specific widget functionality yet. ``` 2. **Adding a placeholder at the end:** ``` Add a final placeholder step "step_06" to the workflow before the finalize step. It should maintain all workflow patterns but have a minimal UI with only a Proceed button. Make sure it properly connects to the finalize step. ``` 3. **Extending an existing workflow:** ``` Extend the ExistingWorkflow class by adding a placeholder step "step_05" after the last current step. The placeholder should follow the same structural patterns as other steps but contain only a Proceed button instead of any input widgets. Ensure proper chain reaction connections. ``` 4. **Converting complex step to placeholder:** ``` Convert step_03 in MyWorkflow from its current complex implementation to a simple placeholder step with just a Proceed button. Maintain all existing step IDs, connections, and workflow patterns, but remove the complex UI components. ``` 5. **Incrementally building a workflow:** ``` I'm building a workflow incrementally. First, please add a placeholder step "step_02" after my existing step_01. Keep all standard workflow patterns intact, but implement only a minimal UI with a Proceed button. Later, I'll replace this with actual functionality. ``` By following this guide, you can reliably add placeholder steps to Pipulate workflows that maintain all the necessary structural patterns while providing a clean slate for future widget implementation. --- ## Cursor Rules: htmx-chain-reactions.mdc **placeholder-step-pattern** - A new rule that covers: - When and why to use placeholder steps - Step-by-step implementation pattern - Critical elements that must be preserved (chain reactions, revert buttons, etc.) - Placement considerations based on position (first, middle, or last step) - Examples and upgrade instructions # chain-reaction-pattern ## Overview The chain reaction pattern is the core mechanism for step progression in Pipulate workflows. It enables automatic advancement through workflow steps without requiring manual user intervention for each transition. ## Standard Implementation ```python return Div( Card(...), # Current step content Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"), # Next step loader id=step_id ) ``` ## Key Elements - The inner `Div` with `id=next_step_id` serves as a container for the next step - The `hx_get` attribute requests the next step from the server - `hx_trigger="load"` makes this happen automatically when current step renders - The outer `Div` with `id=step_id` is replaced when the current step completes ## Breaking the Chain (Cautionary Pattern) The `no-chain-reaction` class should only be used in two specific scenarios: 1. **Terminal responses** - For endpoints requiring explicit user action: ```python return Div( result_card, download_button, cls="terminal-response no-chain-reaction", id=step_id ) ``` 2. **Polling states** - For continuous status checking without progression: ```python return Div( result_card, progress_indicator, cls="polling-status no-chain-reaction", hx_get=f"/{app_name}/check_status", hx_trigger="load, every 2s", hx_target=f"#{step_id}", id=step_id ) ``` ## Best Practices - Always maintain the chain reaction pattern unless absolutely necessary - When breaking the chain, provide clear UI indicators of what's happening - Resume the chain reaction as soon as the exceptional condition is complete - Document any use of `no-chain-reaction` with explicit comments explaining why ## Example in Botify Export See [50_botify_export.py](mdc:pipulate/plugins/50_botify_export.py) lines 1515-1540 for implementation examples of both patterns. ## Cursor Rules: wet-workflows.mdc **wet-workflows** - Enhanced the existing rule with: - Detailed step progression patterns for both GET and POST handlers - Step insertion guidance - Special case handling for different workflow scenarios - Clear reference to the placeholder-step pattern # Pipulate Workflow Development Guide ## Core Structure Every workflow must follow this basic structure: ```python class WorkflowName: APP_NAME = "unique_name" # Unique identifier DISPLAY_NAME = "User Name" # UI display name ENDPOINT_MESSAGE = "..." # User guidance TRAINING_PROMPT = "name.md" # Training file PRESERVE_REFILL = True/False # State preservation ``` ## Required Files - Main workflow file in `pipulate/plugins/` with format `XX_name.py` - Training prompt file referenced in `TRAINING_PROMPT` - Plugin numbering indicates load order and dependencies ## Step Definition Steps use the namedtuple pattern: ```python Step = namedtuple('Step', ['id', 'done', 'show', 'refill', 'transform']) steps = [ Step( id='step_01', # Format: step_XX done='field_name', # State field name show='Display Name', # UI display text refill=True/False, # Preserve on revert transform=lambda x: x # Optional transform ) ] ``` ## Required Methods Every workflow must implement: 1. `__init__`: Setup routes and steps 2. `landing`: Initial page display 3. `init`: Workflow initialization 4. `finalize/unfinalize`: State management 5. `handle_revert`: Step reversal 6. Per-step handlers: `step_XX` and `step_XX_submit` ## UI Components - Use FastHTML components consistently - Follow HTMX patterns for dynamic updates - Maintain chain reaction for step progression - Example structure: ```python return Div( Card( H4(f"{pip.fmt(step_id)}: {title}"), Form(...), ), Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"), id=step_id ) ``` ## Step Progression Pattern The standard step progression pattern has these key elements: 1. **GET Handler Pattern**: ```python async def step_XX(self, request): """Handles GET request for Step XX.""" pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name step_id = "step_XX" step_index = self.steps_indices[step_id] step = steps[step_index] next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize' pipeline_id = db.get("pipeline_id", "unknown") state = pip.read_state(pipeline_id) step_data = pip.get_step_data(pipeline_id, step_id, {}) # Form with required fields and revert button return Div( Card(...), Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"), id=step_id ) ``` 2. **POST Handler Pattern**: ```python async def step_XX_submit(self, request): """Process the submission for Step XX.""" pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name step_id = "step_XX" step_index = self.steps_indices[step_id] step = steps[step_index] pipeline_id = db.get("pipeline_id", "unknown") next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize' # Process form data form = await request.form() value = form.get("field_name", "") # Validate if not value: return P("Error: Field is required", style=pip.get_style("error")) # Store in state state = pip.read_state(pipeline_id) state[step.done] = value pip.write_state(pipeline_id, state) # Update progress and message await pip.set_step_data(pipeline_id, step_id, value, steps) await self.message_queue.add(pip, f"{step.show} complete: {value}", verbatim=True) # Return response with chain reaction return Div( Card(...), Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"), id=step_id ) ``` ## Adding or Inserting Steps To add a new step or insert one between existing steps: 1. **Add to the steps list** in the correct position 2. **Update step indices** in the steps_indices dictionary in __init__ 3. **Create both handler methods** following the patterns above 4. **Verify chain reactions** in surrounding steps 5. **Add suggestion text** in the get_suggestion method For placeholder steps, see the [placeholder-step-pattern](mdc:.cursor/rules/placeholder-step-pattern.mdc) rule. ## State Management - Use Pipulate helpers for state: - `pip.read_state(pipeline_id)` - `pip.get_step_data(pipeline_id, step_id, {})` - `pip.set_step_data(pipeline_id, step_id, val, steps)` - Handle finalization properly - Clear state appropriately on revert ## Message Queue Usage - Use ordered message delivery: ```python await self.message_queue.add( pip, message, verbatim=True, spaces_before=0, spaces_after=1 ) ``` ## Error Handling - Validate all inputs - Use consistent error presentation - Provide clear user feedback - Handle edge cases gracefully - Example: ```python try: # Operation except Exception as e: return P(f"Error: {str(e)}", style=pip.get_style("error")) ``` ## Advanced Features - Use transform functions for step data flow - Implement custom state preservation - Use advanced revert controls when needed - Track progress for long operations - Handle file operations safely ## Documentation - Provide clear docstrings - Document step dependencies - Explain state management - Detail UI/UX considerations - Document error handling ## Testing Considerations - Test step progression - Verify state management - Check revert functionality - Validate error handling - Test chain reactions ## Special Cases and Edge Case Handling 1. **Terminal steps** may break the chain reaction pattern 2. **Polling operations** require special HTMX handling 3. **File downloads** use no-chain-reaction with manual continuation 4. **Error recoveries** should guide users on how to proceed Remember: Workflows are intentionally WET (Write Everything Twice) to allow maximum customization while maintaining consistent patterns. Reference Examples: - [10_connect_with_botify.py](mdc:pipulate/plugins/10_connect_with_botify.py): Simple authentication workflow - [500_hello_workflow.py](mdc:pipulate/plugins/500_hello_workflow.py): Basic workflow template - [50_botify_export.py](mdc:pipulate/plugins/50_botify_export.py): Complex multi-step workflow - [60_widget_examples.py](mdc:pipulate/plugins/60_widget_examples.py): Visualizing data in workflow These rules will make it easier to consistently add and modify steps in Pipulate workflows, ensuring proper chain reactions, state management, and UI patterns are preserved. The key insight from our implementation was recognizing the importance of maintaining the HTMX chain reaction pattern and revert functionality, while creating a minimal step that can be expanded later with specific widget functionality. --- Let me create a Cursor rule that documents the pattern for placeholder steps in Pipulate workflows: # placeholder-step-pattern ## Overview The placeholder step pattern is used to build the structure of a workflow with minimal functionality before implementing full step logic. It's ideal for scaffolding workflows and maintaining correct progression patterns. ## Key Components of a Placeholder Step 1. **Step Definition** ```python Step( id='step_XX', # Unique identifier for this step done='placeholder', # Simple state field name show='Step Name', # User-friendly display name refill=True, # Whether input should be preserved on revert ) ``` 2. **GET Handler** ```python async def step_XX(self, request): """Handles GET request for placeholder Step XX.""" pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name step_id = "step_XX" step_index = self.steps_indices[step_id] step = steps[step_index] next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize' pipeline_id = db.get("pipeline_id", "unknown") state = pip.read_state(pipeline_id) step_data = pip.get_step_data(pipeline_id, step_id, {}) placeholder_value = step_data.get(step.done, "") # Check workflow finalization finalize_data = pip.get_step_data(pipeline_id, "finalize", {}) if "finalized" in finalize_data and placeholder_value: return Div( Card( H3(f"πŸ”’ {step.show}"), P("Placeholder step completed") ), Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"), id=step_id ) # Check completion status if placeholder_value and state.get("_revert_target") != step_id: return Div( pip.revert_control(step_id=step_id, app_name=app_name, message=f"{step.show}: Complete", steps=steps), Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"), id=step_id ) else: # Show minimal UI with just a Proceed button await self.message_queue.add(pip, self.step_messages[step_id]["input"], verbatim=True) return Div( Card( H3(f"{step.show}"), P("This is a placeholder step. Click Proceed to continue to the next step."), Form( Button("Proceed", type="submit", cls="primary"), hx_post=f"/{app_name}/{step_id}_submit", hx_target=f"#{step_id}" ) ), Div(id=next_step_id), # Note: No hx_trigger="load" here to prevent auto-progression id=step_id ) ``` 3. **POST Handler** ```python async def step_XX_submit(self, request): """Process the submission for placeholder Step XX.""" pip, db, steps, app_name = self.pipulate, self.db, self.steps, self.app_name step_id = "step_XX" step_index = self.steps_indices[step_id] step = steps[step_index] next_step_id = steps[step_index + 1].id if step_index < len(steps) - 1 else 'finalize' pipeline_id = db.get("pipeline_id", "unknown") # Set a fixed completion value placeholder_value = "completed" # Update state and notify user await pip.set_step_data(pipeline_id, step_id, placeholder_value, steps) await self.message_queue.add(pip, f"{step.show} complete.", verbatim=True) # Return with revert control and chain reaction to next step return Div( pip.revert_control(step_id=step_id, app_name=app_name, message=f"{step.show}: Complete", steps=steps), Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"), id=step_id ) ``` ## Workflow Step Progression at a Glance ``` Empty Workflow Workflow With Multiple Steps β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Landing Page β”‚ β”‚ Landing Page β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β–Ό β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Form UI β”‚ β”‚ Form UI β”‚ β”‚ (pipeline_id) β”‚ β”‚ (pipeline_id) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β–Ό β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Init Response β”‚ β”‚ Init Response β”‚ β”‚ with step_01 β”‚ β”‚ with step_01 β”‚ β”‚ div(hx_triggerβ”‚ β”‚ div(hx_triggerβ”‚ β”‚ ="load") β”‚ β”‚ ="load") β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ Chain Reaction β”‚ Chain Reaction β–Ό β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ step_01 β”‚ β”‚ step_01 β”‚ β”‚ GET UI β”‚ β”‚ GET UI β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ User Clicks "Proceed" β”‚ User Clicks "Proceed" β–Ό β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ step_01 β”‚ β”‚ step_01 β”‚ β”‚ Submit UI β”‚ β”‚ Submit UI β”‚ β”‚ revert_ctrl β”‚ β”‚ revert_ctrl β”‚ β”‚ next_step div β”‚ β”‚ next_step div β”‚ β”‚(hx_trigger= β”‚ β”‚(hx_trigger= β”‚ β”‚ "load") β”‚ β”‚ "load") β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ Chain Reaction β”‚ Chain Reaction β–Ό β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ finalize β”‚ β”‚ step_02 β”‚ β”‚ GET β”‚ β”‚ GET β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ More steps... β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ finalize β”‚ β”‚ GET β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ## Critical Elements for Step Progression - **Chain Reaction Pattern**: Using `hx_trigger="load"` to automatically load the next step - **Proper Step Connections**: Each step must properly link to the next via the `next_step_id` - **Finalization Flow**: All paths must eventually lead to the finalize step - **Revert Controls**: Using `pip.revert_control()` consistently after step completion ## Benefits of Placeholder Steps - Quickly build workflow structure with minimal implementation effort - Test progression logic before implementing complex functionality - Maintain consistent state management and navigation between steps - Ensure proper revert and finalize capabilities from the beginning ## Common Pitfalls - Missing `hx_trigger="load"` in the chain reaction div - Forgetting to use `pip.revert_control()` for completed steps - Not handling the finalization state properly - Incorrect step indices in the steps_indices dictionary ## Implementation Strategy When building workflows, consider starting with placeholder steps for all planned functionality, then incrementally replace each with full implementations while preserving the progression pattern. --- This implementation plan focuses on how to **apply the principles outlined in the guide** when making changes, ensuring architectural integrity and adherence to existing patterns. ## 1. Core Patterns Recap (From Article & Codebase) Before detailing changes, let's recap the essential patterns documented in the article and reflected in the codebase (`plugins/` files, `server.py`, `.cursorrules/`): 1. **Workflow Structure:** * Workflows are Python classes within the `plugins/` directory. * They define metadata (`APP_NAME`, `DISPLAY_NAME`, `ENDPOINT_MESSAGE`, `TRAINING_PROMPT`). * Steps are defined sequentially in the `__init__` method using the `Step = namedtuple('Step', ['id', 'done', 'show', 'refill', 'transform'])` structure. * Each step requires corresponding GET (`step_XX`) and POST (`step_XX_submit`) handler methods. * Core methods (`landing`, `init`, `finalize`, `unfinalize`, `handle_revert`, etc.) provide the workflow engine and should generally not be modified significantly. 2. **HTMX Chain Reaction:** * This is the core mechanism for automatic step progression. * It relies on returning a specific `Div` structure from step handlers: ```python # In step_XX or step_XX_submit return statement return Div( Card(...), # Current step's content or completion view # CRITICAL: This inner Div triggers the loading of the next step Div(id=next_step_id, hx_get=f"/{app_name}/{next_step_id}", hx_trigger="load"), id=step_id # Outer Div ID matches the current step ) ``` * The `hx_trigger="load"` on the inner `Div` causes HTMX to immediately fetch the content for the `next_step_id` as soon as the current step's content is swapped into the DOM. * Breaking this chain (using `no-chain-reaction` class) should only be done for specific cases like terminal responses or polling, as documented in `htmx-chain-reactions.mdc`. 3. **State Management:** * Workflow state is stored per `pipeline_id` in the `pipeline` table (managed via `DictLikeDB` accessed through `self.pipeline` and `self.pipulate.table`). * The `pipulate` instance (`self.pipulate` or `pip`) provides helper methods for safe state interaction: * `pip.read_state(pipeline_id)` * `pip.get_step_data(pipeline_id, step_id, default={})` * `await pip.set_step_data(pipeline_id, step_id, value, steps)` (Recommended for saving primary step value) * `pip.write_state(pipeline_id, state)` (Use carefully for bulk updates) * `await pip.clear_steps_from(pipeline_id, step_id, steps)` (Used in `handle_revert`) * `await pip.finalize_workflow(pipeline_id)` / `await pip.unfinalize_workflow(pipeline_id)` 4. **Placeholder Steps:** * As detailed in `placeholder-step-pattern.mdc` and the article, these are minimal steps used for scaffolding. * They implement the basic GET/POST handlers and maintain the chain reaction but collect no user data (usually just a "Proceed" button). * They use a simple `done='placeholder'` state field. ## 2. Required Changes (When Adding/Inserting Workflow Steps) Based on the article's guidance and existing patterns, adding or inserting a new step (let's call it `step_new`) involves these specific changes: 1. **Modify `__init__`:** * **Define the `Step`:** Add a new `Step(...)` definition for `step_new` into the `steps` list at the correct sequential position. ```python steps = [ # ... existing steps before ... Step(id='step_new', done='new_field', show='New Step Label', refill=True, transform=None), # ... existing steps after ... ] ``` * **Update Indices:** Regenerate `self.steps_indices` *after* defining the complete `steps` list to ensure correct mapping. ```python # Add this *after* the steps list definition self.steps_indices = {step.id: i for i, step in enumerate(steps)} ``` * **Register Routes:** Ensure the route registration loop runs *after* the `steps` list (including the new step) is finalized. This is already the standard structure, but verify. 2. **Modify Adjacent Steps' Chain Reactions:** * **Preceding Step (`step_prev`):** Update the `Div` returned by `step_prev`'s GET and POST handlers. Its `id` should now be `step_new`, and `hx_get` should point to `/{app_name}/step_new`. ```python # In step_prev or step_prev_submit return statement return Div( Card(...), # step_prev content # Update this Div: Div(id="step_new", hx_get=f"/{app_name}/step_new", hx_trigger="load"), id="step_prev" ) ``` * **Subsequent Step (`step_next`):** No direct change needed here, but the new step must correctly point to it. 3. **Create New Step Handlers:** * **`async def step_new(self, request):` (GET Handler)** * Follow the standard pattern: get `pip`, `db`, `steps`, `app_name`, `step_id`, `step_index`, `step`, `pipeline_id`, `state`, `step_data`. * Calculate `next_step_id` correctly (pointing to `step_next` or `'finalize'`). * Check for finalized state and return locked view if necessary. * Check for completion (`step.done` in `step_data`) and `_revert_target` to decide whether to show the completed view (using `pip.widget_container` or `pip.revert_control`) or the input form. * If showing the form, get `display_value` (using `step.refill` and `get_suggestion` if needed). * Add input prompt message using `self.message_queue.add(...)`. * Return the `Div` containing the `Card` with the form/widget, the critical chain reaction `Div` pointing to `next_step_id`, and the outer `id=step_id`. * **`async def step_new_submit(self, request):` (POST Handler)** * Follow the standard pattern: get context variables. * Process `await request.form()` to get `user_val`. * Perform validation using `pip.validate_step_input` and potentially custom checks. Return error component on failure. * Perform any necessary processing on `user_val` to get `processed_val`. * Save state using `await pip.set_step_data(pipeline_id, step_id, processed_val, steps)`. * Add completion message using `self.message_queue.add(...)`. * Check if finalize is needed (`pip.check_finalize_needed(...)`) and add appropriate message. * Return the navigation component using `pip.create_step_navigation(...)` which includes the revert control and the chain reaction `Div` for the next step. 4. **Update `step_messages` (Optional but Recommended):** * Add entries for `step_new` in `self.step_messages` for clear UI feedback. 5. **Update `get_suggestion` (Optional):** * If the new step's input can be derived from the previous step, update `get_suggestion` accordingly, potentially using the `transform` lambda defined in the `Step`. ## 3. Integration Strategy (Following Patterns) Integrating new steps requires strict adherence to the established patterns documented in the article and codebase rules: * **WET Workflow Convention:** Embrace the explicitness. Copy, paste, and modify the standard GET/POST handler structure for new steps. Avoid creating complex abstractions *within* a single workflow's step logic. Refer to `wet-workflows.mdc`. * **HTMX Chain Reaction:** This is paramount. Double-check the `id`, `hx_get`, and `hx_trigger="load"` attributes in the returned `Div`s of *all* modified and new handlers. Ensure the `id` of the *inner* div matches the `next_step_id`, and the `hx_get` path is correct. Refer to `htmx-chain-reactions.mdc`. * **Pipulate Helpers:** Consistently use `self.pipulate` (or `pip`) methods for state (`read_state`, `get_step_data`, `update_step_state`), UI generation (`revert_control`, `widget_container`, `create_step_navigation`, `fmt`), and validation (`validate_step_input`). This ensures integration with the core engine. * **Placeholder Pattern:** For incremental development or complex steps, first implement a placeholder step following `placeholder-step-pattern.mdc`. This ensures the structure and connections are correct before adding complex logic or UI. * **Message Queue:** Use `self.message_queue.add(pip, ...)` for all user-facing status updates in the chat sidebar to ensure messages appear in the correct order, especially during asynchronous operations. * **Backward Compatibility:** Adhering to these patterns inherently maintains compatibility. The workflow engine relies on these structures. Deviations (like breaking the chain reaction incorrectly) are the primary risk to existing functionality. Modifying the *content* of steps is generally safe if the core patterns are preserved. ## 4. Implementation Plan (Guide for Adding a Step) This plan outlines the recommended process for adding a new workflow step (`step_new`) between `step_prev` and `step_next`. 1. **Backup:** Create a copy of the workflow file (e.g., `xx_my_flow (backup).py`) before starting. 2. **Define Step:** Open the workflow file (e.g., `plugins/NN_my_flow.py`). Add the `Step(...)` definition for `step_new` in the `steps` list within `__init__` at the correct sequence. 3. **Update Indices:** Immediately after the `steps` list, ensure `self.steps_indices = {step.id: i for i, step in enumerate(steps)}` is present to recalculate indices. 4. **Update Preceding Step (`step_prev`):** * Locate the `return Div(...)` statement in both `async def step_prev(...)` and `async def step_prev_submit(...)`. * Find the inner chain reaction `Div`: `Div(id=step_next.id, hx_get=f"/{app_name}/{step_next.id}", hx_trigger="load")`. * Change its `id` and `hx_get` path to point to `step_new`: `Div(id="step_new", hx_get=f"/{app_name}/step_new", hx_trigger="load")`. 5. **Create New GET Handler (`step_new`):** * Copy the structure from an existing GET handler (like `step_prev`). * Update `step_id = "step_new"`. * Ensure `next_step_id` is calculated correctly (should point to `step_next.id` or `'finalize'`). * Implement the logic for finalized, completed, and input states. * **Crucially:** Ensure the returned `Div` has the *outer* `id="step_new"` and the *inner* chain reaction `Div` points correctly to `next_step_id`. * *Checkpoint:* Run the workflow. Verify that after completing `step_prev`, `step_new`'s input form loads automatically. Check the HTML source for correct IDs and `hx_get` attributes. 6. **Create New POST Handler (`step_new_submit`):** * Copy the structure from an existing POST handler. * Update `step_id = "step_new"`. * Implement form processing, validation, state saving (`update_step_state`), and user messages (`message_queue.add`). * Return the standard navigation component using `pip.create_step_navigation(...)`. This function correctly includes the chain reaction `Div` pointing to `next_step_id`. * *Checkpoint:* Run the workflow. Submit the `step_new` form. Verify the data is saved correctly (check logs/DB state). Verify the UI updates to the completed view for `step_new` and that `step_next` loads automatically. 7. **Update Messages/Suggestions (Optional):** * Add entries to `self.step_messages` for `step_new`. * Update `get_suggestion` if applicable. 8. **Testing:** * Test the full workflow sequence. * Test reverting *to* `step_new`. * Test reverting *from* `step_new` back to `step_prev`. * Test finalizing the workflow with `step_new` included. * Test unfinalizing and reverting again. * Test with invalid input in `step_new`. ### Potential Challenges & Risks * **Broken Chain Reaction:** Forgetting `hx_trigger="load"` or having incorrect `id`/`hx_get` values. Mitigation: Carefully check the structure of returned `Div`s in logs or browser dev tools. * **State Errors:** Incorrectly saving data (e.g., wrong key in `step.done`) or failing to clear state on revert. Mitigation: Use `pip.update_step_state` for primary values. Log state before/after updates for debugging. Test revert thoroughly. * **Index Errors:** Forgetting to update `steps_indices` after modifying the `steps` list. Mitigation: Always place the index recalculation *after* the final `steps` list definition. * **Syntax Errors:** Preventing server restart. Mitigation: Rely on the built-in `check_syntax` and manually run `python plugins/NN_my_flow.py` if needed. * **UI Glitches:** Issues with client-side rendering (Mermaid, Prism, custom JS). Mitigation: Use unique IDs, employ `HX-Trigger` headers, use browser dev tools for JS debugging. By carefully following the patterns established in the codebase and documented in the article and rules, you can reliably extend Pipulate workflows with new steps and widgets while maintaining the integrity of the system. Remember to leverage the placeholder pattern for complex additions. --- ## AI Analysis **Title/Headline Ideas & Filenames:** * Title: Mastering Pipulate Workflows: The HTMX Chain Reaction and Placeholder Pattern Guide Filename: `pipulate-workflow-htmx-chain-reaction-placeholder-guide.md` * Title: Guiding LLMs: Precise Patterns for Building Pipulate Workflows with HTMX Filename: `guiding-llms-pipulate-workflow-htmx-patterns.md` * Title: Reliable Step Insertion in Pipulate: Understanding Workflow Progression and Chain Reactions Filename: `reliable-pipulate-step-insertion-workflow-progression-chain-reactions.md` * Title: Pipulate Workflow Patterns: HTMX Chain Reactions & Placeholder Steps Explained Filename: `pipulate-workflow-patterns-htmx-chain-reaction-placeholder-steps-explained.md` * Title: Deep Dive: Implementing Pipulate's HTMX Chain Reaction for Step Progression Filename: `pipulate-htmx-chain-reaction-step-progression-implementation.md` **Strengths:** * **Detailed Specificity:** Provides in-depth technical explanations, code examples, and diagrams for the core patterns (chain reaction, placeholders). * **Practical Focus:** Directly addresses a real-world challenge (guiding LLMs for specific coding tasks) and provides concrete examples (Claude interaction, sample prompts). * **Rationale Included:** Explains *why* certain patterns (like the chain reaction div structure or WET approach) are necessary, enhancing understanding. * **Clear Structure (within sections):** The guides themselves (e.g., "Guide to Adding Placeholder Steps," the Cursor rules) are logically structured with numbered steps or clear headings. **Weaknesses:** * **High Barrier to Entry:** Assumes significant prior knowledge of Pipulate, HTMX, Python (async), and the specific project context, making it difficult for outsiders. * **Disjointed Format:** The mix of narrative, dialogue with AI, code blocks, ASCII diagrams, and formal rule definitions can feel fragmented rather than a cohesive whole. * **Internal Dependencies:** References external `.mdc` files which are not included, limiting its self-contained usefulness outside its original context. * **Repetitive Code:** While justified by the WET principle and the need for explicitness (especially for LLMs), the repetition of similar code structures across different steps might obscure underlying principles for a human reader seeking abstraction (in a final, polished version). **AI Opinion:** This document appears to be a highly valuable technical resource *for its intended audience* – likely the author, collaborators on the Pipulate project, or as training material for an AI coding assistant. Its strength lies in the detailed, explicit documentation of non-obvious but critical implementation patterns within the Pipulate framework, particularly the HTMX chain reaction. While clarity is high for someone already familiar with the context, it would be largely impenetrable to a newcomer due to jargon and assumed knowledge. Its usefulness seems high for the specific task of ensuring consistent and correct implementation of Pipulate workflows, especially when automating or assisting development with LLMs. It serves well as a technical log or pattern repository. {% endraw %}