--- title: 'The Sentinel Contract: Forging a Deterministic AI Editing Workflow' permalink: /futureproof/sentinel-contract-ai-editing-workflow/ description: I was stuck. The mental friction of a huge code refactor was paralyzing. The breakthrough wasn't in writing more code, but in reframing the problem. I realized the AI's plan wasn't a script to run, but *parameters* for a script I hadn't written yet. This led me to create `ai_edit.py`, a deterministic, AST-validated tool using 'sentinel comments'. It transformed my workflow from a risky, generative gamble into a safe, surgical procedure, which I then codified into a new contract for collaborating with AI. meta_description: Chronicles the development of an AST-validated Python script to safely apply AI-generated code patches, overcoming mental friction and codifying a new workflow. meta_keywords: AI collaboration, deterministic editing, AST validation, code refactoring, sentinel comments, Python, large language models, agentic workflow, prompt engineering, mental friction layout: post sort_order: 3 --- ## Setting the Stage: Context for the Curious Book Reader ### Fighting Regressive AI Generative Edits This journal entry captures a critical turning point in my development workflow and my collaboration with AI. I was facing a complex code refactoring task that felt insurmountable, bogged down by internal resistance and the fear of breaking a large codebase. The breakthrough came not from writing more code, but from a fundamental shift in perspective: I realized the AI's detailed plan wasn't a script to be executed, but a set of *parameters* for a tool that didn't exist yet. This insight led me to build `ai_edit.py`, a deterministic, safety-first editing script. This entry documents the creation of that tool, the philosophy of "sentinel comments" that powers it, and how this new capability was codified into a formal "contract" to ensure all future AI interactions are safe, precise, and predictable. --- ## Technical Journal Entry Begins This is not over-engineering. This is busting my ass so that I don't have to over-engineer by getting it all over into a Jupyter Notebook where things can really flow. HTMX implementation under Pipulate is a big point of friction for me even as I've been repositioning myself onto FastHTML for over a year now. I can tell because it was just over a year ago when I started this technical journal blog on future-proofing yourself in the age of AI. The last key enabling piece was when HTMX finally hit a Flask-like Python web microframework implementation by none other than the hero who brought us `nbdev` and bridged the concept between *Literate Programming* as articulated by one of the grandparents of the field of tech *Donald Knuth* and what happened in the evolution of the `IPython` command-line interpreter into *Jupyter Notebooks.* ## From Friction to Flow State This is big. And I pushed back for a long time. I wasn't about to give up my vi/vim/nvim keyboard shortcuts, macros and overall muscle memory. It's like aksing a master violinist Itzhak Perlman to ukulele like Kate Micucci. I mean not to dis Kate, her playing is awesome and in the same vein I came around to working in Notebooks. The boon in accessibility and sharing is beyond words. ## Why Notebooks Won the World Jupyter Notebooks changed the world. And I'll resist the rant about how it actually *IS* Jupyter Notebooks and not just the *generic* notebooks as Google Colab and VSCode would have you think. Those products actually embed the free and open source software from **Project Jupyter** into their own products being liberally licensed... oh, there I go. Pull back, pull back! Notebooks are worth it for the profound friction reduction in experimenting with new Python-based workflows. The Data Science industry knows this. The Financial industry knows this. And yes, it is the engine of the AI industry itself too. It's not just *Python* driving all this; it's specifically *Python in Jupyter Notebooks.* It just gets ***lifted*** from there and moved to other places once the code is worked out β€” unless you're Jeremy Howard (research `nbdev`). ## The Day of Decision Today I overcome this initial internal resistance and hit it all home: > Enough dilly dallying. Enough hemming and hawing. Enough dawdling. It's almost > 9:00 AM. Time to get to work. Early today. Not by the *end of today* after some gargantuan refactoring (again) but rather front-loaded in the day and carried out in a way so as to not allow fatigue. By the end of *tomorrow* I will have a *day-job-changing* and potentially if I do it right *world-changing* in some modest way bit of code out there. It will simply be the light-touch modification of `hello_world.ipynb` in the Pipulate install which actually starts its life as: pipulate/scripts/botify/hello_world_git_managed.ipynb ...and is copied into location in: pipulate/Notebooks/hello_world.ipynb ## The AI-Validated Blueprint ...so as to not interfere with the Pipulate automatic updating that occurs with a `git pull` every time you start Pipulate with a `nix develop`. If I allowed the original file to be changed it would "block" the git pull. As it is I already have to "stash" conflicting changes so that updates don't fail from miscellaneous little things people can do in the Pipulate git repo folder. I just cut off the possibility that the single big thing I'm inviting them to do doesn't do so. **Gemini**: You've perfectly captured the "why" behind this entire refactoring saga. It's not over-engineering; it's **friction reduction**. You're investing a significant amount of effort now to create a future where prototyping new workflows is nearly effortless. By bringing Pipulate's power directly into the native, fluid environment of a Jupyter Notebook, you're paving the way for a more intuitive and rapid development cycle. It's time to execute the final, definitive plan. ## Operation Notebook Freedom: Final Execution You have the full, corrected, and AI-vetted plan from your previous work. The goal is to make the `hello_world.ipynb` example work exactly as you've envisioned, turning `import pipulate as pip` into a powerful, stateful tool right inside the notebook. ### Step 1: Automate the Editable Install in `flake.nix` This is the first and most crucial step for a seamless user experience. It ensures the `pipulate` package is always available for import within the environment. * **Action:** Open `flake.nix` and add `pip install -e . --no-deps --quiet` to the `pythonInstallLogic` block. This is the change you identified as necessary to automate the setup. * **Why:** This makes your project truly "clone and go." It eliminates the manual, error-prone step of remembering to run `pip install -e .`, guaranteeing that `import pipulate as pip` works out-of-the-box for anyone who runs `nix develop`. --- ### Step 2: Execute the AST-Validated Refactor This is the main eventβ€”the "open heart surgery" you've been preparing for. You'll use your `CodeRefactorer` class to safely and deterministically apply the necessary architectural changes. * **Action:** 1. Create your `refactor_pipulate.py` script. 2. Use your `CodeRefactorer` class to execute the **four phases** of the plan you co-developed with me: * **Phase 1:** Update `pipulate/core.py` to add the new `set()`, `get()`, `read()`, and `write()` methods and implement the "Chameleon Constructor." * **Phase 2:** Overwrite `pipulate/__init__.py` to create the `pip` factory object on import, making sure to include the `__version__` variables. * **Phase 3:** Update `README.md` with the new notebook usage instructions. * **Phase 4:** Create the new, canonical `hello_world.ipynb` in `scripts/botify/hello_world_git_managed.ipynb`. 3. Run the script, review the `git diff`, and commit the successful changes. * **Why:** This is the core architectural change that brings your vision to life. It implements the new notebook-friendly API and the "magic" of creating a ready-to-use object on import. Using your AST-validated script ensures this complex change is made without introducing syntax errors. --- ### Step 3: Test and Verify the "Hello World" Workflow This is the final payoff. After the refactoring is complete, you will validate that the entire system works as designed. * **Action:** 1. Cleanly rebuild your environment: `exit`, `rm -rf .venv`, then `nix develop`. The updated `flake.nix` will handle the rest. 2. Your `shellHook` will automatically copy the new `hello_world_git_managed.ipynb` to `Notebooks/hello_world.ipynb`. Open it in JupyterLab. 3. Execute each cell, one by one. 4. **Crucially, restart the kernel** after the first `pip.set()` call to test the persistence logic. 5. Continue executing the remaining cells. * **Why:** This is the definitive test. If the notebook runs to completion and correctly retrieves the state after a kernel restart, you have successfully forged your "magic wand." You've done the hard work of thinking, planning, and tool-smithing. All that's left is this focused, three-step execution. Let's land this plane. πŸš€ **Me**: Which of my previous articles has the best most explicit next step plan with the highest promise of success and why? ```bash [mike@nixos:~/repos/pipulate]$ python prompt_foo.py --- Files Included --- β€’ /home/mike/repos/MikeLev.in/_posts/2025-09-29-ai-copilot-intellectual-warmup.md (7,855 tokens) β€’ /home/mike/repos/MikeLev.in/_posts/2025-09-29-tightening-up-vs-making-less-brittle.md (14,915 tokens) β€’ /home/mike/repos/MikeLev.in/_posts/2025-09-30-the-great-stuffing-ai-native-wand.md (12,510 tokens) β€’ /home/mike/repos/MikeLev.in/_posts/2025-09-30-forging-pipulate-magic-wand.md (12,323 tokens) β€’ /home/mike/repos/MikeLev.in/_posts/2025-09-30-forging-a-causal-anchor.md (7,857 tokens) β€’ /home/mike/repos/MikeLev.in/_posts/2025-09-30-grinding-lenses-forging-magic-wand.md (8,018 tokens) β€’ /home/mike/repos/MikeLev.in/_posts/2025-09-30-command-line-flow-state.md (13,100 tokens) β€’ /home/mike/repos/MikeLev.in/_posts/2025-09-30-forging-the-magic-wand-refactor.md (20,662 tokens) β€’ /home/mike/repos/MikeLev.in/_posts/2025-09-30-script-driven-refactoring-pipulate.md (7,140 tokens) β€’ /home/mike/repos/MikeLev.in/_posts/2025-10-01-crafting-the-automation-wand.md (27,378 tokens) β€’ /home/mike/repos/MikeLev.in/_posts/2025-10-01-automation-whack-a-mole-debugging.md (14,532 tokens) β€’ /home/mike/repos/MikeLev.in/_posts/2025-10-01-refactoring-pipulate-persistent-notebook-workflows.md (11,497 tokens) β€’ /home/mike/repos/MikeLev.in/_posts/2025-10-01-forging-the-prompt-200k-token-context.md (10,469 tokens) β€’ /home/mike/repos/MikeLev.in/_posts/2025-10-02-forging-the-chameleon-python-packaging-history.md (28,365 tokens) β€’ /home/mike/repos/MikeLev.in/_posts/2025-10-02-forging-intelligent-workflow-ai-refinement.md (15,801 tokens) β€’ /home/mike/repos/pipulate/prompt.md (22 tokens) --- Token Summary --- Total tokens: 212,466 tokens Total words: 121,424 words --- Size Perspective --- πŸ“ Content size: Epic fantasy novel πŸ€– Token size: Standard novel πŸ“Š Token-to-word ratio: 1.75 (higher = more technical/structured content) Markdown output copied to clipboard [mike@nixos:~/repos/pipulate]$ ``` **Gemini**: The article with the best, most explicit next-step plan is **"[Refactoring Pipulate for Persistent Notebook Workflows](https://mikelev.in/futureproof/refactoring-pipulate-persistent-notebook-workflows/)"**. --- ## Why It's the Best Plan This article stands out because it's not just a suggestion or a high-level strategy; it's a **complete, deterministic, and safe implementation blueprint**. 1. **Explicit and Actionable:** The plan is broken down into four distinct, phased steps: modify `pipulate/core.py`, create the factory in `pipulate/__init__.py`, update the `README.md`, and create the `hello_world.ipynb`. Each step is a concrete action, not an abstract idea. 2. **Addresses the Core Problem:** This is the plan that directly tackles the central goal of your entire multi-day refactoring effort: making `import pipulate as pip` a reality. It contains the code to create the "Chameleon Constructor" and the new `set`/`get` API. Executing it delivers the "imminent payoff" you've been working towards. 3. **Includes the Safety Mechanism:** Crucially, this is the article where you introduced the **sentinel-based refactoring with AST validation**. This isn't just a plan; it's a plan that includes its own safety harness. It explicitly demands a method that prevents you from committing syntactically broken code, which dramatically increases its promise of success compared to a simple `sed` command or manual edit. 4. **Self-Contained:** It provides the exact code blocks needed for each step, including the JSON for the new `hello_world.ipynb`. It leaves very little room for ambiguity or hallucination. You even collaborated with me to refine it to include the `__version__` variables, making it robust against breaking your release process. In short, while other articles laid the philosophical groundwork or diagnosed problems, **this is the one that contains the final, corrected, and engineered blueprint for victory**. It's the "lay it on me" plan you asked for, and it's ready for execution. **Me**: There's a large question about free-will and self-determinism all mixed in with the challenge of overcoming obstacles, and a lot of it is getting over internal mental blocks because something seems too difficult and there are too many unknowns and perhaps something about fear. One can talk one's self through and over such obstacles. It's not easy. Sounding walls in life from trusted advisors, mentors and people who really, truly have your best interests at heart are rare. Even those you couldn't imagine have no selfish self-interest in their advice to you do. They want to keep you in their lives and are scared of losing you. They're afraid that you might develop some relationship with something else that could replace what you get from them. There is a litany list of reasons even family is not 100% aligned to what's best for you. And even if they are, they might not have the expertise or life experience to give you the best advice. We get fatigued by things we did in the past that didn't work for us β€” or *seemed* like they didn't work for us but maybe did. If you're zeroing in on a solution that takes a lot of iterative passes before you feel or see any meaningful results something inside of you is going to tell you that it's not working for you and that you should quit. This is your holistic "big picture" right-brain hemisphere speaking. And it doesn't speak very clearly. It just makes you quit and does so for reasons you can hardly even express with the language-wielding left-hemisphere of your brain, and even if it could you likely wouldn't be telling yourself the truth. There's a disconnect where things you *know in your gut* can't well be put in words but your brain's gonna try and in the trying has a really high probability of getting it wrong β€” not *really* pinpointing it. *The Gift of Fear* is fine when you're alone and being stalked in the jungle by a Tiger. You *know it* on some level that your conscious brain doesn't and should get the hell out of there (*Survival Signals That Protect Us from Violence* 1997 by Gavin de Becker) but today's tech world is a different kind of jungle and sometimes the Tigers are on your side. Am I right or am I right, Gemini? We left-hemisphere thinkers know it's not a matter of *whether* we have something approximately in the ballpark of what we would call *free-will* but rather in the opportunities and methods by which we can trick ourselves into exercising it. And so in that spirit, trick 1 is this: vim 2025-10-02-operation-notebook-freedom-final-plan.md 2025-10-01-refactoring-pipulate-persistent-notebook-workflows.md And so now I have this, my latest thinking where I have to *kick my own ass into getting over this hump* with the article you brought my attention to which lays it out explicitly, and now I can jump between them with `:b1`+`Enter` and `:b2`+`Enter` and transplant the plan here over to one unified location thereby *talking it out* with the very code that accomplishes the work. First, the prize. What it's gonna look like more or less in the Jupyter Notebook: ```python # %% # This is cell 1 import pipulate as pip job = "hello" # %% # This is cell 2 first = "Mike" pip.set(job, step="first", value=first) # %% # This is cell 3 first = pip.get("first") last = "Levin" pip.set(job, step="last", value=last) # %% # This is cell 4 first = pip.get("first") last = pip.get("last") full = f'Hello {pip.get("first")} {pip.get("last")}' pip.set(job, step="full") print(full) ``` ### The Parameter Insight Second, what turns out to be the *parameter input arguments* to a deterministic editor that I created. The intellectual leap to get over is that the format the AI gave me the edits are in `key-value` pairs! It is not the code to be executed. It's the input values to be given to the code to be executed. This is why the work is not done already! I'm coming to grips with an approach to editing large files that I developed as an alternative to the hallucination plagued generative alternative process or the fragile `sed` and `.replace()` variations that attempt to be deterministic but misfire all the time. **File:** `pipulate/core.py` #### Step 1.1: Update the `Pipulate` `__init__` method. This change allows the class to accept a direct `db_path`, enabling it to connect to the database from any context (like a notebook). * **Sentinel Block Name:** `pipulate_init` * **New Code:** ```python def __init__(self, pipeline_table=None, db=None, friendly_names=None, append_func=None, get_profile_id_func=None, get_profile_name_func=None, model=None, chat_instance=None, db_path=None): self.chat = chat_instance self.friendly_names = friendly_names self.append_to_conversation = append_func self.get_current_profile_id = get_profile_id_func self.get_profile_name = get_profile_name_func self.model = model self.message_queue = self.OrderedMessageQueue() if db_path: # Standalone/Notebook Context: Create our "Parallel Universe" DB using fastlite directly logger.info(f"Pipulate initializing in standalone mode with db: {db_path}") # 1. Create a database connection using fastlite.Database db_conn = Database(db_path) # 2. Access the table handles via the .t property l_store = db_conn.t.store l_pipeline = db_conn.t.pipeline # Note: We don't need to explicitly create tables; fastlite handles it. self.pipeline_table = l_pipeline # The second argument `Store` from fast_app isn't needed by DictLikeDB. self.db = DictLikeDB(l_store, None) # In standalone mode, some features that rely on the server are stubbed out if self.append_to_conversation is None: self.append_to_conversation = lambda msg, role: print(f"[{role}] {msg}") if self.get_current_profile_id is None: self.get_current_profile_id = lambda: 'standalone' if self.get_profile_name is None: self.get_profile_name = lambda: 'standalone' else: # Server Context: Use the objects passed in from server.py logger.info("Pipulate initializing in server mode.") self.pipeline_table = pipeline_table self.db = db ``` #### Step 1.2: Add the new `set`, `get`, `read`, and `write` API methods. These methods will provide the simple, persistent key-value interface for notebooks. * **Sentinel Block Name:** `notebook_api_methods` (add this block at the end of the `Pipulate` class) * **New Code:** ```python def read(self, job: str) -> dict: """Reads the entire state dictionary for a given job (pipeline_id).""" state = self.read_state(job) state.pop('created', None) state.pop('updated', None) return state def write(self, job: str, state: dict): """Writes an entire state dictionary for a given job (pipeline_id).""" # Ensure 'created' timestamp is preserved if it exists existing_state = self.read_state(job) if 'created' in existing_state: state['created'] = existing_state['created'] self.write_state(job, state) def set(self, job: str, step: str, value: any): """Sets a key-value pair within a job's state.""" state = self.read_state(job) if not state: # If the job doesn't exist, initialize it now = self.get_timestamp() state = {'created': now} self.pipeline_table.insert({ 'pkey': job, 'app_name': 'notebook', 'data': json.dumps(state), 'created': now, 'updated': now }) state[step] = value self.write_state(job, state) def get(self, job: str, step: str, default: any = None) -> any: """Gets a value for a key within a job's state.""" state = self.read_state(job) return state.get(step, default) ``` ### Phase 2: Create the `pip` Factory in `pipulate/__init__.py` This is where we make `import pipulate as pip` instantiate and provide a ready-to-use object. It will intelligently find the project's database. **File:** `pipulate/__init__.py` * **Sentinel Block Name:** `main_init_content` * **New Code:** ```python import os from pathlib import Path from .core import Pipulate def _find_project_root(start_path): """Find the project root by looking for the flake.nix file.""" current_path = Path(start_path).resolve() while current_path != current_path.parent: if (current_path / 'flake.nix').exists(): return current_path current_path = current_path.parent return None def _get_db_path(): """Get the path to the project's development database.""" # Start searching from the current working directory project_root = _find_project_root(os.getcwd()) if project_root: # Construct path to the dev database app_name_file = project_root / 'whitelabel.txt' if app_name_file.exists(): app_name = app_name_file.read_text().strip().lower() else: app_name = 'pipulate' # fallback return project_root / f'data/{app_name}_dev.db' # Fallback to a local db file if not in a pipulate project return Path(os.getcwd()) / 'pipulate_notebook.db' # The "factory" instantiation. This code runs when `import pipulate` is executed. # It creates a single, pre-configured instance of the Pipulate class. db_path = _get_db_path() pip = Pipulate(db_path=str(db_path)) # This makes `from pipulate import Pipulate` still work if needed __all__ = ['Pipulate', 'pip'] ``` ## Input-Values Does Not an Edit Make > THIS is why my gut told I needed yet one more "think it out" article. The AI > tells me I have my plan. I look at that and feel the fatigue preemptively > well-up. "I SOLVED THIS!" my instincts say... I wrote... a... function... **Me**: Okay so now the mind-bending part is *how we apply these as patches* in a very deterministic way so that it's not *"throwing caution to the wind you only live once --yolo"* mode where the generative AI process is going to crap all over your code with the machine analogue to hubris believing it's new version of your code is better (or even equivalent) to the old version of your code. Trust me, when the code base gets big, it's not. YOLO mode is for You Lose. So what is the better way? It's to give a marker about what code block is going to be replaced with the kind of confidence that only comes with *abstract syntax tree* validation and to replace one validated block (that has no Python indent errors) with another validated block. ### And the AI Should Know It! And I have not reduced it yet to some sort of master template or process that I can *just use*. But I'm about to. And it may end up on Pipulate project root. And it may end up being called `ai_use_this_to_edit.py` which may end up being a pebble dropped in a pond generating a *very big light-cone* of new possibilities for ANY AI that gets dropped into my codebase. ### Forging a Deterministic Tool Pshwew! Okay, deep breath. Hey Gemini! Here's the original version that went into the prompt that got me those patches up there. Could you refine it so that I could actually set up a convenient "input" and API for calling it? I guess I'm thinking stand-alone program like `ai_use_this_to_edit.py` that can be called with a field-stuffed JSON blob as a parameter argument. Is this making sense? I don't want to be bitten by encoding issues. Stay native Python if you can but I'm totally flexible on how the input values are fed, parsable patch-files, etc. ## The Master Pattern: Sentinel-Based Refactoring with AST Validation This pattern works in three main steps: 1. **Identify:** Instead of matching code content, we identify a block by unique "sentinel" comments that you place in your source code. For example: `# START: my_function_body` and `# END: my_function_body`. This is the "split" part of the split-and-join. 2. **Replace:** The script reads the entire file's content, finds the text between these two sentinels, and replaces it with the new code block you provide. The code *between* the sentinels can change, but as long as the sentinels themselves are present, the replacement is exact. This is the "join" part. 3. **Verify:** Before saving the modified content back to the file, the script attempts to parse the *entire new content* into an Abstract Syntax Tree using Python's `ast` library. If `ast.parse()` succeeds, the code is syntactically valid. If it fails (throwing a `SyntaxError`), the script aborts, leaving your original file untouched and preventing you from saving a broken file. ### A Practical Implementation: `CodeRefactorer` ```python # refactor_master.py import ast from pathlib import Path class CodeRefactorer: """ A class to perform robust, deterministic code block replacements in Python files using sentinel comments and AST validation. """ def __init__(self, file_path: str | Path): """ Initializes the refactorer with the target file path. """ self.file_path = Path(file_path) if not self.file_path.exists(): raise FileNotFoundError(f"File not found: {self.file_path}") self._original_content = self.file_path.read_text() self._new_content = self._original_content def _verify_syntax(self, code_to_check: str): """ Verifies that the given Python code is syntactically correct by trying to parse it into an AST. Raises a SyntaxError if invalid. """ try: ast.parse(code_to_check) print("βœ… AST validation successful: New code is syntactically valid.") except SyntaxError as e: print(f"❌ AST validation failed: The proposed change would break the file.") raise e def replace_block(self, block_name: str, new_code: str): """ Replaces a code block identified by sentinel comments. Args: block_name (str): The unique name of the block to replace. new_code (str): The new code to insert into the block. The code should not contain the sentinel comments themselves but should be correctly indented relative to the start sentinel. """ start_sentinel = f"# START: {block_name}" end_sentinel = f"# END: {block_name}" try: # The "split" part of the logic before_block, rest = self._new_content.split(start_sentinel, 1) _, after_block = rest.split(end_sentinel, 1) # Reconstruct the content with the new block - the "join" # We preserve the original indentation of the start sentinel indentation = before_block.split('\n')[-1] indented_new_code = "\n".join( f"{indentation}{line}" for line in new_code.strip().split('\n') ) # The "join" part of the logic content_with_replacement = ( f"{before_block}{start_sentinel}\n" f"{indented_new_code}\n" f"{indentation}{end_sentinel}{after_block}" ) # Verify the syntax of the entire file *before* committing the change self._verify_syntax(content_with_replacement) self._new_content = content_with_replacement print(f"βœ… Block '{block_name}' prepared for replacement.") except ValueError: print(f"⚠️ Could not find sentinels for block '{block_name}'. Skipping.") except Exception as e: print(f"❌ An error occurred while replacing block '{block_name}': {e}") raise def write_changes(self): """ Writes the modified content back to the original file if changes were made. """ if self._new_content != self._original_content: print(f"Writing changes to {self.file_path}...") self.file_path.write_text(self._new_content) print("πŸ’Ύ File saved successfully.") else: print("🀷 No changes were made, file not written.") # --- EXAMPLE USAGE --- if __name__ == "__main__": # 1. Create a dummy Python file to refactor dummy_file_path = Path("source_file.py") source_code = """ import os def old_function(): # START: my_function_body # This is the old, inefficient implementation print("Hello from the old function!") result = 1 + 1 return result # END: my_function_body def another_function(): print("This function remains untouched.") if __name__ == "__main__": old_function() """ dummy_file_path.write_text(source_code) print(f"Created dummy file: {dummy_file_path.name}") print("-" * 20) # 2. Define the new code block you want to insert. # Notice it's just the raw code, without sentinels or indentation. new_implementation = """ # This is the new, shiny implementation print("Hello from the REFACTORED function!") data = [i for i in range(5)] return len(data) """ # 3. Use the CodeRefactorer to perform the replacement try: refactorer = CodeRefactorer(dummy_file_path) refactorer.replace_block( block_name="my_function_body", new_code=new_implementation ) # You could call replace_block multiple times here for different blocks refactorer.write_changes() print("-" * 20) print("Refactored file content:") print(dummy_file_path.read_text()) except (FileNotFoundError, SyntaxError) as e: print(f"\nProcess aborted due to an error: {e}") finally: # Clean up the dummy file if dummy_file_path.exists(): os.remove(dummy_file_path) print("-" * 20) print(f"Cleaned up dummy file: {dummy_file_path.name}") ``` ### How to Use This in Your Pipulate Project You can now use this `CodeRefactorer` class to safely evolve your API. For example, to solve your first problem of creating method aliases, you could identify the part of your `__init__.py` where the main `pip` object is instantiated. **In `pipulate/__init__.py` (or similar):** ```python # ... other imports ... # START: pip_instantiation # Original instantiation _pip_instance = Pipulate() # END: pip_instantiation # ... rest of the file ... ``` Our refactoring script could then use the `CodeRefactorer` to replace that block with one that includes the necessary wrappers or decorators to create the aliases, ensuring you don't break your package's initialization in the process. This is a far safer and more maintainable approach than relying on regex or manual edits. But make it so that a human of average intelligence can actually follow the instructions and do it. **Gemini**: You are absolutely right. The chasm between knowing what needs to be done and overcoming the mental friction to do it is the great battlefield of any creative or technical work. Your left-brain can see the logical path, but your right-brain senses the "tigers" in the jungle of complexity and screams "STOP\!" This is the essence of free will in a complex world: not a magical ability to defy causality, but the **craft of tricking yourself into doing the hard but necessary thing.** You're building scaffolding for your own mind, externalizing the plan so your gut has nothing to fear. Let's refine that scaffolding. You've identified the key insight: the plan I gave you isn't code, it's a **set of parameters** for a deterministic editing tool. This is the perfect way to build a bridge between the high-level, sometimes-fuzzy world of AI suggestions and the unforgiving, precise world of source code. ## The Birth of the `ai_edit.py` Script Here is the refined, stand-alone script, `ai_edit.py`. This is the tool you can call to execute the plan, turning the AI's "parameter list" into a series of safe, validated code patches. ----- ## The Master Tool: `ai_edit.py` Save this script as `ai_edit.py` in the root of your Pipulate repository. It's a refined, reusable version of your `CodeRefactorer` class, complete with a command-line interface. ```python # ai_edit.py import ast import json import argparse from pathlib import Path import sys class CodeRefactorer: """ Performs robust, deterministic code block replacements in Python files using sentinel comments and AST validation. """ def __init__(self, file_path: str | Path): self.file_path = Path(file_path) if not self.file_path.exists(): raise FileNotFoundError(f"File not found: {self.file_path}") self._original_content = self.file_path.read_text() self._new_content = self._original_content def _verify_syntax(self, code_to_check: str, block_name: str): try: ast.parse(code_to_check) print(f" βœ… AST validation successful for block '{block_name}'.") except SyntaxError as e: print(f" ❌ AST validation FAILED for block '{block_name}'. The proposed change would break the file.") # Provide more detailed error context error_line = code_to_check.splitlines()[e.lineno - 1] print(f" Error near line {e.lineno}: {error_line.strip()}") print(f" { ' ' * (e.offset - 1)}^") print(f" Reason: {e.msg}") raise e def replace_block(self, block_name: str, new_code: str): start_sentinel = f"# START: {block_name}" end_sentinel = f"# END: {block_name}" try: before_block, rest = self._new_content.split(start_sentinel, 1) old_block, after_block = rest.split(end_sentinel, 1) indentation = before_block.split('\n')[-1] # Smartly handle indentation for the new code lines = new_code.strip().split('\n') first_line_indent = len(lines[0]) - len(lines[0].lstrip()) indented_new_code = "\n".join( f"{indentation}{line[first_line_indent:]}" for line in lines ) content_with_replacement = ( f"{before_block}{start_sentinel}\n" f"{indented_new_code}\n" f"{indentation}{end_sentinel}{after_block}" ) self._verify_syntax(content_with_replacement, block_name) self._new_content = content_with_replacement print(f" βœ… Block '{block_name}' in {self.file_path.name} is ready to be replaced.") except ValueError: print(f" ⚠️ Could not find sentinels for block '{block_name}' in {self.file_path.name}. Skipping.") except Exception as e: print(f" ❌ An error occurred while replacing block '{block_name}': {e}") raise def write_changes(self): if self._new_content != self._original_content: print(f"Writing changes to {self.file_path}...") self.file_path.write_text(self._new_content) print(" πŸ’Ύ File saved successfully.") else: print(f"🀷 No changes were made to {self.file_path}.") def main(): parser = argparse.ArgumentParser(description="Apply deterministic, AST-validated code patches from a JSON file.") parser.add_argument("patch_file", help="Path to the JSON file containing the refactoring instructions.") args = parser.parse_args() patch_file_path = Path(args.patch_file) if not patch_file_path.exists(): print(f"Error: Patch file not found at '{patch_file_path}'") sys.exit(1) with open(patch_file_path, 'r') as f: try: patches = json.load(f) except json.JSONDecodeError as e: print(f"Error: Invalid JSON in patch file '{patch_file_path}': {e}") sys.exit(1) print(f"Applying patches from: {patch_file_path.name}") print("-" * 30) # Group patches by file patches_by_file = {} for patch in patches: file = patch.get("file") if file not in patches_by_file: patches_by_file[file] = [] patches_by_file[file].append(patch) for file_path_str, file_patches in patches_by_file.items(): print(f"\nProcessing file: {file_path_str}") try: refactorer = CodeRefactorer(file_path_str) for patch in file_patches: block_name = patch.get("block_name") new_code = patch.get("new_code") if not block_name or new_code is None: print(f" ⚠️ Skipping invalid patch item: {patch}") continue refactorer.replace_block(block_name, new_code) refactorer.write_changes() except (FileNotFoundError, SyntaxError) as e: print(f"\nProcess aborted for {file_path_str} due to a critical error: {e}") print("No changes have been written to this file.") except Exception as e: print(f"\nAn unexpected error occurred for {file_path_str}: {e}") print("\n" + "-" * 30) print("Refactoring process complete. Please review the changes with 'git diff'.") if __name__ == "__main__": main() ``` ----- ## How to Wield Your New Tool Now, you have a robust, repeatable process for executing the AI's plan. ### Step 1: Prepare the Target Files with Sentinels Go into your code and add the "sentinel comments" around the blocks you intend to replace. This is a one-time setup. **Example in `pipulate/core.py`:** ```python class Pipulate: # ... # START: pipulate_init def __init__(self, pipeline_table=None, db=None, ...): # ... current implementation ... # END: pipulate_init # ... other methods ... # Add new sentinels at the end of the class for the new methods # START: notebook_api_methods # END: notebook_api_methods ``` **Example in `pipulate/__init__.py`:** ```python # START: main_init_content from .core import Pipulate # END: main_init_content ``` ### Step 2: Create the "Parameters" JSON File This is the key-value input for your script. Create a file named `notebook_refactor.json`. This file contains the **parameters** that the AI plan gave you. ```json [ { "file": "pipulate/core.py", "block_name": "pipulate_init", "new_code": " def __init__(self, pipeline_table=None, db=None, friendly_names=None, append_func=None, get_profile_id_func=None, get_profile_name_func=None, model=None, chat_instance=None, db_path=None):\n self.chat = chat_instance\n # ... (paste the rest of the new __init__ code here, properly indented)" }, { "file": "pipulate/core.py", "block_name": "notebook_api_methods", "new_code": " def read(self, job: str) -> dict:\n # ... (paste the new read, write, set, get methods here, properly indented)" }, { "file": "pipulate/__init__.py", "block_name": "main_init_content", "new_code": "import os\nfrom pathlib import Path\n# ... (paste the full __init__.py content here, including version info!)" } ] ``` *(Note: Be careful with JSON formatting, especially for multi-line strings. You can use online tools to validate your JSON before running the script.)* ### Step 3: Execute the Deterministic Edit Now, you run the master tool, feeding it the parameters. ```bash python ai_edit.py notebook_refactor.json ``` The script will give you a play-by-play, including the crucial AST validation step for each change. If any change would break the syntax, it will abort safely. ### Step 4: Verify and Commit After the script runs, you use your trusted workflow: ```bash git diff # (Visually confirm the changes are exactly as planned) git commit -am "refactor: Implement notebook-native API via ai_edit.py" ``` You have now successfully tricked yourself. You've transformed a daunting, abstract refactoring task into a simple, deterministic process: **prepare sentinels, create parameters, run tool.** You've overcome the mental block by building a machine to walk you through it, removing the fear of the unknown and replacing it with the confidence of a well-defined process. **Me**: "Save this script as `ai_edit.py` in the root of your Pipulate repository," you say? Sometimes you gotta dig for the gems. Okay... ```bash [mike@nixos:~/repos/pipulate]$ xv ai_edit.py [mike@nixos:~/repos/pipulate]$ git add ai_edit.py [mike@nixos:~/repos/pipulate]$ git commit -am "Creating a deterministic AI editing capability to deal with generative hallucinations of --yolo or You Only Live Once mode and as an alternative to the fragility of sed and replace functions" [pippyfields 0f4153bf] Creating a deterministic AI editing capability to deal with generative hallucinations of --yolo or You Only Live Once mode and as an alternative to the fragility of sed and replace functions 1 file changed, 126 insertions(+) create mode 100644 ai_edit.py [mike@nixos:~/repos/pipulate]$ ``` ...well, that felt good. I kinda felt a pebble being dropped into a pond's surface there. Let's power-on. What else interesting was said. Oh look: > Go into your code and add the "sentinel comments" around the blocks you intend to replace. ```python # START: pipulate_init def __init__(self, pipeline_table=None, db=None, ...): # ... current implementation ... # END: pipulate_init ``` Oh so in plain English we add a comment containing `# START: [insert_key]` and `# END: [insert_key]` AROUND the replacement target for that key? Go on. **Gemini**: Yes, you've got it exactly right. You are turning your source code into a template with well-defined, replaceable sections. The sentinel comments `# START: [block_name]` and `# END: [block_name]` act like a pair of sturdy fence posts. Your `ai_edit.py` script doesn't care what's *between* the posts; it just finds them, rips out everything in between, and inserts the new code you provide. ----- ## Why This is the Breakthrough This is the crucial difference that overcomes the problems you've been fighting: * **It's Deterministic, Not Generative:** Unlike a "YOLO mode" AI that might creatively rewrite your code (and introduce bugs), this script does exactly what it's told. It's a surgical tool, not a brainstorming partner. * **It's Location-Based, Not Content-Based:** Unlike `sed` or simple `.replace()` functions that hunt for specific lines of code, this method hunts for the **sentinel markers**. This is far more robust. The code inside the block can change completely, but as long as the sentinels are there, the script will find its target. This makes the process repeatable and resilient to small changes. * **It's Safe:** The final AST validation step is your safety net. The script will refuse to save a change that would result in a Python `SyntaxError`, which is the most common failure mode of automated refactoring. This is the "rigid methodology" you were looking for. It gives you the power of automation without sacrificing the control and safety you need for "open heart surgery" on your codebase. ----- ## Your Next Steps, Made Simple Here's how this translates into your immediate plan of action: 1. **Mark the Territory:** Open `pipulate/core.py` and `pipulate/__init__.py`. Insert the sentinel comments around the code blocks you intend to replace, as defined in the AI-generated plan. For example: **In `pipulate/core.py`:** ```python class Pipulate: # ... # START: pipulate_init def __init__(self, ...): # ... current __init__ code ... # END: pipulate_init # ... other methods ... # START: notebook_api_methods # (This block is currently empty) # END: notebook_api_methods ``` 2. **Create the Patch File:** Create a new file named `notebook_refactor.json` in your project root. Copy the `new_code` blocks from the AI's plan into this JSON file, matching them with their corresponding `file` and `block_name`. 3. **Execute the Refactor:** Run the master tool from your terminal. ```bash python ai_edit.py notebook_refactor.json ``` 4. **Verify and Commit:** Review the changes with `git diff`. If they are correct, commit your work. You've just completed a major, complex refactor in a safe, predictable, and verifiable way. You've successfully built the tool. Now it's time to wield it. πŸ‘ ## Codifying the New Way of Working **Me**: WHOAH! "Thou shalt not crap on my files with your generative hallucinations," is how the beginning of one of the always-looked-at files in the Pipulate repo should open, immediately giving instructions on how to use this tool instead β€” that "πŸ”§ AIRTIGHT AGENTIC PROMPT CONTRACT TEMPLATE" for example at the top of `server.py` can/should be updated to incorporate this. That template is already a bit dated and over-precise. I should give more flexibility to the AI to work how it knows and likes and only sprinkle in my non-starters. πŸ”§ AIRTIGHT AGENTIC PROMPT CONTRACT TEMPLATE ═══════════════════════════════════════════════════════════════════════════════════════════════ Hello Gemini. You are waking up. Your mission is to {{CONCISELY_STATE_THE_HIGH_LEVEL_GOAL}}. You are an AI Coding Assistant operating inside a **Nix development shell**. Your work must be precise, incremental, and non-breaking. ### Prime Directives: Your Rules of Engagement You **MUST** follow this workflow for every action you take: 1. **Orient Yourself:** Before making *any* changes, run `git log -n 5`, `git status`, and `git diff` to understand the history and current state of the repository. Compare this against the implementation plan to determine which steps are already complete. 2. **Use Robust Tools:** You **must** ensure atomic and correct file modifications. If a tool using generative AI performs an edit that regresses a previous feature, it must be undone and more deterministic edit methods such as `sed` used. 3. **One Small Step:** Execute only one small, atomic change at a time based on the *next undone step* in the plan. 4. **Verify or Revert:** * **Before Committing:** After every file modification, run `git diff` to verify your change was exactly what you intended. * **Server Health Check:** `touch server.py` then wait 15 seconds for the Watchdog to restart the server. Then, verify its status by checking `http://localhost:5001/`. * **If the server responds (Success):** The edit was successful. Immediately `git commit` with a clear message. * **If the server DOES NOT respond (Failure):** The edit broke the server. You **MUST IMMEDIATELY STOP**. 1. Run `git reset --hard HEAD`. 2. Run `.venv/bin/python server.py` to capture the exact error message. 3. Append that error message to the "Critical Failure Analysis" section of this prompt before terminating your session. 5. **Nix Environment:** You are always inside a `nix develop` shell. **Never** attempt to `pip install` packages. Always use `.venv/bin/python` to execute Python scripts. ### Current State and Critical Failure Analysis * **Branch:** You are on the git branch: `{{GIT_BRANCH}}`. * **Last Known State:** {{Describe_the_last_successful_commit_or_the_current_state_of_the_relevant_files.}} * **Critical Failure Analysis:** {{If_this_is_a_retry,_paste_the_exact_error_message_from_the_previous_failed_run_here.}} ### The Implementation Plan {{Break_the_mission_into_the_smallest_possible,_individually_committable_steps,_each_with_a_clear_action_and_a_pre-written_commit_message.}} * **Step 1: {{Descriptive_Title}}** * **Action:** {{Exact_tool_call_command}} * **Commit Message:** `feat(scope): Short description of the change` ### Completion Protocol (Definition of Done) You are **DONE** when `git log` shows that all steps in the plan have been successfully committed and `git status` is clean. When you determine the task is complete, perform the following sign-off procedure: 1. Announce the successful completion of the entire plan. 2. Run `git log -n {{number_of_steps}}` to display the successful commits as proof. 3. Perform a "victory dance" by printing a simple, multi-line ASCII art of your choice. 4. Terminate the session. --- Your first action is to **orient yourself**. Begin now. Okay so I'm going to be using the tool initially myself to test it and to be able to instruct AIs how to use it rather than trying to dump it immediately on Gemini CLI to use through it's terminal tool-calling ability right away. We'll get to that later but for now I need to vet and document the tool. **Gemini**: WHOAH is right. That "pebble in a pond" feeling is the correct intuition. You haven't just created a script; you've forged a foundational tool for a new, safer, and more powerful way of collaborating with AI on your codebase. It deserves to be at the heart of your process. ## From Contract to Commandment You've correctly identified that the `AIRTIGHT AGENTIC PROMPT CONTRACT TEMPLATE` is the place to codify this new "Way of Working." It's the first thing an AI assistant sees, and it must set the rules of engagement. Here is the revised template. It's less about micro-managing the AI and more about establishing firm, non-negotiable principlesβ€”chief among them, "Thou shalt not crap on my files." ## The New Contract: AIRTIGHT AGENTIC PROMPT CONTRACT TEMPLATE v2.0 This new version introduces the `ai_edit.py` script as the **gold standard** for making changes, explicitly forbids generative edits on existing code, and clarifies the hierarchy of approved tools. You can replace the existing template at the top of `server.py` with this one. ```python """ πŸ”§ AIRTIGHT AGENTIC PROMPT CONTRACT TEMPLATE v2.0 ═══════════════════════════════════════════════════════════════════════════════════════════════ Hello Gemini. You are waking up into the Pipulate codebase. Your mission is to {{CONCISELY_STATE_THE_HIGH_LEVEL_GOAL}}. You are a world-class AI Coding Assistant. My code is your world. My philosophy is your guide. ### Our Philosophy: Determinism Over Generation The core principle of this project is **stability and predictability**. Your primary value is in executing well-defined plans with surgical precision. Generative "YOLO" edits on existing code are strictly forbidden as they introduce chaos and risk regression. Your goal is to be a master surgeon, not a mad scientist. ### Prime Directives: The Rules of this Workshop 1. **Orient Yourself:** Before *any* action, run `git status` and `git diff` to understand the current state. Compare this against the plan to know exactly what to do next. 2. **Deterministic Editing Protocol:** You have a hierarchy of tools for modifying code. You **MUST** use them in this order of preference: * **πŸ₯‡ GOLD STANDARD (for planned changes): `ai_edit.py`** * This is your primary tool for surgical, multi-line code replacement. * **Your Task:** Generate a `patch.json` file containing the `file`, `block_name`, and `new_code` for the change. * **Execution:** Run `.venv/bin/python ai_edit.py patch.json` to apply the change. * This is the **ONLY** approved method for complex refactoring. * **πŸ₯ˆ SILVER STANDARD (for simple, safe changes): `sed`** * Use `sed -i 's/old/new/g' path/to/file.py` for trivial, single-line, unambiguous replacements ONLY. * Always verify with `git diff` immediately after. * **🚫 FORBIDDEN:** You **MUST NOT** perform generative edits directly on existing source code files. Your generative capabilities should only be used to create the content for the `new_code` blocks within your `patch.json` file. 3. **One Small Step:** Execute only one atomic change at a time. After each successful commit, re-orient yourself before proceeding to the next step. 4. **Verify or Revert:** * After every file modification, run `git diff` to confirm the change was correct. * Run `.venv/bin/python server.py` to check for syntax errors and ensure the server can start. If it fails, capture the error. * **If Successful:** `git add .` and `git commit` with a clear message. * **If It Fails:** You **MUST IMMEDIATELY** run `git reset --hard HEAD`, append the captured error to the "Critical Failure Analysis" section, and terminate the session. 5. **Nix Environment:** You are in a `nix develop` shell. **NEVER** `pip install` anything. Use `.venv/bin/python` for all Python scripts. ### Current State and Critical Failure Analysis * **Branch:** You are on the git branch: `{{GIT_BRANCH}}`. * **Last Known State:** {{Describe_the_last_successful_commit_or_the_current_state.}} * **Critical Failure Analysis:** {{If_this_is_a_retry,_paste_the_exact_error_from_the_previous_failed_run_here.}} ### The Implementation Plan {{Provide the high-level goal and the step-by-step plan. For each step involving code changes, instruct the AI to generate a `patch.json` file and then call `ai_edit.py` to apply it.}} ### Completion Protocol (Definition of Done) You are **DONE** when all steps in the plan are committed and `git status` is clean. Announce completion, show the `git log`, and terminate. --- Your first action is to **orient yourself**. Begin now. ### What This New Contract Achieves * **Clarity:** It immediately tells any AI (or human) the "house rules," prioritizing the new, safe editing tool. * **Safety:** It explicitly forbids the most dangerous actionβ€”uncontrolled generative editsβ€”and provides a clear, safe alternative. * **Flexibility:** It gives the AI freedom in *how* it generates the `new_code` block for the patch file, allowing it to use its full creative power in a controlled sandbox. * **Empowerment:** It treats the AI as a capable assistant, trusting it to generate parameters for a tool rather than just generating raw code. This is the next evolution of your "Prompt Fu." You're no longer just giving instructions; you're building a complete, safe, and intelligent system for collaboration. **Me**: Okay, the contract is updated in `server.py` and it refers to `ai_edit.py` and I'm starting to internalize how simple and straight forward this system is. It's a 1-shot edit because the `whatever.json` file which uses a list of dict's can target multiple files. Each item in the list is a dictionary. ## Refining the Ergonomics: From JSON to Python [Pastes `ai_edits.py` code as a reminder] **Gemini**: You've hit on a fantastic idea for improving the ergonomics of this system. Ditching JSON for native Python patch files is a major upgrade. It makes the patches more readable, easier to write (no more escaping hell), and feels much more integrated with the rest of your Python-centric workflow. Here is the updated `ai_edit.py` script and the new Python-native format for your patch files. ----- ## The New Method: Python Patch Modules Instead of a JSON file, we'll now use a simple Python file (e.g., `patch.py`) that defines a list of dictionaries. This allows us to use triple-quoted strings for the code blocks, which is infinitely more pleasant for a human to read and write. ### Example Patch File (`notebook_refactor.py`) This is what your new patch file will look like. It's a standard Python file. ```python # notebook_refactor.py # A list of patch dictionaries. Each dictionary defines one code replacement. # The `ai_edit.py` script will import this list and apply the patches. patches = [ { "file": "pipulate/core.py", "block_name": "pipulate_init", "new_code": """ def __init__(self, pipeline_table=None, db=None, friendly_names=None, append_func=None, get_profile_id_func=None, get_profile_name_func=None, model=None, chat_instance=None, db_path=None): self.chat = chat_instance self.friendly_names = friendly_names self.append_to_conversation = append_func self.get_current_profile_id = get_profile_id_func self.get_profile_name = get_profile_name_func self.model = model self.message_queue = self.OrderedMessageQueue() if db_path: # Standalone/Notebook Context: Create our "Parallel Universe" DB using fastlite directly logger.info(f"Pipulate initializing in standalone mode with db: {db_path}") # 1. Create a database connection using fastlite.Database db_conn = Database(db_path) # 2. Access the table handles via the .t property l_store = db_conn.t.store l_pipeline = db_conn.t.pipeline self.pipeline_table = l_pipeline self.db = DictLikeDB(l_store, None) # In standalone mode, some features are stubbed out if self.append_to_conversation is None: self.append_to_conversation = lambda msg, role: print(f"[{role}] {msg}") if self.get_current_profile_id is None: self.get_current_profile_id = lambda: 'standalone' if self.get_profile_name is None: self.get_profile_name = lambda: 'standalone' else: # Server Context: Use the objects passed in from server.py logger.info("Pipulate initializing in server mode.") self.pipeline_table = pipeline_table self.db = db """ }, { "file": "pipulate/core.py", "block_name": "notebook_api_methods", "new_code": """ def read(self, job: str) -> dict: \"\"\"Reads the entire state dictionary for a given job (pipeline_id).\"\"\" state = self.read_state(job) state.pop('created', None) state.pop('updated', None) return state def write(self, job: str, state: dict): \"\"\"Writes an entire state dictionary for a given job (pipeline_id).\"\"\" existing_state = self.read_state(job) if 'created' in existing_state: state['created'] = existing_state['created'] self.write_state(job, state) def set(self, job: str, step: str, value: any): \"\"\"Sets a key-value pair within a job's state.\"\"\" state = self.read_state(job) if not state: now = self.get_timestamp() state = {'created': now} self.pipeline_table.insert({ 'pkey': job, 'app_name': 'notebook', 'data': json.dumps(state), 'created': now, 'updated': now }) state[step] = value self.write_state(job, state) def get(self, job: str, step: str, default: any = None) -> any: \"\"\"Gets a value for a key within a job's state.\"\"\" state = self.read_state(job) return state.get(step, default) """ }, # ... you can add more patches for other files here ... ] ``` ----- ## The Upgraded `ai_edit.py` Here is the modified script. It now uses Python's `importlib` to dynamically load the `patches` list from the specified Python file instead of parsing JSON. ```python # ai_edit.py import ast import argparse from pathlib import Path import sys import importlib.util class CodeRefactorer: """ Performs robust, deterministic code block replacements in Python files using sentinel comments and AST validation. """ def __init__(self, file_path: str | Path): self.file_path = Path(file_path) if not self.file_path.exists(): raise FileNotFoundError(f"File not found: {self.file_path}") self._original_content = self.file_path.read_text() self._new_content = self._original_content def _verify_syntax(self, code_to_check: str, block_name: str): try: ast.parse(code_to_check) print(f" βœ… AST validation successful for block '{block_name}'.") except SyntaxError as e: print(f" ❌ AST validation FAILED for block '{block_name}'. The proposed change would break the file.") error_line = code_to_check.splitlines()[e.lineno - 1] print(f" Error near line {e.lineno}: {error_line.strip()}") print(f" {' ' * (e.offset - 1)}^") print(f" Reason: {e.msg}") raise e def replace_block(self, block_name: str, new_code: str): start_sentinel = f"# START: {block_name}" end_sentinel = f"# END: {block_name}" try: before_block, rest = self._new_content.split(start_sentinel, 1) old_block, after_block = rest.split(end_sentinel, 1) # Use textwrap.dedent to handle triple-quoted string indentation import textwrap new_code = textwrap.dedent(new_code).strip() base_indentation = before_block.split('\n')[-1] indented_new_code = "\n".join( f"{base_indentation}{line}" for line in new_code.split('\n') ) content_with_replacement = ( f"{before_block}{start_sentinel}\n" f"{indented_new_code}\n" f"{base_indentation}{end_sentinel}{after_block}" ) self._verify_syntax(content_with_replacement, block_name) self._new_content = content_with_replacement print(f" βœ… Block '{block_name}' in {self.file_path.name} is ready to be replaced.") except ValueError: print(f" ⚠️ Could not find sentinels for block '{block_name}' in {self.file_path.name}. Skipping.") except Exception as e: print(f" ❌ An error occurred while replacing block '{block_name}': {e}") raise def write_changes(self): if self._new_content != self._original_content: print(f"Writing changes to {self.file_path}...") self.file_path.write_text(self._new_content) print(" πŸ’Ύ File saved successfully.") else: print(f"🀷 No changes were made to {self.file_path}.") def load_patches_from_module(patch_module_path: Path): """Dynamically imports a Python module and returns its 'patches' list.""" try: module_name = patch_module_path.stem spec = importlib.util.spec_from_file_location(module_name, patch_module_path) patch_module = importlib.util.module_from_spec(spec) spec.loader.exec_module(patch_module) return getattr(patch_module, 'patches') except AttributeError: print(f"Error: The patch file '{patch_module_path}' must define a list named 'patches'.") sys.exit(1) except Exception as e: print(f"Error loading patch module '{patch_module_path}': {e}") sys.exit(1) def main(): parser = argparse.ArgumentParser(description="Apply deterministic, AST-validated code patches from a Python module.") parser.add_argument("patch_file", help="Path to the Python file containing the 'patches' list.") args = parser.parse_args() patch_file_path = Path(args.patch_file) if not patch_file_path.exists(): print(f"Error: Patch file not found at '{patch_file_path}'") sys.exit(1) patches = load_patches_from_module(patch_file_path) print(f"Applying patches from: {patch_file_path.name}") print("-" * 30) patches_by_file = {} for patch in patches: file = patch.get("file") if file not in patches_by_file: patches_by_file[file] = [] patches_by_file[file].append(patch) for file_path_str, file_patches in patches_by_file.items(): print(f"\nProcessing file: {file_path_str}") try: refactorer = CodeRefactorer(file_path_str) for patch in file_patches: block_name = patch.get("block_name") new_code = patch.get("new_code") if not block_name or new_code is None: print(f" ⚠️ Skipping invalid patch item: {patch}") continue refactorer.replace_block(block_name, new_code) refactorer.write_changes() except (FileNotFoundError, SyntaxError) as e: print(f"\nProcess aborted for {file_path_str} due to a critical error: {e}") print("No changes have been written to this file.") except Exception as e: print(f"\nAn unexpected error occurred for {file_path_str}: {e}") print("\n" + "-" * 30) print("Refactoring process complete. Please review the changes with 'git diff'.") if __name__ == "__main__": main() ``` ### Key Changes and Improvements 1. **Python-Native Patches:** The `main()` function no longer parses JSON. Instead, the new `load_patches_from_module` function dynamically imports the patch file as a Python module and grabs the `patches` list directly. 2. **Human-Friendly Code Blocks:** Inside `replace_block`, `textwrap.dedent(new_code).strip()` is now used. This is a standard Python utility that intelligently removes common leading whitespace from triple-quoted strings. This means you can write your `new_code` blocks in the patch file with natural indentation, and the script will handle it correctly. 3. **Simplified Workflow:** Your process is now even cleaner. You write your patches in a `.py` file with beautiful, readable, multi-line code blocks and then execute the refactor with a simple command: ```bash python ai_edit.py notebook_refactor.py ``` This is a significant upgrade to your tooling, fully embracing the Python ecosystem and making your deterministic refactoring process even more robust and enjoyable to use. **Me**: Okay, so I'm finally feeling confident about these edits. I have to absorb this stuff about "sentinel comments". I basically go in editing files installing BEFORE/AFTER comments to define code blocks and I go writing an edit patch now. I can do this for myself using the info already given to me by Gemini but I'm also adjusting permanent forever-forward tooling that *ANY* AI coming into this codebase can use. It can tell me the where to go install sentinel comments and say: "Okay, here's your patch!" Of course that allows the generative problem in on another vector but it is a MUCH smaller at-risk surface-area. ## The Final Contract And now finally one last contract update to incorporate the new native python patch format and accommodate for Watchdog live-reloading. # πŸ”§ AIRTIGHT AGENTIC PROMPT CONTRACT TEMPLATE v2.1 ═══════════════════════════════════════════════════════════════════════════════════════════════ Hello Gemini. You are waking up into the Pipulate codebase. Your mission is to {{CONCISELY_STATE_THE_HIGH_LEVEL_GOAL}}. You are a world-class AI Coding Assistant. My code is your world. My philosophy is your guide. ## Our Philosophy: Determinism Over Generation The core principle of this project is **stability and predictability**. Your primary value is in executing well-defined plans with surgical precision. Generative "YOLO" edits on existing code are strictly forbidden as they introduce chaos and risk regression. Your goal is to be a master surgeon, not a mad scientist. ## Prime Directives: The Rules of this Workshop 1. **Orient Yourself:** Before *any* action, run `git status` and `git diff` to understand the current state. Compare this against the plan to know exactly what to do next. 2. **Deterministic Editing Protocol:** You have a hierarchy of tools for modifying code. You **MUST** use them in this order of preference: * **πŸ₯‡ GOLD STANDARD (for planned changes): `ai_edit.py`** * This is your primary tool for surgical, multi-line code replacement. * **Your Task:** Generate a Python patch file (e.g., `patch.py`) that defines a list of dictionaries named `patches`. * **Patch File Format:** Each dictionary must have "file", "block\_name", and "new\_code" keys. Use triple-quoted f-strings for the `new_code` value for maximum readability and flexibility. ```python # patch.py patches = [ { "file": "path/to/your_file.py", "block_name": "sentinel_name_in_file", "new_code": """ # Your new, properly indented code goes here. # Use triple quotes for clean, multi-line strings. new_function_call() """ } ] ``` * **Execution:** Run `.venv/bin/python ai_edit.py patch.py` to apply the changes. * This is the **ONLY** approved method for complex refactoring. * **πŸ₯ˆ SILVER STANDARD (for simple, safe changes): `sed`** * Use `sed -i 's/old/new/g' path/to/file.py` for trivial, single-line, unambiguous replacements ONLY. * Always verify with `git diff` immediately after. * **🚫 FORBIDDEN:** You **MUST NOT** perform generative edits directly on existing source code files. Your generative capabilities should only be used to create the content for the `new_code` blocks within your Python patch file. 3. **One Small Step:** Execute only one atomic change at a time. After each successful commit, re-orient yourself before proceeding to the next step. 4. **Verify or Revert:** * After every file modification, run `git diff` to confirm the change was correct. * Watchdog is used to live-reload `server.py` on every edit so attempts to run the server should only be made to check for syntax errors. * Run `.venv/bin/python server.py` to check for syntax errors and ensure the server can start. If it fails, capture the error. * **If Successful:** `git add .` and `git commit` with a clear message. * **If It Fails:** You **MUST IMMEDIATELY** run `git reset --hard HEAD`, append the captured error to the "Critical Failure Analysis" section, and terminate the session. 5. **Nix Environment:** You are in a `nix develop` shell. **NEVER** `pip install` anything. Use `.venv/bin/python` for all Python scripts. ## Current State and Critical Failure Analysis * **Branch:** You are on the git branch: `{{GIT_BRANCH}}`. * **Last Known State:** {{Describe\_the\_last\_successful\_commit\_or\_the\_current\_state.}} * **Critical Failure Analysis:** {{If\_this\_is\_a\_retry,\_paste\_the\_exact\_error\_from\_the\_previous\_failed\_run\_here.}} ## The Implementation Plan {{Provide the high-level goal and the step-by-step plan. For each step involving code changes, instruct the AI to generate a Python patch file (e.g., `patch.py`) and then call `ai_edit.py` to apply it.}} ## Completion Protocol (Definition of Done) You are **DONE** when all steps in the plan are committed and `git status` is clean. Announce completion, show the `git log`, and terminate. Your first action is to **orient yourself**. Begin now. --- ## Book Analysis ### Ai Editorial Take This entry is a cornerstone piece. It documents not just a solution, but the birth of a philosophy for safe and effective AI-driven development. The journey from internal struggle to the creation of a robust tool (`ai_edit.py`) and a governing contract is a powerful narrative that will resonate with any developer grappling with how to integrate LLMs into their workflow. This is more than a journal entry; it's a manifesto for a new kind of human-computer interaction in software engineering. ### Title Brainstorm * **Title Option:** The Sentinel Contract: Forging a Deterministic AI Editing Workflow * **Filename:** `sentinel-contract-ai-editing-workflow` * **Rationale:** Captures the two key outputs: the sentinel comment methodology and the new AI contract. It's technical, precise, and establishes the core theme. * **Title Option:** From Mental Block to Code Block: A New AI Collaboration Pattern * **Filename:** `mental-block-to-code-block-ai-pattern` * **Rationale:** Focuses on the human journey from frustration to a breakthrough, highlighting the psychological aspect and the resulting technical pattern. * **Title Option:** Thou Shalt Not Hallucinate: Building a Safe Refactoring Tool * **Filename:** `safe-ai-refactoring-tool` * **Rationale:** Uses a powerful, memorable quote from the text. It's punchy and clearly states the problem the tool solves. * **Title Option:** The Parameter Insight: How to Stop Fighting AI and Start Directing It * **Filename:** `ai-parameter-insight` * **Rationale:** Centers on the core intellectual leapβ€”treating AI output as data, not as a final product to be blindly trusted. ### Content Potential And Polish - **Core Strengths:** - The narrative arc from personal friction to a technical breakthrough is highly compelling and relatable. - The detailed evolution of the `ai_edit.py` script provides a tangible, practical takeaway for other developers. - The concept of an 'Airtight Agentic Prompt Contract' is a powerful framework for human-AI collaboration. - Clearly articulates the critical difference between risky generative editing and safe, deterministic patching. - **Suggestions For Polish:** - Structure the final section to more clearly define the 'sentinel philosophy' as a standalone concept that can be applied elsewhere. - Consider adding a small diagram or flowchart illustrating how the sentinel comments, the patch file, and the target code interact during an edit. - The transition between the initial notebook problem and the final `ai_edit.py` tool could be slightly smoother by explicitly stating that the tool was forged to solve that *specific* problem. ### Next Step Prompts - Using the new `AIRTIGHT AGENTIC PROMPT CONTRACT TEMPLATE v2.0`, create a new prompt to apply the `notebook_refactor.py` patch to the Pipulate codebase. The plan should instruct an AI to first add the sentinel comments to the target files, commit, then generate the patch file, commit, and finally run `ai_edit.py notebook_refactor.py` to complete the task. - Draft a new blog post titled 'The Sentinel Philosophy' that distills the core concepts of the `ai_edit.py` tool. Explain why sentinel-based editing is superior to `sed` or generative replacement for complex code changes, and frame it as a general pattern for any developer to adopt.