.In my last post I discussed the implications of higher-level abstractions over HTML in the content that we author on the web. Love them or hate them, shortcodes are a big part of how WordPress has traditionally enabled content authors to engage with such higher-level abstractions. In this post I want to share some of the challenges shortcodes present and why Gutenberg has chosen a different path, what that path is, and what it means for us all.
Let's dive right in! Can you tell me how we should interpret the following snippet? It comes from a hypothetical post_content with a custom shortcode.
[region] [region] [/region]There are actually only two plausible meanings: we have an outer region with a nested and empty inner region; we have an empty region above a non-empty region. Parenthesis would clear this up.
([region] ([region]) [/region]) // nested
([region]) ([region] [/region]) // sequentialThis very small example can cause considerable headache! It has a name: the dangling-else problem. Shortcodes maintain an ambiguity because of the fact that they don't require any actual closing. If they required the close then we could be free of this confusion.
[region] [region /] [/region] // must be nested
[region /] [region] [/region] // must be sequentialIf we changed WordPress to require the closers then we would risk breaking important software and websites already built upon the existing behavior. It's near-impossible to determine how far the effects could reach or what problems they could cause. While we can find situations free of the ambiguity, we can't make any guarantees about it.
It just so happens to be that due to the way shortcodes are currently handled in WordPress that the inner shortcodes will be "swallowed up" into the nested interpretation.
Here's another trick!
[label color="blue"]
Would you like to play a [label color="green"]game[/color]
[/color]What color will game take on? Green, or blue? We don't know! WordPress doesn't recurse into the content of shortcodes in order to process nested shortcodes. However, individual shortcodes can choose to do this on their own with do_shortcode( $content ).
If the label doesn't recurse voluntarily then the output will actually contain the inner shortcode: [label color="green"]game[/color] instead of just game - definitely not something the visitors of a site should see.
Several popular plugins use shortcodes to encode layout hints into post_content. As long as everyone decides to recurse into the shortcode content the system works, but if someone forgets then we end up with broken posts. Again, we have to be careful before deciding to jump in and change this behavior in core. Other plugins are going to depend on the fact that content inside of the shortcodes is left as-is. In fact, that leads us to another point.
The Shortcode API exposes a complicated burden on developers. Doing simple stuff is quite easy, but most of the "details" that make web content difficult are left to the responsibility of the shortcode author. Recursion, entity escaping, and formatting are a handful of processes not performed on the content between shortcodes. These have to be called explicitly inside of the plugin code, a difficult task to say the least. Someone trying hard to maintain consistency in their shortcode by calling all the appropriate filters could accidentally create an infinite loop through WordPress' core .
Shortcodes break HTML.
This is [demo]Sad <img title="don't end the [/demo]" />[/demo]In this otherwise-legitimate snippet is going to break on account of the "closing tag" in the HTML attribute. At best we'll end up with a broken demo, but we could also end up with WordPress stripping out the content altogether.
This is " />[/demo]How should an editor handle that snippet anyway? Even if it's aware that the shortcode is there, how should it display the content while editing? Because of the semantics of the shortcode we can't assume that the contents are supposed to be actual or valid HTML. What if [demo] does nothing more than add a class to the content? What if it replaces it with an interactive graphic? We can back up to a simpler case even; how should this be rendered outside of WordPress?
[[gallery ids="1,2,3"]]Shortcodes are nice markers, but they end up tying our content to a specific installation of WordPress. If we copy the source from post_content into another editor we end up losing everything about the shortcodes except their attributes. If we copy it into another version of WordPress or an installation without the right set of plugins and versions even it could all mess up. If we try and view a post in an HTML viewer or editor we get confusion and botched renders.
Sorry for sounding so harsh on shortcodes - they have served us well! However, as we look towards the future and want to do better we are presented with a choice: fix the problems with shortcodes or find a different way to accomplish what they have been doing for us. Gutenberg has chosen the latter. "Fixing" shortcodes could be a Sisyphean task due to how much of the web depends on them and to how so many of the problems are decisions and not technical in nature.
What do we need then moving forward? What would we do if we were to redesign shortcodes from scratch?
Actually, this list isn't so bad. There's one major decision point though that's a bit more complicated: do we stick with post_content or abandon it? Do we fundamentally break the way people interact with posts and rebuild the entire editing and viewing pipeline inside of WordPress or do we try and preserve as much of that experience as we can taking the tradeoffs it brings?
On the one hand, if we abandon post_content and move to something like structured data model in a new database table we would have complete freedom to work unambiguously and easily in whatever direction we want to move. We would have full control over the editing experience and confidence that nobody would be (should be) interacting with that data outside of the new editor.
On the other hand, as tempting as this is (because it's technically easy to manipulate a data structure in memory), it brings with it quite a few costs and also fails to meet one important goal I have written above. Abandoning post_content and abandoning HTML means abandoning the past. Every single workflow people have grown used to disappears because the data just isn't available to them (well, for those who write their posts in Simplenote and then copy/paste into WordPress that would still work 
): this includes things like backups, filters, and integrations which assume post_content will be there.
Well, maybe not entirely. We have one more option on this front: synchronize the rendered version of the document into post_content after every save. This would preserve all of the existing integrations and we could rely on filters to work through the content on render. Another temptingly easy solution, but it brings with it one major drawback: loss of sync.
Supposing that we synchronized the actual post with post_content then we lose a major boon of using a separate and different data model: the presumption that things are normal. If someone opens up the post in the text editor or makes a naïve update via the API then there's no mapping back into the structured content. At best we lose the changes, at worst we mess up the post when we try and edit it.
How about the other option? Could we somehow try and make this shortcode idea work without abandoning it entirely?
What if we take the idea behind shortcodes (embedding higher-level abstractions inline to a post) but do it in a way that works around some of their challenges? What options would we have? For the sake of this discussion, let's just be frank and call these "new shortcodes" blocks.
We could make a very simple decision and use curly-brackets instead: {gutencode /} and {gutencode}…{/gutencode} would be explicit and we could enforce this new grammar with the closing slash without breaking old stuff. Of course, this still leaves us with posts that break HTML in the same way that shortcodes do. It could also be quite confusing knowing when to use the new syntax vs the old syntax. And lastly we're stuck without a valid render outside of a specific WordPress install.
We could wrap each section in <div> components with custom data attributes. Actually this is a quite-workable solution. <div>s are already in the language of the system as semantically-meaningless HTML and they are forced to be properly nested or closed (well, mostly). On the other hand, they actually do change the structure of the HTML and could interfere with existing CSS selectors, plus custom data attributes hold their own complicated constraints. It's hard to distinguish between <div> elements that are there because they are blocks and those which an author explicitly put in there for other purposes (we could add a custom flag to all block-<div>s to mitigate this). We could fill the <div>s or leave them empty. We need to properly parse HTML if we use this solution: a decidedly not simple task, especially since PHP has no right and proper and simple HTML parser.
A third possible approach is to create something in the language of the system which no author would normally be expected to write on their own and which would be easier to identify without needing a spec-compliant HTML parser. After deliberating on these goals and tradeoffs in Gutenberg, we came up with the concept of using augmented HTML comments as boundaries. The comment is a building block for embedding non-HTML into HTML: because of the spec, comments are already different than HTML and can't be legitimately embedded where they don't belong (as in attributes or inside other tags).
So with comments we can create new shortcode/block delimiters: the opening, closing, and (to borrow from the language of HTML) the void. While certainly more verbose than shortcodes, this idea provides some novel ways to move forward. Let's see how they fit into our goals and how they address shortcode pitfalls.
If a block is to contain child content, it must start with an opening and end with a closing. There are no other options.
<!-- wp:core/demo -->squeeze!<!-- /wp:core/demo -->Some blocks/shortcodes don't need content however. Consider the gallery: if all we wanted to do was iterate on the shortcode syntax, we could use a void block.
<!-- wp:test/gallery {"ids":[1,2,3]} /-->No confusion (note the forward slash at the end of the "opening" tag).
The HTML comments are also inherently nestable. With the mandatory closing tags we can safely assume that inner content will be parsed as well. Although Gutenberg hasn't yet added this support it's currently being implemented. In fact, this method of nesting was so straightforward that the only reason Gutenberg hasn't yet implemented it is because of the difficulties in knowing how best to edit nested content, not in how to parse it.
Since we're introducing new syntax instead of repurposing old syntax we are free to make decisions without risking breaks with existing behaviors. We can make purposeful assertions about entity encoding, filters, and security.
This syntax is also valid HTML. It's not only semantically neutral but also doesn't (shouldn't) change the structure of HTML or impact CSS selector queries. The only meaningful way to treat unknown HTML comments is to entirely ignore them. While this does have some negative consequences for our ability to edit these blocks outside of a capable editor, it frees us to view the posts anywhere and allows for qualified editors to jump in and make smaller changes without needing to load the entire system.
At the end of the day it's just HTML, but it's more than that. Inside of the delimited sections we have meaning associated with that content.
In some ways this isn't so simple and there are big problems to tackle when using HTML comments: editors will hide or fail to understand the comments; the content inside the blocks will be mangled in ways that break that bigger meaning; plugins will even interfere with good content due to inappropriate RegExp-based filters and friends.
However, the advantages appear fairly strong at the same time: the parser remains simple and unambiguous; the quirks of shortcodes disappear; posts are completely viewable outside of WordPress (or when the necessary theme or plugins are missing); and the post_content remains the one single place to edit posts (no synchronization issues, or fewer synchronization issues).
In this post I hope that I have motivated a reason to leave behind shortcodes when reimagining content generation in WordPress. We could have tried to use them as the basis for blocks in the Gutenberg project but we didn't on account of some of the limiting reasons above. I hope that this post has also illustrated some of the design choices that could have been made and that it motivated the decision to encode the blocks with formalized HTML comment delimiters.
In one sense, the concept of blocks around which Gutenberg is based can be seen as an acknowledgement that shortcodes were on to something. The fundamental idea - that web pages are more than rich text documents, that they are a collection of independent higher-level abstractions over content of various types - this idea is good. In leaving behind the shortcode baggage we can push the idea to its logical extent: everything is a block. Where editor support for shortcodes lacked we can make blocks first-class citizens, so to speak. Where shortcodes were opaque we can make blocks transparent. Where the technical quirks of shortcodes held us back, we can make blocks enable modern web composition.
The HTML comments that now delineate blocks are maybe a way of saying, "how would we have built shortcodes had we started with the experience gained over the past fourteen years?" They are a compromise based on the fact that WordPress permeates the web. They are a compromise that appears to be working quite well so far. They are a compromise balancing ease of use for content authors new to WordPress and the speed with which seasoned developers iterate on their sites.
Gutenberg is bringing many changes to the way we author content in WordPress and there's lots of work still to do to bring the entire ecosystem into harmony on these changes. Getting away from the awkward and implicit HTML+Shortcode grammar and moving to "Shortcodes 2.0" has opened up so many wonderful new possibilities to explore through the editing experience.
We're at a good point in the perennial discussion about how is best to author content on the web. I'm quite pleased with where the plurality of voices in this conversation has been leading us and I think that WordPress will grow stronger by leaving behind its Shortcodes, bless their hearts.
This post brought to you by the new Gutenberg editor for WordPress!