Automapping: Explicitly detect map edges #3858

Some time ago I built out my own autotiling system (before trying out and leaning into Tiled's) I came to the conclusion that a lot of features are very context-depedent (like a regex), and I'll just keep finding new match cases that users would want over time.

As far as I got on the particular problem in this issue, I defined rule-specific logic for out-of-bounds tiles as either being "always positive" match, "always negative" match, or "match as if extruded". Again, this property was defined on the "Filter" (equivalent to a Tiled "rule", it's also similar to a Tiled map's MatchOutsideMap property.). As far as I know, Tiled rules are not first class objects that have properties, this may be part of the difficulty in finding a simple solution.

Regardless, I still ended up with a realization that these things are ultimately complex, and all cases can't be captured.

So, line of thinking on my autotile system was to allow custom-defined matching. For Tiled, this would mean adding a new MatchType that is "Custom", and allow a script of some sort to be used. Users could traverse the nearby tiles, define matching parameters and patterns. The script itself might be contained in a correlated property MatchTypeScript. The MatchTypeScript could be used alternatively in place of MatchType. This would catch all cases.

When looking through the Tiled source, it seemed like the roadblock in this would be injecting a script interpreter for this MatchType around here, and all the work involved in verifying and validating the script.

These were my thoughts on the topic at least.

Again, this property was defined on the "Filter" (equivalent to a Tiled "rule", it's also similar to a Tiled map's MatchOutsideMap property.). As far as I know, Tiled rules are not first class objects that have properties, this may be part of the difficulty in finding a simple solution.

Well, in fact since Tiled 1.9, rules can have individual properties by placing rectangle objects over those rules and setting certain properties on them. However, MatchOutsideMap currently remains a per-map property, but if desired it could be made a per-rule option as well along with the OverflowBorder and WrapBorder properties. Neither really provides a solution to this issue, though.

I think another option which hasn't been mentioned here, would be the ability to specify that the outside of a map should be considered equal to a specific tile. Then, you can use that tile to match it, or in some cases, using this option will be enough to make your rules behave as desired on the edges. The main issue with this type of configuration is that there is currently no custom "tile reference" property, though this has already been asked about before (#3235).

I think another option which hasn't been mentioned here, would be the ability to specify that the outside of a map should be considered equal to a specific tile.

I think this adds unnecessary complications when instead the outside of a map could match MatchType "OutsideMap" or something. I don't see much benefit in customising which tile the outside of the map matches, since it's so easy to add a specific special tile as an input option to rules. We already have options for treating the outside as empty (MatchOutsideMap on its own), as extruded (OverflowBorder) and as wrapped (WrapBorder), perhaps this could be a fourth option (maybe "MarkBorder"?).

Edit: Ideally, I'd like for "OutsideMap" to always match tiles outside the map regardless of the OverflowBorder, WrapBorder, etc options, so that I can use both in some cases. For example, rules that tidy up tree trunks might work with OverflowBorder and just pretend the trunk continues downwards past the map edge, but rules that tidy up tree crowns might want to look for tiles outside the map edge specifically and just pretend they crowns are "correct", while extruding wouldn't give correct matches since normal crowns look diffrent from that. But I understand this makes the implementation more difficult. I think making the MatchOutsideMap properties be per-rule would help, since it would allow keeping rules that need different border behaviours in a single map, so my example could have OverflowBorder on the rules map for my trunks, but "OutsideMap" for the tree crown rules.

I think this adds unnecessary complications when instead the outside of a map could match MatchType "OutsideMap" or something.

While an "OutsideMap" matching tile might sound like a good idea from a user's perspective, it's a somewhat different story from the implementation side. Currently, all match types reflect some set of tiles that are either desired or not desired and the input matching function can just loop over those and compare them (so, in this process, the "match type" is no longer relevant at all - it is currently only inspected during the "rule compilation" step).

So, dragging "match type" into the matching code is somewhat involved and likely to affect performance, whereas adding a fallback tile to use for "outside the map" would be relatively straight-forward, heh.

Makes sense D: Maybe it could use a fake "outside the map" tile from a secret Tileset that only Tiled knows about? This could behave like any other tile to the matching code. MatchType "OutsideMap" would match this tile, and the out-of-bounds area would be treated as populated with this tile when the relevant options are set.

Still not ideal though, as I think it would be best to be able to address the out-of-bounds region independently of the "tiles" that "populate" it, allowing mixing both out of bounds matching and the different MatchOutsideMap modes within a single rule.

How about a tile layer (similar to input_ and output_) where you could define overflow tiles that would implicitly be placed there for that rule in the event of an overflow. (Might call it overflow_?)

Imagine a twisty tree 6x8 derived from an input_ using an air tile and tree tile. As it approaches the overflow it would need a complex array of tiles to fill in the overflow shape in order to fit the match, and a simple OverflowBorder, WrapBorder, extrude, or map-or-rule-wide user-defined fill tile wouldn't fit the bill.

This could cause issues when multiple overflow rules could seemingly produce conflicting ideas of what tile exists in the overflow space, placing different "imaginary overflow tiles" to fit their match, creating a "Schrodinger's tile" situation.

How would this proposed overflow_ data relate to the working map, which does not contain these tiles, and what benefit would this have over a single overflow tile? Part of the reason for this feature request is precisely because it's common for rules to define contexts that get cut off by the map border in maps, and some way is needed to identify those situations. Since those details simply aren't in the map, some way to identify parts of the region that might be cut off is sufficient.

Just for clarity, I want to illustrate how a single tile would work to define the out-of-bounds area, whether that's a MatchType "OutsideMap" tile, or a custom tile: Let's say I have an input that matches this tree, and I want to match this tree even if any part of its crown is cut off because it's near the map edge. The bluish region here would be my "out of bounds" area defined by the single tile, any of these tiles would count as a match if they contain the appropriate tree crown tile, or are out of bounds:

This would mean either of these would match, in addition to many other possible matches, while still being more specific than matching just the two trunk tiles:

In practice, I'd probably mark up that entire rule with the out of bounds tile, so that it takes effect when even a single tile from the tree is in the map.

Maybe it could use a fake "outside the map" tile from a secret Tileset that only Tiled knows about? This could behave like any other tile to the matching code. MatchType "OutsideMap" would match this tile, and the out-of-bounds area would be treated as populated with this tile when the relevant options are set.

Yeah, that's an interesting idea, that should make the comparison only a little bit more complicated given that we'd still need to also support the case where tiles outside the map are treated as empty.

To come back about a section from your first post:

Another important issue is that if matching the borders is done using regular input layers, it can be unclear how this interacts with other input tiles at the same location. Unlike other tiles, I feel like the map border stuff should AND with the other tiles, not OR like input layers currently are.

In your above example of the tree, it seems you're suggesting to match as usual, considering "outside map OR part of tree" a match. Or was the reference to "edge" matching something else than "outside of map" matching? In any case I hope we can treat the layers the same, with "input" layers doing OR and "inputnot" layers doing AND.

In your above example of the tree, it seems you're suggesting to match as usual, considering "outside map OR part of tree" a match.

For that example rule, that is correct. For another rule, such as one defining a frame for a map or map-edge cleanup, I may have the outside-the-map tile as the only input at those locations.

As for that bit from the OP: I think there I was considering rules that I'd want to only match the border, while also checking for specific overflows (most useful with WrapBorder) - there are situations where one wants to look for those matches while also making sure the rule matches only at the border. But there are also situations like that tree example where one wants to match either at a border or not at a border. Thinking about it now, just treating it as regular tiles and ORing would probably be fine, and is closer to what's desired in most cases. The effect of AND can be accomplished by checking the border of another (possibly non-existent/dummy) layer instead of the layer where we check for specific tiles.

As for making sure Empty matches also work, could it perhaps be made that when MatchOutsideMap is true but the other two options aren't, MatchType Empty is treated as matching Empty and the secret OutsideMap tile? I guess that wouldn't really help in the wrap and extrude cases though, since you'd still need the OutsideMap tile to match outside map and the overflow/wrap tiles... Edit: Since borders are shared by all layers and there are no plans to support layer offset compensation, maybe any time Automapping encounters the MatchType "OutsideMap" tile, it should "move" it to a separate input layer that matches a dummy layer populated with empty inside the map and with the special tile outside the map? This would allow the regular input layers to match tiles as they always have, and the extra Tiled-generated layers would handle matching map bounds.

I wanted to revive this Feature suggestion understanding that there's probably ways to get around this by now, but there are some powerful use cases when we consider some of the more commonly available tools in engines like Godot.

A bit of context: For use cases like making a Metroidvania (that's me!), we use an addon called "MetSys" which functionally sets up the connections between rooms. These connections start and end almost exactly where the player exits the end of a viewport, so it doesn't leave a lot of room for error, but it works really well.

Let's say you have a room that is 2x3 viewports worth of size. Once you cross any "opening" on any of the edges of said map, you'll transition to the next room that's set up on that MetSys structure towards that side.

The use cases:

• Being able to auto-detect edges in Automapping would simplify setting up rooms to behave well with systems like MetSys and any other technology that uses viewport exits to transition the player to the next room.
• Also, there's always the fear of not having enough space for screen shake purposes, so detecting the edge of the map helps format the art correctly not only inside the room for screen shake, but for connecting to the next one over in terms of a fully visible map or image.
• Edge detection also helps us with "data tiles" like respawn positions for when the player takes environment damage that sends them back to where they were last safely standing.

Like I mentioned, I'm sure there are some ways around this now that a lot of time has passed since last this was discussed, but I wanted to bring up the possible evolution of the opportunity since Tiled has become our primary tool for so much and the more we can do in it automatically, the safer we are from small human errors that cause tons of rework!

Thanks as always!

EDIT: To be more precise, in function, being able to draw the borders of the map without some complex/manual process is what I'm envisioning for my personal use case, but I do see more value than that. I don't just need to draw the whole border, but also need to clear exits and make sure that the border of the map is X pixels close to the nearest viewport, etc. I've been able to do all of this but I have to draw the border of the map manually for now, which is not too much to ask for but definitely doesn't scale well over time compared to the power of Automapping.

I'm sure there are some ways around this now that a lot of time has passed since last this was discussed

This issue isn't that old, Automapping hasn't received any substantial new features since. The current way to detect map edges is the same as it I described in the OP. So, this feature would add no true new functionality, this is just about making that easier, more intuitive, and not require an extra layer to accomplish. I imagine that's why it hasn't been a high priority.

Edit: Oh, and scripting can also be used to add the borders, and has been able to since it was added years ago xP But scripting can get around a lot of missing features and shouldn't count ;D

Agree that there's workarounds, so I'm happy to just throw the 2 cents in to bubble the value of the reduced friction!

How would you like this to actually work, i.e. how do you expect to set up your rules for working with map edges? I've listed a bunch of options in the OP, but all of them have major downsides. Do you have a preference, or maybe some new ideas?

Yes, I didn't want to be presume I'd know better than you guys that have dealt with this for longer than me and already had the conversation but I can see how I could've been more helpful with some extra ideas to add more value.

For me, I see some options (once again, I primarily care about friction reduction, full automation, and ease of training other team members, etc.)

• A Property in the AutoTileRules map that treats the outside edge of the map as a new unique tile in the AutoMap Rules tileset (I think I'm very close or identical to one of the options you proposed with this one)
• An option at the Map level or at the Layer level that auto-draws the edge of the map minus some offset/inside pixels. For example: If I have a 25x25 map, I'd like to be able to make a layer and tell it "draw a box of the Map Size -2 tiles of height and -4 tiles of width with this tile from this Tileset" (I imagine scripting can do this, but yet again, friction is where I'm focusing on)
• An Automap tile that you can put as an Input that represents the entire inside border of the map (so the paintable edge).

An Automap tile that you can put as an Input that represents the entire inside border of the map (so the paintable edge).

I'm surprised I hadn't thought of this, actually. One of my options comes very close, but it didn't occur to me that Tiled could treat this differently, so InsideMap always only matches inside the map regardless of what out-of-bounds options are set. So this would basically be a one-tile version of my two-match-type suggestion. Edit: Ah wait no, because you still need to specify outside-the-map as a tile, otherwise it'll just be doing a plain match on the layer :/

This would, however, still require MatchOutsideMap or other out-of-bounds option to actually do anything, as without those options set, all tiles within a match region will be inside the map. But, intuitively, if you want to detect the map edge, you might not think that you actually need to look outside the map to contrast the inside with it.

Auto-drawing tiles along the borders is something that can already be done with a script, perhaps that's worth looking into? You can package a script with your Project or whatever else you're delivering to your team members, and it shouldn't be much more friction than learning another Automapping feature, no matter how well the latter is designed. Things like dealing with transitions are really probably best handled by scripts rather than Automapping in general, since there's usually logic to it beyond just tiles.

I think you're right, scripting is the solution here, but this is definitely something that I believe a lot of users that are venturing into AutoMapping and are making (lets say) platformers or MVs are going to not find intuitive at all. Even if I can understand the limitations, the verbiage of the current options makes it seem to me that we already have a way to detect the edge as simply as it sounds, but there's clear complications as you're presenting them. I'd agree with one of your prior statements, everything could be resolved by a script, but probably that'll up the barrier of entry a bit more than we might think.