4 Mar
2023
4 Mar
'23
1:31 a.m.
On Fri, Mar 3, 2023 at 2:06 PM Grant Taylor via COFF <coff(a)tuhs.org> wrote:
Thank you all for very interesting and engaging
comments & threads to
chase / pull / untangle.
I'd like to expand / refine my original question a little bit.
On 3/2/23 11:54 AM, Grant Taylor via COFF wrote:
The thing about backreferences is that they're not representable in
the regular languages because they require additional state (the thing
the backref refers to), so you cannot construct a DFA corresponding to
them, nor an NDFA simulator (this is where Freidl gets things wrong!);
you really need a pushdown automata and then you're in the domain of
the context-free languages. Therefore, "regexps" that use back
references are not actually regular expressions.
Yet, popular engines support them...but how? Well, pretty much all of
them use a backtracking implementation, which _can_ be exponential in
both time and space.
Now, that said, there are plenty of REs, even some with backrefs,
that'll execute plenty fast enough on backtracking implementations; it
really depends on the expressions in question and the size of strings
you're trying to match against. But you lose the bounding guarantees
DFAs and NDFAs provide.
I'd like some thoughts ~> input on
extended regular expressions used
with grep, specifically GNU grep -e / egrep.
While some reading of the references that Clem provided I came across
multiple indications that back-references can be problematic from a
performance stand point.
So I'd like to know if all back-references are problematic, or if very
specific back-references are okay. Suppose I have the following two lines:
aaa aaa
aaa bbb
Does the following RE w/ back-reference introduce a big performance penalty?
(aaa|bbb) \1
As in:
% echo "aaa aaa" | egrep "(aaa|bbb) \1"
aaa aaa
I can easily see how a back reference to something that is not a fixed
length can become a rabbit hole. But I'm wondering if a back-reference
to -- what I think is called -- an alternation (with length fixed in the
RE) is a performance hit or not.
Well, it's more about the implementation strategy than the specific
expression here. Could this become exponential? I don't think this one
would, no; but others may, particularly if you use Kleene closures in
the alternation.
This _is_ something that appears in the wild, by the way, not just in
theory; I did a change to Google's spelling service code to replace
PCRE with re2 precisely because it was blowing up with exponential
memory usage on some user input. The problems went away, but I had to
rewrite a bunch of the REs involved.
- Dan C.