Grant - check out Russ Cox's web page on this very subject: Implementing Regular Expressions <https://streaklinks.com/BaglRcA5OeZepwiX_AELMUI5/https%3A%2F%2Fswtch.com%2F%7Ersc%2Fregexp%2F> ᐧ On Thu, Mar 2, 2023 at 1:55 PM Grant Taylor via COFF &lt;coff(a)tuhs.org&gt; wrote:

On Thu, Mar 2, 2023 at 1:55 PM Grant Taylor via COFF &lt;coff(a)tuhs.org&gt; wrote: Well, obviously the former matches any sequence 3 of alpha-numerics/underscores at the beginning of a string, while the latter only matches abbreviations of months in the western calendar; that is, the two REs are matching very different things (the latter is a strict subset of the former). But I suspect you mean in a more general sense. ...do you really want to match a space, a colon and a single digit 11 times in a single string? Using character classes would greatly simplify what you're trying to do. It seems like this could be simplified to (untested) snippet: ( [1-9]|[12][[0-9]]|3[01]) [0-2][0-9]:[0-5][0-9]:[0-5][0-9] For this, I'd probably eschew `[:digit:]`. Named character classes are for handy locale support, or in lieu of typing every character in the alphabet (though we can use ranges to abbreviate that), but it kind of seems like that's not coming into play here and, IMHO, `[0-9]` is clearer in context. It's not clear to me that dates, in their generality, can be matched with regular expressions. Consider leap years; you'd almost necessarily have to use backtracking for that, but I admit I haven't thought it through. `\w` is a GNU extension; I'd probably avoid it on portability grounds (though `\b` is very handy). The thing about regular expressions is that they describe regular languages, and regular languages are those for which there exists a finite automaton that can recognize the language. An important class of finite automata are deterministic finite automata; by definition, recognition by such automata are linear in the length of the input. However, construction of a DFA for any given regular expression can be superlinear (in fact, it can be exponential) so practically speaking, we usually construct non-deterministic finite automata (NDFAs) and "simulate" their execution for matching. NDFAs generalize DFAs (DFAs are a subset of NDFAs, incidentally) in that, in any non-terminal state, there can be multiple subsequent states that the machine can transition to given an input symbol. When executed, for any state, the simulator will transition to every permissible subsequent state simultaneously, discarding impossible states as they become evident. This implies that NDFA execution is superlinear, but it is bounded, and is O(n*m*e), where n is the length of the input, m is the number of nodes in the state transition graph corresponding to the NDFA, and e is the maximum number of edges leaving any node in that graph (for a fully connected graph, that would m, so this can be up to O(n*m^2)). Construction of an NDFA is O(m), so while it's slower to execute, it's actually possible to construct in a reasonable amount of time. Russ's excellent series of articles that Clem linked to gives details and algorithms. In practical terms? Basically, don't worry about it too much. Egrep will generate an NDFA simulation that's going to be acceptably fast for all but the weirdest cases. - Dan C.

Stuff Received wrote in &lt;c4328f0f-f304-e131-a9f3-32ce01fa6814(a)riddermarkfarm.ca&gt;: |On 2023-03-02 14:23, Clem Cole wrote: |> Grant - check out Russ Cox's web page on this very subject: Implementing |> Regular Expressions |> <https://streaklinks.com/BaglRcA5OeZepwiX_AELMUI5/https%3A%2F%2Fswtch.co\ |> m%2F%7Ersc%2Fregexp%2F> | |Clem, why are you linking through streaklinks.com? I do not want to be unfriendly; (but) i use firefox-bin (mozilla-compiled) and my only extension is uMatrix that i have been pointed to, and i can only recommend it highly to anyone (though the "modern" web mostly requires to turn off tracking protection and numerous white flags in uMatrix to work), maybe even to those who simply put their browser into a container. Anyhow, uMatrix gives you the following, and while i have not tried to selectively click me through to get to the target, i could have done so: streak.com www.streak.com cloudflare.com cdnjs.cloudflare.com d3e54v103j8qbb.cloudfront.net facebook.net connect.facebook.net google.com www.google.com ajax.googleapis.com storage.googleapis.com intercom.io widget.intercom.io licdn.com snap.licdn.com pdst.fm cdn.pdst.fm producthunt.com api.producthunt.com sentry-cdn.com js.sentry-cdn.com website-files.com assets.website-files.com assets-global.website-files.com google-analytics.com www.google-analytics.com googletagmanager.com www.googletagmanager.com Randomized links i find just terrible. IETF started using randomized archive links, which are mesmerising; most often mailman archive links give you a bit of orientation by themselves, isn't that more appealing. --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt)

On 3/2/23 4:01 PM, Stuff Received wrote: Here's a direct link to the page that I landed on when following Clem's link: Link - Implementing Regular Expressions - https://swtch.com/~rsc/regexp/ I didn't pay attention to Clem's link beyond the fact that I got to the desired page without needing to tilt at my various filtering plugins. Though the message I'm replying to has caused a few brain cells to find themselves in confusion ~> curiosity. -- Grant. . . . unix || die

On Thu, Mar 2, 2023 at 8:06 PM Grant Taylor via COFF &lt;coff(a)tuhs.org&gt; wrote: I guess what I'm saying is, match what you want to match and don't sweat the small stuff. Not exactly. :-) What I understand you to mean, based on this and the rest of your note, is that you want to find a good division point between overly specific, complex REs and simpler, easy to understand REs that are less specific. The danger with the latter is that they may match things you don't intend, while the former are harder to maintain and (arguably) more brittle. I can sympathize. For the purposes of grep/egrep, that'll be a logical "line" of text, terminated by a newline, though the newline itself isn't considered part of the text for matching. I believe the `-z` option can be used to set a NUL byte as the "line" terminator; presumably this lets one match strings with embedded newlines, though I haven't tried. "string" in this context is the input you're attempting to match against. `egrep` will attempt to match your pattern against each "line" of text it reads from the files its searching. That is, each line in your log file(s). But consider what `[ :[:digit:]]{11}` means: you've got a character class consisting of space, colon and a digit; {11} means "match any of the characters in that class exactly 11 times" (as opposed to other variations on the '{}' syntax that say "at least m times", "at most n times", or "between n and m times"). But that'll match all sorts of things that don't look like 'dd hh:mm:ss': term% egrep '[ :[:digit:]]{11}' 11111111111 11111111111 111111111 1111111111111 1111111111111 :::::::::::::::: :::::::::::::::: aaaa bbbbb aaaa bbbbb term% (The first line is my typing; the second is output from egrep except for the short line of 9 '1's, for which egrep had no output. That last two lines are matching space characters and egrep echoing the match, but I'm guessing gmail will eat those.) Note that there are inputs with more than 11 characters that match; this is because there is some 11-character substring that matches the RE in those lines. In any event, I suspect this would generally not be what you want. But if nothing else in your input can match the RE (which you might know a priori because of domain knowledge about whatever is generating those logs) then it's no big deal, even if the RE was capable of matching more things generally. Ah. I suspect this relies on domain knowledge about the format of log lines to match reliably. Otherwise it could match, `___ 123 456:789` which is probably not what you are expecting. Sure. One nice thing about `egrep` et al is that you can put the REs into a file and include them with `-f`, as opposed to having them all directly on the command line. Typo. :-) far. That seems reasonable. Aside: I found the note on it's website amusing: Brought to you by the UK's best gambling sites! "Only gamble with what you can afford to lose." Yikes! I'd proceed with caution here; it also seems to be in the FreeBSD and DragonFly ports collections and Homebrew on the Mac (but so is GNU grep for all of those). things. Yeah. IMHO `\w` is too general for what you're trying to do. Basically, a regular expression is a regular expression if you can build a machine with no additional memory that can tell you whether or not a given string matches the RE examining its input one character at a time. I think that's about right. Sure thing! - Dan C.

On 3/2/23 8:04 PM, Dan Cross wrote: ACK You got it. Fair enough. That's also sort of what I thought might be the case. *nod* Yep, I'm well aware of the that. That's one of the reasons that I'm interested in coming up with a more precise regular expression ... without being overly complex. Yep. Here's an example of the full RE: ^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ postfix/msa/smtpd\[[[:digit:]]+\]: timeout after STARTTLS from [._[:alnum:]-]+\[[.:[:xdigit:]]+\]$ As you can see the "[ :[:digit:]]{11}" is actually only a sub-part of a larger RE and there is bounding & delimiting around the subpart. This is to match a standard message from postfix via standard SYSLOG. Yep. Though said domain knowledge isn't anything special in and of itself. Yep. logcheck makes extensive use of many files like this to do it's work. ACKK Thank you for the logic CRC. Um ... that's concerning. Fair enough. My use case is on Linux where GNU egrep is a thing. I think that `\w` is a good primer, but not where I want things to end up long term. I /think/ that I could build a complex nested tree of switch statements to test each character to see if things match what they should or not. Though I would need at least one variable / memory to hold absolutely minimal state to know where I am in the switch tree. I think a number to identify the switch statement in question would be sufficient. So I'm guessing two bytes of variable and uncounted bytes of program code. Thank you again Dan. :-) -- Grant. . . . unix || die

On Fri, Mar 3, 2023 at 5:59 AM Ralph Corderoy &lt;ralph(a)inputplus.co.uk&gt; wrote: I'm afraid I must sound a note of caution about Friedl's book. Russ Cox alludes to some of the problems in the "History and References" section of his page (https://swtch.com/~rsc/regexp/regexp1.html) that was linked earlier, and he links to this post: http://regex.info/blog/2006-09-15/248 The impression is that Friedl shows wonderfully how to _use_ regular expressions, but does not understand the theory behind their implementation. It is certainly true that today what many people refer to as "regular expressions" are not in fact regular (and require a pushdown automata to implement, putting them somewhere between REs and the context-free languages in terms of expressiveness). Personally, I'd stick with Russ's stuff, especially as `egrep` is the target here. - Dan C.

On Thu, Mar 2, 2023 at 10:53 PM Grant Taylor via COFF &lt;coff(a)tuhs.org&gt; wrote: Oh, for sure; to be clear, it was obvious that in the earlier discussion the original was just part of something larger. FWIW, this RE seems ok to me; the additional context makes it unlikely to match something else accidentally. It needn't be special. The point is simply that there's some external knowledge that can be brought to bear to guide the shape of the REs. In this case, you know that log lines won't begin with `___ 123 456:789` or other similar junk. Kinda. The "machine" in this case is actually an abstraction, like a Turing machine. The salient point here is that REs map to finite state machines, and in particular, one need not keep (say) a stack of prior states when simulating them. Note that even in an NDFA simulation, where one keeps track of what states one may be in, one doesn't need to keep track of how one got into those states. Obviously in a real implementation you've got the program counter, register contents, local variables, etc, all of which consume "memory" in the conventional sense. But the point is that you don't need additional memory proportional to anything other than the size of the RE. DFA implementation could be implemented entirely with `switch` and `goto` if one wanted, as opposed to a bunch of mutually recursive function calls, NDFA simulation similarly except that you need some (bounded) additional memory to hold the active set of states. Contrast this with a pushdown automata, which can parse a context-free language, in which a stack is maintained that can store additional information relative to the input (for example, an already seen character). Pushdown automata can, for example, recognize matched parenthesis while regular languages cannot. Anyway, sorry, this is all rather more theoretical than is perhaps interesting or useful. Bottom line is, I think your REs are probably fine. `egrep` will complain at you if they are not, and I wouldn't worry too much about optimizing them: I'd "stop" whenever you're happy that you've got something understandable that matches what you want it to match. - Dan C.

On Fri, Mar 3, 2023 at 11:12 AM Dave Horsfall &lt;dave(a)horsfall.org&gt; wrote: If I may, I'd like to point out something fairly subtle here that, I think, bears on the original question (paraphrased as, "where does one draw the line between concision and understandability?"). Note Dave's class to match the first letter of the month: `[JFMAMJJASOND]`. One may notice that a few letters are repeated (J, M, A), and one _could_ shorten this to: `[JFMASOND]`. But I can see a serious argument where that may be regarded as a mistake; in particular, the original is easy to validate by just saying the names of the month out loud as one scans the list. For the shorter version, I'd worry that I would miss something or make a mistake. The lesson here is keep it simple and don't over-optimize! Too true. A few years ago, Rob Pike gave a talk about lexing in Go that bears on this that's worth a listen: https://www.youtube.com/watch?v=HxaD_trXwRE - Dan C.

Thank you all for very interesting and engaging comments & threads to chase / pull / untangle. I'd like to expand / refine my original question a little bit. On 3/2/23 11:54 AM, Grant Taylor via COFF wrote: While some reading of the references that Clem provided I came across multiple indications that back-references can be problematic from a performance stand point. So I'd like to know if all back-references are problematic, or if very specific back-references are okay. Suppose I have the following two lines: aaa aaa aaa bbb Does the following RE w/ back-reference introduce a big performance penalty? (aaa|bbb) \1 As in: % echo "aaa aaa" | egrep "(aaa|bbb) \1" aaa aaa I can easily see how a back reference to something that is not a fixed length can become a rabbit hole. But I'm wondering if a back-reference to -- what I think is called -- an alternation (with length fixed in the RE) is a performance hit or not. Now to read and reply to the many good comments that people have shared. :-) -- Grant. . . . unix || die

On 3/3/23 6:42 AM, Ralph Corderoy wrote: You are mostly correct. The motivation for this thread is very much so wanting to learn "how best to use today's regular expression implementations". However there is also the part of me that wants to have a little bit of understanding behind why the former is the case. It seems like I need to find another copy of Friedl's book. -- My current copy is boxed up for a move nearly 1k miles away. :-/ It sounds like I'm coming from a similar position of "what is the best* way to process this corpus" more than "what is the underlying theory behind what I'm wanting to do". :-) I'm learning that I'm more of a technician that wants to know how to use the existing tools to the best of his / their ability. While having some interest in theory behind things. Yep. That's the position that I would be in if someone were paying me to write the REs that I'm writing. This all sounds interesting to me, and like something I might add to my collection of books. But it also sounds like something that will be an up hill read and vast learning opportunity. Yep. Even inventorying and keeping track of the books can be time consuming. -- Thankfully I took some time to do exactly that and have access to that information on the super computer in my pocket. -- Grant. . . . unix || die

On Fri, Mar 3, 2023 at 2:06 PM Grant Taylor via COFF &lt;coff(a)tuhs.org&gt; wrote: The thing about backreferences is that they're not representable in the regular languages because they require additional state (the thing the backref refers to), so you cannot construct a DFA corresponding to them, nor an NDFA simulator (this is where Freidl gets things wrong!); you really need a pushdown automata and then you're in the domain of the context-free languages. Therefore, "regexps" that use back references are not actually regular expressions. Yet, popular engines support them...but how? Well, pretty much all of them use a backtracking implementation, which _can_ be exponential in both time and space. Now, that said, there are plenty of REs, even some with backrefs, that'll execute plenty fast enough on backtracking implementations; it really depends on the expressions in question and the size of strings you're trying to match against. But you lose the bounding guarantees DFAs and NDFAs provide. Well, it's more about the implementation strategy than the specific expression here. Could this become exponential? I don't think this one would, no; but others may, particularly if you use Kleene closures in the alternation. This _is_ something that appears in the wild, by the way, not just in theory; I did a change to Google's spelling service code to replace PCRE with re2 precisely because it was blowing up with exponential memory usage on some user input. The problems went away, but I had to rewrite a bunch of the REs involved. - Dan C.

Hi Grant, You could measure the number of CPU instructions and experiment. $ echo xyzaaa aaaxyz >f $ ticks() { LC_ALL=C perf stat -e instructions egrep "$@"; } $ $ ticks '(aaa|bbb) \1' <f xyzaaa aaaxyz Performance counter stats for 'egrep (aaa|bbb) \1': 2790889 instructions:u 0.009146904 seconds time elapsed 0.009178000 seconds user 0.000000000 seconds sys $ Bear in mind that egreps differ, even within GNU egrep, say, over time. $ LC_ALL=C perf stat -e instructions egrep '(aaa|bbb) \1' f xyzaaa aaaxyz ... 2795836 instructions:u ... $ LC_ALL=C perf stat -e instructions perl -ne '/(aaa|bbb) \1/ and print' f xyzaaa aaaxyz ... 2563488 instructions:u ... $ LC_ALL=C perf stat -e instructions sed -nr '/(aaa|bbb) \1/p' f xyzaaa aaaxyz ... 610213 instructions:u ... $ -- Cheers, Ralph.

Hi Grant, For those regexp engines which test each alternative in turn, ordering the months most-frequent first would give a slight win. :-) It really is a rabbit hole once you start. Typically not worth entering, but it can be fun if you like that kind of thing. You could develop your regexps to find lines of interest and then flip them about, e.g. egrep's -v, to see what lines are missed and consider if any are interesting. Repeat. But this happens at development time. Or at run time, you can have a ‘loose’ regexp to let all expected lines in through the door and then match with one or more ‘tight’ regexps, baulking if none do. There's no right answer in general. -- Cheers, Ralph.

On Mon, Mar 6, 2023 at 5:02 AM Ed Bradford &lt;egbegb2(a)gmail.com&gt; wrote: This reminds me of something that I wanted to bring up. Perhaps one _could_ define a sufficiently rich regular expression that one could match a number of date formats. However, I submit that one _should not_. REs may be sufficiently powerful, but in all likelihood what you'll end up with is an unreadable mess; it's like people who abuse `sed` or whatever to execute complex, general purpose programs: yeah, it's a clever hack, but that doesn't mean you should do it. Pick the right tool for the job. REs are a powerful tool, but they're not the right tool for _every_ job, and I'd argue that once you hit a threshold of complexity that'll be mostly self-evident, it's time to move on to something else. As for large vs small REs.... When we start talking about differences of orders of magnitude in size, we start talking about real performance implications; in general an NDFA simulation of a regular expression will have on the order of the length of the RE in states, so when the length of the RE is half a million symbols, that's half-a-million states, which practically speaking is a pretty big number, even though it's bounded is still a pretty big number, and even on modern CPUs. I wouldn't want to poke that bear. - Dan C.

On Mon, Mar 06, 2023 at 04:01:51PM -0500, Dan Cross wrote: Dan, I agree with you. I ran a software company for almost 20 years and the main thing I contributed was "lets be dumb". Lets write code that is easy to read, easy to bug fix. Smart engineers love to be clever, they would be the folks that wrote those long RE that worked magic. But that magic was something they understood and nobody else did. Less is more. Less is easy to support.

Hi Dan, It sounds to me like an "optimizer" is needed. There is alreay a compiler that uses FA's. Is someone else going to create a program to look for dates without using regular expressions? Today, I write small-sized RE's. If I write a giant RE, there is nothing preventing the owner of RE world to change how they are used. For instance. Compile your RE and a subroutine/function is produced that performs the RE search. RE is a *language*, not necessarily an implementation. At least that is my understanding. Ed On Mon, Mar 6, 2023 at 3:02 PM Dan Cross &lt;crossd(a)gmail.com&gt; wrote: -- Advice is judged by results, not by intentions. Cicero

On Mon, Mar 6, 2023 at 11:01 PM Ed Bradford &lt;egbegb2(a)gmail.com&gt; wrote: Actually, I believe you. I'm sure that with enough effort, it _is_ possible to make a 50K RE that is understandable to mere mortals. But it begs the question: why bother? The answer to that question, in my mind, shows the difference between a clever programmer and a pragmatic engineer. I submit that it's time to reach for another tool well before you get to an RE that big, and if one is still considering such a thing, one must really ask what properties of REs and the problem at hand one thinks lend itself to that as the solution. I'm not sure what you're referring to here, though you were replying to me. There are a couple of different threads floating around: 1. Writing really big regular expressions: this is probably a bad idea. Don't do it (see below). 2. Writing a recognizer for dates. Yeah, the small REs you have for that are fine. If you want to extend those to arbitrary date formats, I think you'll find it starts getting ugly. 3. Optimizing regular expressions. You're still bound by the known theoretical properties of finite automata here. Many people have already done so. :-) I'm not sure I understand what you mean. The theory here is well-understood: we know recognizers for regular languages can be built from DFAs, that run in time linear in the size of their inputs, but we also know that constructing such a DFA can be exponential in space and time, and thus impractical for many REs. We know that NDFA simulators can be built in time and space linear in the length of the RE, but that the resulting recognizers will be superlinear at runtime, proportional to the product of the length of input, number of states, and number edges between states in the state transition graph. For a very large regular expression, that's going to be a pretty big number, and even on modern CPUs won't be particularly fast. Compilation to native code won't really help you. There is no "owner of RE world" that can change that. If you can find some way to do so, I think that would qualify as a major breakthrough in computer science. Regular expressions describe regular languages, but as I mentioned above, the theory gives the currently understood bounds for their performance characteristics. It's kinda like the speed of light in this regard; we can't really make it go faster. - Dan C.

On 3/7/23 4:39 AM, Ralph Corderoy wrote: I want to agree, but then I run into things like this: ^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ postfix(/smtps)?/smtpd\[[[:digit:]]+\]: disconnect from [._[:alnum:]-]+\[[.:[:xdigit:]]+\]( helo=[[:digit:]]+(/[[:digit:]]+)?)?( ehlo=[[:digit:]]+(/[[:digit:]]+)?)?( starttls=[[:digit:]]+(/[[:digit:]]+)?)?( auth=[[:digit:]]+(/[[:digit:]]+)?)?( mail=[[:digit:]]+(/[[:digit:]]+)?)?( rcpt=[[:digit:]]+(/[[:digit:]]+)?)?( data=[[:digit:]]+(/[[:digit:]]+)?)?( bdat=[[:digit:]]+(/[[:digit:]]+)?)?( rset=[[:digit:]]+(/[[:digit:]]+)?)?( noop=[[:digit:]]+(/[[:digit:]]+)?)?( quit=[[:digit:]]+(/[[:digit:]]+)?)?( unknown=[[:digit:]]+(/[[:digit:]]+)?)?( commands=[[:digit:]]+(/[[:digit:]]+)?)?$ Which is produced by this m4: define(`DAEMONPID', `$1\[DIGITS\]:')dnl define(`DATE', `\w{3} [ :[:digit:]]{11}')dnl define(`DIGIT', `[[:digit:]]')dnl define(`DIGITS', `DIGIT+')dnl define(`HOST', `[._[:alnum:]-]+')dnl define(`HOSTIP', `HOST\[IP\]')dnl define(`IP', `[.:[:xdigit:]]+')dnl define(`VERB', `( $1=DIGITS`'(/DIGITS)?)?')dnl ^DATE HOST DAEMONPID(`postfix(/smtps)?/smtpd') disconnect from HOSTIP`'VERB(`helo')VERB(`ehlo')VERB(`starttls')VERB(`auth')VERB(`mail')VERB(`rcpt')VERB(`data')VERB(`bdat')VERB(`rset')VERB(`noop')VERB(`quit')VERB(`unknown')VERB(`commands')$ I only consider myself to be an /adequate/ m4 user. Though I've done some things that are arguably creating new languages. I personally find the generated regular expression to be onerous to read and understand, much less modify. I would be highly dependent on my editor's (vim's) parenthesis / square bracket matching (%) capability and / or would need to explode the RE into multiple components on multiple lines to have a hope of accurately understanding or modifying it. Conversely I think that the m4 is /largely/ find and replace with a little syntactic sugar around the definitions. I also think that anyone that does understand regular expressions and the concept of find & replace is likely to be able to both recognize patterns -- as in "VERB(...)" corresponds to "( $1=DIGITS`'(/DIGITS)?)?", that "DIGITS" corresponds to "DIGIT+", and that "DIGIT" corresponds to "[[:digit:]]". There seems to be a point between simple REs w/o any supporting constructor and complex REs with supporting constructor where I think it is better to have the constructors. Especially when duplication comes into play. If nothing else, the constructors are likely to reduce one-off typo errors. The typo will either be everywhere the constructor was used, or similarly be fixed everywhere at the same time. Conversely, finding an unmatched parenthesis or square bracket in the RE above will be annoying at best if not likely to be more daunting. I feel like there is an analogy between machine code and assembly language as well as assembly language and higher level languages. My understanding is that the computer industry has vastly agreed that the higher level language is easier to understand and maintain. That may very well be. But what will be more maintainable / easier to correct in the future; adding `{2}` when necessary or changing the value of `1` to `2`? I think this is an example of tradeoff of not strictly required to make something more maintainable down the road. Sort of like fleet vehicles vs non-fleet vehicles. I think this is another example of the maintainability. I'm also replying to only the COFF mailing list. There is a reasonable chance that the COFF mailing list and / or your account therein is configured to minimize duplicates meaning the COFF mailing list won't send you a copy if it sees your subscribed address as receiving a copy directly. I personally always prefer the mailing list copy and shun the direct copies. I think that the copy from the mailing list keeps the discussion on the mailing list and avoids accidental replies bypassing the mailing list. -- Grant. . . . unix || die

On Sat, Mar 04, 2023 at 10:15:33AM +0000, Ralph Corderoy wrote: My own recommendation for this is to get a dedicated ebook reader. It will feel a bit clumsy to have both a cretinphone and another thing with you, but at least the thing is doing the job. At least, mine keeps cropping across pages. Also, the e-ink/epaper display of ebook reader is not supposed to screw your eyes and/or circadian rhythms (not that I know anything specific, but I find it very strange that people shine blue light into their eyes for extended periods of time and do not even quietly protest - well, perhaps it is akin to what goes between human and a dog, they become alike to each other, now, when a human has cretinphone...). Or, if it just one pdf to read, then you should be fine reading it on bigger screen. HTH -- Regards, Tomasz Rola -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did "rm -rif" on the programmer's home ** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:tomasz_rola@bigfoot.com **

Thank you for the very useful comments. However, I disagree with you about the RE language. While I agree all RE experts don't need that, when I was hiring and gave some software to a new hire (whether an experienced programmer or a recent college grad) simply handing over huge RE's to my new hire was a daunting task to that person. I wrote that stuff that way to help remind me and anyone who might use the python program. I don't claim success. It does help me. When you say '{1}' is redundant, I think I did that to avoid any possibility of conflicts with the next string that is concatentated to the *Y_* (e.g. '*' or '+' or '{4,7}'). I am embarrassed I did not communicate that in the code. I had to think about it for a couple of hours before I recalled the "why". I will fix that. (it would be difficult to discuss this RE if I had to write "(19\d\d|20[01]\d|202" + "[0-" + lastYearRE + "]" + ") rather than just *Y_*). My initial thoughts on naming were I wanted the definition to be defined in exactly one place in the software. Python and the BTL folks told me to never use a constant in code. Always name it. Hence, I gave it a name. Each name might be used in multiple places. They might be imported. You are correct, the expression is unbalanced. I tried to remove the text2bytes(lastYearRE*)* call so the expression in this email was all text. I failed to remove the trailing *)* when I removed the call to text2bytes(). My hasty transcriptions might have produced similar errors in my email. Recall, my focus was on any file of any size. I'm on Windows 10 and an m1 MacBook. Python works on both. I don't have a Linux machine or enough desktop space to host one. I'm also mildly fed-up with virtual machines. Friedl taught me one thing. Most RE implementations are different. I'm trying to write a program that I could give to anyone and could reliably find a date (an RE) in any file. YYYY, MM, DD, HR, MI, SE, TH are words my user could use in the command line or in an options dialog. LAT and LON might also be possibilities. CST, EST, MST, PST, ... also. A 500 gigabyte archive or directory/folder of pictures and movies would be a great test target. I very much appreciate your comments. If this discussion is boring to others, I would be happy to take it to emails. I like your program. My experience with RE, grep, python, and sed suggests that anything but gnu grep and sed might not work due to the different implementations. I've been out of the Unix software business for 30 years after starting work at BTL in the 1970s and working on Version 6. I didn't know "printf" was now built into bash! That was a surprise. It's an incremental improvement, but doesn't compare with f-strings in python. *The interactive interpreter for python should have* *a "bash" mode?!* Does grep use a memory mapped file for its search, thereby avoiding all buffering boundaries? That too, would be new information to me. The additional complexity of dealing with buffering is more than annoying. Do you have any thoughts on how to verify a program that uses RE's. I've given no thought until now. My first thought for dates would be to write a separate module that simply searched through the file looking for 4 numbers in a row without using RE's, recording the offsets and 16 characters after and 1 character before in a python list of (offset,str) of tuples, ddddList, and using *dddd**List* as a proxy for the entire file. I could then aim my RE's at *ddddList*. *[A list of tuples in python* *is wonderful! !]* It seems to me '*' and '+' and {x,y} are the performance hogs in RE's. My RE's avoid them. One pass, I think, should suffice. What do you think? I haven't "archived" my 350 GB of pictures and movies, but one pass over all files therein ought to suffice, right? Two different programs that use different algorithms should be pretty good proof of correctness wouldn't you think? My RE's have no stars or pluses. If there is a mismatch before a match, give up and move on. On my Windows 10 machine, I have cygwin. Microsoft says my CPU doesn't have a TPM and the specific Intel Core I7 on my system is not supported so Windows 11 is not happening. Microsoft is DOS personified. (An unkind editorial remark about the low quality of software coming from Microsoft.) Anyway, I thank you again for your patience with me and your observations. I value your views and the other views I've seen here on coff(a)tuhs.org. I welcome all input to my education and will share all I have done so far with anyone who wants to collaborate, test, or is just curious. GOAL: run python program from an at-cost thumb drive that: reaps all media files from a user specified directory/folder tree and Adds files to the thumb drive. *Adds files* means Original file system is untouched Adds only unique files (hash codes are unique) Creates on the thumb drive a relative directory wherein the original file was found Prepends a "YYYY-MM-DD-" string to the filename if one can be found (EXIF is great shortcut). Copies srcroot/relative_path/oldfilename to thumbdrive/relative_path/YYYY-MM-DD-oldfilename or thumbdrive/relative_path/0000-oldfilename. Can also incrementally add new files by just scanning anywhere in any other computer file system or any other computer. Must work on Mac, Windows, and Linux What I have is a working prototype. It works on Mac and Windows. It doesn't do the date thing very well, and there are other shortcomings. I have delivered exactly one Christmas present to my favorite person in the world - a 400 GB SSD drive with all our pictures and media we have ever taken. The next things are to *add *more media and *re-unique-ify* (check) what is already present on the SSD drive and *improve the proper choice of "YYYY-MM-DD-" prefix* to filenames. I am retired and this is fun. I'm too old to want to get rich. Ed Bradford Pflugerville, TX egbegb2(a)gmail.com On Tue, Mar 7, 2023 at 5:40 AM Ralph Corderoy &lt;ralph(a)inputplus.co.uk&gt; wrote: -- Advice is judged by results, not by intentions. Cicero

On Tue, Jun 20, 2023 at 11:02:33AM -0500, Michael Parson wrote: Hello Michael, I am not challenging your choices (to each his/her own), but to add some alternative, my own preferences go toward: a. have sd card slot in a reader (I mean hardware with e-ink, not some app on a phone). This means a card can be slipped into the box without opening it. This means the box is not water-proof. However, I had a look inside and I suspect it can still be water-prooved with duct tape, if someone wants it so much. b. so far I was rather happy with Linux custom made by manufacturer, but not an Android - I am yet to try Android based ebook reader (but maybe not too fast). Phones with A* are rather lousy at keeping their batteries loaded, I wonder how eink devices fare - do they, too, need to be recharged every day? My reader is recharged every 2-3 weeks, when batt drops to about 70%, while I keep using it at least every second day for few hours at a time. I had once (many years go, when I was to buy my first reader) a dream of browsing web pages with it. However, built in browser in non-A* reader proved to be lousy, equally lousy to browser in A* phones that I have tried. So, my current ereader was never connected to the net because I see no point. Of course each model nowadays comes with wi-fi, it just does not add anything useful so no need to even configure it on home router etc. Nowadays, I would rather convert whatever to pdf or epub and upload to the reader. Reading wikipedia pages printed to pdf saved me plenty of grief, as opposed to trying them in a (builtin) browser. I suspect elinks could look much better, but trying this requires some free time (compiling/porting, uploading). As a side note, I have observed that some pdfs do not render too well on my reader - it seems that they make this software "too new" to be solid & fast nowadays. Same pdfs converted to djvu read like a dream, however. Having more than few supported book formats is nice. My reader also comes with BT, possibly meant to connect headphones but perhaps usable for BT keyboard. Might be a thing to try in a future (or not), I mention it to let others know there may be such an option in case they care about it (I really do not, but I do not make those things so what can I do...). HTH -- Regards, Tomasz Rola -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did "rm -rif" on the programmer's home ** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:tomasz_rola@bigfoot.com **

On Thu, Jun 22, 2023 at 10:45:43AM -0500, Michael Parson wrote: [...] Hello again and sorry for the late answer. Busy, hardware glitches, tired, etc. There is no use pattern as such. It was just very convenient to have one. When I bought my first reader, PocketBook 622 (or something like it, a good decade ago), it only had 2GB of flash. Of this, about half was already prefilled with free ebooks, free photos to see in some app. I decided to keep all this, which left me with .LT. 1GB for my use. If I wanted just epubs, this was more than enough to have few thousands files. But, an atlas of human anatomy from Internet Archive was ca. 8MB, Lisp 1.5 manual from bitsavers was another 9MB and if I wanted some more, space would quickly ran out. An 8GB card solved the problem. Later on, a member of family fell in love with e-reader, too. But, in her hands they died rather quickly. So we struck a deal - she will buy me a new one and get mine, which after a year of use was still mostly like new. I just replugged SD card and it was it. Now I have two broken Pocketbooks 622 in a drawer, waiting for their chance - just replace their displays. Later on, I bought me an Inkpad (again from Pocketbook, probably I am a fanboy). This one had about 7.8'' display, many more inky pixels and not too big cpu, so was a bit sluggish. But I liked it. Alas, it entered nirvana as a result of botched upgrade. If I am to believe internet, it can be pulled back into this world with the help of specially crafted serial cable and kermit. So, later on I bought me a Pocketbook 633 (definitely a fanboy!). This one can display 256 colors on outer LCD display and 16 levels of grey on inner e-ink display, which combines into 4096 colors. I wanted to give it a try, but if this specimmen dies, I will go back to normal e-ink b/w. The colors are there, but nothing outstanding. Ok for office kind of documents - pie charts, graphs etc. Space photographs, ukiyo-e, n nono no. Besides, if I use color, it powers on the LCD layer and consumes significantly more power than when it goes with e-ink only. For some documents, color can be turned off, for icons on main page, they show in color. Each time I just replug the card. So next time, I guess I will be looking for another reader with a slot. As of what you write about using cable and or BT for file transfer, it is all true but if reader does not work anymore, it might be hard to get files back. Or impossible. I have a smartphone, just to make a photo sometimes, nothing more, and I store it all on sdcard, again. This one phone does not want to emulate pendrive when I plug it into my Linux box, I had to install ftp server on it (phone). If it dies, I will unplug the card. Easy as a pie. No need to spend days looking for some bloody program which maybe is Windows-XP-only or maybe just been dropped from manufacturer support website. So I guess my next phone should better have a socket for sdcard, too. This is the future - if I have a need to support, I need to support it by me. Investing money in something that may turn into a brick and get my stuff with it - I do not think so. Too bad if some manufacturer does not want to play along, but they come by a dozen so maybe I can find what I want :-)... [...] Well, somehow all readers without sdcard tout about them being waterproof. Like I wanted to read in water. [...] I bought me a cheap mechanical clock from China (say, 17 bucks). It only needs a "mechanical recharge" once a day and I have to turn it back 5 minutes once a week to keep it somewhat accurate. Good enough for me. [...] Well I went the other way - bought some pencils to see how it goes with paper notes. Color crayons, some other thingies for making circles and lines (compass, ruler). Now, seems I will have to make some kind of notebook (in a future, not now) because what I see in my shops somehow does not convince me to buy (yeah I know about moleskines and leichturms, but this is not what I want, not sure what I want). Right now, it is just a normal crappy school notebook but with nice cover :-) . "Dropbox baaad, shoe box good" :-) -- Regards, Tomasz Rola -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did "rm -rif" on the programmer's home ** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:tomasz_rola@bigfoot.com **