Why is this syntax for patterns with Optional values not supported?

Bella: 2 days ago

I want to express Optional[PatternTest[Pattern[a,Blank[]],t],1] concisely.

a : _?t : 1 is not quite this but works: Optional[Pattern[a,PatternTest[Blank[],t]],1]. However I want to get rid of the first :.

I started with a_?t : 1 which unfortunately turns into a_?(t : 1), i.e. PatternTest[Pattern[a,Blank[]],Pattern[t,1]].

So I tried to go with (a_?t) : 1 but this gives Syntax::tsntxi: "(a_?t):1" is incomplete; more input is needed..

The only way to that works seems to be to state Optional explicitly, e.g. (a_?t)~Optional~1.

Why is (a_?t) : 1 not parsed as Optional, or at least as Pattern (which is wrong here though) but instead not parsed at all?


Is : : a special operator (at least as far as parsing is concerned), like /: :=?

Caleb: 2 days ago

The Documentation states (emphasis is mine):

  • Pattern (:): s:obj represents the pattern object obj, assigned the name s.

  • Optional (:): p:v is a pattern object that represents an expression of the form p, which, if omitted, should be replaced by v.

I think that the Documentation page for Optional is misleading since it states that the infix operator for Optional is : what is wrong. I'll show why I think so.

  1. First of all when you double click the symbol : in the valid pattern object s_: v you get selected not the whole pattern as one should expect if the symbol : would be the operator between s_ and v (and what happens for the case s : v irrespectively of how complicated pattern v is), but the construct _: is selected! This clearly indicates that _: is the actual operator here. So this behavior shows that in this case for Optional the corresponding operator is _: and not the symbol : alone.

  2. Citing the Documentation for Optional:

    The form s_:v is equivalent to Optional[s_,v]. This form is also equivalent to s:_:v. There is no syntactic ambiguity since s must be a symbol in this case.

    So "s must be a symbol" what explains the error message Syntax::tsntxi: "(s_):v:1" is incomplete; more input is needed. produced for input (s_) : v : 1. Note that the same message is produced for the input (s_) : v. Why this happens? The only explanation again is that for Optional the valid operator is _: and for Pattern is :, so this expression should be interpreted as Pattern[s_, v] what isn't valid because the name of the pattern must be a symbol (see expanded explanation of how the parser works at the bottom of this answer). Of course the message is vague and doesn't say this explicitly but when we add at the beginning sym: we get a valid pattern object sym : (s_) : v:

     sym : (s_) : v // FullForm
    Optional[Pattern[sym, Pattern[s, Blank[]]], v]

    Note that pattern sym : s_: v is parsed differently:

     sym : s_: v // FullForm
    Pattern[sym, Optional[Pattern[s, Blank[]], v]]

    The latter case is consistent with that _: is the infix operator for Optional. The former suggests that there is a special syntactic form which is parsed as Optional[Pattern[s, p], v]: it is the form s : p : v where s must be a symbol. In this form two coupled operators : are parsed as a whole for creating this complicated expression which isn't simply Optional or Pattern but a combination of them both.

  3. There is a documented limitation of the parser for Optional when it is used without the second argument (i.e. in the meaning of Default, emphasis is mine):

    Optional[s_h] represents a function which can be omitted, but which, if present, must have head h. There is no simpler syntactic form for this case.

    In the view of the previous considerations, the immediately obvious reason for this limitation is that the operator for Default is _. and for Optional is _:, hence the forms s_h. and s_h: (as well as (s_h). (s_h):) can't be used as equivalents for Optional[s_h].

    What isn't obvious is the fact that the two-argument form is nevertheless supported (as another special case):

    s_h: v // FullForm
    Optional[Pattern[s, Blank[h]], v]

    Note that the form (s_h) : v produces the error we already know: Syntax::tsntxi: "(s_h):v" is incomplete; more input is needed. The explanation is that there is another special short syntactic form s_h: v where the symbol : is coupled with preceding symbol _ and together they are interpreted as the complex construct Optional[Pattern[s, Blank[h]], v] which is neither Optional nor Pattern but a combination of them.

    Note also that when we double-click on the symbol : we get selected s_h: without v. This supports the interpretation that in the input s_h: v the symbols _ and : form single syntactic construct which can't be divided into separate operators _ and :. But then why the syntax s_h: (short form for Optional[s_h]) isn't supported? I think the reason may be simply aesthetic: the Mathematica (Wolfram) Language is supposed to be as transparent and readable as possible, and introducing syntax like s_h: moves it in the direction of esoteric programming languages ( similar to regular expressions ( what certainly is undesirable. Personally I find the current situation already unfortunate because the above-mentioned supported forms with entirely different meanings are too much similar to each other and hence they are hard to read/interpret.

From my point of view it would be much better to clearly document the actual operators which can be used as short syntactic forms for Pattern, Optional and Default (which correspondingly are :, _: and _.). I see no reasons to introduce the above-mentioned ambiguities and confusion.

Why is (a_?t) : 1 not parsed as Optional, or at least as Pattern (which is wrong here though) but instead not parsed at all?

Since we speak about a short syntactic input form, we should ask ourselves how the parser handles such special cases? There is a number of such special forms which can be confused with each other (these forms are described in the previous section), so the parser should have simple rules which allow to decide which interpretation it has to accept in this particular case. If it interprets : as Pattern, then the first argument must be a symbol what isn't the case here, so this interpretation is rejected. The parser doesn't see here the construct _:, so it can't interpret it as Optional either. Since the symbol : is alone, there are no more possible interpretations and the parser reports the syntactic error.

Note that the equivalent syntax Pattern[a_?t, 1] is accepted and even evaluated without warnings. The reason is simple: since at the top level it is written in the FullForm syntax there is no above-mentioned ambiguity (whether is it Pattern or Optional?), hence the parser simply takes this expression verbatim. Only when such expression is passed to a function like ReplaceAll we get an error message from the evaluator (not from the parser!):

1 /. Pattern[a_, x_] :> {a, x}

Pattern::patvar: First element in pattern Pattern[a_,x_] is not a valid pattern name. >>