from_name: Bryan Ford

subject: [Wikitech-l] In CVS

body:
Jan Hidders wrote:
>On Monday 16 August 2004 19:35, Magnus Manske wrote:
>> So, what you'd need is an EBNF representation or something?

Mind if I jump in and suggest a substitution for the "or something"
alternative? :)  Check out "parsing expression grammar" on wikipedia (and in
more detail on the external links that article leads to).  Although at the
moment you probably won't find a parser generator that'll generate the PHP
code you want from it, if the primary goal here is formal specification then
that's not such an issue - and in any case, unlike (LA)LR CFGs, parsing
expression grammars tend to be very easy to convert manually into working
parsers.

>Yes, that is all I ask. A precise formal grammar in whatever notation you
>like
>but preferrably in the input format of bison. If only because that would
>document what it is that your parser exactly does. Note there are some
>requirements:
>- it should accept all possible input strings
>- it should ideally do all the necessary parsing i.e. from the parse tree we
>should be able to generate the output with a single tree-walk, and
>- it should be unambiguous (and even LALR(1)) or have an explicit conflict
>resolution rule

The first and third requirements above are likely to be _very_ difficult to
achieve at the same time in a CFG paradigm, because of the limited lookahead
and extremely rudimentary disambiguation facilities of LR-class parser
generators.  LR parser generators are designed for languages that were
designed for LR parser generators; they tend to be difficult or impossible to
use for more freeform languages such as wikitext without making some serious
compromises or horrible hacks.  Not to discourage you from trying, though; I
just want to point out an alternative that you might consider when the
shift-reduce and reduce-reduce conflicts start becoming unbearable. :)

One caveat, though: I'm not exactly unbiased, since I wrote much of the
aforementioned stuff on parsing expression grammars. :)

OK, I'll shut up now.

Cheers,
Bryan


list: wikipedia-l

source: pipermail-mbox

from_name: Constans, Camille (C.C.)

body:
Maybe you should take a look at http://www.archive.org for ja.wikipedia.com

--
Shaihulud

>-----Message d'origine-----
>De : wikipedia-l-bounces at Wikimedia.org
>[mailto:wikipedia-l-bounces at Wikimedia.org]De la part de Tomos at
>Wikipedia
>Envoy? : mardi 17 ao?t 2004 11:04
>? : wikipedia-l at wikimedia.org
>Objet : [Wikipedia-l] Does anyone knows the beginning of ja?
>
>
>Hello.
>
>An email arrived at japanese wikipedia's press contact
>mailinglist regarding Web Creation Award. It is an award
>given to people who made significant difference in the web
>in Japan. The host of the award is an association of
>advertisers, though the recipients of the awards are not
>limited to marketing campaign planners. Wikipedia is under
>consideration because someone made a recommendation. :-)
>
>As a part of selection process, they would like to do
>some interview. The person who emailed us expressed that
>he would like to find the person who started the Japanese
>Wikipedia, someone who came up with the idea, plan, or
>determination. Who exactly this is, I think, is somewhat up
>to interpretation. Is there any strong candidate?
>
>I recall reading early Wikipedia-l posting (perhaps by Larry
>and/or Jason)
>announcing the launch of several non-English Wikipedia
>circa May 2001. Japanese Wikipedia was among the first
>ones. Is there anybody who remembers how that happened?
>Was multilingualization of Wikipedia an idea of Jimmy
>and Larry from the beginning? Or was there someone else
>who suggested/ pushed for it? Is there any reason why
>Japanese was among the first?
>
>In the early days, Aoineko was kind of "stationed" when
>there was only several edits a week, and welcomed the
>logged on users (there were only several back then) if
>I understand correct. Brion was there to fix bugs and
>take care of other technical aspects, too.  The first
>article seems to have been written by RoseParks (phonemes
>of Japanese). But there are only fragmental traces and
>virtually nobody knows those early days. Could somebody
>inform us of people who made significant contributions?
>
>Thank you in advance for your help.
>
>Regards,
>
>
>Tomos
>
>More info. is available on the award at here (in Japanese):
>
>http://award.wab.ne.jp/award/index.html
>
>_________________________________________________________________
>FREE pop-up blocking with the new MSN Toolbar - get it now!
>http://toolbar.msn.click-url.com/go/onm00200415ave/direct/01/
>
>_______________________________________________
>Wikipedia-l mailing list
>Wikipedia-l at Wikimedia.org
>http://mail.wikipedia.org/mailman/listinfo/wikipedia-l
>


list: wikitech-l

from_name: Timwi

subject: [Wikitech-l] Re: An alternate parser (IT WORKS! :))

body:
Magnus Manske wrote:

> I agree with Eckhart, though, that you should use (X)HTML tags where
> possible. All conversion you don't do in your parser will have to be
> done in PHP later :-(

Gaahhrr! You are still thinking too much in terms of conversion.  (The
current Parser.php, and your recent product, are technically speaking
converters, not parsers.)

The XML I produce is a representation of the parse tree.  Its purpose is
*NOT* to be output.  Its purpose is only to represent the parse tree.
There will be a PHP routine that reads this XML and produces a native
PHP structure from it.  Then another routine will do the post-processing
on that structure.  Finally, yet another routine will walk the structure
and produce the output (XHTML).

There is no conceptual similarity between the XML representing the parse
tree and the XML representing the rendered page.  Correspondingly, none
(I repeat, *NONE*) of the XML representing the parse tree is used
directly for the XHTML.

> It is also fun to try to confuse the parser with things like
> *##*#**#* one line
> *#*##*'*''* the other

This would indeed confuse a converter... but not a parser. :-)

<list type='bullet'><listitem><list type='numbered'><listitem><
list type='numbered'><listitem><list type='bullet'><listitem><list type=
'numbered'><listitem><list type='bullet'><listitem><list type='bullet'><
listitem><list type='numbered'><listitem><list type='bullet'><listitem>o
ne line</listitem></list></listitem></list></listitem></list></listitem>
</list></listitem></list></listitem></list></listitem></list><list type=
'bullet'><listitem><list type='numbered'><listitem><list type='numbered'
 ><listitem><list type='bullet'><listitem>'*''* the other</listitem></lis
t></listitem></list></listitem></list></listitem></list></listitem></lis
t></listitem></list>

Check for me if this is correct?  :-)

> Bold and italics should come along well, also links and templates,
> which, in my manual parser (now obsolete) actually use the same routine.

Actually, bold and italics are a problem.  Haven't you noticed the
modifications I made to the current converter to handle them?  When I
first came here, it was just two regular expressions, thus generating
something like Text, which, obviously, is
invalid HTML.  I had to replace it with a finite state machine to make
that work.  The next problem is that in French, if you input the
following: "l'''homme''", then you don't want the ''' to mean "bold";
instead, you want "l'homme".  Notice, especially, that the
apostrophe needs to be *before* the .  Doing that in a formal
grammar is going to be tricky.  :-)

> Table markup might be a bitch, though.

Not really; it seems to me that the table markup is pretty well-suited
to LALR(1).  (Very much unlike the list markup.)

> Do you check for valid HTML within nowiki tags? What does
> <nowiki><badtag></nowiki> do?

Again, you are thinking too much in terms of conversion.  Your above
example produces:

<paragraph><extension
name="nowiki"><badtag></extension></paragraph>

The post-processing step is then free to do anything it likes with the
contents of the "extension" tag.  The correct behaviour, of course,
would be to output it HTML-escaped.

Timwi


from_name: Magnus Manske

body:
Timwi wrote:

>
> By now I have written a real parser (it actually works!! It outputs
> correct XML!! I'm totally excited about it!) and I'm afraid it's not
> *trivial* to translate to PHP (though I'm sure I could do it easily).

Cool! You definitely beat me to that one - I barely got a glance at the
bison manual ;-)

I agree with Eckhart, though, that you should use (X)HTML tags where
possible. All conversion you don't do in your parser will have to be
done in PHP later :-(

For testing, I suggest the first 10 pages on
http://en.wikipedia.org/wiki/Special:Longpages
If it survives these, it's good to go :-)

It is also fun to try to confuse the parser with things like
*##*#**#* one line
*#*##*'*''* the other

Bold and italics should come along well, also links and templates,
which, in my manual parser (now obsolete) actually use the same routine.
Table markup might be a bitch, though.

Do you check for valid HTML within nowiki tags? What does
<nowiki><badtag></nowiki> do?

Magnus

>
> So far, my parser can do headings, paragraphs, lists (only * and #
> yet), pres (lines beginning with a space), and "extensions"
> (everything enclosed in <sometag>...</sometag>, including <nowiki>,
>
, <math>, etc.). I'll give an example below. Please send me input
> files to test my parser on so I can see if it handles all possible
> situations correctly, including broken input mark-up!
>
> I'm going to do bold/italics and links next.
>
> Timwi
>
>
> --- Input file ---
>
> == This is a heading! ==
>
> The parser works like this:
>
>  yylex();
>  yyparse();
>
>  outputXML();
>
> == Another heading! ==
> The individual steps are:
> * yylex: Tokenisation
> * yyparse: Parsing (*)
> ** this is the exciting part!
> * outputXML: XMLisation
>
> I can prove that <math>2 < 1</math>!
>
> --- Output (the newlines are inserted by my mailer) ---
>
>
<heading level='2'> This is a heading!
> </heading><paragraph>The parser works like
> this:</paragraph><preblock><preline>yylex();</preline><preline>yyparse();</preline><preline></preline><preline>outputXML();</preline></preblock><heading
> level='2'> Another heading! </heading><paragraph>The individual steps
> are:</paragraph><list type='bullet'><listitem>yylex:
> Tokenisation</listitem><listitem>yyparse: Parsing (*)<list
> type='bullet'><listitem>this is the exciting
> part!</listitem></list></listitem><listitem>outputXML:
> XMLisation</listitem></list><paragraph>I can prove that <extension
> name="math">2 < 1</extension>!</paragraph>

>
> --- Manually beautified output ---
>
>

>  <heading level='2'> This is a heading! </heading>
>  <paragraph>The parser works like this:</paragraph>
>  <preblock>
>   <preline>yylex();</preline>
>   <preline>yyparse();</preline>
>   <preline></preline>
>   <preline>outputXML();</preline>
>  </preblock>
>  <heading level='2'> Another heading! </heading>
>  <paragraph>The individual steps are:</paragraph>
>  <list type='bullet'>
>   <listitem>yylex: Tokenisation</listitem>
>   <listitem>yyparse: Parsing (*)
>    <list type='bullet'>
>     <listitem>this is the exciting part!</listitem>
>    </list>
>   </listitem>
>   <listitem>outputXML: XMLisation</listitem>
>  </list>
>  <paragraph>
>   I can prove that <extension name="math">2 < 1</extension>!
>  </paragraph>
>

>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at wikimedia.org
> http://mail.wikipedia.org/mailman/listinfo/wikitech-l


subject: [Wikitech-l] Re: In CVS

body:
Jan Hidders wrote:

> It is my experience that you never really know until you have actually
> looked. We also have the option of going with the usual disambiguation
> rules in yacc/bison.

The default disambiguation rule for shift/reduce conflicts in bison is
to do the shift.  So far, I have crafted my grammar in such a way that
this is always the correct thing to do.  I think this is always possible.

The default disambiguation rule for reduce/reduce conflicts is to reduce
the rule that is mentioned earlier in the source file.  So far, I have
not had a reduce/reduce conflict, but clearly, whenever this
disambiguation rule produces something other than the desired result,
all you need to do is to change the order of the rules.

Timwi


subject: [Wikitech-l] Re: CVS problems & snapshot

body:
Brion Vibber wrote:

> For those of you who want to work on the current source but aren't on
> the project on SF, here's a fresh checkout of CVS head branch with
> phase3, extensions, and wiki2xml modules:
> http://download.wikimedia.org/mediawiki/mediawiki-cvs-2004-08-17.tar.bz2

And, of course, you can subscribe to the mediawiki-cvs at wikipedia.org
mailing list (http://mail.wikipedia.org/mailman/listinfo/mediawiki-cvs)
to get the patches we apply mailed to you. You can use the "patch" tool
under UNIX to apply these patches to your local copy.


subject: [Wikitech-l] Re: In CVS

body:
Bryan Ford wrote:

>>- it should accept all possible input strings
>>- it should ideally do all the necessary parsing i.e. from the parse tree we
>>  should be able to generate the output with a single tree-walk, and
>>- it should be unambiguous (and even LALR(1)) or have an explicit conflict
>>  resolution rule
>
> The first and third requirements above are likely to be _very_ difficult to
> achieve at the same time in a CFG paradigm, because of the limited lookahead
> and extremely rudimentary disambiguation facilities of LR-class parser
> generators.  LR parser generators are designed for languages that were
> designed for LR parser generators; they tend to be difficult or impossible to
> use for more freeform languages such as wikitext without making some serious
> compromises or horrible hacks.

Can you provide a more specific example of this?  So far, I have not
encountered a situation where accepting all possible input strings would
be hard to do.  In fact, all I need to do to ensure this is to allow the
"text" non-terminal to contain any token.

Also, I have not found it difficult to craft the grammar in such a way
that bison's default disambiguation rules (conflict resolution rules)
produce the correct result.  There does not seem to be any real need to
have the grammar be unambiguous.

> I just want to point out an alternative that you might consider when the
> shift-reduce and reduce-reduce conflicts start becoming unbearable. :)

So far, I have not had a reduce/reduce conflict, and, as I said, the
shift/reduce conflicts are not a problem.

Timwi


list: wikien-l

from_name: Christiaan Briggs

subject: [WikiEN-l] When can an article be considered public domain?

body:
Sorry it was meant to be a joke. Obviously not very funny. :(

Christiaan

On Aug 17, 2004, at 4:09 am, jcecropia at mail.com wrote:

> ----- Original Message -----
> From: Christiaan Briggs <christiaan at yurkycross.co.uk>
> Date: Mon, 16 Aug 2004 09:49:09 +0100
> To: English Wikipedia <wikien-l at Wikipedia.org>
> Subject: Re: [WikiEN-l] When can an article be considered public
> domain?
>
>> On 14 Aug 2004, at 6:39 am, Jim Cecropia wrote:
>>
>>> First IANAL
>>
>> Would such jargon stand up in a court of law?
>>
>> Christiaan
>>
> In terms of what? I am not representing myself as a lawyer and thereby
> stating that what I am saying may be knowledgeable, but it's not legal
> advice, so take it FWIW. Anyone who went to court and said, "I did
> this because a guy on the internet told me so" is not likely to do too
> well anyway.
>
> Even if I WERE a lawyer, I would still have to say it is not legal
> advice, because no lawyer-client relationship exists.
>
> IA-(still)-NAL, C


body:
Jan Hidders wrote:

> Timwi wrote:
>
>> Magnus Manske wrote:
>>
>>> It is also fun to try to confuse the parser with things like
>>> *##*#**#* one line
>>> *#*##*'*''* the other
>>
>> This would indeed confuse a converter... but not a parser. :-)
>>
>> <article>
>>   <list type='bullet'>
>>     <listitem>
>>       <list type='numbered'>
>>       <listitem> [...]
>
> No no no. Only the *'s and #'s at the beginning of a line indicate an item.

Uhm? Yes, I know? So what's the problem?


from_name: Jan Hidders

subject: [Wikitech-l] In CVS

body:
Delirium wrote:

> I can't prove it at the moment, but after having played around with
> wikisyntax for a while, I'm *pretty* sure it's not LALR(1).  There's
> too much reuse of symbols that requires potentially unbounded
> lookahead to resolve ambiguities.  Even if it were, the LALR(1)
> grammar would be so unintuitive and unlike the functional intent of
> wikisyntax that it wouldn't be very useful as a reference.

It is my experience that you never really know until you have actually
looked. We also have the option of going with the usual disambiguation
rules in yacc/bison.

> I did some searching for general LR(n) parsers (some call themselves
> GLR) parsers a while back, and none are too great.  The two most
> promising are Elkhound from Berkeley
> [http://www.cs.berkeley.edu/~smcpeak/elkhound/] and Packrat parsing
> [http://www.pdos.lcs.mit.edu/~baford/packrat/].  The latter is much
> cleaner, and uses a "parsing expression grammar" (PEG), basically a
> CFG with ordered productions, to resolve ambiguity.  However, it uses
> O(n) RAM where n is the size of the file to be parsed.  Elkhound is
> more efficient, but I wasn't able to figure out how exactly to build
> something clean out of it.

You missed one. bison. :-) And indeed that is also an option that we
have: go with an GLR parser.

-- Jan Hidders


from_name: xmlizer

subject: [Wikitech-l] Re: disambiguation enhancement

references:
<cflt0k$e7u$1@sea.gmane.org>	<200408150805.30306.smolensk@eunet.yu>	<411F0351.40208@pobox.com>	<200408160634.54714.smolensk@eunet.yu>	<cfqfq6$fr9$1@sea.gmane.org>
	<cfsdrj$bl1$1@sea.gmane.org>

body:
Timwi a ?crit :

> xmlizer wrote:
>
>> What about defining <tt>#DISAMBIGUATION</tt> and <tt>#STUB</tt> and to
>> insert them respectively into Template:Disambig or Mod?le:Homonymie,
>> and Template:Stub or Mod?le:?bauche
>
>
> I am rather opposed to anything that would give any particular meaning
> to "disambiguation pages" or "stubs", as they are Wikipedia-centric
> concepts.  The software should allow either server administrators or
> even the users themselves to define custom such classifications.  People
> in the install the wiki software for different purposes, or even the
> sister projects (Wiktionary, Wikisource, etc.) should be able to come up
> with and use other classifications, and they should not be burdened with
> concepts like "disambiguation page" that don't make any sense in the
> context of their content.
>
> Timwi
>

That's why I said to use some keyword like #REDIRECT
These keywords doesn't have to have a special meaning instead these
keywords has to be inserted in the page (or the template) so as to have
a special behaviour

Before, in the database you have to declare the keywords and when the
parser find a keyword it flag the cur_kind the corresponding bit
Example
  -------------------------
| keyword  | bit_position |
  -------------------------
| REDIRECT |      0       |
| STUB     |      1       |
| DISAMBIG |      2       |
| FOOBAR   |      3       |
  -------------------------

Then it could be used in the cur table with  the cur_kind column

Secondly you could add to link to these page
... if the value of cur_kind =
1110b
or
... if the value of cur_kind is 1

This behaviour is indepedent of STUB, DISAMBIG etc... so could be used
by any wiki and it's up to the admin of the wiki to define keywords
allowed to be used

Xmlizer


from_name: Manuel VACELET

subject: [Wikitech-l] Re: Embed mediawiki ?

body:
Rowan Collins wrote:
> I think "embed" means combining them such that certain features would
> work seamlessly as though it was all one application - most
> importantly, I would expect, logins.
Yes, it's is exactly what I want.

> Basically, there doesn't seem to be a lot to stop you - if you do want
> to combine the database with something else the same issues apply:
> make sure table names don't conflict, formats match appropriately,
> etc. Other than that, it's really a case of customising skins and
> interface features to make it look and feel like part of the same
> thing [hopefully fairly straight-forward], and adding any particular
> magic features you want for your own needs.
Ok, thanks for your help.
I will begin to read mediawiki architecture :)

--
# VACELET Manuel        manuel.vacelet(at)st(dot)com #
# Tel:  042 6089                  +33 (0)476 92 6089 #
# STMicroelectronics  - Central R&D DAIS             #
# 850, rue Jean Monet - 38926 CROLLES CEDEX - FRANCE #


subject: [Wikitech-l] Re: In CVS

body:
Jan Hidders wrote:

> Timwi wrote:
>
>> MediaWiki's syntax is perfectly LALR(1), except for only one very
>> important exception: lists.
>
> ?? Could you explain that last remark a bit more?

I am not sure what you are asking: do you want to know why I think
MediaWiki's syntax is LALR(1), or do you want to know why lists are not?

I cannot give a definite answer to the first question other than "it has
worked for me so far, and I have already thought about some of the
things that are yet to come, and have not come up with any problems".

As for the second question, the problem with lists is that one token
(e.g. '*') at the beginning of several lines can represent the same item:

* One
*# Two
*# Three
* Four

Although there are four lines and four '*'s, there are only three list
items in the outermost list, with the second item containing a nested
list spanning two lines.  The inner list essentially has a '*' in the
middle of it to indicate that it does not end here.  In other words,
there is no clearly-indicated beginning or end to the inner list, and
the inner list has tokens inside it that do not belong to it.

I have solved this currently by first parsing it as

<listblock>
  <listline><listbullet /> One</listline>
  <listline><listbullet /><listnumbered /> Two</listline>
  <listline><listbullet /><listnumbered /> Three</listline>
  <listline><listbullet /> Four</listline>
</listblock>

and then calling a separate routine that transforms this into the
desired structure.

Notice that this does not defeat the purpose of a parser, in case you
are thinking this.  The above example uses "one" as an example for the
contents of a "listline", but in reality it can contain bold, italics,
links, etc. and the good thing about using a parser is that I have one
node in my structure that represents this whole text block, already
fully parsed.  I do not need to care what it contains, and I do not need
to worry that rearranging the nodes representing the list will in any
way conflict or interfere with the contents of the node representing the
text block.

I love parsing.  Does it show?  :-)

Timwi


subject: [Wikitech-l] In CVS

body:
Bryan Ford wrote:

>Jan Hidders wrote:
>
>
>>On Monday 16 August 2004 19:35, Magnus Manske wrote:
>>
>>
>>>So, what you'd need is an EBNF representation or something?
>>>
>>>
>
>Mind if I jump in and suggest a substitution for the "or something"
>alternative? :)  Check out "parsing expression grammar" on wikipedia (and in
>more detail on the external links that article leads to).
>
You still need a formal grammar to start with. :-)

>Although at the moment you probably won't find a parser generator that'll generate the PHP
>code you want from it, if the primary goal here is formal specification then
>that's not such an issue - and in any case, unlike (LA)LR CFGs, parsing
>expression grammars tend to be very easy to convert manually into working
>parsers.
>
>
Did you just say "formal specification"? :-) Without a parser generator
there is no hope that we will have somewhere in the documentation a
correct, precise and complete specification of the wikisyntax.

-- Jan Hidders


body:
Magnus Manske wrote:

> I agree with Eckhart, though, that you should use (X)HTML tags where
> possible. All conversion you don't do in your parser will have to be
> done in PHP later :-(

When using the XML parser translating the tag names will be trivial. The
choice of the tag names is the least of your problems.

-- Jan Hidders


subject: [Wikitech-l] Re: In CVS

body:
Timwi wrote:

> MediaWiki's syntax is perfectly LALR(1), except for only one very
> important exception: lists.

?? Could you explain that last remark a bit more?

-- Jan Hidders


subject: [Wikitech-l] Re: In CVS

body:
Timwi wrote:

> Michael Becker wrote:
>
>> I'm going to break out my compiler books and start working on a CFG
>> for the wiki code
>
>
> I am already working on this.

Yes. Impressive. But do you already have a grammar that describes what
your parser parses?

-- Jan Hidders


subject: [Wikitech-l] Re: In CVS

body:
Jan Hidders wrote:

> Timwi wrote:
>
>> Michael Becker wrote:
>>
>>> I'm going to break out my compiler books and start working on a CFG
>>> for the wiki code
>>
>> I am already working on this.
>
> Yes. Impressive. But do you already have a grammar that describes what
> your parser parses?

I have indeed been told by a friend that it would be a better idea to
write down the BNF for the grammar before starting to write the yacc
file.  So I tried.  But then, when I finally did the yacc file, I
realised that it was not such a good idea after all.  In fact, it was a
waste of time.  You need to be able to test the grammar while you are
working on it, and you want to be able to tweak certain parts before
writing certain other parts that depend on them.

This is quite different from planning a software project, where you
sketch your system and its modules in a very rough outline first.
Rather, when writing the yacc file, you are actually writing the
grammar.  With BNF, you would also be writing the grammar, but without
the benefit of being able to test it.  In particular, in BNF you
wouldn't be able to described transformations like the one I needed to
do for lists.  You would instead rack your brains over how to parse the
lists directly, never coming to a conclusion (because it is probably
impossible).

If you are still interested, please find below the BNF I originally
wrote.  However, the current yacc file I use is quite different from it.

I haven't even done links yet, and I already know that the below grammar
will not work for links, because it will treat the pipe character as
text inside the double square brackets.

Timwi

article ::= e | block | article NEWLINE+ block

block ::= paragraph | heading | bulletedlist | numberedlist | preblock |
comment

paragraph ::= nonemptytext | paragraph NEWLINE nonemptytext

comment ::= EMPTYCOMMENT | BEGINCOMMENT TEXT ENDCOMMENT
           | BEGINCOMMENT TEXT | BEGINCOMMENT

heading ::= headinglevel1 | headinglevel2 | headinglevel3
           | headinglevel4 | headinglevel5 | headinglevel6
headinglevel1 ::= HEADING1 nonemptytext ENDHEADING?
headinglevel2 ::= HEADING2 nonemptytext ENDHEADING?
headinglevel3 ::= HEADING3 nonemptytext ENDHEADING?
headinglevel4 ::= HEADING4 nonemptytext ENDHEADING?
headinglevel5 ::= HEADING5 nonemptytext ENDHEADING?
headinglevel6 ::= HEADING6 nonemptytext ENDHEADING?

bulletedlist ::= LISTBULLET text
                | bulletedlist NEWLINE LISTBULLET text
                | bulletedlist NEWLINE NEWLINE LISTBULLET text

numberedlist ::= LISTNUMBERED text
                | numberedlist NEWLINE LISTNUMBERED text
                | numberedlist NEWLINE NEWLINE LISTNUMBERED text

preblock ::= PRELINE text | preblock NEWLINE* PRELINE text

nonemptytext ::= intext | intext nonemptytext
text ::= e | nonemptytext

intext ::= TEXT | EXTENSION | comment | dblsqbr | PIPE | templateinclusion
          | templatevariable | bold | italics | APOSTROPHE | DOUBLEQUOTE
              # Things that are invalid syntax but we need to allow:
              | ENDCOMMENT | OPENDBLSQBR CLOSEDBLSQBR | OPENLINK
CLOSEDBLSQBR

dblsqbr ::= linkorimageorcategory | forcedlink

linkorimageorcategory ::= OPENDBLSQBR linkinside CLOSEDBLSQBR
forcedlink            ::= OPENLINK linkinside CLOSEDBLSQBR
linkinside            ::= nonemptytext | linkinside PIPE nonemptytext


body:
Jan Hidders wrote:

> Oops. My mistake. For some reason I thougt that only the first * in the
> line was relevant.

You thought that lists could not be nested at all?  :-)

Timwi


body:
Timwi wrote:

> Magnus Manske wrote:
>
>> I agree with Eckhart, though, that you should use (X)HTML tags where
>> possible. All conversion you don't do in your parser will have to be
>> done in PHP later :-(
>
>
> Gaahhrr! You are still thinking too much in terms of conversion.  (The
> current Parser.php, and your recent product, are technically speaking
> converters, not parsers.)

Indeed.

>> It is also fun to try to confuse the parser with things like
>> *##*#**#* one line
>> *#*##*'*''* the other
>
>
> This would indeed confuse a converter... but not a parser. :-)
>
> <article>
>   <list type='bullet'>
>     <listitem>
>       <list type='numbered'>
>       <listitem> [...]

No no no. Only the *'s and #'s at the beginning of a line indicate an item.

-- Jan Hidders


subject: [Wikitech-l] Re: An alternate parser

body:
Nikola Smolenski wrote:

> On Monday 16 August 2004 08:10, Krzysztof Kowalczyk wrote:
>
>>No-one can know that unless the new parser is written and benchmarked
>>against the current one.
>
> I think that I can know: if it doesn't use regexps but just goes through
> entire text character-by-character, it will be MUCH slower.

I suppose I can count myself lucky, then, that flex does use regular
expressions? :-)

Seriously, though: That's quite a bold statement of yours. I don't think
that it generally applies. Regular expressions just go through the
entire text, too. In fact, several regular expressions would do it
several times; a proper parser would do it only once.

Regular expressions in PHP are faster than character-by-character in PHP
because the regular expressions are executed in C code.

Timwi


subject: [Wikitech-l] Re: Two markets (was Re: An alternate parser)

body:
Nikola Smolenski wrote:

> It works the other way around - if someone want to include contents from other
> wiki into Wikipedia, there might also be problems.

Wikipedia does not tend to copy huge amounts of data from other
MediaWiki-based wikis. Individual articles can very easily be changed by
hand, and big databases we import from other sites (like the Unihan
character stuff they imported into Wiktionary) tends to require
automated conversion anyway.

Timwi


body:
Timwi wrote:

> Jan Hidders wrote:
>
>> Timwi wrote:
>>
>>> Magnus Manske wrote:
>>>
>>>> It is also fun to try to confuse the parser with things like
>>>> *##*#**#* one line
>>>> *#*##*'*''* the other
>>>
>>>
>>> This would indeed confuse a converter... but not a parser. :-)
>>>
>>> <article>
>>>   <list type='bullet'>
>>>     <listitem>
>>>       <list type='numbered'>
>>>       <listitem> [...]
>>
>>
>> No no no. Only the *'s and #'s at the beginning of a line indicate an
>> item.
>
>
> Uhm? Yes, I know? So what's the problem?

Oops. My mistake. For some reason I thougt that only the first * in the
line was relevant.

I have manually checked your output now and as far as I can see it is
correct.

-- Jan Hidders


subject: [Wikipedia-l] UTF-8 for nl:wikipedia

body:
Ask when you want, I prefer the week-end, I have more spare time :)

Shaihulud

>
>I have been informed by Walter that nl:wikipedia had decideded quite
>some time ago to move to UTF-8. At the time it was not such a
>good idea
>as UTF-8 does take more room.
>
>However, as many other wikipedia like de: and es: have made the
>conversion, it is a good idea to ask for the nl:wikipedia to be
>converted. We have asked if there are people on nl: that have opposing
>views, there were none.
>
>As nl:wikipedia is a very active environment, some prior
>notice would be
>appreciated.
>
>Thanks,
>    GerardM
>_______________________________________________
>Wikipedia-l mailing list
>Wikipedia-l at Wikimedia.org
>http://mail.wikipedia.org/mailman/listinfo/wikipedia-l
>


subject: [Wikitech-l] RE: [Wikipedia-l] UTF-8 for nl:wikipedia


from_name: Gerard Meijssen

subject: [Wikipedia-l] UTF-8 for nl:wikipedia

body:
I have been informed by Walter that nl:wikipedia had decideded quite
some time ago to move to UTF-8. At the time it was not such a good idea
as UTF-8 does take more room.

However, as many other wikipedia like de: and es: have made the
conversion, it is a good idea to ask for the nl:wikipedia to be
converted. We have asked if there are people on nl: that have opposing
views, there were none.

As nl:wikipedia is a very active environment, some prior notice would be
appreciated.

Thanks,
    GerardM


subject: [Wikitech-l] UTF-8 for nl:wikipedia


from_name: Peter Gervai

subject: [Wikipedia-l] personal threats again

body:
On Tue, Aug 17, 2004 at 03:08:58PM +0430, Roozbeh Pournader wrote:

> What is still very weird, is that he used IP addresses from many
> different places, from Thailand to the United States. My question is:
> how could he?

By the way this makes him probably a criminal in most civilised countries
since breaking into others' machines and using unauthorised resources are
crimes in most places. At least you can earn a good jail time doing it in
Europe.

Not that it's useful unless he's identified, but maybe good to know.

my 2 cents.


subject: [Wikitech-l] Re: In CVS

references:
<200408161255.AA121111114@jumpingjackweb.com>	<cfseuj$bl1$4@sea.gmane.org>	<4121C9B8.3050702@ua.ac.be>
	<cfsl6j$rsa$1@sea.gmane.org>

body:
Timwi wrote:

> Jan Hidders wrote:
>
>> Timwi wrote:
>>
>>> Michael Becker wrote:
>>>
>>>> I'm going to break out my compiler books and start working on a CFG
>>>> for the wiki code
>>>
>>>
>>> I am already working on this.
>>
>>
>> Yes. Impressive. But do you already have a grammar that describes
>> what your parser parses?
>
> I have indeed been told by a friend that it would be a better idea to
> write down the BNF for the grammar before starting to write the yacc
> file.  So I tried.  But then, when I finally did the yacc file, I
> realised that it was not such a good idea after all.  In fact, it was
> a waste of time.  You need to be able to test the grammar while you
> are working on it, and you want to be able to tweak certain parts
> before writing certain other parts that depend on them.

I hadn't realized yet you were already using yacc. All I wanted was to
see the grammar you are using now, by which I simply mean the yacc input.

-- Jan Hidders


subject: [Wikitech-l] Re: UTF-8 for nl:wikipedia

body:
>
>
>Gerard Meijssen wrote:
>> some time ago to move to UTF-8. At the time it was not such
>a good idea
>> as UTF-8 does take more room.
>
>More room? UTF-8 does not use more memory, if that's what you
>mean. HTML
>entities (like Ӓ) use 5 up to 7 bytes, while a character
>in UTF-8
>uses at most 4 bytes.

It wasnt the pb, to convert nl.wikipedia, we had to uncompress the old table, which once uncompressed take much more space.

Shaihulud


from_name: Jimmy (Jimbo) Wales

subject: [WikiEN-l] Re: you are autoblocking me

body:
Cantus wrote:
> This is stupid. Everytime I try to edit an article logged in as Cantus
> (just to see if the block has expired), my IP is getting blocked for
> another 24 hours. When my original blocking time expires, I won't be
> able to edit, because the IP was reblocked for another 24 hours.

I recommend that you not do that, then.

> If this is not fixed now, I'm talking to Jimbo.

I'm listening, but I recommend just relaxing for a little while, and this
will sort itself out soon enough.

--Jimbo


from_name: Brion Vibber

subject: [Wikitech-l] XML: Part 2

body:
Magnus Manske wrote:
>   // If var already defined, make array
>   eval('$test=isset('.$wgXMLobject->tree.'->'.$name.');');

In case you didn't know, you _can_ write code like this:

   $test = isset( ${$wgXMLobject->tree}->$name );

While it's amazingly ugly and probably a bad idea anyway;) it's going to
be more efficient than the even uglier use of an eval() that has to
parse and compile entire PHP statements at runtime. (For the above case,
about three times faster.)

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 253 bytes
Desc: OpenPGP digital signature
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20040817/a0835e6c/attachment.pgp>


subject: [Wikitech-l] Wikicommons

body:
Gerard Meijssen wrote:
> Please let us start the Wikicommons and add functionality when we think
> of some, when we program some.

Erik M?ller proposed specific things for WikiCommons. When he gets some
code written, I'm sure we'll be happy to put it online.

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 253 bytes
Desc: OpenPGP digital signature
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20040817/4d6de29b/attachment.pgp>


from_name: Daniel Mayer

subject: [Wikipedia-l] Brazilian Portuguese

body:
--- Felipe Sanches <felipe.sanches at gmail.com> wrote:

> Wikipedia new language request.
> There are 182.1 million people in Brazil who use Brazilian Portuguese
> as their main language.
> Brazilian Portuguese IS NOT the same as Portuguese (domain
> pt.wikipedia.org) and it still isn't on the list
> (http://meta.wikimedia.org/wiki/Complete_list_of_language_wikis_available)

American English isn't the same as British English nor is Mexican Spanish the
same as Spanish Spanish.

So you are going to have to justify balkanizing the Portuguese Wikipedia, IMO.
A vote on that wiki (which is rather large and active) should be conducted.

-- mav

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com


from_name: Geoffrey Burling

subject: [WikiEN-l] Milestone

body:
On Tue, 17 Aug 2004, David Gerard wrote:

> On 08/17/04 14:34, Pete/Pcb21 wrote:
>
> > When we passed 100,000 , 200,000 , 300,000 etc articles there was a fair
> >   bit of fanfare.
> > I think it is only right therefore to announce that en passed the
> > 100,000,000 word mark earlier this month.
> > This is about the same size as Columbia, Encarta Deluxe and EB put
> > together, and if printed would weigh about the same as George Bush.
>
> I award you the job of writing the Slashdot story, using that exact
> weight measurement ;-)
>
Much as I like that unit of measure, we do not define in [[George Walker Bush]] just how much he weighs. That might then be a problem. (A spokesman for EB
says, ``100 million words, & Wikipedia lacks an entry for George Bush's
weight. And they expect us to believe they have all of the facts.")

But not going to add it to the article. It's currently under protection.
(Why am I not surprised?)
>
> > Across all languages we reached this milestone back in January, and are
> > now up to 220,000,000 words.
>
Geoff


from_name: Andre Engels

subject: [Wikipedia-l] Does anyone knows the beginning of ja?

body:
On Tue, 17 Aug 2004 09:03:58 +0000 "Tomos at Wikipedia"
<wiki_tomos at hotmail.com> wrote:

>I recall reading early Wikipedia-l posting (perhaps by Larry and/or Jason)
>announcing the launch of several non-English Wikipedia
>circa May 2001. Japanese Wikipedia was among the first
>ones. Is there anybody who remembers how that happened?
>Was multilingualization of Wikipedia an idea of Jimmy
>and Larry from the beginning? Or was there someone else
>who suggested/ pushed for it? Is there any reason why
>Japanese was among the first?

As far as I know, it was Brion, who used it as a kind of 'experimenting
area' or 'proof of concept' for creating a UTF-8 Wikipedia.

>In the early days, Aoineko was kind of "stationed" when
>there was only several edits a week, and welcomed the
>logged on users (there were only several back then) if
>I understand correct. Brion was there to fix bugs and
>take care of other technical aspects, too.  The first
>article seems to have been written by RoseParks (phonemes
>of Japanese). But there are only fragmental traces and
>virtually nobody knows those early days. Could somebody
>inform us of people who made significant contributions?

 From that I guess it would not be bad to consider Aoineko the one who
started the Japanese Wikipedia. That also has the advantage of having
someone who actually speaks Japanese and is still active on Wikipedia.

Andre Engels


body:
Tomos at Wikipedia wrote:
> Japanese Wikipedia was among the first
> ones. Is there anybody who remembers how that happened?
> Was multilingualization of Wikipedia an idea of Jimmy
> and Larry from the beginning?

>From the very beginning, yes.  Japanese in particular was and is an
interest of mine because my wife was born in Tokyo and is
half-Japanese.  But of course, I am not responsible for any edits
there, and others must take all the credit.

I would be *keenly* interested in visiting Japan to accept the award,
if there's a ceremony or whatever, even if it has to be at my own
expense.


from_name: Stan Shebs

subject: [Wikitech-l] Wikicommons

body:
Brion Vibber wrote:

> Gerard Meijssen wrote:
>
>> Please let us start the Wikicommons and add functionality when we
>> think of some, when we program some.
>
>
> Erik M?ller proposed specific things for WikiCommons. When he gets
> some code written, I'm sure we'll be happy to put it online.

Do we really have to be blocked on coding? The situation with
image duplication is really bad now, and it will only get worse
as people want to illustrate articles in the non-English WPs.
Is there a subset of amazing-wikicommons-world that we can start
with, perhaps by creating it empty and adding files manually?

(I'm assiduously studying the software myself, not quite up to
hacking it yet.)

Stan


from_name: Stephan Walter

subject: [Wikitech-l] Re: UTF-8 for nl:wikipedia

body:
Gerard Meijssen wrote:
> some time ago to move to UTF-8. At the time it was not such a good idea
> as UTF-8 does take more room.

More room? UTF-8 does not use more memory, if that's what you mean. HTML
entities (like Ӓ) use 5 up to 7 bytes, while a character in UTF-8
uses at most 4 bytes.

Regards,
Stephan

--
Stephan.Walter at epfl.ch  --  http://lart.info/~stw/  --  PGP: B2421799


subject: [Wikitech-l] Re: In CVS

body:
Timwi wrote:

> As for the second question, the problem with lists is that one token
> (e.g. '*') at the beginning of several lines can represent the same item:
>
> * One
> *# Two
> *# Three
> * Four
>
> Although there are four lines and four '*'s, there are only three list
> items in the outermost list, with the second item containing a nested
> list spanning two lines.  The inner list essentially has a '*' in the
> middle of it to indicate that it does not end here.  In other words,
> there is no clearly-indicated beginning or end to the inner list, and
> the inner list has tokens inside it that do not belong to it.

Ah, of course. Never mind LALR(1). Recognizing a single block (items
that all belong to the same list)  is not possible with a context-free
language. I think I'll keep proving that one as an exercise for my
students. :-) But that tells us that a fancier parser generator is not
going to solve that problem for us.

> I have solved this currently by first parsing it as
>
> <listblock>
>  <listline><listbullet /> One</listline>
>  <listline><listbullet /><listnumbered /> Two</listline>
>  <listline><listbullet /><listnumbered /> Three</listline>
>  <listline><listbullet /> Four</listline>
> </listblock>
>
> and then calling a separate routine that transforms this into the
> desired structure.

Afterwards? Could you not have a global variable that contains a stack
of bullettypes (of the preceding line) and do this while parsing? I
suspect you now have to materialize the parse tree, no?

> I love parsing.  Does it show?  :-)

It does. :-)

-- Jan Hidders


subject: [Wikipedia-l] Brazilian Portuguese

body:
Daniel Mayer wrote:
> American English isn't the same as British English nor is Mexican
> Spanish the same as Spanish Spanish.
>
> So you are going to have to justify balkanizing the Portuguese
> Wikipedia, IMO.

I agree completely about the need to justify.

http://www.jrdias.com/jrd-portugal-brasil.htm
looks to be an excellent resource for English speakers to learn
about the issue.

I don't know enough to make a valid judgment, but it seems that the
differences may be substntial.

http://www.necco.ca/faq_what_clients_need_to_know.htm
is linked from that earlier page, and gives an overview.

--Jimbo


from_name: millosh

subject: [Wikipedia-l] Re: Request for the second Serbian Wikipedia

body:
>> I suppose so, though I believe both Serbian and Croatian usually
>> "transliterate" foreign names, so "George Bush" becomes "Dz^ordz^
>> Bus^", whether in Latin or in Cyrillic Script.
>
> AFAIK only Serbian does that, both in cyrillic and in latin script.
> Croatian says "Georg Bush" while Serbian says "?or? Bu?".

Actually, we say D?ord? Bu?, not ?or? Bu?... One more thing: Croatians just write "Georg Bush", but say the same as in English ;)

Some foreign names are transcribed, some are not. Usually, we do not transcribe technical abbreviations (BGP, SMTP, POP, WWW, etc.), but we do that with cultural (DNA is in Serbian DNK, because acid in Serbian is 'kiselina'). GNU is the same in English and in Serbian, but GNU FDL is "GNU-ova LSD" or "Gnuova LSD" (Free Document License in Serbian is "Licenca za slobodnu dokumentaciju").

It is possible to have "group page". For example, page "BGP" would have just links for BGP in Cyrillic script and BGP in Latin script.


subject: [Wikitech-l] Wikicommons

body:
Stan Shebs wrote:
> Brion Vibber wrote:
>> Gerard Meijssen wrote:
>>> Please let us start the Wikicommons and add functionality when we
>>> think of some, when we program some.
>>
>> Erik M?ller proposed specific things for WikiCommons. When he gets
>> some code written, I'm sure we'll be happy to put it online.
>
> Do we really have to be blocked on coding? The situation with
> image duplication is really bad now, and it will only get worse
> as people want to illustrate articles in the non-English WPs.

Since there is not yet code to support pulling images in an organized
fashion from a separate site (WikiCommons), I don't see how putting up
another wiki would change anything right now -- everything would have to
be uploaded again on all the other wikis to be used.

Could you elaborate on how this would work?

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 253 bytes
Desc: OpenPGP digital signature
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20040817/262f8568/attachment.pgp>


references:
<cflt0k$e7u$1@sea.gmane.org>	<200408150805.30306.smolensk@eunet.yu>	<411F0351.40208@pobox.com>	<200408160634.54714.smolensk@eunet.yu>	<cfqfq6$fr9$1@sea.gmane.org>	<cfsdrj$bl1$1@sea.gmane.org>
	<cfsgav$h4d$1@sea.gmane.org>

body:
xmlizer wrote:

> Before, in the database you have to declare the keywords and when the
> parser find a keyword it flag the cur_kind the corresponding bit
> Example
>  -------------------------
> | keyword  | bit_position |
>  -------------------------
> | REDIRECT |      0       |
> | STUB     |      1       |
> | DISAMBIG |      2       |
> | FOOBAR   |      3       |
>  -------------------------
>
I just realized that we could also store the "protected" status there.
It would actually save memory in the database ;-)

Magnus


subject: [Wikitech-l] Wikicommons

body:
Brion Vibber wrote:

> Stan Shebs wrote:
>
>> Brion Vibber wrote:
>>
>>> Gerard Meijssen wrote:
>>>
>>>> Please let us start the Wikicommons and add functionality when we
>>>> think of some, when we program some.
>>>
>>>
>>> Erik M?ller proposed specific things for WikiCommons. When he gets
>>> some code written, I'm sure we'll be happy to put it online.
>>
>>
>> Do we really have to be blocked on coding? The situation with
>> image duplication is really bad now, and it will only get worse
>> as people want to illustrate articles in the non-English WPs.
>
>
> Since there is not yet code to support pulling images in an organized
> fashion from a separate site (WikiCommons), I don't see how putting up
> another wiki would change anything right now -- everything would have
> to be uploaded again on all the other wikis to be used.
>
> Could you elaborate on how this would work?

I don't know, that's why I'm asking. :-) Presumably there has to be
some coding, but the proposal I remember seeing was pretty elaborate.
For instance, a low-budget start might be to recognize when an Image:
reference has no local file, and substitute an external-link-like=
connection into the embryonic wikicommons.

For that matter, it wouldn't hurt just to create the wikicommons
site, upload a few pics to it and start experimenting, without
them necessarily being available to WP. There are some multilingual
issues (multiple image description pages? redirects for different
languages to use?) that could be worked out in the prototype, then
that will give some semi-real-world experience to guide what to
concentrate on for a real implementation.

But this is all coming from a MediaWiki noob, don't take it too
seriously!

Stan


from_name: Roozbeh Pournader

subject: [Wikipedia-l] personal threats again

body:
On Sun, 15 Aug 2004 08:11:39 -0700, Jimmy (Jimbo) Wales
<jwales at wikia.com> wrote:
> I don't know if there has been a solution to the lack of sysops in the
> Persian wikipedia while I was hiding from the hurricane, but if that's
> a problem I recommend just making a few people sysops on a temporary
> emergency basis and then let the usual methods be used to ratify.

Well, I made one other person a systop, we reverted the attacker's
edits many times and blocked random IPs he used for the standard 24
hour period, and the attacker seems to have stopped for almost a day
now.

What is still very weird, is that he used IP addresses from many
different places, from Thailand to the United States. My question is:
how could he?

roozbeh


from_name: Gabriel Wicke

subject: [Wikitech-l] Re: In CVS

body:
On Tue, 2004-08-17 at 10:29 +0100, Timwi wrote:
> Bryan Ford wrote:
>  LR parser generators are designed for languages that were
> > designed for LR parser generators; they tend to be difficult or impossible to
> > use for more freeform languages such as wikitext without making some serious
> > compromises or horrible hacks.
>
> Can you provide a more specific example of this?  So far, I have not
> encountered a situation where accepting all possible input strings would
> be hard to do.  In fact, all I need to do to ensure this is to allow the
> "text" non-terminal to contain any token.
>
> Also, I have not found it difficult to craft the grammar in such a way
> that bison's default disambiguation rules (conflict resolution rules)
> produce the correct result.  There does not seem to be any real need to
> have the grammar be unambiguous.

I agree, i didn't find any problems so far while working on the parser
at http://moinmoin.wikiwikiweb.de/NewWikiParser. Works fine so far, but
doesn't build the DOM tree yet (only returns simple text currently). My
plan is to use cDomlette for its low memory footprint and good dom
manipulation performance, haven't worked on it in the last two weeks
though.

An alternative to BisonGen might be the 'Gold parser
generator' (http://www.devincook.com/goldparser/) that does more
languages than just C and python, especially C# and Java. The module
wrapping would need to be done manually in that case, using swig or
something. Also the generator isn't really open source and only runs on
win. It can use normal BNF grammar though, not the more verbose BisonBen
xmlized one.

My current code:
http://dl.aulinx.de/wiki.bgen
http://dl.aulinx.de/wikishort.py
http://dl.aulinx.de/wikishort.c
--
Gabriel Wicke


from_name: Ray Saintonge

subject: [Wikipedia-l] Brazilian Portuguese

body:
Daniel Mayer wrote:

>--- Felipe Sanches <felipe.sanches at gmail.com> wrote:
>
>
>>Wikipedia new language request.
>>There are 182.1 million people in Brazil who use Brazilian Portuguese
>>as their main language.
>>Brazilian Portuguese IS NOT the same as Portuguese (domain
>>pt.wikipedia.org) and it still isn't on the list
>>(http://meta.wikimedia.org/wiki/Complete_list_of_language_wikis_available)
>>
>>
>American English isn't the same as British English nor is Mexican Spanish the
>same as Spanish Spanish.
>
>So you are going to have to justify balkanizing the Portuguese Wikipedia, IMO.
>A vote on that wiki (which is rather large and active) should be conducted.
>
>
I don't think that a vote would help.  It will only show one group to be
dominant, and may just divide that community even more.  Accepting the
other side's way of expressing itself is worth a lot more.

Ec


subject: [WikiEN-l] Milestone

body:
When we passed 100,000 , 200,000 , 300,000 etc articles there was a fair
   bit of fanfare.

I think it is only right therefore to announce that en passed the
100,000,000 word mark earlier this month.

This is about the same size as Columbia, Encarta Deluxe and EB put
together, and if printed would weigh about the same as George Bush.

Across all languages we reached this milestone back in January, and are
now up to 220,000,000 words.

Qualit?tsoffensive, anyone?

Pete/Pcb21


subject: [Wikitech-l] Re: An alternate parser

body:
On Sun, 2004-08-15 at 23:32 +0100, Timwi wrote:
> Magnus Manske wrote:
> >
> > I would prefer if this initiative wouldn't lead to a fork, partial or
> > not. IIRC, a parser-generator uses some language syntax file (in EBNF or
> > whatever is the flavor of the month now). If this is a more-or-less
> > standard format, we won't need a single parser generator that generates
> > both C and PHP; we could do with two separate ones which can read the
> > same format.

There are a few that do this already- BisonGen does both a pure python
parser and a C python module from a single spec file. The 'Gold parser
generator' does many languages from one grammar and multiple action code
files. See links in my earlier post.

Of course, the action code will always be language-specific, there might
be even different versions (using different dom implementations or just
printing plain xml text) in one language. If having the action code in a
single file or one file per language is better depends on the number of
implemented languages imo- more than two supported languages will be
messy in an xml syntax file like the BisonGen one.
--
Gabriel Wicke


from_name: Felipe Sanches

subject: [Wikipedia-l] Brazilian Portuguese

body:
Wikipedia new language request.
There are 182.1 million people in Brazil who use Brazilian Portuguese
as their main language.
Brazilian Portuguese IS NOT the same as Portuguese (domain
pt.wikipedia.org) and it still isn't on the list
(http://meta.wikimedia.org/wiki/Complete_list_of_language_wikis_available)


subject: [Wikitech-l] An alternate parser

body:
On Fri, 2004-08-13 at 20:46 +0200, Magnus Manske wrote:
> Warning: Yeat Another Crazy Idea of Mine ahead. If you're sick of these
> (by bitter experience;-) delete this mail *now*.
>
> Still here? Great!
>
> OK, we all know that the current parser, while working, is not the final
> word. It is kinda slow due to multi-pass, the source is confusing, and
> there are some persistant bugs in it, like the template malfunctions.
>
> I therefore suggest a new structure:
> 1. Preprocessor
> 2. Wiki markup to XML
> 3. XML to (X)HTML

This is what i'm writing currently, except that the parser will return a
dom tree instead of the xml dump of it. Saves another parse step before
postpocessing (template replacement, link status updates etc and the
final xslt transform). Besides being able to save the dom tree as xml at
any stage it's also possible to pickle the python object, which is a bit
faster to wake up than xml.

Caveat: Based on python's xml features, don't know a lot about php dom
implementations.
--
Gabriel Wicke


from_name: Jens Ropers

subject: [Wikitech-l] Re: [WikiEN-l] WP:(...)

body:
> Date: Tue, 17 Aug 2004 09:23:20 +0100
> From: Timwi <timwi at gmx.net>
> To: wikitech-l at wikimedia.org
>
> Jens Ropers wrote:
>
>> We don't actually have to move anything
>
> If you want to introduce a new namespace (or namespace alias) called
> "X", for example, you *do* have to move all the pages whose title
> begins
> with "X:", or else they will become inaccessible.

Err... no, actually. It's just a matter of additional checks and
PRECEDENCE when resolving. (See my previous mail and below.)

> This is because an article "WP:SB" is actually an article with the
> title
> "WP:SB" in the article namespace, while an article "Wikipedia:SB" is
> actually an article with the title "SB" in the Wikipedia namespace.  If
> you introduce a new namespace (or namespace alias) called "WP", the
> article [[WP:SB]] will still be in the article namespace, but the
> server
> will start looking for it in the new namespace and not find it.
>
> Timwi

Things COULD be implemented as follows:

Let's assume we're looking for "Wikipedia:Sandbox".
Now let's assume we're entering "WP:Sandbox".
Assuming name resolution as I am proposing were implemented, the
following would happen:
FIRST the system would do name resolution as usual: This means if any
article of "WP:Sandbox" existed in the article namespace, then this
article would be resolved to.
ONLY IF NO article named "WP:Sandbox" exists AND an article of
"Wikipedia:Sandbox" exists, THEN "WP:Sandbox" will redirect to
"Wikipedia:Sandbox".
IF no article of "Wikipedia:Sandbox" existed, THEN it would show the
standard "do you want to create an article?"-screen -- but this for
"WP:Sandbox" (!)

As I said in my previous post, it's strictly a matter of precedence, a
question of WHEN you check what.
The option of whether to expand "WP" to "Wikipedia" will ONLY get
checked AFTER ordinary article namespace resolution for WP:xyz has
failed, but BEFORE the "do you want to create this article"-screen is
shown.

All that said, I could understand why you might be weary of
implementing such a solution -- it might be unnecessarily confusing to
future maintainers of the system.

An alternative would be to create a bot that will sift through the
Wikipedia database and automatically create ''individual'' redirects
following the above rationale and order of checks. This might be a
preferred option.

Thanks and regards,
Jens Ropers

There are two types of IT techs: The ones who watch soap operas and the
ones who watch progress bars.
http://www.ropersonline.com/elmo/#108681741955837683


subject: [Wikitech-l] Re: Wikitech-l Digest, Vol 13, Issue 48

body:
>
> Message: 6
> Date: Tue, 17 Aug 2004 10:49:14 +0100
> From: Timwi <timwi at gmx.net>
>
> (...)
>
> I love parsing.  Does it show?  :-)
>
> Timwi
>

Say, you wouldn't have worked with sendmail in the past? :)

Thanks and regards,
Jens Ropers

There are two types of IT techs: The ones who watch soap operas and the
ones who watch progress bars.
http://www.ropersonline.com/elmo/#108681741955837683


from_name: Ævar Arnfjörð Bjarmason

subject: [Wikitech-l] Re: UTF-8 for nl:wikipedia

body:
He is probably confusing it with UTF-16 or UTF-32

On Tue, 17 Aug 2004 14:22:04 +0200, Stephan Walter
<stephan.walter at epfl.ch> wrote:
> Gerard Meijssen wrote:
> > some time ago to move to UTF-8. At the time it was not such a good idea
> > as UTF-8 does take more room.
>
> More room? UTF-8 does not use more memory, if that's what you mean. HTML
> entities (like Ӓ) use 5 up to 7 bytes, while a character in UTF-8
> uses at most 4 bytes.
>
> Regards,
> Stephan
>
> --
> Stephan.Walter at epfl.ch  --  http://lart.info/~stw/  --  PGP: B2421799
>
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at wikimedia.org
> http://mail.wikipedia.org/mailman/listinfo/wikitech-l
>


from_name: James R. Johnson

subject: [Wikipedia-l] Wikicommons

body:
A wikicommons would be very nice to have, a place to store all the pictures
and sound files we find, and then link to them in the wikipedia/wiktionary,
etc. - that'd save space in the wikis for more articles, etc.

James

-----Original Message-----
From: wikipedia-l-bounces at Wikimedia.org
[mailto:wikipedia-l-bounces at Wikimedia.org] On Behalf Of Gerard Meijssen
Sent: Tuesday, August 17, 2004 4:25 PM
To: wikitech-l at Wikipedia.org; wikipedia-l at Wikimedia.org
Subject: [Wikipedia-l] Wikicommons

Hoi,
There has been much talk about starting a repository for all kinds of media.
Pictures etc  that have a license compatible with the ideas of wikimedia.
The idea lives to have one repository for all pictures so that we need to
store it only once.
The idea lives to make Wikicommons ??  the location where the userid's for
the wikimedia projects may live.

Anyway, Wikicommons does not exist and we have all these harddrives full of
our digital snapshots.that may eventually end up on wikisomewhere.
Wouldn't it be nice to start Wikicommons and have a place to dump these
pictures now for people to find?

Yes, we cannot use Wikicommons pictures in the diverse projects yet, but now
we do not have the pictures either. What is stopping us from creating this
space and get something going ??

Personally I have nice pictures from within the Vatican, a scull of a Homo
erectus etc etc. I have been to many musea and I have pictures a plenty from
Berlin, Prague, Budapest, Brussels, Oostvaardersplassen. I am sure that I am
not the only one with interesting pictures that are usefull to others.

Please let us start the Wikicommons and add functionality when we think of
some, when we program some.

Thanks,
    GerardM
_______________________________________________
Wikipedia-l mailing list
Wikipedia-l at Wikimedia.org
http://mail.wikipedia.org/mailman/listinfo/wikipedia-l


from_name: Magnus Manske

subject: [Wikitech-l] XML: Part 2

body:
As we are now progressing towards XML generation, what to do with it? I
altered a little code from the PHP page about the XML parser to arrange
the XML in a treelike structure. Don't know where to put it, so I'll
just dump it below ;-)

<?

# Three global functions, sorry guys
function wgXMLstartElement($parser, $name, $attrs) {
   global $wgXMLobject;

   // If var already defined, make array
   eval('$test=isset('.$wgXMLobject->tree.'->'.$name.');');
   if ($test) {
     eval('$tmp='.$wgXMLobject->tree.'->'.$name.';');
     eval('$arr=is_array('.$wgXMLobject->tree.'->'.$name.');');
     if (!$arr) {
       eval('unset('.$wgXMLobject->tree.'->'.$name.');');
       eval($wgXMLobject->tree.'->'.$name.'[0]=$tmp;');
       $cnt = 1;
     }
     else {
       eval('$cnt=count('.$wgXMLobject->tree.'->'.$name.');');
     }

     $wgXMLobject->tree .= '->'.$name."[$cnt]";
   }
   else {
     $wgXMLobject->tree .= '->'.$name;
   }
   if (count($attrs)) {
       eval($wgXMLobject->tree.'->attr=$attrs;');
   }
}

function wgXMLendElement($parser, $name) {
   global $wgXMLobject;
   // Strip off last ->
   for($a=strlen($wgXMLobject->tree);$a>0;$a--) {
       if (substr($wgXMLobject->tree, $a, 2) == '->') {
           $wgXMLobject->tree = substr($wgXMLobject->tree, 0, $a);
           break;
       }
   }
}

function wgXMLcharacterData($parser, $data) {
   global $wgXMLobject;

   eval($wgXMLobject->tree.'->data=\''.$data.'\';');
}

# Here's the class that generates a nice tree
class wikiXML
    {
    function init_object ()
        {
        global $wgXMLobject ;
        $wgXMLobject->tree = '$wgXMLobject->xml';
        $wgXMLobject->xml = '';
        }

    function scanFile ( $filename )
        {
        $this->init_object () ;
        $xml_parser = xml_parser_create();
        xml_set_element_handler($xml_parser, "wgXMLstartElement",
"wgXMLendElement");
        xml_set_character_data_handler($xml_parser, "wgXMLcharacterData");
        if (!($fp = fopen($filename, "r"))) {
           die("could not open XML input");
        }

        while ($data = fread($fp, 4096)) {
           if (!xml_parse($xml_parser, $data, feof($fp))) {
               die(sprintf("XML error: %s at line %d",
                   xml_error_string(xml_get_error_code($xml_parser)),
                   xml_get_current_line_number($xml_parser)));
           }
        }
        xml_parser_free($xml_parser);
        }

    function scanString ( &$data )
        {
        $this->init_object () ;
        $xml_parser = xml_parser_create();
        xml_set_element_handler($xml_parser, "wgXMLstartElement",
"wgXMLendElement");
        xml_set_character_data_handler($xml_parser, "wgXMLcharacterData");
           if (!xml_parse($xml_parser, $data, true)) {
               die(sprintf("XML error: %s at line %d",
                   xml_error_string(xml_get_error_code($xml_parser)),
                   xml_get_current_line_number($xml_parser)));
           }
        xml_parser_free($xml_parser);
        }

    function is_list ( &$xml )
        {

        }

    function parseit ( &$xml )
        {
        print_r ( $xml ) ;
        }

}

$w = new wikiXML ;
$filename = 'sample.xml';
$w->scanFile ( $filename ) ;
$w->parseit ( $wgXMLobject->xml ) ;
return 0;

?>


from_name: Christian Semrau

subject: [Wikipedia-l] Double database entries

body:
Hello wikipedians all around the world!

I am Christian aka SirJective from the german WP, and want to promote some
knowledge about a database flaw that can be handled by sysops.

Did you ever ask yourselves why the "What links here" page of some articles
lists articles that don't link there?
Or found articles listed on Special:Shortpages that are not short (at least not
as short as listed on that page)?
Or articles listed on Special:Lonelypages that are not orphaned?

One possible solution is that these articles have duplicates, entries in the
database tables with the exact same title. Only one of these duplicates is
visible, but all of them contribute to automatically generated article lists.

These doubled entries are probably created, when a user wants to save a new
article and clicks "Save page" several times. Instead of ignoring the request or
saving the requests as different revisions of one article, the server creates
new articles. The technical details are being discussed at wikitech-l under the
same subject.

The doubled database entries can be removed by sysops who delete and undelete
these articles. All but the newest copy will then be marked as older revisions
of one article. At the german WP this worked nearly perfectly; some redirects
were missed, probably by deleting the redirection target, not the redirect page
itself.

I (and some users before me) created some lists of double database entries for
different languages, and all databases I checked have those entries. I uploaded
lists for en:, de: and nl: wikipedia. See
http://meta.wikipedia.org/wiki/Multiple_titles for some more information and
links to the lists.

You could run SQL queries to get lists of double entries on your WP, but I
suggest downloading a dump, see http://download.wikimedia.org, and running the
query on this dump. I would be happy to create these lists for other WPs, as
soon as some sysop from the other WP is willing to work on this problem.

Christian aka SirJective


from_name: rose.parks at att.net

body:
Hi-

     Yes, I did write the first article on Japanese, but it was on the English wikipedia. It is located, in much enhanced form at:

http://en.wikipedia.org/wiki/Japanese_Language

It is one of the first 100 or so articles in the English Wikipedia. I did not, however, have anything to do with the Japanese wikipedia.

I hope this helps...

                       As Ever,

                           Ruth Ifcher

                             User:RoseParks

--

-------------- Original message from "Tomos at Wikipedia" : --------------

> Hello.
>
> An email arrived at japanese wikipedia's press contact
> mailinglist regarding Web Creation Award. It is an award
> given to people who made significant difference in the web
> in Japan. The host of the award is an association of
> advertisers, though the recipients of the awards are not
> limited to marketing campaign planners. Wikipedia is under
> consideration because someone made a recommendation. :-)
>
> As a part of selection process, they would like to do
> some interview. The person who emailed us expressed that
> he would like to find the person who started the Japanese
> Wikipedia, someone who came up with the idea, plan, or
> determination. Who exactly this is, I think, is somewhat up
> to interpretation. Is there any strong candidate?
>
> I recall reading early Wikipedia-l posting (perhaps by Larry and/or Jason)
> announcing the launch of several non-English Wikipedia
> circa May 2001. Japanese Wikipedia was among the first
> ones. Is there anybody who remembers how that happened?
> Was multilingualization of Wikipedia an idea of Jimmy
> and Larry from the beginning? Or was there someone else
> who suggested/ pushed for it? Is there any reason why
> Japanese was among the first?
>
> In the early days, Aoineko was kind of "stationed" when
> there was only several edits a week, and welcomed the
> logged on users (there were only several back then) if
> I understand correct. Brion was there to fix bugs and
> take care of other technical aspects, too. The first
> article seems to have been written by RoseParks (phonemes
> of Japanese). But there are only fragmental traces and
> virtually nobody knows those early days. Could somebody
> inform us of people who made significant contributions?
>
> Thank you in advance for your help.
>
> Regards,
>
>
> Tomos
>
> More info. is available on the award at here (in Japanese):
>
> http://award.wab.ne.jp/award/index.html
>
> _________________________________________________________________
> FREE pop-up blocking with the new MSN Toolbar ? get it now!
> http://toolbar.msn.click-url.com/go/onm00200415ave/direct/01/
>
> _______________________________________________
> Wikipedia-l mailing list
> Wikipedia-l at Wikimedia.org
> http://mail.wikipedia.org/mailman/listinfo/wikipedia-l


from_name: Tim Starling

subject: [Wikitech-l] Re: replaceLinkHolders

body:
Brion Vibber wrote:
> Jens Frank wrote:
>
>> Congratulations! That sounds really fast! Now it's really faster than
>> the 1.3.1 renderer?
>
>
> Unfortunately no; it takes out a big chunk but it's still slower. This
> brings the full Village Pump page load time on my Linux box from 1501ms
> down to 901 ms, compared to 340 ms on 1.3.1.
>
> (Both with memcached and parser cache active; times are median total
> per-hit load time measured with 'ab -n 10' on the local host.)
>

That's probably because parser cache hits in 1.3 are extremely fast,
requiring no parsing or link lookup. In the placeholder scheme, I
thought it may be advantageous to save the placeholders to the parser
cache and perform link lookups even on cache hits. This means that the
parser cache doesn't need to be invalidated when linked articles are
created or deleted. I thought this would be more efficient based on the
assumption that replaceLinkHolders() would be very fast. If it is
unacceptably slow, we can replace the link holders with anchor tags
before we save them into the parser cache, thereby maintaining fast hit
service speed at the expense of keeping a lower cache hit rate.

Note that the code to avoid invalidating the parser cache when linked
articles are created or deleted has not yet been written, but it
shouldn't be hard to get it in before 1.4 if it will be advantageous.

-- Tim Starling


from_name: David Gerard

subject: [WikiEN-l] Re: you are autoblocking me

body:
On Tue, 17 Aug 2004 14:21:00 -0400, Cantus Vetustus <cantus at gmail.com> wrote:

> Look:
>
> 13:44, Aug 17, 2004, David Gerard blocked #7771 (expires 13:44, Aug
> 18, 2004) (Autoblocked because your IP address has been recently used
> by "'''Cantus'''". The reason given for Cantus's block is as follows:
> "'''[[WP:3RR]] in violation of Arbitration Committee ruling'''".)
>
> This is stupid. Everytime I try to edit an article logged in as Cantus
> (just to see if the block has expired), my IP is getting blocked for
> another 24 hours. When my original blocking time expires, I won't be
> able to edit, because the IP was reblocked for another 24 hours.
>
> You see, USER:CANTUS had an specific 24-hour block time. Any IPs
> USER:CANTUS uses under the *USER:CANTUS ACCOUNT* should only get
> blocked until the original 24-hour block of the *USER:CANTUS ACCOUNT*
> expires. Reblocking those IPs used by USER:CANTUS under the
> *USER:CANTUS ACCOUNT* is unfairly prolonguing the block for
> USER:CANTUS.
>
> It's an entirely different thing if USER:CANTUS, not logged in under
> the *USER:CANTUS ACCOUNT*, evaded the ban and actually edited an
> article with a sock puppet account or with an anonymous IP. But this
> was not the case here. In this case, USER:CANTUS was *under* the
> *USER:CANTUS ACCOUNT*, and only clicked the Edit button to see if he
> could edit, and his IP is getting reblocked for another 24-hours.

Yes, that is the system automatically resetting the clock on you as
long as you try to evade the block. If you stop trying for 24 hours,
it'll stop resetting the clock.

> If this is not fixed now, I'm talking to Jimbo.
> Sincerely,
> Cantus.

I've cc'd this to wikien-l, which should alert Jimbo to your
satisfaction. Also the developers, in case this isn't how the
autoblock is supposed to work.

(Note to all: this was after I blocked Cantus for 24 hours for
violating the Arbitration Committee verdict on his reverts - that if
he violated [[WP:3RR]], he was subject to a block. See [[Template
talk:Protected]] for the reversions in question, and see history of
[[Template:Protected]] for Cantus' edit warring.)

- d.


list: foundation-l

from_name: Michael Snow

subject: [Foundation-l] Re: Milestone

body:
Pete/Pcb21 wrote:

> When we passed 100,000 , 200,000 , 300,000 etc articles there was a
> fair   bit of fanfare.
>
> I think it is only right therefore to announce that en passed the
> 100,000,000 word mark earlier this month.
>
> This is about the same size as Columbia, Encarta Deluxe and EB put
> together, and if printed would weigh about the same as George Bush.
>
> Across all languages we reached this milestone back in January, and
> are now up to 220,000,000 words.
>
> Qualit?tsoffensive, anyone?
>
> Pete/Pcb21

Speaking of milestones, aren't we getting pretty close to one million
articles covering all languages? A nice handy round number probably
worthy of some fanfare. Are we planning a press release or other special
efforts to publicize it, either externally or internally? Sometime in
September looks like the target date, given our statistical direction.

--Michael Snow


subject: [WikiEN-l] Re: Milestone


list: wikimedia-l

subject: [Foundation-l] Re: Milestone


body:
Brion Vibber wrote:
> <evil tempter>The XHTML 2.0 draft has an <l> element for this...</evil>

Your post is not valid XML and cannot be displayed.

Errors:
line 1:
     error (303): parsing error: parse error, expecting `EQUALS'
line 1:
     error (1222): bogus attribute: >
line 1:
     error (1102): tag uses GI for an undeclared element: evil
line 1:
     error (1150): enclosing tag undefined or lacks content model; can't
check child: (CharData)
line 1:
     error (1102): tag uses GI for an undeclared element: l
line 1:
     error (1150): enclosing tag undefined or lacks content model; can't
check child: (CharData)
line 1:
     error (1103): end tag uses GI for an undeclared element: evil
line 1:
     error (1098): end tag doesn't match its start tag: evil
line 1:
     error (1150): enclosing tag undefined or lacks content model; can't
check child: l
line 1:
     error (780): missing end tag: evil
line 1:
     error (402): EOF encountered; no doctype declaration found: evil

-- Tim Starling


subject: [WikiEN-l] New English Wikipedia Mailing List Admin needed

body:
On 08/17/04 12:56, Poor, Edmund W wrote:

> Folks, it's been fun administering the mailing list of the English
> Wikipedia, but I no longer have the time to do a good job.
> I'm resigning effective Friday, August 20th. If anyone wants to be
> trained as my replacement, I'll be glad to help (there's not much to it,
> really).

*raises hand* I can easily check once a day, possibly more often.
What's it involve? Feel free to email me directly :-)

- d.


subject: [WikiEN-l] Milestone

body:
On 08/17/04 14:34, Pete/Pcb21 wrote:

> When we passed 100,000 , 200,000 , 300,000 etc articles there was a fair
>   bit of fanfare.
> I think it is only right therefore to announce that en passed the
> 100,000,000 word mark earlier this month.
> This is about the same size as Columbia, Encarta Deluxe and EB put
> together, and if printed would weigh about the same as George Bush.

I award you the job of writing the Slashdot story, using that exact
weight measurement ;-)

> Across all languages we reached this milestone back in January, and are
> now up to 220,000,000 words.

- d.


subject: [WikiEN-l] Milestone

body:
On Tue, 17 Aug 2004 15:34:10 +0100, Pete/Pcb21
<pete_pcb21_wpmail at pcbartlett.com> wrote:
> When we passed 100,000 , 200,000 , 300,000 etc articles there was a fair
>    bit of fanfare.
>
> I think it is only right therefore to announce that en passed the
> 100,000,000 word mark earlier this month.
>
> This is about the same size as Columbia, Encarta Deluxe and EB put
> together, and if printed would weigh about the same as George Bush.
How many kilos, what kind of paper and how many words pr/page ? (I
have no idea how heavy George Bush is (Is this the new measurement for
weight in the spirit of "football fields" and "vw beatles"?))
>
> Across all languages we reached this milestone back in January, and are
> now up to 220,000,000 words.
>
> Qualit?tsoffensive, anyone?
>
> Pete/Pcb21
>
> _______________________________________________
> WikiEN-l mailing list
> WikiEN-l at Wikipedia.org
> http://mail.wikipedia.org/mailman/listinfo/wikien-l
>


subject: [Wikitech-l] CVS problems & snapshot

body:
Wouldnt this be a nice time to set up svn.wikimedia.org to go along
with the nice bugzilla.wikimedia.org? ;=)

On Tue, 17 Aug 2004 00:37:52 -0700, Brion Vibber <brion at pobox.com> wrote:
> One of SourceForge's anonymous CVS mirrors has been down since August
> 12, naturally the one that serves us. This has broken:
> * anonymous CVS checkouts
> * ViewCVS
> * nightly backups of the CVS repository
>
> Developer CVS is still working; the anonymous servers just mirror that,
> so we're still working on things ok but it's a pretty annoying service
> outage. SF's status page says they expect it to be fixed sometime on
> August 13 *cough*. We'll see... :P
>
> For those of you who want to work on the current source but aren't on
> the project on SF, here's a fresh checkout of CVS head branch with
> phase3, extensions, and wiki2xml modules:
> http://download.wikimedia.org/mediawiki/mediawiki-cvs-2004-08-17.tar.bz2
>
> -- brion vibber (brion @ pobox.com)
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at wikimedia.org
> http://mail.wikipedia.org/mailman/listinfo/wikitech-l
>
>
>


from_name: Jan Hidders

subject: [Wikitech-l] XML: Part 2

body:
Magnus Manske wrote:

> As we are now progressing towards XML generation, what to do with it?
> I altered a little code from the PHP page about the XML parser to
> arrange the XML in a treelike structure. Don't know where to put it,
> so I'll just dump it below ;-)

Using the eval function makes it a bit hard to understand and I suspect
it isn't very efficient. How about the following:

<?

//  the base class for an element
class element {
  var $name = "";
  var $attrs = array();
  var $children = array();

  function myPrint() {
    echo "<UL>";
    echo "<LI> <B> Name: </B> $this->name";
    // print attributes
    echo "<LI> <B> Attributes: </B>";
    foreach ($this->attrs as $name => $value) {
      echo "$name => $value; " ;
    }
    // print children
    foreach ($this->children as $child) {
       if ( is_string($child) ) {
         echo "<LI> $child";
       } else {
         $child->myPrint();
       }
    }
    echo "</UL>";
  }

}

$ancStack = array();    // the stack with ancestral elements
$allElem = array();     // the stack with all elements in the tree

// Three global functions needed for parsing, sorry guys
function wgXMLstartElement($parser, $name, $attrs) {
   global $ancStack, $allElem;

   $newElem = new element;
   $newElem->name = $name;
   $newElem->attrs = $attrs;
   array_push($allElem, $newElem);
   $newIndex = count($allElem)-1;
   // add to parent if parent exists
   if ( $ancStack ) {
     array_push($ancStack[count($ancStack)-1]->children,
&$allElem[$newIndex]);
   }
   // push new element itself on ancestor stack
   array_push($ancStack, &$allElem[$newIndex]);
}

function wgXMLendElement($parser, $name) {
  global $ancStack;

  // pop element of stack
  array_pop($ancStack);
}

function wgXMLcharacterData($parser, $data) {
  global $ancStack;
  // add to parent if parent exists
  if ( $ancStack ) {
    array_push($ancStack[count($ancStack)-1]->children, $data);
  }
}

# Here's the class that generates a nice tree
class xml2php {

   function &scanFile( $filename ) {
       global $allElem;

       $xml_parser = xml_parser_create();
       xml_set_element_handler($xml_parser, "wgXMLstartElement",
"wgXMLendElement");
       xml_set_character_data_handler($xml_parser, "wgXMLcharacterData");
       if (!($fp = fopen($filename, "r"))) {
          die("could not open XML input");
       }
       while ($data = fread($fp, 4096)) {
          if (!xml_parse($xml_parser, $data, feof($fp))) {
              die(sprintf("XML error: %s at line %d",
                  xml_error_string(xml_get_error_code($xml_parser)),
                  xml_get_current_line_number($xml_parser)));
          }
       }
       xml_parser_free($xml_parser);

       return $allElem[0];
   }

}

$w = new xml2php;
$filename = 'sample.xml';
$result = $w->scanFile( $filename );
$result->myPrint();

return 0;

?>


subject: [WikiEN-l] WP:(...)

body:
From: WikiEN-l Digest, Vol 13, Issue 50:
Timwi <timwi at gmx.net> wrote:

> Message: 4
> Date: Tue, 17 Aug 2004 09:25:32 +0100
> To: wikien-l at wikipedia.org
>
> Jens Ropers wrote:
>
>> We don't actually have to move anything

(my explanation why that is and how precedence can ensure things are
resolved right anyway was removed here)

> If you want to introduce a new namespace (or namespace alias), you *do*
> have to move all pages whose title begins with the new namespace name
> followed by a colon, or else they will become inaccessible.  For the
> reasons, please see my reply on the technical list.

I've already sent a (politely dissenting) reply to the tech list. :-)
-- Jens


from_name: Gerard Meijssen

subject: [Wikipedia-l] Wikicommons

body:
Hoi,
There has been much talk about starting a repository for all kinds of
media. Pictures etc  that have a license compatible with the ideas of
wikimedia. The idea lives to have one repository for all pictures so
that we need to store it only once.
The idea lives to make Wikicommons ??  the location where the userid's
for the wikimedia projects may live.

Anyway, Wikicommons does not exist and we have all these harddrives full
of our digital snapshots.that may eventually end up on wikisomewhere.
Wouldn't it be nice to start Wikicommons and have a place to dump these
pictures now for people to find?

Yes, we cannot use Wikicommons pictures in the diverse projects yet, but
now we do not have the pictures either. What is stopping us from
creating this space and get something going ??

Personally I have nice pictures from within the Vatican, a scull of a
Homo erectus etc etc. I have been to many musea and I have pictures a
plenty from Berlin, Prague, Budapest, Brussels, Oostvaardersplassen. I
am sure that I am not the only one with interesting pictures that are
usefull to others.

Please let us start the Wikicommons and add functionality when we think
of some, when we program some.

Thanks,
    GerardM


subject: [Wikitech-l] Wikicommons


subject: [Wikitech-l] XML: Part 2

body:
On Tuesday 17 August 2004 21:38, Jan Hidders wrote:
> Magnus Manske wrote:
> > As we are now progressing towards XML generation, what to do with it?
> > I altered a little code from the PHP page about the XML parser to
> > arrange the XML in a treelike structure. Don't know where to put it,
> > so I'll just dump it below ;-)
>
> Using the eval function makes it a bit hard to understand and I suspect
> it isn't very efficient. How about the following: [...]

Here is a better version that doesn't require an array that stores all
elements in the tree:

<?

//  the base class for an element
class element {
  var $name = "";
  var $attrs = array();
  var $children = array();

  function myPrint() {
    echo "<UL>";
    echo "<LI> <B> Name: </B> $this->name";
    // print attributes
    echo "<LI> <B> Attributes: </B>";
    foreach ($this->attrs as $name => $value) {
      echo "$name => $value; " ;
    }
    // print children
    foreach ($this->children as $child) {
       if ( is_string($child) ) {
         echo "<LI> $child";
       } else {
         $child->myPrint();
       }
    }
    echo "</UL>";
  }

}

$ancStack = array();    // the stack with ancestral elements

// Three global functions needed for parsing, sorry guys
function wgXMLstartElement($parser, $name, $attrs) {
   global $ancStack, $rootElem;

   $newElem = new element;
   $newElem->name = $name;
   $newElem->attrs = $attrs;
   array_push($ancStack, $newElem);
   // add to parent if parent exists
   $nrAncs = count($ancStack)-1;
   if ( $nrAncs > 0 ) {
     array_push($ancStack[$nrAncs-1]->children, &$ancStack[$nrAncs]);
   } else {
     // make extra copy of root element and alias it with the original
     array_push($ancStack, &$ancStack[0]);
   }
}

function wgXMLendElement($parser, $name) {
  global $ancStack, $rootElem;

  // pop element of stack
  array_pop($ancStack);
}

function wgXMLcharacterData($parser, $data) {
  global $ancStack, $rootElem;

  // add to parent if parent exists
  if ( $ancStack ) {
    array_push($ancStack[count($ancStack)-1]->children, $data);
  }
}

# Here's the class that generates a nice tree
class xml2php {

   function &scanFile( $filename ) {
       global $ancStack;

       $xml_parser = xml_parser_create();
       xml_set_element_handler($xml_parser, "wgXMLstartElement",
"wgXMLendElement");
       xml_set_character_data_handler($xml_parser,
"wgXMLcharacterData");
       if (!($fp = fopen($filename, "r"))) {
          die("could not open XML input");
       }
       while ($data = fread($fp, 4096)) {
          if (!xml_parse($xml_parser, $data, feof($fp))) {
              die(sprintf("XML error: %s at line %d",
                  xml_error_string(xml_get_error_code($xml_parser)),
                  xml_get_current_line_number($xml_parser)));
          }
       }
       xml_parser_free($xml_parser);

       // return the remaining root element we copied in the beginning
       return $ancStack[0];
   }

}

$w = new xml2php;
$filename = 'sample.xml';
$result = $w->scanFile( $filename );
$result->myPrint();

return 0;

?>


subject: [Wikitech-l] Wikicommons

body:
Brion Vibber wrote:

> Gerard Meijssen wrote:
>
>> Please let us start the Wikicommons and add functionality when we
>> think of some, when we program some.
>
>
> Erik M?ller proposed specific things for WikiCommons. When he gets
> some code written, I'm sure we'll be happy to put it online.
>
> -- brion vibber (brion @ pobox.com)
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Wikitech-l mailing list
>Wikitech-l at wikimedia.org
>http://mail.wikipedia.org/mailman/listinfo/wikitech-l
>
What is the timeline, why are we waiting? When there is  a repository,
what is stopping Erik in adding functionality? There are loads of
pictures that are not uploaded as pictures _without_ a story are orphans
and get deleted. Respectable usefull pictures that just do not have a
story yet.

Why wait ? How long ? What for ?

Thanks,
  GerardM


subject: [WikiEN-l] GWB weight

body:
According to the www:
"At his August 2001 physical, Bush was 6 feet 0 inches tall, and weighed
189.75 lbs"

189.75 lbs ~ 86kg

The Second Edition of the OED weighs 62.6kg and consists of 59 million
words on 21,730 pages. (http://www.oed.com/about/facts.html).

So assuming the printed WP had the same size volume and type of binding
as the OED, it follows after a little calculation that WP would weigh
the same as GWB if about 3350 words were put on each page. This is
possible if you use a size 8 font and big (OED sized) pages.

The heaviest President was William Taft (1909-1913), at 322lbs. If we
are allowed this weight, we could probably use size 10.

Pete


subject: [Wikitech-l] Wikicommons

body:
Brion Vibber wrote:

> Stan Shebs wrote:
>
>> Brion Vibber wrote:
>>
>>> Gerard Meijssen wrote:
>>>
>>>> Please let us start the Wikicommons and add functionality when we
>>>> think of some, when we program some.
>>>
>>>
>>> Erik M?ller proposed specific things for WikiCommons. When he gets
>>> some code written, I'm sure we'll be happy to put it online.
>>
>>
>> Do we really have to be blocked on coding? The situation with
>> image duplication is really bad now, and it will only get worse
>> as people want to illustrate articles in the non-English WPs.
>
>
> Since there is not yet code to support pulling images in an organized
> fashion from a separate site (WikiCommons), I don't see how putting up
> another wiki would change anything right now -- everything would have
> to be uploaded again on all the other wikis to be used.
>
> Could you elaborate on how this would work?
>
> -- brion vibber (brion @ pobox.com)
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Wikitech-l mailing list
>Wikitech-l at wikimedia.org
>http://mail.wikipedia.org/mailman/listinfo/wikitech-l
>
To start off, you would be right what is the point but to have a place
where you can dump pictures. Pictures that you need to upload to the
appropriate wikipedia in order to use them. (I hope that it can be
organised in such a way that we can at least link to the wikicommon
pictures).

The resulting mess can be organised in articles or categories around a
subject. All pictures without an apropriate license get deleted. When
some code arrives, it will make the stuff more usefull. Untill that time
we can publish the stuff and have a change for it to be usefull.

The good thing for the programmer(s) will be that there will be a
testing ground and that code can be added and it will be used. Because
there is the opportunity. Another advantage is that WikiCommons will not
be academic anymore. More people can contribute to it, it is the open
source way of doing things.

Thanks,
  GerardM


from_name: Daniel Mayer

subject: [Wikipedia-l] Brazilian Portuguese

body:
--- "James R. Johnson" <modean52 at comcast.net> wrote:

> How do we find out the results of a vote on a wiki?

You link to the result page.

> And who votes?

The people who participate on the Portuguese wiki would be the ones to vote.
Since they speak Portuguese, they, not non-speakers, are the ones best suited
to decide if the differences between any of their dialects are sufficiently
incompatible to warrent a separate wiki.

-- mav

__________________________________
Do you Yahoo!?
Yahoo! Mail - 50x more storage than other providers!
http://promotions.yahoo.com/new_mail


subject: [Wikipedia-l] Brazilian Portuguese

body:
--- "James R. Johnson" <modean52 at comcast.net> wrote:

> How do we find out the results of a vote on a wiki?

You link to the result page.

> And who votes?

The people who participate on the Portuguese wiki would be the ones to vote.
Since they speak Portuguese, they, not non-speakers, are the ones best suited
to decide if the differences between any of their dialects are sufficiently
incompatible to warrent a separate wiki.

-- mav

__________________________________
Do you Yahoo!?
Yahoo! Mail is new and improved - Check it out!
http://promotions.yahoo.com/new_mail


subject: [WikiEN-l] Re: Milestone

body:
--- Michael Snow <wikipedia at earthlink.net> wrote:
> Speaking of milestones, aren't we getting pretty close to one million
> articles covering all languages? A nice handy round number probably
> worthy of some fanfare. Are we planning a press release or other special
> efforts to publicize it, either externally or internally? Sometime in
> September looks like the target date, given our statistical direction.

We already have a 500,000 article milestone press release on meta at
http://meta.wikimedia.org/wiki/Wikimedia's_first_press_release

All we need to do is update the numbers and contacts. But I'm sure other
improvements will be made as well.

I coordinated that press release - anybody want to corrdinate the new one? If
not, then I'll do it.

Oh, and there was a major drift in the article number count. I'm not sure if
that bug has been fixed yet. Hm - I just checked and the number on the Main
Page is now fairly close to the number on the WikiStats page. But then, they
could both be wrong....

-- mav

__________________________________
Do you Yahoo!?
Yahoo! Mail is new and improved - Check it out!
http://promotions.yahoo.com/new_mail


subject: [Wikipedia-l] Brazilian Portuguese

body:
How do we find out the results of a vote on a wiki?  And who votes?

James

-----Original Message-----
From: wikipedia-l-bounces at Wikimedia.org
[mailto:wikipedia-l-bounces at Wikimedia.org] On Behalf Of Daniel Mayer
Sent: Tuesday, August 17, 2004 4:41 PM
To: Felipe Sanches; wikipedia-l at Wikimedia.org
Subject: Re: [Wikipedia-l] Brazilian Portuguese

--- Felipe Sanches <felipe.sanches at gmail.com> wrote:

> Wikipedia new language request.
> There are 182.1 million people in Brazil who use Brazilian Portuguese
> as their main language.
> Brazilian Portuguese IS NOT the same as Portuguese (domain
> pt.wikipedia.org) and it still isn't on the list
> (http://meta.wikimedia.org/wiki/Complete_list_of_language_wikis_availa
> ble)

American English isn't the same as British English nor is Mexican Spanish
the same as Spanish Spanish.

So you are going to have to justify balkanizing the Portuguese Wikipedia,
IMO.
A vote on that wiki (which is rather large and active) should be conducted.

-- mav

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com _______________________________________________
Wikipedia-l mailing list
Wikipedia-l at Wikimedia.org
http://mail.wikipedia.org/mailman/listinfo/wikipedia-l


subject: [Foundation-l] Proposed Project: Old English Wikipedia

body:
Hello,
    I would also like to propose an  Anglo-Saxon  Wikipedia  and Wiktionary
.  Its ISO Code is ang.   The language has already been added to several
definitions in the Wiktionary, and I personally have begun translating the
interface of the Wikipedia into Anglo-Saxon.  What would anyone recommend to
get this wikipedia done?

James


from_name: Ewen Caroline

subject: [Wikipedia-l] Wikicommons

body:
And it will save time looking for pictures where you can never be sure about
the license etc. I'm sure, lots of us (myself included) have holiday pics
from various places on their computer, so let's share them.

Caroline, LB.

On Tue, 17 Aug 2004 16:27:58 -0400, James R. Johnson wrote
> A wikicommons would be very nice to have, a place to store all the pictures
> and sound files we find, and then link to them in the
> wikipedia/wiktionary, etc. - that'd save space in the wikis for more
> articles, etc.
>
> James
>
> -----Original Message-----
> From: wikipedia-l-bounces at Wikimedia.org
> [mailto:wikipedia-l-bounces at Wikimedia.org] On Behalf Of Gerard Meijssen
> Sent: Tuesday, August 17, 2004 4:25 PM
> To: wikitech-l at Wikipedia.org; wikipedia-l at Wikimedia.org
> Subject: [Wikipedia-l] Wikicommons
>
> Hoi,
> There has been much talk about starting a repository for all kinds
> of media. Pictures etc  that have a license compatible with the
> ideas of wikimedia. The idea lives to have one repository for all
> pictures so that we need to store it only once. The idea lives to
> make Wikicommons ??  the location where the userid's for the
> wikimedia projects may live.
>
> Anyway, Wikicommons does not exist and we have all these harddrives
> full of our digital snapshots.that may eventually end up on
> wikisomewhere. Wouldn't it be nice to start Wikicommons and have a
> place to dump these pictures now for people to find?
>
> Yes, we cannot use Wikicommons pictures in the diverse projects yet,
> but now we do not have the pictures either. What is stopping us from
> creating this space and get something going ??
>
> Personally I have nice pictures from within the Vatican, a scull of
> a Homo erectus etc etc. I have been to many musea and I have
> pictures a plenty from Berlin, Prague, Budapest, Brussels,
> Oostvaardersplassen. I am sure that I am not the only one with
> interesting pictures that are usefull to others.
>
> Please let us start the Wikicommons and add functionality when we
> think of some, when we program some.
>
> Thanks,
>     GerardM
> _______________________________________________
> Wikipedia-l mailing list
> Wikipedia-l at Wikimedia.org
> http://mail.wikipedia.org/mailman/listinfo/wikipedia-l
>
> _______________________________________________
> Wikipedia-l mailing list
> Wikipedia-l at Wikimedia.org
> http://mail.wikipedia.org/mailman/listinfo/wikipedia-l

--
Open WebMail Project (http://openwebmail.org)


subject: [Wikipedia-l] Wikicommons

body:
Andrew Lih wrote:
> FYI, you might be interested in a similar new Open Source Media
> project started by Marc Canter and J.D. Lasica, which can be found
> here: http://www.newmediamusings.com/blog/2004/08/open_media_the_.html

Thanks for pointing this out Andrew. I've created a page for people to
give ideas on how we might collaborate with them at
http://meta.wikimedia.org/wiki/Wikimedia_Commons/Collaboration_with_Open_Media.
For those with a login to the wiki on Socialtext, there is a
discussion page on how they might collaborate with various partners at
http://www.socialtext.net/openmedia/index.cgi?partners.

Angela.