subject:
[Wikitech-l] Changing the "Powered by" and "GNU FDL" images at the
	bottom of pages.

body:
Is it possible to get the source files for these two images somewhere?:

http://en.wikipedia.org/style/images/poweredby_mediawiki_88x31.png
http://en.wikipedia.org/style/images/gnu-fdl.png

I would like to alter the text for the Icelandic Wikipedia, change
"Powered by" to "Kn?i? af" and "GNU FDL FREE DOC LICENCE" to some
equivalent in Icelandic,

This would be the only part that cannot currently be translated right
(not even de. has it), could some system similar to the one with
Image:Wiki.png be put in place so that it would be possible to replace
these with translated versions?

Of course the source files for them would be pretty essential to that,
is it possible to get them?


list: wikitech-l

source: pipermail-mbox

from_name: Magnus Manske

subject: [Wikitech-l] wiki2xml : Deeds following words

body:
Hello all,

after so much talk about an alternate parser, here it is! Well, half of
it, anyway.

Attached (only 8KB), you'll find my manual C++ source (GPL) to convert
wiki markup to XML.

Remarks:
* This was hacked in a few days, so don't expect it to work perfectly,
although it does surprisingly well so far.
* It doesn't check for invalid user XML constructs (only partially
implemented so far)
* It doesn't recognize nowiki and pre tags
* Some small wikimarkup is not supported (ISBNs, for example)
All of these problems are due to to early stages of this project, not
because of "can't do".

* At the moment, it reads the source from the file "test.txt". Switching
this to piped command line input is dead easy, though.
* Because of strangeness in the Firebird XML display, I have to replace
"—" with something else ("!mdash;" in this case...), otherwise it
doesn't render as valid XML.

I have tested it against several "live" wikipedia articles, and
benchmarked a few on my laptop:
* List_of_state_leaders_in_1839 : 200 pages per second (link-heavy page)
* Operation_Downfall : 380 pages per second ("normal" page of
above-average length)
* List_of_Baronies : 3 pages per second (source is 233KB...)
* Sian_Lloyd : 2300 pages per second (stub)

For an average page (in number of links, tables, overall length etc.), I
estimate above 500 conversions per second on an off-the-shelf laptop
(with stuff running in the background). This ain't bad...

I am certain that further tweaking could enhance this some more.

I have not yet found a page that doesn't output correct XML; however, it
crashes on [[Results of the Canadian federal election, 2004]], our
longest page, for unknown reasons. I assume it voted for the other party ;-)

I would this consider that a promising start. Please give it a try, and
maybe look into the PHP XML-parser.

Magnus


from_name: Eckhart Wörner

body:
Hello Magnus,

Magnus Manske schrieb:
> after so much talk about an alternate parser, here it is! Well, half of
> it, anyway.
>
> Attached (only 8KB), you'll find my manual C++ source (GPL) to convert
> wiki markup to XML.

wasn't there something about attachments? ;-)

At least, i didn't get the source.

Eckhart


from_name: Brion Vibber

subject: [Wikitech-l] replaceLinkHolders (was: External diff engine)

body:
Brion Vibber wrote:
> Clearly there's still room for improvement in replaceLinkHolders.

I've added some smaller profiling cunks in that functions:

OutputPage::replaceLinkHolders           2 1330.494 665.247 83.280%
OutputPage::replaceLinkHolders-replace   2  934.327 467.163 58.483%
OutputPage::replaceLinkHolders-check     2  245.321 122.660 15.355%
OutputPage::replaceLinkHolders-construct 2  141.510  70.755  8.858%
OutputPage::replaceLinkHolders-match     2    8.755   4.377  0.548%

The big 'replace' section is just this line:
$this->mBodytext = str_replace( $search, $replace, $this->mBodytext );

where $search and $replace are arrays with hundreds of items and
$this->mBodytext is a couple hundred kb of text.

(Note that the function is being called twice -- once in Article::view()
and again in OutputPage::output(). The second one is relatively quick
[21ms] since the page view has already done its thing. The first call
dominates, at least for a large page with many links like this.)

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 253 bytes
Desc: OpenPGP digital signature
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20040816/4ad28bd0/attachment.pgp>


from_name: Michael Becker

subject: [Wikitech-l] In CVS (was: wiki2xml : Deeds following words)

body:
I'm going to break out my compiler books and start working on a CFG for the wiki code based on this parser. It should not be too hard to change it if we decide not to use xml as an intermediate language. Since I know there are people out there with more experience than me, I won't be offended if someone beats me to this...

On Mon, 16 Aug 2004 15:00:51 +0200, Emmanuel Engelhart <emmanuel at engelhart.org> wrote:
> I was very happy, because you coded this first parser very fast. And, I support
> the idea of making a real wiki parser. But, it seems that you didn't use
> flex/bison or any parser generator. I think continuing in this direction, would
> be a very bad idea. The best specialists use parser generators, and we are not
> so good like parsing specialists...
>
> Emmanuel
>
> Selon Magnus Manske <magnus.manske at web.de>:
>
>
>
> > I have now created a new module in the CVS : "wiki2xml". Check it out ;-)
> >
> > Magnus
> >
> >
> > Emmanuel Engelhart wrote:
> >
> > >>Try http://www.magnusmanske.de/wiki2xml.zip
> > >>
> > >>
> > >>
> > >
> > >My 'unzip' command can't unzip this file.... strange.
> > >
> > >Emmanuel Engelhart
> > >_______________________________________________
> > >Wikitech-l mailing list
> > >Wikitech-l at wikimedia.org
> > >http://mail.wikipedia.org/mailman/listinfo/wikitech-l
> > >
> > >
> > _______________________________________________
> > Wikitech-l mailing list
> > Wikitech-l at wikimedia.org
> > http://mail.wikipedia.org/mailman/listinfo/wikitech-l
> >
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at wikimedia.org
> http://mail.wikipedia.org/mailman/listinfo/wikitech-l
>

--
Michael Becker


subject: [Wikitech-l] External diff engine

body:
Magnus Manske wrote:
> Yup, that function's new (didn't see it a few days ago). Seems the most
> time is spent in database queries. Not much to do about that except
> caching, which we do. We could collect the links to use one big query
> instead of several ones, maybe?

It already collects it into one big query, and most of the time in the
large-page-with-many-links case is spent in a single str_replace() call.

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 253 bytes
Desc: OpenPGP digital signature
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20040816/ef5b64e5/attachment.pgp>


subject: [Wikitech-l] replaceLinkHolders

body:
Jens Frank wrote:
> Do you also have figures for release 1.3.1 rendering the same page?

Median page view time for this page on 1.3.1 with $wgEnableParserCache
is 346ms as measured with ab on the same machine.

At this point a dominant time sink is loading and parsing the PHP files
(I haven't installed a PHP cache module on this machine), and skin
building output and buffering/sending of raw output become significant
as well.

Major offenders from profiling output:

-total                  1 277.770 277.770 100.000%
Setup.php               1 151.931 151.931 54.697%
Skin::outputPage        1 102.973 102.973 37.071%
OutputPage::output      1 103.143 103.143 37.132%
Skin::doAfterContent    1 63.289 63.289 22.785%
wfMsg                  53 50.454  0.952 18.164%
Skin::quickBar          1 34.159 34.159 12.297%
Skin::doBeforeContent   1 27.438 27.438  9.878%
Article::view           1 18.326 18.326  6.597%

So clearly we've gone backwards on at least some cases by doing link
replacement after the parser cache. (A very short page with no links
renders slightly faster with 1.4-cvs than on 1.3.1.)

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 253 bytes
Desc: OpenPGP digital signature
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20040816/52bf1356/attachment.pgp>


subject: AW: [Wikitech-l] wiki2xml : Deeds following words

body:
<curse mode='censored'>!"?$%&/()</curse>

Try http://www.magnusmanske.de/wiki2xml.zip

Magnus

Eric P?hlsen wrote:

>
>
>
>>At least, i didn't get the source.
>>
>>Eckhart
>>
>>
>>
>
>I guess the list does not allow attachments
>
>Eric
>
>_______________________________________________
>Wikitech-l mailing list
>Wikitech-l at wikimedia.org
>http://mail.wikipedia.org/mailman/listinfo/wikitech-l
>
>


from_name: Emmanuel Engelhart

body:
>
> Try http://www.magnusmanske.de/wiki2xml.zip
>

My 'unzip' command can't unzip this file.... strange.

Emmanuel Engelhart


from_name: Tim Starling

subject: [Wikitech-l] Two markets (was Re: An alternate parser)

body:
Magnus Manske wrote:
> I would prefer if this initiative wouldn't lead to a fork, partial or
> not.

What's wrong with forking? We have two different markets for our
software, and these markets have conflicting needs. Why not produce a
customised product for each one?

-- Tim Starling


from_name: Jimmy (Jimbo) Wales

subject: [Wikitech-l] Wiki branches, voting

body:
Andre Engels wrote:
> >The majority has no choice but to respect the viewpoint of the
> >minority, because the minority can always just _edit the article_.
>
> That sounds very idealistic, but is it realistic? Do you see this happen
> on any pages?

Yes, I see it everywhere, all the time.  It is the primary mode of
operation on the website.

> Hardly. For one thing, there's usually not 10 people working
> on a page, but 2 or 3, who either agree or do not. And secondly, either
> people want to compromise, or they do not. If they do, they'll vote for
> compromise solutions too.

>From a game-theoretic point of view, this is not likely to be true.

People are often *willing* to compromise, if the system makes it
necessary, but are equally *willing* to force the majority view, if
the system makes it possible.

> I really don't see why 9 people voting for A and 1 for B, then choosing A
> is a worse way of getting compromise than 9 people reverting B to A and 1
> reverting A to B, and then choose B when that 1 person happens to be the
> most determined and A otherwise.

But this last is very rarely the way articles are edited.  It is a
reasonable description of at least *some* edit wars, but even *most*
edit wars do not work in this way for long.  There is no stable
equilibrium until the A's and the B's find a way to "go meta", and
that's what normally happens.

> And what is the incentive now?

If you want your text to survive, then whether you are in the majority
or the minority, you must find a way to present it such that all sides
are satisfied.  That is _the_ central incentive in the Wikipedia
system.

And this is why Wikipedia is as good as it is, because the software
forces people to live together, and has no mechanisms to permit anyone
(majority or minority) to force through something which is not a
compromise.

You can find clear examples of this in action on virtually every
article on the site, even articles where edit wars have broken out and
gone on for a very long time, due to some people refusing to
compromise.  But in most cases, even cases with edit wars, the fact
that it takes a long time to reach equilibrium is natural, not
dysfunctional, because it can be very hard to find a wording which
really is neutral.

It really is quite easy to find good examples of how our steady
avoidance of voting on articles has forced people to compromises that
would not have otherwise been reached.

--Jimbo


list: foundation-l

from_name: Andre Engels

subject: [Foundation-l] Request for starting Wikicommons

body:
About one month ago, I asked about WikiCommons, and Eloquence answered that
he would not object against early creation any more. I have let it pass
because of my holidays, but now would like to get started with this, hoping
to have something set up when I start my new job on October 1.

Thus, I would ask for the Wiki to be set up, proposed address:
commons.wikimedia.org, perhaps with a redirect from commons.wikipedia.org.

I would also like to know how far attempts at getting a new upload form have
now come.

Once it has been set up, I would like to invite everyone to come over to
join in early policy discussion and get an upload basis, as well as to join
with Eloquence in his work to improve the Mediawiki software for this
purpose (see [[meta:Wikimedia Commons]]).

People who have already shown interest in the project will be contacted by
me once it is set up, as will be the various (other) Wikimedia projects.

Andre Engels


list: wikimedia-l


body:
I used 7zip in zip mode, which seems to be incompatible to zip mode :-(

New version up, try it now. All good things...

Magnus

Emmanuel Engelhart wrote:

>>Try http://www.magnusmanske.de/wiki2xml.zip
>>
>>
>>
>
>My 'unzip' command can't unzip this file.... strange.
>
>Emmanuel Engelhart
>_______________________________________________
>Wikitech-l mailing list
>Wikitech-l at wikimedia.org
>http://mail.wikipedia.org/mailman/listinfo/wikitech-l
>
>


subject: [Wikitech-l] A donation

body:
Microneil Research Corporation,
http://www.microneil.com/About.html

is donating the following hardware to Wikipedia:

>12 ea 4U rackmount computers, each bundled with a 1/4 rack containing
>an APC rackmount UPS. Typical CPU = Intel P3 w/512M Ram (Kingston),
>ASUS motherboard, 40GB+ EIDE hard drive. These are basic units, but
>capable as web servers & open for expansion.

12 times 4U is rather a LOT of physical space, so we will have trouble
using these "as-is" with our current configuration.  We should
brainstorm as to what to do with these exactly.

One possibility would be to use some of them as webservers, some of
them as offsite backup machines, and some of them as chassis with an
eye towards upgrades; they are weak machines as they are, but
nonetheless potentially useful for some things.

--Jimbo


body:
I have now created a new module in the CVS : "wiki2xml". Check it out ;-)

Magnus

Emmanuel Engelhart wrote:

>>Try http://www.magnusmanske.de/wiki2xml.zip
>>
>>
>>
>
>My 'unzip' command can't unzip this file.... strange.
>
>Emmanuel Engelhart
>_______________________________________________
>Wikitech-l mailing list
>Wikitech-l at wikimedia.org
>http://mail.wikipedia.org/mailman/listinfo/wikitech-l
>
>


from_name: Bryan Ford

subject: [Wikitech-l] Wiki branches, voting

body:
Dear Wikitechies,

Although I'm fairly new to Wikipedia and WikiMedia, I've been interested in
and working on ideas relating to participatory/democratic collaborative
systems for a while.  I happened across Marc Girod's suggestion on the "A
Better Wikipedia" page for adding some form of branch support as an
alternative to content freezing or other similar "hard security" measures.
Since this was right in line with some of my own thinking recently, I decided
to invest a little time to try to work out the idea in a little more detail
and explore some of the design issues and challenges.  So I've written up one
possible, preliminary design and put it on my user page, here:

	http://meta.wikimedia.org/wiki/User:Brynosaurus

The most important goals this design tries hard to embody are (a) providing a
clean, robustly, and democratic solution to most of the existing edit war and
vandalism issues, and (b) being as simple (from the user's perspective) and
minimally intrusive as possible to the standard wiki workflow, especially for
the vast majority of wiki pages that aren't highly controversial.

This is somewhat ambitious and long-term stuff, not something I'm expecting
anyone to jump up and implement immediately or slip into a minor release.
I'm interested in working on it myself if there is support for the idea, but
for starters I'd just like to gauge interest and get some initial feedback.

Thanks very much!
Bryan


body:
I was very happy, because you coded this first parser very fast. And, I support
the idea of making a real wiki parser. But, it seems that you didn't use
flex/bison or any parser generator. I think continuing in this direction, would
be a very bad idea. The best specialists use parser generators, and we are not
so good like parsing specialists...

Emmanuel

Selon Magnus Manske <magnus.manske at web.de>:

> I have now created a new module in the CVS : "wiki2xml". Check it out ;-)
>
> Magnus
>
>
> Emmanuel Engelhart wrote:
>
> >>Try http://www.magnusmanske.de/wiki2xml.zip
> >>
> >>
> >>
> >
> >My 'unzip' command can't unzip this file.... strange.
> >
> >Emmanuel Engelhart
> >_______________________________________________
> >Wikitech-l mailing list
> >Wikitech-l at wikimedia.org
> >http://mail.wikipedia.org/mailman/listinfo/wikitech-l
> >
> >
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at wikimedia.org
> http://mail.wikipedia.org/mailman/listinfo/wikitech-l
>


subject: [Wikitech-l] In CVS

body:
Emmanuel Engelhart wrote:

>I was very happy, because you coded this first parser very fast. And, I support
>the idea of making a real wiki parser. But, it seems that you didn't use
>flex/bison or any parser generator. I think continuing in this direction, would
>be a very bad idea. The best specialists use parser generators, and we are not
>so good like parsing specialists...
>
>
I agree. The reasons I didn't use a parser generator are
* I was already working on this when I wrote my initial mail to the
list, and I didn't think of a parser generator then
* I don't know s**t about bison or flex

I *did* continue to develop this, because I believe we can use it as a
black box. It already works OK (except for a headings bug, which I'm
about to fix now), and someone (hint!;-) could start working on
integrating it into a new PHP structure that will trake the XML and
output (X)HTML. If we later replace my manual version with an
automatically generated one, it should not matter at all to the rest of
the parser.

Magnus


from_name: xmlizer

subject: [Wikitech-l] Re: In CVS

body:
Magnus Manske a ?crit :

>
>
> Emmanuel Engelhart wrote:
>
>> I was very happy, because you coded this first parser very fast. And,
>> I support
>> the idea of making a real wiki parser. But, it seems that you didn't use
>> flex/bison or any parser generator. I think continuing in this
>> direction, would
>> be a very bad idea. The best specialists use parser generators, and we
>> are not
>> so good like parsing specialists...
>>
>>
> I agree. The reasons I didn't use a parser generator are
> * I was already working on this when I wrote my initial mail to the
> list, and I didn't think of a parser generator then
> * I don't know s**t about bison or flex
>
> I *did* continue to develop this, because I believe we can use it as a
> black box. It already works OK (except for a headings bug, which I'm
> about to fix now), and someone (hint!;-) could start working on
> integrating it into a new PHP structure that will trake the XML and
> output (X)HTML. If we later replace my manual version with an
> automatically generated one, it should not matter at all to the rest of
> the parser.
>
> Magnus

I think it is the time to start [[Wikimedia:WikiSyntax]] to contain the
EBNF  wikisyntax to use in Flex/Bison, Lex/Yacc, JavaCC, etc....

See http://bugzilla.wikipedia.org/show_bug.cgi?id=7

I've start a very very little stub on
http://fr.wikipedia.org/wiki/Wikip%C3%A9dia:WikiSyntaxe

PS : Of course i totally agree on the model

         Wikisyntax      (MUST STAY A NON XML SYNTAX)
             |
        | Parsing |
             |
     | XMLTreeDumping |
             |
            XML         (Cached)
             |
     | Postprocessing | (for templates, magics, and links, and options)
             |
| XSLT-like-Tansformation |
             |
      -------|-------------- - - - - - - - -
      |      |             |                |
  (X)HTML   PDF     XML/MATHML/SVG        ?????

Xmlizer


from_name: Eric Pöhlsen

body:
>
> At least, i didn't get the source.
>
> Eckhart
>

I guess the list does not allow attachments

Eric


subject: [Wikitech-l] Re: disambiguation enhancement

body:
Nikola Smolenski a ?crit :

> On Sunday 15 August 2004 08:31, Brion Vibber wrote:
>
>>Nikola Smolenski wrote:
>>
>>>How costly it would be to change type of cur_is_redirect to int?
>>
>>It's presently a tinyint, with valid values from -128 to 127. Altering
>>its meaning would require being careful not to break existing
>>assumptions, however, and neither 'disambiguation' nor 'stub' are imho
>>useful or objectively calculable.
>
>
> I'm also not sure how useful it is, but it is objectively calculable :)
>
> $kind=substr($article,"{{disambig}})<<2|substr($article,"{{stub}}")<<1|substr($article,"#REDIRECT");
> // (more or less...)
What about other langages
fr : {{disambig}} --> {{Homonymie}}
fr : {{stub}} --> {{?bauche}}

What about defining <tt>#DISAMBIGUATION</tt> and <tt>#STUB</tt> and to
insert them respectively into Template:Disambig or Mod?le:Homonymie, and
Template:Stub or Mod?le:?bauche

???


from_name: Ashar Voultoiz

references:
<cflt0k$e7u$1@sea.gmane.org>
	<cfm2uj$b8f$1@sea.gmane.org>	<200408150805.30306.smolensk@eunet.yu>
	<cfn1ve$nuu$1@sea.gmane.org>

body:
Timwi wrote:
<snip>
> For completely general stuff, one could use the 'links' table to see
> what templates a page includes.
>
> Then you could have features like "Average length of pages that include
> a given template" (and on Wikipedia, we would type 'dab' and 'stub' into
> the field, etc.).
>
> So far you haven't made your actual suggestion, or told us what your
> original idea was. An idea should ideally not start out as an
> implementation plan. :) You should first have an idea of what should
> change in the user interface. If you describe that to us, maybe someone
> can come up with an implementation plan that doesn't require huge
> changes to the DB.
>
> Timwi

Hello Timiw,

I will code my draft using the link table. The disambiguation template
name can then be set in the Language.php file.

--
Ashar Voultoiz


from_name: Jan Hidders

subject: [Wikitech-l] In CVS

body:
Magnus Manske wrote:

> I *did* continue to develop this, because I believe we can use it as a
> black box. It already works OK (except for a headings bug, which I'm
> about to fix now), and someone (hint!;-) could start working on
> integrating it into a new PHP structure that will trake the XML and
> output (X)HTML. If we later replace my manual version with an
> automatically generated one, it should not matter at all to the rest
> of the parser.

I think that for that to be true there would first have to be consensus
about the interface of the black box, which means that it should be clear
(1) what the formal grammar of the mark-up is and
(2) what the corresponding XML is.
I think (2) could be straightforward (simply the syntax tree in XML
form, with e.g. non-terminals as tag names) but for (1) it is not just a
matter of creating a grammar that accepts all current contents, but you
need to decide how far you want to go with the functional decomposition,
i.e., how detailed you want the syntax tree to be. One extreme would be
to let all parsing be done by the parser, and no more matching should be
necessary in the translation afterwards. There are also some other vey
deep and fundamental decisions that have to be made there. For example,
do we accept all possible input or do we only accept well-formed markup?
In the first case the grammar becomes very complex and the
transformation very hard because it has to correct the non-wellformed
input. In the second case the grammar and the transformation become much
simpeler but you would give the user a harsh SYNTAX ERROR if the input
is not well-formed and possibly you would discover that most of the
current pages are in fact not well-formed.

It is hard to have a discussion over (1) if you are not already using
bison/yacc because ideally your grammar should be LALR(1) or nearly so
with only a few ambiguities and it is nearly impossibl to analyse that
without these tools.

-- Jan Hidders


from_name: Ivan Krstic

subject: [Wikitech-l] External diff engine

body:
Brion Vibber wrote:
> Plain page view:    2798ms
> Classic diff:       4856ms (2798ms + 2058ms)
> Diff with wikidiff: 3342ms (2798ms +  544ms)

That's a very long time. Is there any chance you can post some more
fine-grained timings (perhaps profiling output) for these pages? What of
the three seconds is network latency, what's overhead, and what's parsing?

If this page renders in 1501ms with memcached and the parser cache, then
the machine is either choking under too much load, or something's
horribly wrong. Getting something out of RAM should most certainly not
take a second and a half, and if it does, most of the current
discussions about a new parser should take the form of "we need a new
parser because the current one is ugly", as opposed to "because the
current one is slow".

This is not to say rewriting the parser the Right Way will not enhance
its speed somewhat, but if 1501ms is your minimum baseline, it doesn't
seem like there's too much room for improvement.

Ivan


subject: [Wikitech-l] In CVS

body:
Jan Hidders wrote:

> Magnus Manske wrote:
>
>> I *did* continue to develop this, because I believe we can use it as
>> a black box. It already works OK (except for a headings bug, which
>> I'm about to fix now), and someone (hint!;-) could start working on
>> integrating it into a new PHP structure that will trake the XML and
>> output (X)HTML. If we later replace my manual version with an
>> automatically generated one, it should not matter at all to the rest
>> of the parser.
>
>
> I think that for that to be true there would first have to be
> consensus about the interface of the black box, which means that it
> should be clear
> (1) what the formal grammar of the mark-up is and
> (2) what the corresponding XML is.
> I think (2) could be straightforward (simply the syntax tree in XML
> form, with e.g. non-terminals as tag names) but for (1) it is not just
> a matter of creating a grammar that accepts all current contents, but
> you need to decide how far you want to go with the functional
> decomposition, i.e., how detailed you want the syntax tree to be. One
> extreme would be to let all parsing be done by the parser, and no more
> matching should be necessary in the translation afterwards. There are
> also some other vey deep and fundamental decisions that have to be
> made there. For example,  do we accept all possible input or do we
> only accept well-formed markup? In the first case the grammar becomes
> very complex and the transformation very hard because it has to
> correct the non-wellformed input. In the second case the grammar and
> the transformation become much simpeler but you would give the user a
> harsh SYNTAX ERROR if the input is not well-formed and possibly you
> would discover that most of the current pages are in fact not
> well-formed.
>
> It is hard to have a discussion over (1) if you are not already using
> bison/yacc because ideally your grammar should be LALR(1) or nearly so
> with only a few ambiguities and it is nearly impossibl to analyse that
> without these tools.
>
I think one point is out of the question : The parser should accept even
malformed source, just as the current PHP-based parser does now (or
ought to do;-).

For the rest, I take the more practical approach here; (1) is what is
allowed today, for the time being. I don't need a formal grammar for the
manual parser, I hack away until it does what it should do. Once we
replace my black box with a lexx/bison black box, we have to worry about
the grammar.

In the meantime, this blackbox will do fine, meaning, it will suffice to
write everything after the conversion to XML. If the black box doesn't
work with all articles during this initial phase - so what? We'll just
*assume* that it works correctly, leave problematic articles out during
testing, and bulid a XML->XHTML component that works. By the time this
is done, we should have that grammar finalized and can replace the black
box with a bison/whatever version.

I just don't want to lose the momentum by establishing the grammar
before doing anything else. There is plenty we can do already, based on
what we have.

Magnus


from_name: Tomasz Wegrzanowski

subject: [Wikitech-l] In CVS

body:
On Mon, Aug 16, 2004 at 04:44:26PM +0200, Magnus Manske wrote:
> >It is hard to have a discussion over (1) if you are not already using
> >bison/yacc because ideally your grammar should be LALR(1) or nearly so
> >with only a few ambiguities and it is nearly impossibl to analyse that
> >without these tools.
> >
> I think one point is out of the question : The parser should accept even
> malformed source, just as the current PHP-based parser does now (or
> ought to do;-).
>
> For the rest, I take the more practical approach here; (1) is what is
> allowed today, for the time being. I don't need a formal grammar for the
> manual parser, I hack away until it does what it should do. Once we
> replace my black box with a lexx/bison black box, we have to worry about
> the grammar.
>
> In the meantime, this blackbox will do fine, meaning, it will suffice to
> write everything after the conversion to XML. If the black box doesn't
> work with all articles during this initial phase - so what? We'll just
> *assume* that it works correctly, leave problematic articles out during
> testing, and bulid a XML->XHTML component that works. By the time this
> is done, we should have that grammar finalized and can replace the black
> box with a bison/whatever version.
>
> I just don't want to lose the momentum by establishing the grammar
> before doing anything else. There is plenty we can do already, based on
> what we have.

Code first, think later - that's the only way which works.
Too many promising projects died because of wasting too much time
on discussion before there was any code.


subject: [Wikitech-l] replaceLinkHolders

body:
Magnus Manske wrote:
> I have checked in the dirtiest hack of the week to replace that
> str_replace :-)
>
> Total rendering time is down 10% on my machine.

Neat! But now that I look at it, it can be sped up a lot just by using a
regular expression:

Original str_replace method:
  1    4260.353   4260.353     44.304%

Magnus' scary hack:
  1    423.490    423.490      7.385%

Using preg_replace_callback:
  1     21.842     21.842      0.354%

(Times are on my PowerBook so not directly comparable to the previously
posted tests from my Linux box.)

I think the str_replace does something stupid like comparing _every one_
of the hundreds of link keys individually to each character position in
the string. :P  Using a regexp cuts down the comparisons and doesn't
change behavior.

(The current link replacement code does Weird Things on invalid links
such as things that look like nested links. This needs to be fixed
farther up the chain to avoid creating such monstrosities.)

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 253 bytes
Desc: OpenPGP digital signature
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20040816/00f32820/attachment.pgp>


subject: [Wikitech-l] Re: Two markets (was Re: An alternate parser)

body:
Nikola Smolenski wrote:
> On Monday 16 August 2004 05:57, Tim Starling wrote:
>>What's wrong with forking? We have two different markets for our
>>software, and these markets have conflicting needs. Why not produce a
>>customised product for each one?
>
>
> In this case, interoperability. If parsers diverge, it might happen that other
> MediaWiki wikis couldn't include Wikipedia's contents, and vice versa.

That's correct, any wiki wishing to use Wikipedia's contents without any
conversion or risk of problems would have to be running the heavy-duty
parser branch. But most MediaWiki wikis do not include Wikipedia's
contents, and do not require interoperability.

Brion Vibber wrote:
 > It's additional work to maintain two versions of the code.

Krzysztof Kowalczyk wrote:
 > Possible dilution of dev resources, for one. If there are N developers
 > working on unified code-based, in the worst-case scenario after the
 > split each project might end up with, say, N/2 developers, testers,
 > users etc.

I'm not talking about completely splitting the project, only a small
proportion of the work would be duplicated. And look at the benefits! It
would require extra work, but it would produce a better package, for
both Wikimedia and non-Wikimedia users.

Since when has putting in work for tangible gain been a bad thing?

-- Tim Starling


list: wikipedia-l

from_name: Peter Shaw

subject: [Wikipedia-l] Re: Text-to-speech

body:
On Saturday 14 August 2004 00:19, Timwi wrote:
> Well, personally, I find it rather disappointing, and I have heard
> better ones even several years ago.  However, it would certainly be
> pretty cool to have Wikipedia articles as sound files.

Where?


subject: [Wikipedia-l] Text-to-speech

body:
On Friday 13 August 2004 08:24, Magnus Manske wrote:
> I just found an AT&T demo site for text-to-speech synthesis:
>
> http://www.research.att.com/projects/tts/demo.html
>
> which appears to generate *way* better speech than I got from other
> (local installation) demos.

The reason why it's so good, is that they don't generate the speech from
scratch but instead concatenate recorded speech pieces. This also means that
the program is rather huge, and that listening to text at a speed different
from recording will not sound very good. See "Concatenative synthesis" on
http://en.wikipedia.org/wiki/Speech_synthesis for more details.

Peter Shaw


subject: [Wikitech-l] In CVS

body:
Magnus Manske wrote:

> Jan Hidders wrote:
>
>> I think that for that to be true there would first have to be
>> consensus about the interface of the black box, which means that it
>> should be clear
>> (1) what the formal grammar of the mark-up is and
>> (2) what the corresponding XML is.
>> [...]  do we accept all possible input or do we only accept
>> well-formed markup?
>
> I think one point is out of the question : The parser should accept
> even malformed source, just as the current PHP-based parser does now
> (or ought to do;-).

I'm happy either way as long as the decision is taken in full awareness
of the increase in complexity of the code.

> For the rest, I take the more practical approach here; (1) is what is
> allowed today, for the time being. I don't need a formal grammar for
> the manual parser, I hack away until it does what it should do.

But until you know the grammar you do not know what "it should do".
Until you know the grammar you do not know what XML to produce, what the
DTD is and what it means. That you can parse most of the current
articles and produce well-formed XML is virtually meaningless unless you
can tell the programmer of the code that processes the XML what the DTD
is and what it means.

> Once we replace my black box with a lexx/bison black box, we have to
> worry about the grammar.

It's not that simple. You cannot generate a parser for *any* formal
grammer, so there is probably going to be some tweaking of the grammar
(apart from other changes such as moving some of the parsing from or to
the transformation phase) which will mean that probably the output XML
will also change. Note that by "hacking away" as you are doing now you
are in fact already implicitly choosing a grammar by the way that you
produce your XML. Unless you make those choices explicit it's hard to
tell if they are the right ones.

> I just don't want to lose the momentum by establishing the grammar
> before doing anything else.

We don't have to. Note that we do not need *the* grammar to start. It
would probably be better anyway to take a gradual refinement approach by
which I mean here the following:

1. You start with a very simple grammar that basically chops the source
into big semantic chuncks. So you get a very small parse tree with leafs
that contain large pieces of unparsed text. These leaves are transformed
by the old code and then combined into to the final output.

2. You test if for the most pages you generate the same output as
before. If you are not happy with this you redefine the grammar and
corresponding transformation until you are.

3. Until the leaves no longer contain unparsed text you refine the
syntax such that the parse tree becomes bigger and the leaves smaller,
then go back to step 2.

4. Remove all old transformation code.

If you tell me what grammar you would like to start with I (and others)
can probably help you with the bison code.

-- Jan Hidders


subject: [Wikitech-l] Wiki branches, voting

body:
On Mon, 16 Aug 2004 08:42:44 -0700 "Jimmy (Jimbo) Wales" <jwales at wikia.com>
wrote:

>Let's imagine that 90% of the people editing a particular page are of
>one opinion, and 10% are of a different opinion.  So long as the
>participants are all reasonable and friendly, this poses no obstacle
>to the production of an article that is satisfactory to all -- this is
>the normal wiki way of producing quality encyclopedic content.
>
>The majority has no choice but to respect the viewpoint of the
>minority, because the minority can always just _edit the article_.

That sounds very idealistic, but is it realistic? Do you see this happen on
any pages? Hardly. For one thing, there's usually not 10 people working on a
page, but 2 or 3, who either agree or do not. And secondly, either people
want to compromise, or they do not. If they do, they'll vote for compromise
solutions too. If they don't, we just get the normal Wikipedia way of giving
the page to the person with either most audacity or the longest breath.

I really don't see why 9 people voting for A and 1 for B, then choosing A is
a worse way of getting compromise than 9 people reverting B to A and 1
reverting A to B, and then choose B when that 1 person happens to be the
most determined and A otherwise.

>If branching and voting are introducted, we've destroyed the incentive
>structure that makes wiki work.  The 90% now has no reason to respect
>the viewpoint of the minority, because the vote is so heavily
>one-sided that the minority can be safely ignored.

And what is the incentive now? I'd rather go along with the majority than
with the person who happens to be loudest about it, or who happens to make
the first page on the subject, or who has the strongest opinion.

Andre Engels


from_name: Sj

subject: [Wikitech-l] A donation

body:
On Mon, 16 Aug 2004 14:22:34 -0700, Jimmy (Jimbo) Wales
<jwales at wikia.com> wrote:

> Microneil Research Corporation,
> http://www.microneil.com/About.html
> is donating the following hardware to Wikipedia:
>
> >12 ea 4U rackmount computers, each bundled with a 1/4 rack containing
> >an APC rackmount UPS. Typical CPU = Intel P3 w/512M Ram (Kingston),
> >ASUS motherboard, 40GB+ EIDE hard drive. These are basic units, but
> >capable as web servers & open for expansion.
>
> 12 times 4U is rather a LOT of physical space, so we will have trouble
> using these "as-is" with our current configuration.  We should
> brainstorm as to what to do with these exactly.

That's great news!  We could make a little beowulf cluster out of them.
I would also like to see a dedicated stats machine, which regularly
grabs a dump of
the various databases and runs statistics and many other static
queries (such as
allpages, longpages, "new users", etc.) for each project.

+sj+


body:
On Mon, 16 Aug 2004 22:10:48 +0000, David Gerard
<fun at thingy.apana.org.au> wrote:
> On 08/16/04 12:13, Andre Engels wrote:
>
> > Thus, I would ask for the Wiki to be set up, proposed address:
> > commons.wikimedia.org, perhaps with a redirect from commons.wikipedia.org.
>
> Yes please. Seconded. Eagerly awaited.
>
> (I do like the idea of somewhere I can put all the pics of buildings,
> monuments and so on that I took a pic of because I thought they'd be a
> nice idea for Wikipedia if I ever write the article ...)

*AOL*   This would be excellent.

Upload form : The text of the new upload form is available in 8 languages now,
with partial translations into 5 more, although all need to be updated
to conform
to the latest changes to the German original.
   http://meta.wikimedia.org/wiki/Translation/New_upload_form

I don't know who is working on implementing the code changes involved.

 ~sj~


subject: [Wikitech-l] Distributing wikimedia hardware

body:
On Mon, 16 Aug 2004 09:03:24 -0700, Jimmy (Jimbo) Wales
<jwales at wikia.com> wrote:
>
> 3.  We are widely beloved ... it seems likely that if we pay
> the cost of a machine, they would be happy to host on a permanent
> basis a slave db machine which we could keep synced to have a constant
> realtime backup.

At some point soon, we will be backing up many terabytes of archives
and historical data.  Would it make sense to have an offsite location
dedicated to preserving these backups and regular snapshots of the db,
in case there is ever data corruption which goes unnoticed and is replicated
to all active slaves/mirrors?

That location could also maintain a synced mirror of the current projects
and key filesystem content, along with a few machines available for
configuration as temporary web and db servers, in case something terrible
happened at the colo facility.

+sj+


from_name: Magnus Manske

subject: [Wikitech-l] In CVS

body:
Jan Hidders wrote:

> Magnus Manske wrote:
>
>> For the rest, I take the more practical approach here; (1) is what is
>> allowed today, for the time being. I don't need a formal grammar for
>> the manual parser, I hack away until it does what it should do.
>
>
> But until you know the grammar you do not know what "it should do".
> Until you know the grammar you do not know what XML to produce, what
> the DTD is and what it means. That you can parse most of the current
> articles and produce well-formed XML is virtually meaningless unless
> you can tell the programmer of the code that processes the XML what
> the DTD is and what it means.

Well, the tags we can use are HTML (limited set, most of them nested), and :
http://en.wikipedia.org/wiki/Wikipedia:How_to_edit_a_page

So, we can create internal and external links. Internal links can have
parameters and sub-links, as can templates. IMHO, that pretty much
defines what the XML should look like, or should be able to describe.
Give my parser thingy a try, have a look at the output, and tell me what
the XML it outputs cannot describe that out MediaWiki parser can
(current bugs excluded:-).

If I know the input format (wiki markup) and the desired output (XML), I
can create a black box that can do the translation.

Remember, we're not talking about the grammar of C# here. This is wiki
markup. Links, bold, italics, lists, and tables, that's about it . I
might not be able to solve a traveling salesman problem with 10000
cities in my head, but with four cities, it *is* rather trivial.

In the wiki2xml ware, I don't even bother with a parsing tree, just a
little recursive call for nested links (images) and some lists for the
table markup. Yeah! :-)

>
>> Once we replace my black box with a lexx/bison black box, we have to
>> worry about the grammar.
>
>
> It's not that simple. You cannot generate a parser for *any* formal
> grammer, so there is probably going to be some tweaking of the grammar
> (apart from other changes such as moving some of the parsing from or
> to the transformation phase) which will mean that probably the output
> XML will also change. Note that by "hacking away" as you are doing now
> you are in fact already implicitly choosing a grammar by the way that
> you produce your XML. Unless you make those choices explicit it's hard
> to tell if they are the right ones.

Good judement comes from experience. Experience comes from bad
judgement. ;-)

>
>> I just don't want to lose the momentum by establishing the grammar
>> before doing anything else.
>
>
> We don't have to. Note that we do not need *the* grammar to start. It
> would probably be better anyway to take a gradual refinement approach
> by which I mean here the following:
>
> 1. You start with a very simple grammar that basically chops the
> source into big semantic chuncks. So you get a very small parse tree
> with leafs that contain large pieces of unparsed text. These leaves
> are transformed by the old code and then combined into to the final
> output.
>
> 2. You test if for the most pages you generate the same output as
> before. If you are not happy with this you redefine the grammar and
> corresponding transformation until you are.
>
> 3. Until the leaves no longer contain unparsed text you refine the
> syntax such that the parse tree becomes bigger and the leaves smaller,
> then go back to step 2.
>
> 4. Remove all old transformation code.
>
> If you tell me what grammar you would like to start with I (and
> others) can probably help you with the bison code.

So, what you'd need is an EBNF representation or something? I've never
worked with bison or the like. Is there a Windows version, by the way
(yes, I *have* Linux, but currently I'm mostly under W).

Magnus


from_name: Delirium

subject: [Foundation-l] Re: Copyright issues...walking on thin ice

body:
Ray Saintonge wrote:

> We have too little detail on the printing and publication plans for
> the print edition.  The danger with an edition that infringes
> copyright revolves around injunctive measures that would prevent its
> distribution.  By limiting press runs to what can be distributed in
> one month that would limit those damages.  In going from version 1.0
> to version 1.1 it would be easier to adjust  for any copyright
> complaint that might have been received.

That could be a problem, as frequent small print runs are much more
expensive than infrequent large print runs.

-Mark


subject: [Wikitech-l] External diff engine

body:
Brion Vibber wrote:

> Ivan Krstic wrote:
>
>> Brion Vibber wrote:
>>
>>> Plain page view:    2798ms
>>> Classic diff:       4856ms (2798ms + 2058ms)
>>> Diff with wikidiff: 3342ms (2798ms +  544ms)
>>
>>
>> That's a very long time. Is there any chance you can post some more
>> fine-grained timings (perhaps profiling output) for these pages? What
>> of the three seconds is network latency, what's overhead, and what's
>> parsing?
>
>
> Ah, you noticed it's kind of slow? :)

<snip>depressing time statistic</snip>

If there ever was need for a C++ parser... ;-)

Magnus


body:
Jimmy (Jimbo) Wales wrote:

>With images, the separate authorship and the independence of the work
>is transparently obvious.  The text is not derived from the image, nor
>is the image derived from the text.  No one modifies GNU FDL text and
>turns it into an image, or vice-versa.  (Barring hypothetical edge
>cases that are beyond the scope of this discussion.)
>
>
I don't think the edge cases are that infrequent, but IANAL of course.
For example, if I were to have a collection of GFDL'd pictures of
flowers, and you took that collection of flowers and used them to
publish an illustrated book of flowers, I would consider your book,
including its textual content, a derived work of my collection of images.

-Mark


from_name: Jens Frank

subject: [Wikitech-l] External diff engine

body:
On Mon, Aug 16, 2004 at 08:29:39PM +0200, Magnus Manske wrote:
> Brion Vibber wrote:
>
> >Ivan Krstic wrote:
> >
> >>Brion Vibber wrote:
> >>
> >>>Plain page view:    2798ms
> >>>Classic diff:       4856ms (2798ms + 2058ms)
> >>>Diff with wikidiff: 3342ms (2798ms +  544ms)
> >>
> >>
> >>That's a very long time. Is there any chance you can post some more
> >>fine-grained timings (perhaps profiling output) for these pages? What
> >>of the three seconds is network latency, what's overhead, and what's
> >>parsing?
> >
> >
> >Ah, you noticed it's kind of slow? :)
>
> <snip>depressing time statistic</snip>
>
> If there ever was need for a C++ parser... ;-)

Well, OutputPage::replaceLinkHolders is not part of the parser.
It's the replacing of wikilinks by HTML links, which, if I
understand your concept correctly, would be done during the
XML->XHTML translation, in PHP.

Regards,

JeLuF


from_name: Ray Saintonge

body:
Delirium wrote:

> Ray Saintonge wrote:
>
>> We have too little detail on the printing and publication plans for
>> the print edition.  The danger with an edition that infringes
>> copyright revolves around injunctive measures that would prevent its
>> distribution.  By limiting press runs to what can be distributed in
>> one month that would limit those damages.  In going from version 1.0
>> to version 1.1 it would be easier to adjust  for any copyright
>> complaint that might have been received.
>
> That could be a problem, as frequent small print runs are much more
> expensive than infrequent large print runs.

Agreed.  But we would need more concrete figures about the price breaks
to permit an optimization of the situation.

Ec


from_name: Philip Newton

subject: [Wikipedia-l] Re: Request for the second Serbian Wikipedia

references:
<200408081744.i78Hirxf020318@mx1.vdial.verat.net>	<cf5p9f$mss$1@sea.gmane.org>	<20040808181648.GC27170@tomk32.homelinux.org>	<20040808181648.GC27170-KXMOcH7pGWW/RHH9vPZxQGD2FQJk+8+b@public.gmane.org>	<4116A914.4000404@telus.net>
	<8ndvh0p0o5r9mqukac5nqnfm7he80jfg1k@4ax.com>
	<8ndvh0p0o5r9mqukac5nqnfm7he80jfg1k-e09XROE/p8c@public.gmane.org>
	<411FB8E2.9030706@pobox.com>

body:
On Sun, 15 Aug 2004 12:26:26 -0700, Brion Vibber wrote:

> Philip Newton wrote:
> > On Sun, 08 Aug 2004 15:28:36 -0700, Ray Saintonge wrote:
> >>I suppose this all depends on there being a reversible one-to-one
> >>relationship between the two versions of Serbian.
> >
> > This is possible, to the best of my knowledge.
> >
> > Some sounds which have one letter in Cyrillic have two letters in Latin,
> > but AFAIK those two-letter sequences always map to that one Cyrillic
> > letter, to the translation is unambiguous in both directions.
>
> Except, of course, when you have foreign words and names to deal with.

I suppose so, though I believe both Serbian and Croatian usually
"transliterate" foreign names, so "George Bush" becomes "Dz^ordz^ Bus^",
whether in Latin or in Cyrillic Script.

Cheers,
Philip


from_name: Jama Poulsen

subject: [Wikipedia-l] Text-to-speech

body:
On Mon, Aug 16, 2004 at 05:59:23PM +0000, Peter Shaw wrote:
> The reason why it's so good, is that they don't generate the speech from
> scratch but instead concatenate recorded speech pieces. This also means that
> the program is rather huge, and that listening to text at a speed different
> from recording will not sound very good. See "Concatenative synthesis" on
> http://en.wikipedia.org/wiki/Speech_synthesis for more details.

Interesting, I did not realize they used such a technique and was
wondering why the pronunciations sounded more natural than eg.
the Festival engine.

What I would like to see is some sort of streaming text-to-speech
server system which can use the Speex codec (http://www.speex.org)
as output. Shifting through the text could be done based on approx.
calculations with the length of the text and the configured voice
speed. Client side interaction (voice configuration, streaming
quality, text selections, pointer indication feedback, bookmarks,
history, etc.) would ideally be taken care via some sort of unified
API, eg. using Speech Dispatcher (http://www.freebsoft.org/speechd).

More speech synthesis links: http://debianlinux.net/multimedia.html#speech

Jama Poulsen


list: wikien-l

subject: [WikiEN-l] When can an article be considered public domain?

body:
jcecropia at mail.com wrote:

>In terms of what? I am not representing myself as a lawyer and thereby stating that what I am saying may be knowledgeable, but it's not legal advice, so take it FWIW. Anyone who went to court and said, "I did this because a guy on the internet told me so" is not likely to do too well anyway.
>
At least it would have the court to ROTFL.

Ec


body:
On Mon, Aug 16, 2004 at 12:33:27PM -0700, Brion Vibber wrote:
> Brion Vibber wrote:
> >Clearly there's still room for improvement in replaceLinkHolders.
>
> I've added some smaller profiling cunks in that functions:
>
> [snip]

Do you also have figures for release 1.3.1 rendering the same page?

Regards,

JeLuF


subject: [Wikitech-l] External diff engine

body:
Jens Frank wrote:

>On Mon, Aug 16, 2004 at 08:29:39PM +0200, Magnus Manske wrote:
>
>
>>Brion Vibber wrote:
>>
>>
>>
>>>Ivan Krstic wrote:
>>>
>>>
>>>
>>>>Brion Vibber wrote:
>>>>
>>>>
>>>>
>>>>>Plain page view:    2798ms
>>>>>Classic diff:       4856ms (2798ms + 2058ms)
>>>>>Diff with wikidiff: 3342ms (2798ms +  544ms)
>>>>>
>>>>>
>>>>That's a very long time. Is there any chance you can post some more
>>>>fine-grained timings (perhaps profiling output) for these pages? What
>>>>of the three seconds is network latency, what's overhead, and what's
>>>>parsing?
>>>>
>>>>
>>>Ah, you noticed it's kind of slow? :)
>>>
>>>
>><snip>depressing time statistic</snip>
>>
>>If there ever was need for a C++ parser... ;-)
>>
>>
>
>Well, OutputPage::replaceLinkHolders is not part of the parser.
>It's the replacing of wikilinks by HTML links, which, if I
>understand your concept correctly, would be done during the
>XML->XHTML translation, in PHP.
>
>
Yup, that function's new (didn't see it a few days ago). Seems the most
time is spent in database queries. Not much to do about that except
caching, which we do. We could collect the links to use one big query
instead of several ones, maybe?

Magnus


from_name: Anthony DiPierro

subject: [WikiEN-l] Re: [Foundation-l] Request for starting Wikicommons

body:
 > (I do like the idea of somewhere I can put all the pics of buildings,
> monuments and so on that I took a pic of because I thought they'd be a
> nice idea for Wikipedia if I ever write the article ...)

Why not just use http://en.wikipedia.org/wiki/Wikipedia:List_of_images ?


subject: [Wikitech-l] External diff engine

body:
Brion Vibber wrote:

> Magnus Manske wrote:
>
>> Yup, that function's new (didn't see it a few days ago). Seems the
>> most time is spent in database queries. Not much to do about that
>> except caching, which we do. We could collect the links to use one
>> big query instead of several ones, maybe?
>
>
> It already collects it into one big query, and most of the time in the
> large-page-with-many-links case is spent in a single str_replace() call.

Didn't see your previous mail, or looked at the function very carefully.

I still didn't, so is every occurrence in $string unique? If so, why not use
$this->mBodytext = str_replace( $search, $replace, $this->mBodytext , 1);
(note the ",1"; maybe it should be "count($search)", I'm not sure)

Thus, if one replacement is shown, it won't have to search the rest of
the string.

Magnus


from_name: David Gerard

body:
On 08/16/04 12:13, Andre Engels wrote:

> Thus, I would ask for the Wiki to be set up, proposed address:
> commons.wikimedia.org, perhaps with a redirect from commons.wikipedia.org.
> Once it has been set up, I would like to invite everyone to come over to
> join in early policy discussion and get an upload basis, as well as to
> join with Eloquence in his work to improve the Mediawiki software for
> this purpose (see [[meta:Wikimedia Commons]]).
> People who have already shown interest in the project will be contacted
> by me once it is set up, as will be the various (other) Wikimedia projects.

Yes please. Seconded. Eagerly awaited.

(I do like the idea of somewhere I can put all the pics of buildings,
monuments and so on that I took a pic of because I thought they'd be a
nice idea for Wikipedia if I ever write the article ...)

- d.


subject: [Wikitech-l] External diff engine

body:
Please ignore last mail, mis-read the manual ;-(

Magnus

Magnus Manske wrote:

> Brion Vibber wrote:
>
>> Magnus Manske wrote:
>>
>>> Yup, that function's new (didn't see it a few days ago). Seems the
>>> most time is spent in database queries. Not much to do about that
>>> except caching, which we do. We could collect the links to use one
>>> big query instead of several ones, maybe?
>>
>>
>>
>> It already collects it into one big query, and most of the time in
>> the large-page-with-many-links case is spent in a single
>> str_replace() call.
>
>
> Didn't see your previous mail, or looked at the function very carefully.
>
> I still didn't, so is every occurrence in $string unique? If so, why
> not use
> $this->mBodytext = str_replace( $search, $replace, $this->mBodytext , 1);
> (note the ",1"; maybe it should be "count($search)", I'm not sure)
>
> Thus, if one replacement is shown, it won't have to search the rest of
> the string.
>
> Magnus
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at wikimedia.org
> http://mail.wikipedia.org/mailman/listinfo/wikitech-l


from_name: Brion Vibber

subject: [Wikitech-l] replaceLinkHolders

body:
Jens Frank wrote:
> Congratulations! That sounds really fast! Now it's really faster than
> the 1.3.1 renderer?

Unfortunately no; it takes out a big chunk but it's still slower. This
brings the full Village Pump page load time on my Linux box from 1501ms
down to 901 ms, compared to 340 ms on 1.3.1.

(Both with memcached and parser cache active; times are median total
per-hit load time measured with 'ab -n 10' on the local host.)

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 253 bytes
Desc: OpenPGP digital signature
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20040816/4c67d4cf/attachment.pgp>


subject: [Wikitech-l] In CVS

body:
Jan Hidders wrote:

>On Monday 16 August 2004 19:35, Magnus Manske wrote:
>
>
>>So, what you'd need is an EBNF representation or something?
>>
>>
>
>Yes, that is all I ask. A precise formal grammar in whatever notation you like
>but preferrably in the input format of bison. If only because that would
>document what it is that your parser exactly does. Note there are some
>requirements:
>- it should accept all possible input strings
>- it should ideally do all the necessary parsing i.e. from the parse tree we
>should be able to generate the output with a single tree-walk, and
>- it should be unambiguous (and even LALR(1)) or have an explicit conflict
>resolution rule
>
>
I can't prove it at the moment, but after having played around with
wikisyntax for a while, I'm *pretty* sure it's not LALR(1).  There's too
much reuse of symbols that requires potentially unbounded lookahead to
resolve ambiguities.  Even if it were, the LALR(1) grammar would be so
unintuitive and unlike the functional intent of wikisyntax that it
wouldn't be very useful as a reference.

I did some searching for general LR(n) parsers (some call themselves
GLR) parsers a while back, and none are too great.  The two most
promising are Elkhound from Berkeley
[http://www.cs.berkeley.edu/~smcpeak/elkhound/] and Packrat parsing
[http://www.pdos.lcs.mit.edu/~baford/packrat/].  The latter is much
cleaner, and uses a "parsing expression grammar" (PEG), basically a CFG
with ordered productions, to resolve ambiguity.  However, it uses O(n)
RAM where n is the size of the file to be parsed.  Elkhound is more
efficient, but I wasn't able to figure out how exactly to build
something clean out of it.

So, good luck. =]

-Mark


from_name: Roman Maurer

body:
Philip Newton pravi:

> I suppose so, though I believe both Serbian and Croatian usually
> "transliterate" foreign names, so "George Bush" becomes "Dz^ordz^ Bus^",
> whether in Latin or in Cyrillic Script.

AFAIK only Serbian does that, both in cyrillic and in latin script.
Croatian says "Georg Bush" while Serbian says "?or? Bu?".
--
	Pozdrav,
	  Roman


body:
xmlizer wrote:
<snip>
>> $kind=substr($article,"{{disambig}})<<2|substr($article,"{{stub}}")<<1|substr($article,"#REDIRECT");
>> // (more or less...)
>
> What about other langages
> fr : {{disambig}} --> {{Homonymie}}
> fr : {{stub}} --> {{?bauche}}

They will be localised through LanguageFr.php and then in the code we
can just use:
  wfmsg("disambigtemplate");
  wfmsg("stubtemplate");
and that will translate things easily :o))

> What about defining <tt>#DISAMBIGUATION</tt> and <tt>#STUB</tt> and to
> insert them respectively into Template:Disambig or Mod?le:Homonymie, and
> Template:Stub or Mod?le:?bauche

Useless. If an article is already flagged with {{stub}} it's not needed
to add a keyword in [[template:stub]].

--
Ashar Voultoiz


from_name: jcecropia at mail.com

body:
----- Original Message -----
From: Christiaan Briggs <christiaan at yurkycross.co.uk>
Date: Mon, 16 Aug 2004 09:49:09 +0100
To: English Wikipedia <wikien-l at Wikipedia.org>
Subject: Re: [WikiEN-l] When can an article be considered public domain?

> On 14 Aug 2004, at 6:39 am, Jim Cecropia wrote:
>
> > First IANAL
>
> Would such jargon stand up in a court of law?
>
> Christiaan
>
In terms of what? I am not representing myself as a lawyer and thereby stating that what I am saying may be knowledgeable, but it's not legal advice, so take it FWIW. Anyone who went to court and said, "I did this because a guy on the internet told me so" is not likely to do too well anyway.

Even if I WERE a lawyer, I would still have to say it is not legal advice, because no lawyer-client relationship exists.

IA-(still)-NAL, C

--
___________________________________________________________
Sign-up for Ads Free at Mail.com
http://promo.mail.com/adsfreejump.htm


body:
----- Original Message -----
From: Christiaan Briggs <christiaan at yurkycross.co.uk>
Date: Mon, 16 Aug 2004 09:51:41 +0100
To: English Wikipedia <wikien-l at Wikipedia.org>
Subject: Re: [WikiEN-l] When can an article be considered public domain?

> On 14 Aug 2004, at 6:39 am, Jim Cecropia wrote:
>
> > To go to an extreme, you obviously cannot quote an entire article and
> > then make some comments and call it fair use.
>
> Could you quote the whole thing verbatim and sign it author unknown?
>
> Christiaan
>

I wouldn't. Suppose I write an article for a local newspaper, signed, copyright notice and all (just so there's no doubt I take it seriously). Someone likes the article and posts it on his web site, no copyright, no author's name. It has just been pirated. If someone else then copies it off the web site thinking it's public domain, it's still a violation of my copyright.

You can always quote points from the article for critical analysis, something like: "In an unsigned article at www.blahblah.com, it is said that 'the Caballah is originally of Indo-Siamese origin before the year 650 CE,' but it is well known that this is a factual error, because..."

And so on...

It is almost never safe to quote an entire anything without knowing where you stand.

Cheers, C
> _______________________________________________
> WikiEN-l mailing list
> WikiEN-l at Wikipedia.org
> http://mail.wikipedia.org/mailman/listinfo/wikien-l

--
___________________________________________________________
Sign-up for Ads Free at Mail.com
http://promo.mail.com/adsfreejump.htm


from_name: Yann Forget

body:
Hi,

Le Monday 16 August 2004 14:13, Andre Engels a ?crit :
> About one month ago, I asked about WikiCommons, and Eloquence answered that
> he would not object against early creation any more. I have let it pass
> because of my holidays, but now would like to get started with this, hoping
> to have something set up when I start my new job on October 1.
>
> Thus, I would ask for the Wiki to be set up, proposed address:
> commons.wikimedia.org, perhaps with a redirect from commons.wikipedia.org.
>
> I would also like to know how far attempts at getting a new upload form
> have now come.
>
> Once it has been set up, I would like to invite everyone to come over to
> join in early policy discussion and get an upload basis, as well as to join
> with Eloquence in his work to improve the Mediawiki software for this
> purpose (see [[meta:Wikimedia Commons]]).
>
> People who have already shown interest in the project will be contacted by
> me once it is set up, as will be the various (other) Wikimedia projects.
>
> Andre Engels

Yes, I think that would be great.

Yann

--
http://www.non-violence.org/ | Site collaboratif sur la non-violence
http://www.forget-me.net/ | Alternatives sur le Net
http://fr.wikipedia.org/ | Encyclop?die libre
http://www.forget-me.net/pro/ | Formations et services Linux


from_name: Jan Hidders

subject: [Wikitech-l] In CVS

body:
On Monday 16 August 2004 19:35, Magnus Manske wrote:
>
> So, what you'd need is an EBNF representation or something?

Yes, that is all I ask. A precise formal grammar in whatever notation you like
but preferrably in the input format of bison. If only because that would
document what it is that your parser exactly does. Note there are some
requirements:
- it should accept all possible input strings
- it should ideally do all the necessary parsing i.e. from the parse tree we
should be able to generate the output with a single tree-walk, and
- it should be unambiguous (and even LALR(1)) or have an explicit conflict
resolution rule

Just having the first two would already be nice, we could then work from there
towards a grammar that satisifies the final requirement. To begin with you
could simply give the grammar of what you think your current parser does.

> I've never  worked with bison or the like. Is there a Windows version, by
> the way  (yes, I *have* Linux, but currently I'm mostly under W).

Yes there is:

 http://gnuwin32.sourceforge.net/packages/bison.htm

Are you familiar with the XML parser in PHP? That should give you an idea of
how to place the code in the grammar.

-- Jan Hidders


body:
Roman Maurer pravi:
> AFAIK only Serbian does that, both in cyrillic and in latin script.
> Croatian says "Georg Bush" while Serbian says "?or? Bu?".

Croatian writes "George Bush" (with an "e"), of course.  Sorry.
--
	Pozdrav,
	  Roman


subject: [Wikitech-l] replaceLinkHolders

body:
I have checked in the dirtiest hack of the week to replace that
str_replace :-)

Total rendering time is down 10% on my machine.

Magnus

Brion Vibber wrote:

> Jens Frank wrote:
>
>> Do you also have figures for release 1.3.1 rendering the same page?
>
>
> Median page view time for this page on 1.3.1 with $wgEnableParserCache
> is 346ms as measured with ab on the same machine.
>
> At this point a dominant time sink is loading and parsing the PHP
> files (I haven't installed a PHP cache module on this machine), and
> skin building output and buffering/sending of raw output become
> significant as well.
>
> Major offenders from profiling output:
>
> -total                  1 277.770 277.770 100.000%
> Setup.php               1 151.931 151.931 54.697%
> Skin::outputPage        1 102.973 102.973 37.071%
> OutputPage::output      1 103.143 103.143 37.132%
> Skin::doAfterContent    1 63.289 63.289 22.785%
> wfMsg                  53 50.454  0.952 18.164%
> Skin::quickBar          1 34.159 34.159 12.297%
> Skin::doBeforeContent   1 27.438 27.438  9.878%
> Article::view           1 18.326 18.326  6.597%
>
> So clearly we've gone backwards on at least some cases by doing link
> replacement after the parser cache. (A very short page with no links
> renders slightly faster with 1.4-cvs than on 1.3.1.)
>
> -- brion vibber (brion @ pobox.com)
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Wikitech-l mailing list
>Wikitech-l at wikimedia.org
>http://mail.wikipedia.org/mailman/listinfo/wikitech-l
>


from_name: Rowan Collins

subject: [Wikitech-l] Bugzilla bug report migration

body:
On Fri, 13 Aug 2004 19:18:27 -0700, Brion Vibber <brion at pobox.com> wrote:
> Timwi's started on migrating still-open bug reports from the sourceforge
> tracker to Bugzilla, which will bring us these amazing benefits:

Just one slight hiccup - I've noticed that at least a couple of bugs
have actually been migrated *twice*: for instance, bugs 98 and 115
both come from http://sourceforge.net/tracker/index.php?func=detail&aid=977535&group_id=34373&atid=411192

Oddly, this doesn't seem to be two people not co-ordinating properly:
both are credited to Timwi. I'm guessing a bot is involved somewhere
in the process, so perhaps it would be worth pausing and working out
why this is happening (there were two instances just among the very
small number of bugs I was tracking at SF, so there may be quite a bit
of it about...)

Meanwhile, thanks very much for the "MediaZilla": I've not done much
MediaWiki work yet, but the clunkiness of the SourceForge tracker
certainly didn't encourage me any. Hurrah for 'zillas!

--
Rowan Collins BSc
[IMSoP]


subject: [Wikitech-l] CVS problems & snapshot

body:
One of SourceForge's anonymous CVS mirrors has been down since August
12, naturally the one that serves us. This has broken:
* anonymous CVS checkouts
* ViewCVS
* nightly backups of the CVS repository

Developer CVS is still working; the anonymous servers just mirror that,
so we're still working on things ok but it's a pretty annoying service
outage. SF's status page says they expect it to be fixed sometime on
August 13 *cough*. We'll see... :P

For those of you who want to work on the current source but aren't on
the project on SF, here's a fresh checkout of CVS head branch with
phase3, extensions, and wiki2xml modules:
http://download.wikimedia.org/mediawiki/mediawiki-cvs-2004-08-17.tar.bz2

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 253 bytes
Desc: OpenPGP digital signature
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20040817/9307b43d/attachment.pgp>


body:
> Just one slight hiccup - I've noticed that at least a couple of bugs
> have actually been migrated *twice*: for instance, bugs 98 and 115
> both come from http://sourceforge.net/tracker/index.php?func=detail&aid=977535&group_id=34373&atid=411192
>
> Oddly, this doesn't seem to be two people not co-ordinating properly:
> both are credited to Timwi. I'm guessing a bot is involved somewhere
> in the process, so perhaps it would be worth pausing and working out
> why this is happening (there were two instances just among the very
> small number of bugs I was tracking at SF, so there may be quite a bit
> of it about...)

OK, I now officially *love* BugZilla: I had the cunning idea of
searching for "This bug has been migrated to MediaZilla:" in the text
of bugs (in MediaZilla) on the grounds that these are probably all
duplicate migrations (there was one where it was just erroneous).
There are quite a lot. But thankfully, bugzilla nicely took me through
them one by one without me even telling it to load the next bug -
nice!

So, they should now all be resolved as duplicates appropriately - just
don't do it again! ;) Now, with my good deed for the night done, I'm
off to bed: I'm flying off on holiday tomorrow...

--
Rowan Collins BSc
[IMSoP]


subject: [Wikitech-l] Re: An alternate parser (IT WORKS! :))

body:
Eckhart W?rner wrote:
> I think this is very impressive.
> But why do you try to avoid (x)html-like tag names?
[snip]
>>  <preblock>
>>   <preline>yylex();</preline>
>>   <preline>yyparse();</preline>
>>   <preline></preline>
>>   <preline>outputXML();</preline>
>>  </preblock>
>
> Well, here is some difficulty, as xhtml has no way to mark as a line.

<evil tempter>The XHTML 2.0 draft has an <l> element for this...</evil>

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 253 bytes
Desc: OpenPGP digital signature
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20040817/cd910e2a/attachment.pgp>


subject: [WikiEN-l] WP:(...)

body:
On Fri, 13 Aug 2004, Jens Ropers wrote (to WikiEN-L):

> Can I suggest we implement a "cover all" redirect, whereby ALL
> [[Wikipedia:Some stuff here]] kind of links can alternatively be
> written [[WP:Some stuff here]]?

This has been suggested before, and I must say I'm in favour, but it
was turned down on the grounds that it's too Wikipedia-centric to
warrant changing the code. However, I think other installations might
be pleased to have some kind of "namespace alias" system (in this
example, we would make "WP" an alias for "Wikipedia"). It might even
be useful for some of the non-English 'pedias, since they could have
the English namespace names as aliases for localised ones. [Of course,
in that scenario, it would be better to have a "replace on save"
system, rather than one where the links look different but go to the
same place, but it might be handy either way...]

> Not all "Wikipedia:(...)" pages have "own" their redirects (meaning
> redirects such as WP:SB), but it still seems like a good idea to ALWAYS
> allow the "Wikipedia:" bit to be abbreviated as "WP:" -- so that if I'm
> writing e.g. [[WP:Dispute resolution]], this would automatically
> redirect/be expanded to [[Wikipedia:Dispute resolution]]. Again, I'm
> asking whether we can do the WP: to Wikipedia: redirecting/expanding
> stuff on a GENERIC basis. (I know it /could/ be done on an individual
> article basis, but that's not my point.

And this is even more useful for namespaces with longer names, of
course: Wikipedia_talk: is just a pain to type!

> It would help, be really convenient and it would not get in the way of
> anything. Pages can still have actual own redirects such as [[WP:SB]].

Of course, if it were a real aliasing system such as I am imagining, a
page "called" [[WP:SB]] would actually be called [[Wikipedia:SB]], but
I don't see that that would matter much - we'd just have to move all
pages called WP:xxx to Wikipedia:xxx before activating the alias (with
a bot, or a database script). This would have the added benefit of
taking these shortcuts out of the main article namespace, where they
are likely to confuse people mirroring our content.

BTW, I've re-addressed this to what I hope is a more appropriate list,
and moved wikien-l to a CC. Also, I'm off on holiday, so having
stirred things up, I'll leave you to fight this one out between
yourselves ;)

--
Rowan Collins BSc
[IMSoP]


subject: [Wikitech-l] Re: [WikiEN-l] WP:(...)


from_name: Austin Hair

subject: [Wikipedia-l] personal threats again

body:
> What is still very weird, is that he used IP addresses from many
> different places, from Thailand to the United States. My question is:
> how could he?

This is commonly accomplished by means of proxy servers -- programs
running on host machines which allow Internet traffic to be piped
through as if the request came from the proxy host, rather than the
actual origin.  An all-purpose redirect, if you will.

These days, most of the proxies used for this purpose are nothing more
than backdoored Windows PCs; desktop computers which have, at some point
or another, been compromised by a trojan horse.  This is one of the many
reasons the use of Microsoft Windows should be considered to be actively
harmful -- but now I'm editorializing.

> roozbeh

--
Austin D. Hair <austin at austinhair.org>
http://en.wikipedia.org/wiki/User:Callidus
http://www.austinhair.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.wikimedia.org/pipermail/wikipedia-l/attachments/20040817/7f8a6b80/attachment.pgp


from_name: Jens Ropers

subject: [WikiEN-l] WP:(...)

body:
CB low

> Date: Tue, 17 Aug 2004 01:10:02 +0100
> Rowan Collins wrote:
>
> On Fri, 13 Aug 2004, Jens Ropers wrote (to WikiEN-L):
>
>> Can I suggest we implement a "cover all" redirect, whereby ALL
>> [[Wikipedia:Some stuff here]] kind of links can alternatively be
>> written [[WP:Some stuff here]]?
>
> [snip]
>
>> Not all "Wikipedia:(...)" pages have "own" their redirects (meaning
>> redirects such as WP:SB), but it still seems like a good idea to
>> ALWAYS
>> allow the "Wikipedia:" bit to be abbreviated as "WP:" -- so that if
>> I'm
>> writing e.g. [[WP:Dispute resolution]], this would automatically
>> redirect/be expanded to [[Wikipedia:Dispute resolution]]. Again, I'm
>> asking whether we can do the WP: to Wikipedia: redirecting/expanding
>> stuff on a GENERIC basis. (I know it /could/ be done on an individual
>> article basis, but that's not my point.
>
> And this is even more useful for namespaces with longer names, of
> course: Wikipedia_talk: is just a pain to type!
>
>> It would help, be really convenient and it would not get in the way of
>> anything. Pages can still have actual own redirects such as [[WP:SB]].
>
> Of course, if it were a real aliasing system such as I am imagining, a
> page "called" [[WP:SB]] would actually be called [[Wikipedia:SB]], but
> I don't see that that would matter much - we'd just have to move all
> pages called WP:xxx to Wikipedia:xxx before activating the alias (with
> a bot, or a database script).

We don't actually have to move anything -- it's just a matter of
precedence during the URL resolution process:
If the WP: to Wikipedia: conversion ONLY kicked in AFTER all other
option have been tried (but before the "Wikipedia doesn't have such a
page yet - do you want to create it"-screen got resorted to, then it
would just work.
Pity I can't code.

Thanks and regards,
Jens Ropers

There are two types of IT techs: The ones who watch soap operas and the
ones who watch progress bars.
http://www.ropersonline.com/elmo/#108681741955837683


subject: [Wikitech-l] Re: [WikiEN-l] WP:(...)


from_name: Poor, Edmund W

subject: [WikiEN-l] New English Wikipedia Mailing List Admin needed

body:
Folks, it's been fun administering the mailing list of the English
Wikipedia, but I no longer have the time to do a good job.

I'm resigning effective Friday, August 20th. If anyone wants to be
trained as my replacement, I'll be glad to help (there's not much to it,
really).

Ed Poor
Wikien-l admin


from_name: Jimmy (Jimbo) Wales

body:
Yes, all the things you say "make sense".  :-) It's really just a
matter of priorities.  I remember making negative comments about such
suggestions when we had only 2 or 3 machines, basically arguing that
worrying about destruction of the colo was pointless when we could
hardly function with the loss of a single *machine*.

But with 14 (15?) machines in service now, and with 10 more on the
way, with 12 4U machines soon to arrive as well (though not to be
pressed into service right away until we figure out what to do with
them exactly), and with the soon-to-exist Paris data center, it does
start to make some sense to think about geographical diversity.

I'm not sure how important this is, really.  But it is at least worth
thinking about.

Sj wrote:

> On Mon, 16 Aug 2004 09:03:24 -0700, Jimmy (Jimbo) Wales
> <jwales at wikia.com> wrote:
> >
> > 3.  We are widely beloved ... it seems likely that if we pay
> > the cost of a machine, they would be happy to host on a permanent
> > basis a slave db machine which we could keep synced to have a constant
> > realtime backup.
>
> At some point soon, we will be backing up many terabytes of archives
> and historical data.  Would it make sense to have an offsite location
> dedicated to preserving these backups and regular snapshots of the db,
> in case there is ever data corruption which goes unnoticed and is replicated
> to all active slaves/mirrors?
>
> That location could also maintain a synced mirror of the current projects
> and key filesystem content, along with a few machines available for
> configuration as temporary web and db servers, in case something terrible
> happened at the colo facility.
>
> +sj+
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at wikimedia.org
> http://mail.wikipedia.org/mailman/listinfo/wikitech-l
>


from_name: Nikola Smolenski

body:
On Monday 16 August 2004 09:46, Tim Starling wrote:
> Nikola Smolenski wrote:
> > On Monday 16 August 2004 05:57, Tim Starling wrote:
> >>What's wrong with forking? We have two different markets for our
> >>software, and these markets have conflicting needs. Why not produce a
> >>customised product for each one?
> >
> > In this case, interoperability. If parsers diverge, it might happen that
> > other MediaWiki wikis couldn't include Wikipedia's contents, and vice
> > versa.
>
> That's correct, any wiki wishing to use Wikipedia's contents without any
> conversion or risk of problems would have to be running the heavy-duty
> parser branch. But most MediaWiki wikis do not include Wikipedia's
> contents, and do not require interoperability.

I see, this is actually a secret plot against Wikinfo ;)

It works the other way around - if someone want to include contents from other
wiki into Wikipedia, there might also be problems.


references:
<cflt0k$e7u$1@sea.gmane.org> <200408160634.54714.smolensk@eunet.yu>
	<cfqfq6$fr9$1@sea.gmane.org>

body:
On Monday 16 August 2004 16:20, xmlizer wrote:
> Nikola Smolenski a ?crit :
> > On Sunday 15 August 2004 08:31, Brion Vibber wrote:
> >>Nikola Smolenski wrote:
> >>>How costly it would be to change type of cur_is_redirect to int?
> >>
> >>It's presently a tinyint, with valid values from -128 to 127. Altering
> >>its meaning would require being careful not to break existing
> >>assumptions, however, and neither 'disambiguation' nor 'stub' are imho
> >>useful or objectively calculable.
> >
> > I'm also not sure how useful it is, but it is objectively calculable :)
> >
> > $kind=substr($article,"{{disambig}})<<2|substr($article,"{{stub}}")<<1|su
> >bstr($article,"#REDIRECT"); // (more or less...)
>
> What about other langages
> fr : {{disambig}} --> {{Homonymie}}
> fr : {{stub}} --> {{?bauche}}
>
> What about defining <tt>#DISAMBIGUATION</tt> and <tt>#STUB</tt> and to
> insert them respectively into Template:Disambig or Mod?le:Homonymie, and
> Template:Stub or Mod?le:?bauche

No need for that, this could easily be localised.


subject: [Wikitech-l] Re: An alternate parser

body:
On Monday 16 August 2004 08:10, Krzysztof Kowalczyk wrote:
> On Mon, 16 Aug 2004 06:51:21 +0200, Nikola Smolenski <smolensk at eunet.yu>
wrote:
> > One question: wouldn't that PHP parser be slower than current parser, and
> > how much?
>
> No-one can know that unless the new parser is written and benchmarked
> against the current one.

I think that I can know: if it doesn't use regexps but just goes through
entire text character-by-character, it will be MUCH slower.


from_name: Jens Frank

subject: [Wikitech-l] replaceLinkHolders

body:
On Mon, Aug 16, 2004 at 05:04:13PM -0700, Brion Vibber wrote:
> Magnus Manske wrote:
> >I have checked in the dirtiest hack of the week to replace that
> >str_replace :-)
> >
> >Total rendering time is down 10% on my machine.
>
> Neat! But now that I look at it, it can be sped up a lot just by using a
> regular expression:
>
> Original str_replace method:
>  1    4260.353   4260.353     44.304%
>
> Magnus' scary hack:
>  1    423.490    423.490      7.385%
>
> Using preg_replace_callback:
>  1     21.842     21.842      0.354%
>

Congratulations! That sounds really fast! Now it's really faster than
the 1.3.1 renderer?

JeLuF


subject: [Wikitech-l] Re: CVS problems & snapshot

body:
Timwi wrote:
> And, of course, you can subscribe to the mediawiki-cvs at wikipedia.org
> mailing list (http://mail.wikipedia.org/mailman/listinfo/mediawiki-cvs)
> to get the patches we apply mailed to you. You can use the "patch" tool
> under UNIX to apply these patches to your local copy.

Note that not all patches in the emails are complete; every once in a
while the mail comes with "...2574 lines snipped..." or such and doesn't
include binary files.

But it's a big help.

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 253 bytes
Desc: OpenPGP digital signature
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20040817/960670d1/attachment.pgp>


from_name: Timwi

body:
Nikola Smolenski wrote:
> On Monday 16 August 2004 00:32, Timwi wrote:
>
>>*However*, at least in the lexer I have written so far, there isn't
>>anything spectacularly impressive. This one will be trivial to keep a C
>>and PHP version synchronised of. As for the parser itself, I won't make
>>any promises just yet, but from what I've seen, it seems similarly
>>promising.
>
> One question: wouldn't that PHP parser be slower than current parser, and how
> much?

As I said, I didn't make any promises. :-) By now I have written a real
parser (it actually works!! It outputs correct XML!! I'm totally excited
about it!) and I'm afraid it's not *trivial* to translate to PHP (though
I'm sure I could do it easily).

So far, my parser can do headings, paragraphs, lists (only * and # yet),
pres (lines beginning with a space), and "extensions" (everything
enclosed in <sometag>...</sometag>, including <nowiki>,
, <math>,
etc.). I'll give an example below. Please send me input files to test my
parser on so I can see if it handles all possible situations correctly,
including broken input mark-up!

I'm going to do bold/italics and links next.

Timwi

--- Input file ---

== This is a heading! ==

The parser works like this:

  yylex();
  yyparse();

  outputXML();

== Another heading! ==
The individual steps are:
* yylex: Tokenisation
* yyparse: Parsing (*)
** this is the exciting part!
* outputXML: XMLisation

I can prove that <math>2 < 1</math>!

--- Output (the newlines are inserted by my mailer) ---

<heading level='2'> This is a heading! </heading><paragraph>The
parser works like
this:</paragraph><preblock><preline>yylex();</preline><preline>yyparse();</preline><preline></preline><preline>outputXML();</preline></preblock><heading
level='2'> Another heading! </heading><paragraph>The individual steps
are:</paragraph><list type='bullet'><listitem>yylex:
Tokenisation</listitem><listitem>yyparse: Parsing (*)<list
type='bullet'><listitem>this is the exciting
part!</listitem></list></listitem><listitem>outputXML:
XMLisation</listitem></list><paragraph>I can prove that <extension
name="math">2 < 1</extension>!</paragraph>

--- Manually beautified output ---

  <heading level='2'> This is a heading! </heading>
  <paragraph>The parser works like this:</paragraph>
  <preblock>
   <preline>yylex();</preline>
   <preline>yyparse();</preline>
   <preline></preline>
   <preline>outputXML();</preline>
  </preblock>
  <heading level='2'> Another heading! </heading>
  <paragraph>The individual steps are:</paragraph>
  <list type='bullet'>
   <listitem>yylex: Tokenisation</listitem>
   <listitem>yyparse: Parsing (*)
    <list type='bullet'>
     <listitem>this is the exciting part!</listitem>
    </list>
   </listitem>
   <listitem>outputXML: XMLisation</listitem>
  </list>
  <paragraph>
   I can prove that <extension name="math">2 < 1</extension>!
  </paragraph>


body:
Tim Starling wrote:

> Krzysztof Kowalczyk wrote:
> > Possible dilution of dev resources, for one. If there are N developers
> > working on unified code-based, in the worst-case scenario after the
> > split each project might end up with, say, N/2 developers, testers,
> > users etc.
>
> I'm not talking about completely splitting the project, only a small
> proportion of the work would be duplicated.

The parser is a pretty significant proportion, and Krzysztof's argument
still applies to that portion.

Mind you, though, I'm not taking either side in this issue. I tend to
agree more with you, Tim, but I won't take any responsibility :)

Timwi


subject: [Wikitech-l] Re: Bugzilla bug report migration

body:
Rowan Collins wrote:

> On Fri, 13 Aug 2004 19:18:27 -0700, Brion Vibber <brion at pobox.com> wrote:
>
>>Timwi's started on migrating still-open bug reports from the sourceforge
>>tracker to Bugzilla, which will bring us these amazing benefits:
>
> Just one slight hiccup - I've noticed that at least a couple of bugs
> have actually been migrated *twice*: for instance, bugs 98 and 115
> both come from http://sourceforge.net/tracker/index.php?func=detail&aid=977535&group_id=34373&atid=411192

I noticed that too. Yes, it's my fault. But I'm not particularly worried
about this: it is better to have each bug report twice than to miss one
of them.  Currently I have 247 e-mails in my inbox, at least 200 of
which are e-mails from MediaZilla, and I am planning to sift through all
of them.  I have already marked a number of bugs as duplicates, there is
no reason I can't do it again :-)

> I'm guessing a bot is involved somewhere
> in the process, so perhaps it would be worth pausing and working out
> why this is happening

I thought about doing this, but I think it will be more productive to
spend time on actually closing the duplicates now.  After all the bugs
are migrated, there will no longer be a need for the migration script,
so finding bugs in it is pointless.

> Meanwhile, thanks very much for the "MediaZilla": I've not done much
> MediaWiki work yet, but the clunkiness of the SourceForge tracker
> certainly didn't encourage me any. Hurrah for 'zillas!

I am very glad you like it :-)

Timwi


body:
Rowan Collins wrote:

> So, they should now all be resolved as duplicates appropriately - just
> don't do it again! ;)

Oh, you have already closed them!  Thank you!  That's very useful!

> Now, with my good deed for the night done, I'm
> off to bed: I'm flying off on holiday tomorrow...

Have good fun!  You really deserve it now :-)

Timwi


subject: [Wikitech-l] Re: [X] Yes i want to delete this,

body:
Ray Saintonge wrote:

> A little safety valve is always a good idea.  One additonal keystroke
> should not seriously make demands on a person's.

I would normally agree with you if we were talking only about deleting
individual pages.  However, when deleting quite a number of pages, that
checkbox can become quite an annoyance.

Maybe someone (I?) should write a feature whereby sysops can specify a
list of article titles and mass-delete them with one single "safety
valve".  Then there should be one button for sysops to undo such a
mass-deletion quickly in order to have a safeguard against sysops acting
in bad faith.

Timwi


subject: [Wikitech-l] Re: Double database entries

body:
Magnus Manske wrote:

> Timwi wrote:
>
>> Magnus Manske wrote:
>>
>>> There's a "INSERT ... ON DUPLICATE KEY UPDATE" function in MySQL
>>> which should avoid double creation.
>>
>> Ah, but that would over-write the older 'cur' row, no?
>>
> Yes, but AFAIK, this "double creation" only happens if a new article is
> submitted and saved twice (thrice etc.) in rapid-fire mode. So, it would
> basically overwrite the information with the same information again.

Good point.

Timwi


body:
xmlizer wrote:

> What about defining <tt>#DISAMBIGUATION</tt> and <tt>#STUB</tt> and to
> insert them respectively into Template:Disambig or Mod?le:Homonymie, and
> Template:Stub or Mod?le:?bauche

I am rather opposed to anything that would give any particular meaning
to "disambiguation pages" or "stubs", as they are Wikipedia-centric
concepts.  The software should allow either server administrators or
even the users themselves to define custom such classifications.  People
in the install the wiki software for different purposes, or even the
sister projects (Wiktionary, Wikisource, etc.) should be able to come up
with and use other classifications, and they should not be burdened with
concepts like "disambiguation page" that don't make any sense in the
context of their content.

Timwi


references:
<cflt0k$e7u$1@sea.gmane.org>	<cfm2uj$b8f$1@sea.gmane.org>	<200408150805.30306.smolensk@eunet.yu>	<cfn1ve$nuu$1@sea.gmane.org>
	<cfqfrh$fka$1@sea.gmane.org>

body:
Ashar Voultoiz wrote:

> I will code my draft using the link table. The disambiguation template
> name can then be set in the Language.php file.

Better to let the user specify a template name, in case people on a
different wiki using our software want to use your feature using several
different templates.

Timwi


from_name: Tomos at Wikipedia

subject: [Wikipedia-l] Does anyone knows the beginning of ja?

body:
Hello.

An email arrived at japanese wikipedia's press contact
mailinglist regarding Web Creation Award. It is an award
given to people who made significant difference in the web
in Japan. The host of the award is an association of
advertisers, though the recipients of the awards are not
limited to marketing campaign planners. Wikipedia is under
consideration because someone made a recommendation. :-)

As a part of selection process, they would like to do
some interview. The person who emailed us expressed that
he would like to find the person who started the Japanese
Wikipedia, someone who came up with the idea, plan, or
determination. Who exactly this is, I think, is somewhat up
to interpretation. Is there any strong candidate?

I recall reading early Wikipedia-l posting (perhaps by Larry and/or Jason)
announcing the launch of several non-English Wikipedia
circa May 2001. Japanese Wikipedia was among the first
ones. Is there anybody who remembers how that happened?
Was multilingualization of Wikipedia an idea of Jimmy
and Larry from the beginning? Or was there someone else
who suggested/ pushed for it? Is there any reason why
Japanese was among the first?

In the early days, Aoineko was kind of "stationed" when
there was only several edits a week, and welcomed the
logged on users (there were only several back then) if
I understand correct. Brion was there to fix bugs and
take care of other technical aspects, too.  The first
article seems to have been written by RoseParks (phonemes
of Japanese). But there are only fragmental traces and
virtually nobody knows those early days. Could somebody
inform us of people who made significant contributions?

Thank you in advance for your help.

Regards,

Tomos

More info. is available on the award at here (in Japanese):

http://award.wab.ne.jp/award/index.html

_________________________________________________________________
FREE pop-up blocking with the new MSN Toolbar ? get it now!
http://toolbar.msn.click-url.com/go/onm00200415ave/direct/01/


subject: [Wikitech-l] Re: In CVS

body:
Delirium wrote:

> I can't prove it at the moment, but after having played around with
> wikisyntax for a while, I'm *pretty* sure it's not LALR(1).

MediaWiki's syntax is perfectly LALR(1), except for only one very
important exception: lists.

Timwi


subject: [Wikitech-l] Re: In CVS (was: wiki2xml : Deeds following words)

body:
Michael Becker wrote:

> I'm going to break out my compiler books and start working on a CFG for the wiki code

I am already working on this.


subject: [Wikitech-l] Re: [WikiEN-l] WP:(...)

body:
Jens Ropers wrote:

> We don't actually have to move anything

If you want to introduce a new namespace (or namespace alias) called
"X", for example, you *do* have to move all the pages whose title begins
with "X:", or else they will become inaccessible.

This is because an article "WP:SB" is actually an article with the title
"WP:SB" in the article namespace, while an article "Wikipedia:SB" is
actually an article with the title "SB" in the Wikipedia namespace.  If
you introduce a new namespace (or namespace alias) called "WP", the
article [[WP:SB]] will still be in the article namespace, but the server
will start looking for it in the new namespace and not find it.

Timwi


subject: [WikiEN-l] WP:(...)

body:
Jens Ropers wrote:

> We don't actually have to move anything

If you want to introduce a new namespace (or namespace alias), you *do*
have to move all pages whose title begins with the new namespace name
followed by a colon, or else they will become inaccessible.  For the
reasons, please see my reply on the technical list.


subject: [Wikipedia-l] Re: Text-to-speech

body:
Peter Shaw wrote:

> On Saturday 14 August 2004 00:19, Timwi wrote:
>
>>Well, personally, I find it rather disappointing, and I have heard
>>better ones even several years ago.  However, it would certainly be
>>pretty cool to have Wikipedia articles as sound files.
>
> Where?

The "better one" I am talking about was an old DOS application called
"SBTalker", and it came with the Sound Blaster Pro 4.  Needless to say,
it no longer works under Windows.


subject: [Wikitech-l] Re: phase3/includes OutputPage.php,1.165,1.166

body:
Brion Vibber wrote:

> * Split a couple "?>" instances in string literals -- these hose some syntax highlighters

I think you should add a comment to explain this. Otherwise in future
people will be likely to undo this, not thinking anything of it.

Timwi


body:
Eckhart W?rner wrote:

> I think this is very impressive.
> But why do you try to avoid (x)html-like tag names?

Because I am outputting XML that represents the parse tree. It is not
XHTML and does not conceptually have anything in common with the XHTML
that is supposed to come out at the end. And because I want to prevent
right from the onset that people get confused with stuff.  It should be
as easy as possible to tell from looking at something whether it is the
parse tree or the XHTML.

Timwi


from_name: Eckhart Wörner

body:
Hello Timwi,

I think this is very impressive.
But why do you try to avoid (x)html-like tag names?

Timwi schrieb:
>

>  <heading level='2'> This is a heading! </heading>

This is a heading

>  <paragraph>The parser works like this:</paragraph>

The parser works like this:

>  <preblock>
>   <preline>yylex();</preline>
>   <preline>yyparse();</preline>
>   <preline></preline>
>   <preline>outputXML();</preline>
>  </preblock>

Well, here is some difficulty, as xhtml has no way to mark as a line.

[...]
>  <list type='bullet'>
>   <listitem>yylex: Tokenisation</listitem>
>   <listitem>yyparse: Parsing (*)
>    <list type='bullet'>
>     <listitem>this is the exciting part!</listitem>
>    </list>
>   </listitem>
>   <listitem>outputXML: XMLisation</listitem>
>  </list>

yylex: Tokenisation

yyparse: Parsing (*)

this is the exciting part!

outputXML: XMLisation

>   I can prove that <extension name="math">2 < 1</extension>!
>  </paragraph>
>

The reasons for (x)html tag names are:
* They are very short.
* They are well-known by thousands of people.
* There should be no problem with them for possible conflicts with xhtml
are avoided by the namespace.

Eckhart