subject: [Wikitech-l] caching question

body:
Jan said that I have caching turned off, which surprised me because I thought
it was on.  Now I've looked at the code and I still think it is on.

wikiSettings.php:
$useCachedPages = true ;

wikiLocalSettings.php:
# $useCachedPages = false; # Disable page cache

(This is commented out, right?)

-----------------

Playing with benchmarking, grabbing a normal article 100 times:

(I know that this type of benchmarking is not very scientific, since conditions
may change on the live server due to someone else doing something big at the same
time, etc.  But I think it gives an indication.)

As the site is running:
/apache/bin/ab -n100 -c1 http://www.wikipedia.com/wiki/Alabama
Requests per second:    0.95 [#/sec] (mean)

Now I will set $useCachedPages to false by uncommenting the line in wikiLocalSettings.php.

/apache/bin/ab -n100 -c1 http://www.wikipedia.com/wiki/Alabama
Requests per second:    0.97 [#/sec] (mean)

So I see no material difference.

How can I easily tell if caching is actually on or off?  Am I doing something wrong
here?


list: wikipedia-l

source: pipermail-mbox

from_name: Larry Sanger

subject: [Wikipedia-l] Announcement about my involvement in Wikipedia and Nupedia

body:
Dear everyone,

After a lot of thinking and negotiating, I've arrived at an important
decision and it's also important that I convey this decision to you.

For reasons I'll let Jimbo explain if he likes--it had *nothing* to do
with my job performance or with any personal animosity--I was placed on
half-time pay in January, and as of February 1, I am no longer a Bomis
employee.  There is no ill-will between Jimbo and I, and I wish him and
Bomis all the best.  This means that any work I do for the 'pedias, for
the near future anyway, I am doing as an unpaid volunteer, the same as the
rest of you.

This means two things.  First, I've got to get a job that will pay the
bills, and second, I'm going to have to reduce my participation in the
project quite a bit (over my pre-January participation).

I want to stress that I am *not* simply quitting altogether.  I can't see
abandoning the projects I started--they're too important.  I hate to think
of what might become of the projects if I were to quit altogether.  But
Wikipedia has become *largely* self-managing, something I'm tickled pink
about, and Nupedia has been moving so slowly anyway that I don't need to
spend a lot of time managing that.  (Ironically, I might have more time
for Nupedia, now that I'm not a paid employee. :-) )  Moreover, Bomis
might well start selling ads on Wikipedia sometime within the next few
months, and revenue from those ads might make it possible for me to come
back to my old job.  That would be great.  I've liked this job very much,
and I'm willing to do some work to help make it pay for itself.

Now that I'm on the job market, if you want to see my resume, let me know
at lsanger at nupedia.com.  I'm willing to work as a consultant or as an
employee--or, possibly, even as a partner.  I have quite a few ideas for
Internet projects that are potentially profitable and/or extremely
productive and interesting.  Among the cards I have to play are proposals
to big Internet portals, thinktanks, and magazines, to spearhead these
projects, much as I've spearheaded Nupedia and Wikipedia.  If you'd like
to discuss these project ideas confidentially on a serious business level,
please let me know.  Of course, I'm interested in less ambitious ways of
making money, too.  :-)

One last thing.  Please don't use this announcement as an excuse to say,
"Fine, I'm quitting then."  As you can see, *I* am *not* quitting, even
though prudence about my employment situation might dictate otherwise.  I
have decided to tough it out and join the ranks of volunteers myself.  If
anything, I hope this will make you even more inspired than ever to help:
"By golly, if Larry doesn't quit even though he's out of a job, and
volunteers his now-limited free time, I will too!"  That's what I hope
you'll think.  Just remember: Wikipedia is going full steam ahead.  We've
got nearly 25,000 articles in just a year!  We're getting ever-increasing
amounts of traffic and we have a very large, fairly competent (in some
cases, brilliant) base of contributors.  I will still be around to answer
questions, offer opinions, and occasionally kick ass over bad articles
:-).  As for Nupedia, now that I've finally made this decision, I do plan
sometime in the near future to make that final push to rewriting the
policy guidelines with the recent vote in mind.  Magnus Manske is champing
at the bit to write the code (Nunupedia, he calls it) that will implement
the new Nupedia system.  I do think that once the guidelines and the code
are up and running, Nupedia is going to experience growth of a sort it's
never seen before--one or two orders of magnitude greater.  Once we make
it easier and more automatic to submit Wikipedia articles to the new
Nupedia system, I think things are *really* going to take off for Nupedia.

Our forging ahead surely doesn't *require* my full-time paid involvement.
It would make it a lot easier and faster if I were still involved in such
a capacity, but I don't think it's absolutely necessary.

So--stay the course!

Best,
Larry


from_name: Mark Christensen

subject: [Wikipedia-l] Re: Simple XML ideas

body:
> Why? This would keep the software and the interface simple and more
> consistent. Look at the links to namespaces at the bottom of pages.
> Sometimes there is none, sometimes one, sometimes two. Why is that?
> And why on earth is the link to a discussion page called
> a namespace? And why is there no Talk page for SpecialPages?

I think constancy is not the solution, it's the problem.

The special namespaces contain pages which are functionally different from
standard wikipedia pages.  Talk namespaces are for discussion of articles,
and user namespaces are for interaction with other wikipedians.   These are
NOT encyclopedia articles, and I think it behooves us to help anybody who
comes to wikipedia to gain a clear and distinct understanding of this point
as quickly as possible.

I would actually suggest that we try somehow to change the "look and feel"
of the special namespaces, so that people will just SEE the distinction as
they browse through our site.  The fundamental problem I have with using ()
both to disambiguate terms within the wikipedia, and to denote special pages
which are not encyclopedia articles, is that we don't want people to
accidentally miss this distinction and start using the standard encyclopedia
article namespace for user pages, or for talking about articles, etc.

Making the distinction as clear as possible is and important goal -- in my
opinion more important than simplifying the way namespaces are handled in
the code.

That said, I think we can and should look at ways to make the current
interface more self explanatory, so that the links help users make the
necessary distinction between wikipedia and the necessary wikipedia related
pages.

Yours
Mark


subject: [Wikipedia-l] Re: Simple XML ideas

body:
> -----Original Message-----
> From: Magnus Manske [mailto:Magnus.Manske at epost.de]
> Sent: Wednesday, February 13, 2002 10:26 AM
> To: wikipedia-l at nupedia.com
> Subject: RE: [Wikipedia-l] Re: Simple XML ideas
>
> I agree that the special namespaces should be distinguished
> more from the encyclopedia pages. Again, this is more a
> layout problem than anything else. How about marking the
> namespace part of the title in red, of italics?

I agree that changing the layout of the special
pages will go a long way to solving this problem.
But I also think we may want to add some text
near the links to other namespaces, explaining
things.

That said, I don't think just changing the color
of the title is enough to make it immediately
visually apparent that you are looking at a different
kind of page.  I think we need something visually
bigger -- a slight change in the background color,
or the addition of a subtle background image,
or re-arranging the layout of special pages in a more
obvious way.

Beyond that, I do think the text for the link to a
Talk: namespace should be more descriptive. But, I
also don't believe users need a longwinded description
of the _concept_ of namespaces, to be able to grasp
them, this is because people use namespaces intuitively
all the time.  ("Bob" means something different for me
at work than it does at a family picnic.)  What they
need is visual cues which will help them to make use
of this intuitive knowledge.

Yours
Mark


list: wikitech-l

from_name: Brion Vibber

subject: [Wikitech-l] Slow parser

body:
On mer, 2002-02-13 at 07:36, Magnus Manske wrote:
> I just ran "ab n=10" for an atricle (with cache turned off) and deactivated
> some functions to see where the slow parts are.
>
> Full rendering : 4.99 sec
> removeHTMLtags turned off : 3.319 sec

How much time does parseContents() take?

> It seems removeHTMLtags is responsible for 1/3 of the *total* runtime, which
> includes apache, php calling, and a thousand other things that can't be
> avoided.

Well, it can be made much more efficient... As Jan has hinted, explode()
is a killer, and I can take that out.

> So, if these HTML tags are *never* used anyway, why can't we replace them
> with < and > just prior to saving an edited article?

I just have two objections:

First, it violates the principal of least surprise; the user doesn't get
the same thing upon a re-edit that they left during the last edit. This
is particularly annoying for people who are putting complicated tables
into articles (cf. [[Beryllium]] and [[Periodic Table]]) -- if they do
one thing wrong, POOF! Half their table <tags> suddenly turn into
<tags> and instead of fixing one tiny mistake, they fix one tiny
mistake AND change a lot of <>s back into <>s.
  Conclusion: bad for users.

Second, enforcing the limited subset HTML is just a part of the wiki
parsing. Doing that on save is fine, but is basically doing half the
parsing job and caching that, then doing the other half when we display
the page. Why stop there, when we could just parse the wiki-specific
code while we're at it and save the final result?
  Conclusion: what exactly is our goal here? To save processing time on
page load? This is most effectively done by caching the completely
parsed version, both HTML and wiki -> HTML.

> I'll be gone tomorrow until Saturday, and I doubt I can hack it today, so
> it's up to you...

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] Slow parser

body:
On mer, 2002-02-13 at 09:00, Jan Hidders wrote:
> From: "Magnus Manske" <Magnus.Manske at epost.de>
> >
> > I'll be gone tomorrow until Saturday, and I doubt I can hack it today, so
> > it's up to you...
>
> I wouldn't dare, that's Brion's turf. :-) But my remarks for parsing also
> apply here. You guys seem to be really fond of the explode operator, but
> it's really not a very good idea.

I know, I know, explode is bad. I'll replace it with something cleaner,
it was just a shortcut to what should have been a walk-through-the-
string-stopping-when-you-find "<" thing.

> PS. At the risc of comitting social suicide, let me admit that, yes, at one
> point in my life I have been teaching compiler theory. But it is certainly
> not my real area of expertise.

(shudder)

-- brion vibber (brion @ pobox.com)


subject: [Wikipedia-l] Faster!  Thanks!

body:
Just as the announcements page says, Wikipedia is a lot faster now.
Thanks to everyone who made it possible.  A quick-loading Recent Changes
page is a beautiful thing.

Larry


subject: [Wikitech-l] Faster!  Thanks!


from_name: Jan Hidders

subject: [Wikipedia-l] Re: Simple XML ideas

body:
From: "Axel Boldt" <axel at uni-paderborn.de>
>
> I think the two are different: the software enforced name spaces are
> there to separate the encyclopedia proper from all the cruft
> surrounding it.

Why do we want to do that? Because some pages are generated? That
is more elegantly solved with macro's like {{{MostWanted}} and
{{{SpecialPages}}}. So why then?

> Your parenthesis notation serves mainly to distinguish
> several concepts with the same name: [[Cardinal (bird)]], [[Cardinal
> (person)]]. I don't think people want to or should be required to
> universally slap on "(person)" to every title that describes a person.

I don't see how that is implied by what I said. All I am saying is that we
could replace names like "special:MostWanted" with something like
"MostWanted (Wikipedia)" and "user_talk:AxelBoldt" with
"Axel Boldt (Talk)" (or, better: "Talk (Axel Boldt)").
Of course you would then reserve certain namespaces such as Wikipedia
so people couldn't add terms in those namespaces, or even make all those
pages uneditable for non-sysops, but otherwise they
would be treated equally.

Why? This would keep the software and the interface simple and more
consistent. Look at the links to namespaces at the bottom of pages.
Sometimes there is none, sometimes one, sometimes two. Why is that?
And why on earth is the link to a discussion page called
a namespace? And why is there no Talk page for SpecialPages?

In my scheme you would have the same rules for every page:
1. at the bottom of page "X" there is a discussion link to "Talk (X)" except
    when "X" is already of the form "Talk (Y)".
2. at the bottom of page "X (Y)" there is a link to namespace "Y"

Or is that too simple? :-)

-- Jan Hidders


subject: [Wikipedia-l] Subsections

body:
From: "Larry Sanger" <lsanger at nupedia.com>
>
> I just formulated an interesting hypothesis, which might explain why some
> people are attracted to ideas like subpages and subsections, despite the
> many clear disadvantages of them.  Sorry, this is going to be a bit of a
> ramble. [...]

Hm, I believe you have actually convinced me. Can I summarize you as
follows:
we don't want people to build their own little hierarchies in this
encyclopedia and
have some links between them, but we want an interlinked web of relatively
self-contained small articles that give a sketch of the context necessary to
understand them and link to other articles that describe the context in a
general
way or parts of the context in more detail.

Correct?

-- Jan Hidders


subject: [Wikitech-l] Cache vs. other-language links

body:
From: "Brion Vibber" <brion at pobox.com>
>
> I noticed that the other-language links (links in the form [[fr:Japon]]
> [[en:Japan]] [[eo:Japanio]] etc which are hidden in the article body but
> listed by language name in the header bar, pointing to the article on
> the current subject in the other-language wikis) are vanishing on cached
> pages, because they're scanned and listed during the wiki->html link
> parsing which of course doesn't occur when loading a cached page.

Can I suggest we simply stop with the whole caching thing? It complicates
things unnecesarily. Keeping the code simple should be one of our top
priorities. Jimbo doesn't have it turned on at the moment anyway, and
Wikipedia seems to be fine on non-generated pages. And I expect that we can
do really a lot of optimization on the generated pages comparable with
Recent Changes (which is also not cached at the moment), and there is a
whole bunch of very inefficient (esp. in terms of memory use) programming
going on in the current parser.

> Alternatively, we might have a separate database which contains nothing
> but lists of connected articles. This could facilitate keeping the
> other-language links consistent; [...]

*sigh* It's a very nice idea, but currently I don't believe that phpwiki is
really out of the woods yet. First the current functionality has to be
correct, efficient and the code has to be well-organized and documented. And
only then can we start thinking about such fancy extensions of
functionality.

-- Jan Hidders


subject: [Wikipedia-l] Re: Simple XML ideas

body:
> Well, it seems that the battle is over, and I lost. :-)
> I'm still not really  convinced because I feel that you
> are seeing threats that are not really  there. Was
> consistency ever a real problem when we still had
> subpages? Yes, some articles contained pieces of
> discussion that belonged on the Talk pages, but that
> was usually solved in time.

I want to be clear about one thing.  My argument has less to
do with how user and talk pages are differentiated by the
software than it does with the point of having these extra
namespaces -- we want them to seem different to users, so
they know they are in a different part of the site.

This could be done while using the Talk:XYZ format, and
it could also be done using XYZ the (Talk) format, but I
think it is important to make the conceptual distinction
between disambiguation of terms, and the creation of a
penumbra of wikipedia related materials which are not
encyclopedia articles.  If we were doing this from the
beginning, I think we'd be far more willing to do it your
way, but fortunately or unfortunately we are already
working with live code that uses Talk:XYZ style
namespaces.

Another issue I just thought about is that we've
discussed at great length when originally thinking
through the idea of namespaces it that we might like to
at some point create a "Stable:XYZ" namespace, which
will contain locked versions of wikipedia articles
which have been in some special way validated by a group
of experts in that field.  These articles might very
well include () style disambiguation, at some point
we'd clearly want a stable version of
"Lincoln (Nebraska)." This and this would
would make using () style namespaces for the stable
branch of the wikipedia "codebase" very confusing.

Yours
Mark


from_name: Magnus Manske

subject: [Wikipedia-l] Re: Simple XML ideas

body:
> I don't see how that is implied by what I said. All I am saying is that we
> could replace names like "special:MostWanted" with something like
> "MostWanted (Wikipedia)" and "user_talk:AxelBoldt" with
> "Axel Boldt (Talk)" (or, better: "Talk (Axel Boldt)").
> Of course you would then reserve certain namespaces such as Wikipedia
> so people couldn't add terms in those namespaces, or even make all those
> pages uneditable for non-sysops, but otherwise they
> would be treated equally.

But, then you want to filter all non-encyclopedia articles from your search
(should become a standard option on searches). Then what? Now, all you have
to do is say "no articles with : in them". How do you filter something like
"Talk (Axel Boldt)"? All articles starting with "Talk ("? And then the same
thing for wikipedia? And move the user pages back into the normal namespace?
Or "User (Axel Boldt)"? Like "User (computer)"?

Somehow, I fail to see the beauty of that.

> Why? This would keep the software and the interface simple and more
> consistent. Look at the links to namespaces at the bottom of pages.
> Sometimes there is none, sometimes one, sometimes two. Why is that?
> And why on earth is the link to a discussion page called
> a namespace? And why is there no Talk page for SpecialPages?

Looks more like a layout problem than anything else. At the time I installed
namespaces, listing all of 'em "en blocc" seemed easiest. No reason not to
change that to something more readable.

> In my scheme you would have the same rules for every page:
> 1. at the bottom of page "X" there is a discussion link to "Talk
> (X)" except
>     when "X" is already of the form "Talk (Y)".
> 2. at the bottom of page "X (Y)" there is a link to namespace "Y"
>
> Or is that too simple? :-)

That must be it! ;)

Magnus


body:
From: "Magnus Manske" <Magnus.Manske at epost.de>
>
> The parser has to be brought up to speed. I'll also have a look into
> connecting the PHP script with the C++ parser I wrote (did I mention 0.05
> secs for rendering "Signal transduction", with fetching it from the
> database, searching the database for existing topics, and adding the
> "framework"?;)

As I said I'd rather keep it in PHP, but it's your project of course. Does
your parser put any requirements on the syntax. Should it be LL(1) or
LALR(1)? Are you going to use yacc, or is it just a simple recursiev descent
parser?

What we could improve in PHP for example is that the current parser parses
the string paragraph by paragraph. (But please don't use the function
explode() for that because that is a memory killer.) Most replace-functions
could be limited to only one paragraph and the rest can be dealt with by
making the parser a little context-sensitive. Standard Wiki matrup is
supposed to be limited to one paragraph anyway. The HTML markup is a bit
harder, but there you can remember the nesting depth and type of nesting,
and once you see that that tags are not balanced you go back in the string
and replace the < and > with entities. This will be expensive but it is an
exception, so it won't hurt.

-- Jan Hidders


subject: [Wikipedia-l] Re: Simple XML ideas

body:
From: "Magnus Manske" <Magnus.Manske at epost.de>
>
> But, then you want to filter all non-encyclopedia articles from your
search
> (should become a standard option on searches). Then what?  Now, all you
have
> to do is say "no articles with : in them". How do you filter something
like
> "Talk (Axel Boldt)"? All articles starting with "Talk ("?

.. and ending with ")". Exactly.

> And then the same thing for wikipedia?

Yes, everying ending with "(Wikipedia)" is filtered out if you choose so.
Why do you think that is a problem?

> And move the user pages back into the normal namespace?

No, you would have "Axel Boldt (User)". The namespaces "(Wikipedia)" and
"(User)" would probably then be special restricted namespaces that replace
"special:" and "user:".

> > Why? This would keep the software and the interface simple and more
> > consistent. Look at the links to namespaces at the bottom of pages.
> > Sometimes there is none, sometimes one, sometimes two. Why is that?
> > And why on earth is the link to a discussion page called
> > a namespace? And why is there no Talk page for SpecialPages?
>
> Looks more like a layout problem than anything else. At the time I
installed
> namespaces, listing all of 'em "en blocc" seemed easiest. No reason not to
> change that to something more readable.

How about user friendliness? Is that not a good reason? But the names was
only one point. What matters is that it should be either obvious or easy to
explain why these links are there an what they mean. I believe we can do
better than this.

-- Jan Hidders


from_name: Jimmy Wales

subject: [Wikitech-l] update: new MySQL search index committed to CVS

in_reply_to: <02ae01c1b4e5$8e9abaa0$3020a98f@uia.ac.be>; from hidders@uia.ua.ac.be on Thu, Feb 14, 2002 at 12:24:16AM +0100

body:
One cute trick that I have often used is to calculate the
"uselessness" of a particular word.  A word is semantically more
useless if it appears more often.  This has really dramatic empirical
results for the better, especially on small datasets.  (Maybe on
really big ones, too, but I've never played with those.)

Thus if someone searches for 'John Malkovich' they get a good result,
because 'John' is not weighted so heavily -- it's a more useless word
because it appears more often in the search set.  But 'Malkovich', now
you're talking, there's a word that _means something_.


subject: [Wikipedia-l] Re: Simple XML ideas

body:
> I would actually suggest that we try somehow to change the "look and feel"
> of the special namespaces, so that people will just SEE the distinction as
> they browse through our site.

I think that's what is bugging me about the "()", in a way. Talk pages are
not just a variant of encyclopedia articles, they're something else
entirely, even though they use the same interface and syntax. The hardwired
":" character makes a clear distinction between the pages types, while "()"
does not. There are already some articles with "()" in the name, and I'm
afraid the talk, wikipedia and user pages would blend into each other again
if we stop using the ":".

> The fundamental problem I have with using ()
> both to disambiguate terms within the wikipedia, and to denote
> special pages
> which are not encyclopedia articles, is that we don't want people to
> accidentally miss this distinction and start using the standard
> encyclopedia
> article namespace for user pages, or for talking about articles, etc.

I agree that the special namespaces should be distingushed more from the
encyclopedia pages. Again, this is more a layout problem than anything else.
How about marking the namespace part of the title in red, of italics?

The namespace display should also be updated. The link to the Talk namespace
should probably get a special position, or an icon or something.

I think Larry agrees with me that hardwireing is a *good* thing to sepetate
the actual articles from the "surrounding" pages. That was *one* of the
reasons we made meta.wikipedia.com, right?

Magnus


subject: [Wikitech-l] Slow parser

body:
I just ran "ab n=10" for an atricle (with cache turned off) and deactivated
some functions to see where the slow parts are.

Full rendering : 4.99 sec
removeHTMLtags turned off : 3.319 sec

It seems removeHTMLtags is responsible for 1/3 of the *total* runtime, which
includes apache, php calling, and a thousand other things that can't be
avoided.

So, if these HTML tags are *never* used anyway, why can't we replace them
with < and > just prior to saving an edited article?

I'll be gone tomorrow until Saturday, and I doubt I can hack it today, so
it's up to you...

Magnus


subject: [Wikitech-l] about changing edited text

body:
On mer, 2002-02-13 at 17:00, Chuck Smith wrote:
> My first thought when I read this was to have two
> seperate versions of the page.  One for the server and
> one for the editor.  Basically when someone saves a
> page, this creates an html page and saves a seperate
> wiki page.  This way all processing would be done only
> when someone saves a page rather than everytime
> someone loads a page?
>
> Is there a flaw in my reasoning other than the
> database doubling in size?  I guess it depends on
> which is more important: space or speed?

Nothing wrong with it, except that that's pretty much what we do already
by keeping a cached copy that's been completely converted from wiki
markup to HTML.

I'm not sure what advantage would be gotten out of storing a version
that has had HTML tags worked over, but still needs the wiki code
converted into HTML every time we load it. We get more speed by caching
the completely parsed version, or more storage savings by reparsing it
every time and not storing anything but the the editable text.

Storing the HTML-munged version seems like the worst of both worlds. :)
I'm willing to be convinced, though, if there's a really good reason I
haven't thought of or if I'm misunderstanding the problem.

-- brion vibber (brion @ pobox.com)


subject: [Wikipedia-l] Re: Simple XML ideas

body:
From: "Larry Sanger" <lsanger at nupedia.com>
>
> I think your solution is actually less user-friendly.  We simply need to
> retain namespaces (they work fine, for the most part) and provide unique,
> user-friendly templates (I guess) for pages in different namespaces.

Well, it seems that the battle is over, and I lost. :-) I'm still not really
convinced because I feel that you are seeing threats that are not really
there. Was consistency ever a real problem when we still had subpages? Yes,
some articles contained pieces of discussion that belonged on the Talk
pages, but that was usually solved in time. Also in my proposal the pages
would not exactly behave the same: The talk pages would not have a Talk
link; below non-Talk pages there would be link to a corresponding Talk page;
the system pages would not have an edit link; the user page is where you
arrive when you click your name after the login; and finally there is of
course the contents of the page itself (people read you know :-)). Giving
them a different background color is something that I would not have a
problem with. But the buttons, links, menus, et cetera should basically look
the same, be in the same place and do the same things although some might be
missing or disabled. This is lesson #1 from user interface design: the
interface should not be modal.

Let me make one other final remark: What I am missing a bit at the moment is
an obvious place to put my personal stuff. It's now all in
Jan_Hidders/Whatever because that is where the script put it, but actually
it should be in user:Jan_Hidders/Whatever, or should it? Yes it should,
because it is clearly not an encyclopedia article so it should be in the
user: namespace. Would it be an interesting idea to push new users in the
right direction by, dare I say it, reintroducing subpages (no, please, don't
walk away, hear me out) but limited to home pages only?

Anyway, I had my day in court and I'll leave it at that. I'll be happy to
help implement whatever you guys suggest.

-- Jan Hidders


subject: [Wikitech-l] Slow parser

body:
From: "Magnus Manske" <Magnus.Manske at epost.de>
>
> I'll be gone tomorrow until Saturday, and I doubt I can hack it today, so
> it's up to you...

I wouldn't dare, that's Brion's turf. :-) But my remarks for parsing also
apply here. You guys seem to be really fond of the explode operator, but
it's really not a very good idea.

-- Jan Hidders

PS. At the risc of comitting social suicide, let me admit that, yes, at one
point in my life I have been teaching compiler theory. But it is certainly
not my real area of expertise.


body:
From: "Jimmy Wales" <jwales at bomis.com>

> Jan Hidders wrote:
> > Can I suggest we simply stop with the whole caching thing? It
complicates
> > things unnecesarily. Keeping the code simple should be one of our top
> > priorities. Jimbo doesn't have it turned on at the moment anyway,
>
> I have no strong opinion about this, but I wanted to say that I
> thought I did have it turned on.  If it's off, that's a mistake.

My mistake. I saw that the language links worked, but hadn't realized I was
looking at the page just after I edited it. I see now that you do have it on
because upon reloading the page the language links are missing, so I get
apparently the cached page.

> Tell you what, I'll benchmark with and without, on the live server,
> and report the numbers.

Yes, that would be very very welcome.

-- Jan Hidders


subject: [Wikipedia-l] Re: Simple XML ideas

body:
> Also in my proposal the pages
> would not exactly behave the same: The talk pages would not have a Talk
> link; below non-Talk pages there would be link to a corresponding
> Talk page;
> the system pages would not have an edit link; the user page is where you
> arrive when you click your name after the login; and finally there is of
> course the contents of the page itself (people read you know :-)). Giving
> them a different background color is something that I would not have a
> problem with. But the buttons, links, menus, et cetera should
> basically look
> the same, be in the same place and do the same things although
> some might be
> missing or disabled. This is lesson #1 from user interface design: the
> interface should not be modal.

I don't see anything that can't be made with namespaces as well, mainly by
altering the display of the namespaces. Instead of ":Main Page" on
"Talk:Main Page", it could read "Back to the article".

> Let me make one other final remark: What I am missing a bit at
> the moment is
> an obvious place to put my personal stuff. It's now all in
> Jan_Hidders/Whatever because that is where the script put it, but actually
> it should be in user:Jan_Hidders/Whatever, or should it? Yes it should,
> because it is clearly not an encyclopedia article so it should be in the
> user: namespace. Would it be an interesting idea to push new users in the
> right direction by, dare I say it, reintroducing subpages (no,
> please, don't walk away, hear me out) but limited to home pages only?

We,, the conversion script just couldn't figure out what is user page and
what is article, so all of it got converted the same way. Fell free to
create "User:Jan Hidders/Whatever", just remember to link back to your user
page (~~~).

> Anyway, I had my day in court and I'll leave it at that. I'll be happy to
> help implement whatever you guys suggest.

I am not judging you, and neither is the honorable Larry :]

Magnus


subject: [Wikitech-l] caching question

body:
From: "Jimmy Wales" <jwales at bomis.com>
>
> How can I easily tell if caching is actually on or off?  Am I doing
something wrong
> here?

By the following bug. If a page contains a link like [[fr:test]] then there
should be a special link "Other languages : French (Francais)" on top of the
page just below "Watch this article for me". In the cached version you don't
see this link, in the "fresh" version you see it. You have my permission to
edit and try it

http://www.wikipedia.com/wiki/Jan_Hidders/Sandbox

You will see the link just after the edit, but not if you revisit the page
(first visit another page otherwise you resubmit the edit).

If caching is turned off, then you should always see this link.

-- Jan Hidders


from_name: Chuck Smith

subject: [Wikitech-l] about changing edited text

body:
> > So, if these HTML tags are *never* used anyway,
> why can't we replace them
> > with < and > just prior to saving an edited
> article?
>
> I just have two objections:
>
> First, it violates the principal of least surprise;
> the user doesn't get
> the same thing upon a re-edit that they left during
> the last edit. This
> is particularly annoying for people who are putting
> complicated tables
> into articles (cf. [[Beryllium]] and [[Periodic
> Table]]) -- if they do
> one thing wrong, POOF! Half their table <tags>
> suddenly turn into
> <tags> and instead of fixing one tiny mistake,
> they fix one tiny
> mistake AND change a lot of <>s back into <>s.
>   Conclusion: bad for users.
>
> Second, enforcing the limited subset HTML is just a
> part of the wiki
> parsing. Doing that on save is fine, but is
> basically doing half the
> parsing job and caching that, then doing the other
> half when we display
> the page. Why stop there, when we could just parse
> the wiki-specific
> code while we're at it and save the final result?
>   Conclusion: what exactly is our goal here? To save
> processing time on
> page load? This is most effectively done by caching
> the completely
> parsed version, both HTML and wiki -> HTML.

My first thought when I read this was to have two
seperate versions of the page.  One for the server and
one for the editor.  Basically when someone saves a
page, this creates an html page and saves a seperate
wiki page.  This way all processing would be done only
when someone saves a page rather than everytime
someone loads a page?

Is there a flaw in my reasoning other than the
database doubling in size?  I guess it depends on
which is more important: space or speed?

Chuck

=====
Venu al la senpaga, libera enciklopedio
esperanta reta! http://eo.wikipedia.com/
====
Junuloj!  Filadelfio, Usono 15an-17an de Februaro
http://unumondo.com/cgi-bin/wiki.pl?Filadelfia_JES

_________________________________________________________
Do You Yahoo!?
Informaci?n de Estados Unidos y Am?rica Latina, en Yahoo! Noticias.
Vis?tanos en http://noticias.espanol.yahoo.com


subject: [Wikipedia-l] Page titles to be deleted

body:
In point of fact, sometimes explanations are helpful.  I know this from
experience.

Larry

On Wed, 13 Feb 2002, Ian wrote:

> Really, if you have to explain why it should be deleted then it probably
> shouldn't. All the canidates I've seen for delation have been common
> sense.
>
> Ian Monroe
> http://mlug.missouri.edu/~eean/
>
> On Mon, 11 Feb 2002, Larry Sanger lsanger at nupedia.com XXXXXXXXXXXXXXXXXXXXX wrote:
>
> > Sounds good to me.  In "voting" for deletion, of course it's important to
> > be able explain why one believes the page should be deleted.
> >
> > Larry
> >
> > On Mon, 11 Feb 2002, Magnus Manske wrote:
> >
> > > I thought about having an easy way of expressing "this page is obsolete and
> > > should be deleted". Currently, we have "page titles to be deleted", but it
> > > takes time and effort to go there and edit.
> > >
> > > The obvious solution:
> > > * For logged-in users (troll prevention!), have a link in the sidebar "Mark
> > > this page for deletion" (a shorter title might fit better, but I can't think
> > > of any; maybe a shredder logo?;)
> > > * This adds the current page to "log:Pages to be deleted" (or
> > > "wikipedia:Pages to be deleted")
> > > ** Alternatively, we mark the article in the database, so the page is "PTBD"
> > > page is generated on-the-fly, ensuring it doesn't list pages that already
> > > *have* been deleted
> > > * Additionally, these pages are sorted by votes (descending, the "most
> > > hated" on top;)
> > > * Optionally, list the users that voted for deletion of that page
> > >
> > > Then, a sysop (=Larry) comes along, checks out the list and cleans up behind
> > > us thoughtless page-creators by permanently deleting the pages he deems
> > > obsolete.
> > >
> > > Before I implement any of that, any ideas, comments, flames? ;)
> > >
> > > Magnus
> > >
> > > [Wikipedia-l]
> > > To manage your subscription to this list, please go here:
> > > http://www.nupedia.com/mailman/listinfo/wikipedia-l
> > >
> >
> > [Wikipedia-l]
> > To manage your subscription to this list, please go here:
> > http://www.nupedia.com/mailman/listinfo/wikipedia-l
> >
>
>
> [Wikipedia-l]
> To manage your subscription to this list, please go here:
> http://www.nupedia.com/mailman/listinfo/wikipedia-l
>


from_name: Ian

subject: [Wikipedia-l] Page titles to be deleted

body:
Really, if you have to explain why it should be deleted then it probably
shouldn't. All the canidates I've seen for delation have been common
sense.

Ian Monroe
http://mlug.missouri.edu/~eean/

On Mon, 11 Feb 2002, Larry Sanger lsanger at nupedia.com XXXXXXXXXXXXXXXXXXXXX wrote:

> Sounds good to me.  In "voting" for deletion, of course it's important to
> be able explain why one believes the page should be deleted.
>
> Larry
>
> On Mon, 11 Feb 2002, Magnus Manske wrote:
>
> > I thought about having an easy way of expressing "this page is obsolete and
> > should be deleted". Currently, we have "page titles to be deleted", but it
> > takes time and effort to go there and edit.
> >
> > The obvious solution:
> > * For logged-in users (troll prevention!), have a link in the sidebar "Mark
> > this page for deletion" (a shorter title might fit better, but I can't think
> > of any; maybe a shredder logo?;)
> > * This adds the current page to "log:Pages to be deleted" (or
> > "wikipedia:Pages to be deleted")
> > ** Alternatively, we mark the article in the database, so the page is "PTBD"
> > page is generated on-the-fly, ensuring it doesn't list pages that already
> > *have* been deleted
> > * Additionally, these pages are sorted by votes (descending, the "most
> > hated" on top;)
> > * Optionally, list the users that voted for deletion of that page
> >
> > Then, a sysop (=Larry) comes along, checks out the list and cleans up behind
> > us thoughtless page-creators by permanently deleting the pages he deems
> > obsolete.
> >
> > Before I implement any of that, any ideas, comments, flames? ;)
> >
> > Magnus
> >
> > [Wikipedia-l]
> > To manage your subscription to this list, please go here:
> > http://www.nupedia.com/mailman/listinfo/wikipedia-l
> >
>
> [Wikipedia-l]
> To manage your subscription to this list, please go here:
> http://www.nupedia.com/mailman/listinfo/wikipedia-l
>


subject: [Wikitech-l] about changing edited text

body:
On mer, 2002-02-13 at 17:24, Jimmy Wales wrote:
> Brion Vibber wrote:
> > I'm not sure what advantage would be gotten out of storing a version
> > that has had HTML tags worked over, but still needs the wiki code
> > converted into HTML every time we load it. We get more speed by caching
> > the completely parsed version, or more storage savings by reparsing it
> > every time and not storing anything but the the editable text.
>
> It's worth noting that on the live server, I see no material difference when
> I turn caching on or off.

Interesting. I have to wonder whether this means caching is for some
reason not working at all... It seems to be disabled and/or broken at
the moment, unless someone sneaked in and fixed the other-languages bug
while I wasn't looking.

I ran "ab -n 10" on a couple pages running on my test server with
various states: caching on, caching off w/ no removeHTMLtags() call,
caching off with the old removeHTMLtags() code, and caching off with my
new as-yet unoptimized but more secure version of removeHTMLtags(). The
pages per second figures from three trials each:

Beryllium (large HTML table, various other tags)
* cached 2.06 2.06 2.16
* none   0.94 0.95 0.95
* old    0.90 0.90 0.89
* new    0.47 0.48 0.48

Esperanto-wiki mainpage: (a few <b>, <i>, and <font> tags)
* cached 3.26 3.13 3.47
* none   1.84 1.83 1.76
* old    1.82 1.80 1.80
* new    1.58 1.62 1.58

> Also, space is really cheap these days.  And we're not in any immediate danger
> of running out of it.

Very true.

-- brion vibber (brion @ pobox.com)


body:
L.S.,

I have just committed a rewritten version of the search engine to CVS. It is
now based upon the fulltext indexes as is offered by MySQL. In order to be
able to use this the database schema has to be extended a little. The table
types have changed to myISAM (MySQL own version of ISAM), there is an extra
redundant column, and two fulltext indexes have to be added.. The commands
to do this have been added to "updSchema.sql", just uncomment the commands
you still have to do. The resulting database scheme should be as in
"wikipedia.sql".

This new search engine is somewhat a mixed blessing.

First the good news
+ The main reason for introducing it is that the search queries were
reported as the slowest by the database, so they are eating up a lot of
database resources. So not only will it make searches faster but the whole
of Wikipedia will probably benefit from its introduction.
+ The search engine tries to estimate the relevance of pages wrt. to the
given search words and it is in this order that they are presented to the
user. Not sorted alphabetically, as now.
+ With the arrival of MySQL4 there will be new search possibilities such as
boolean searches and natural language searches.

And now for the bad news.
- I had to introduce an extra redundant column contain a duplicate of
cur_title. This is because the fulltext index cannot be defined on binary
columns.
- If you give the search engine multiple words it will also regards multiple
occurrences of one word already very relevant. So the term "Larry Sanger"
will also lead you to pages with only a lot of "Larry" on it.
- It does not search on small words of three letters or less. So the therm
"war" gives you zero results.
- It searches in the raw HTML so it doesn't know that G?del and Gödel
are the same.

That's it for now. My next task will be the MostWanted Page.

-- Jan Hidders


subject: [Wikitech-l] about changing edited text

body:
On mer, 2002-02-13 at 22:20, Brion Vibber wrote:
> I ran "ab -n 10" on a couple pages running on my test server with
> various states: caching on, caching off w/ no removeHTMLtags() call,
> caching off with the old removeHTMLtags() code, and caching off with my
> new as-yet unoptimized but more secure version of removeHTMLtags(). The
> pages per second figures from three trials each:
>
> Beryllium (large HTML table, various other tags)
> * cached 2.06 2.06 2.16
> * none   0.94 0.95 0.95
> * old    0.90 0.90 0.89
> * new    0.47 0.48 0.48
>
> Esperanto-wiki mainpage: (a few <b>, <i>, and <font> tags)
> * cached 3.26 3.13 3.47
> * none   1.84 1.83 1.76
> * old    1.82 1.80 1.80
> * new    1.58 1.62 1.58

I've eliminated most of the loops from removeHTMLtags(), which is now
only a couple of percent slower than the old function which let
JavaScript through. Which is to say, still at >90% the performance of
not doing live HTML tag removal, but still below a completely parsed
cache.

Beryllium
* cached 2.20 2.13 2.15
* none   1.00 0.98 0.98
* newest 0.92 0.92 0.93

Esperanto-wiki mainpage
* cached 3.50 3.25 3.49
* none   1.87 1.89 1.87
* newest 1.86 1.82 1.80

Still room to optimize though,

-- brion vibber (brion @ pobox.com)


from_name: Neil Harris

subject: [Wikipedia-l] New contribution - would this be useful?

body:
I have just uploaded a file, 1037c-output.txt

It contains the text from Federal Standard 1037C, after the following
operations have been performed:

* stripping of HTML markup (which has left some nasties like 1012 for
1012 )
* tracing of sources based on tags in text
* stripping of items that contain stuff from non-Federal Govt. sources
like NATO or CCITT...
* stripping of bracketed abbreviations from titles
* generated an introductory phrase where possible
* added paragraph breaks in generally sensible places
* added a heading like this {{ title }}
* general wikification (term in bold, etc.)
* classified articles by adding tags like
      # REDIRECT [[whatever]]
      #ONELINER for one-line articles < 120 chars
      #NONTRIVIAL for articles > 200 chars

Approx breakdown (this might be a few versions old)

1241 REDIRECTs
  411 ONELINERs
  799 mid-length items
2783 NONTRIVIALs

It is now in a form where not too much more programming is needed to
shove some or all of it into the Wikipedia.

Would this be useful?

Neil


subject: [Wikitech-l] about changing edited text

body:
On ?a?, 2002-02-14 at 01:33, Jan Hidders wrote:
> From: "Brion Vibber" <brion at pobox.com>
> > Interesting. I have to wonder whether this means caching is for some
> > reason not working at all... It seems to be disabled and/or broken at
> > the moment, unless someone sneaked in and fixed the other-languages bug
> > while I wasn't looking.
>
> It works, otherwise we wouldn't have seen the language-link bug.

Well, I know it used to work on the live server (and the bug was visible
there) and it works now on my test machine (and the bug is visible
here), but right now I go to the live server and I *don't* see the
language bug. Which means either Jimbo left it disabled, or something
mysterious crept into the code that I don't know about.

Oh hey, I just noticed another bug! Wikilinks in the article change
style (lose underline) after an other-language link has been parsed.
That's probably my fault when I switched things over to the style-sheet
code, I'll fix it.

> I'm not the
> least surprised that caching doesn't help much. With large datasets like
> Wikipedia the bottlenecks are usually the database or programming that
> doesn't keep an eye on the big O time/space complexity of its behavior.

Very true. Which is why I don't think removeHTMLtags() is such a big
barrier that we have to do it only on article save. :)

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] about changing edited text

body:
On ?a?, 2002-02-14 at 02:47, Brion Vibber wrote:
> Oh hey, I just noticed another bug! Wikilinks in the article change
> style (lose underline) after an other-language link has been parsed.
> That's probably my fault when I switched things over to the style-sheet
> code, I'll fix it.

Got it. Also threw in the look-for-otherlanguage-links-when-cached fix,
and put in the rest of the URL in $wikiInterwiki so that links to
meta.wikipedia.com in the form [[m:Article Title]] work instead of
directing to the mainpage there.

-- brion vibber (brion @ pobox.com)


in_reply_to: <3C6B0E71.7060201@tonal.clara.co.uk>; from usenet@tonal.clara.co.uk on Thu, Feb 14, 2002 at 01:10:09AM +0000

body:
Neil Harris wrote:
> I have just uploaded a file, 1037c-output.txt
>
> It contains the text from Federal Standard 1037C, after the following
> operations have been performed:

Just in case other people are as clueless as me, what this is:
GLOSSARY OF TELECOMMUNICATION TERMS

Most of the terms appear to be short definitions of things like
"harmful interference"
http://www.its.bldrdoc.gov/fs-1037/dir-017/_2541.htm

The copyright situation looks good: this is a publication of the
U.S. Federal Government, as a standard, and therefore by law not
copyright.

The content, I'll leave the judgment to other people.

--Jimbo


subject: [Wikitech-l] about changing edited text

in_reply_to: <1013683642.7981.0.camel@rdaneel.moisty.org>; from brion@pobox.com on Thu, Feb 14, 2002 at 02:47:21AM -0800

body:
Brion Vibber wrote:
> Well, I know it used to work on the live server (and the bug was visible
> there) and it works now on my test machine (and the bug is visible
> here), but right now I go to the live server and I *don't* see the
> language bug. Which means either Jimbo left it disabled, or something
> mysterious crept into the code that I don't know about.

I left it disabled because it wasn't helping (didn't seem to be anyway) and
because of the bug it introduced, although that's pretty minor.  I can turn it
back on.

But, like you, I'm wondering if maybe it wasn't _really_ on.  I will
test for that today by reproducing the bug to "prove" the caching is
on.

--Jimbo


from_name: wikipedia at epoptic.org

subject: [Wikipedia-l] Search Engine bug report

body:
I have overhauled the [[Cologne]] article, working around Wiki bugs and
getting it to render much better.  Once the search engine updates, we'll
see if it fixes the problem there too.

    Foo
    Bar

does not render properly.

Foo
Bar

renders fine -- the initial indentation spaces cause the screw up.

--the Epopt

-----Original Message-----
From: wikipedia-l-admin at nupedia.com
[mailto:wikipedia-l-admin at nupedia.com]On Behalf Of Gareth Owen
Sent: Thursday, February 14, 2002 04:12
To: wikipedia-l at nupedia.com
Subject: [Wikipedia-l] Search Engine bug report

Type "Miles Davis" into the search engine.
View the source and look for "Cologne"

  08

      Cologne

Now I don't know why, but there is an unclosed empty
 entity in
there,
which is causing the search results to not render at all on Netscape 3
and 4.
--
Gareth Owen
"Wikipedia does rock.  By the count on the "brilliant prose" page, there
 are 14 not-bad articles so far" -- Larry Sanger (12 Jan 2001)

[Wikipedia-l]
To manage your subscription to this list, please go here:
http://www.nupedia.com/mailman/listinfo/wikipedia-l


subject: [Wikitech-l] Boolean search

body:
Boolean searching would be cool.  If not that, then at least "search
within results."

Larry


subject: [Wikitech-l] about changing edited text

body:
From: "Brion Vibber" <brion at pobox.com>
>
> Interesting. I have to wonder whether this means caching is for some
> reason not working at all... It seems to be disabled and/or broken at
> the moment, unless someone sneaked in and fixed the other-languages bug
> while I wasn't looking.

It works, otherwise we wouldn't have seen the language-link bug. I'm not the
least surprised that caching doesn't help much. With large datasets like
Wikipedia the bottlenecks are usually the database or programming that
doesn't keep an eye on the big O time/space complexity of its behavior.

-- Jan Hidders


subject: [Wikipedia-l] Re: Simple XML ideas

body:
From: "Mark Christensen" <mchristensen at HTEC.com>
>
> [...] These articles might very
> well include () style disambiguation, at some point
> we'd clearly want a stable version of
> "Lincoln (Nebraska)." This and this would
> would make using () style namespaces for the stable
> branch of the wikipedia "codebase" very confusing.

Ah, now that I *do* find a good argument.

-- Jan Hidders


from_name: Gareth Owen

subject: [Wikipedia-l] Search Engine bug report

body:
Type "Miles Davis" into the search engine.
View the source and look for "Cologne"

  08

      Cologne

Now I don't know why, but there is an unclosed empty
 entity in there,
which is causing the search results to not render at all on Netscape 3 and 4.
--
Gareth Owen
"Wikipedia does rock.  By the count on the "brilliant prose" page, there
 are 14 not-bad articles so far" -- Larry Sanger (12 Jan 2001)


subject: [Wikitech-l] Sourcing in wikiTextEn.php later

body:
On ?a?, 2002-02-14 at 11:48, Axel Boldt wrote:
> With my new checkin permission, I went to work happy as a clam.
>
> I added a README file and the GPL license in COPYING.

Yay!

> The file wikiTextEn.php uses the $THESCRIPT variable, and it therefore
> has to be included at the end of wikiSettings.php, after
> wikiLocalSettings.php has been read.

Oops! Thanks for catching that.

> I made a new variable
> $wikiLanguage which can be set in wikiLocalSettings and which
> determines which wikiText file is included later.

Note that we should always include wikiTextEn.php before another
language file, as it defines a number of common things that will rarely
have to be changed ($wikiInterwiki and $wikiOtherLanguages, which
arguably should be defined elsewhere, and the stub wikiRecode*
functions) and provides a default English fallback for any new strings
that are added to the software but haven't been translated yet.

Also, I'd prefer to use the standard lowercase language codes (that's
what we should be putting in the Content-language header etc), and do a
ucfirst() for loading the file. Hope you don't mind, I've checked in
such a change.

I've also checked in the Esperanto translation file (wikiTextEo.php) as
an example. For that matter, we're just about ready to go live with the
new software on eo.wikipedia.com if Jimbo has no objections. There are
some bugs in the old software that are... well... bugging us (certain
UTF-8 sequences don't work in page titles, and capitalization is all
wrong). Nobody ever looks at the test server (ain't it the truth?), but
putting up the new software on the live server should get us a
reasonably active set of users who should notice most remaining i18n
bugs before we get ready to let it loose on other languages.

I'll toss in the internationalized convertWiki2SQL.php when I've checked
that it works with the latest version of the database structure.

-- brion vibber (brion @ pobox.com)


subject: [Wikipedia-l] Re: Simple XML ideas

body:
From: "Magnus Manske" <Magnus.Manske at epost.de>
>
> We,, the conversion script just couldn't figure out what is user page and
> what is article, so all of it got converted the same way. Fell free to
> create "User:Jan Hidders/Whatever", just remember to link back to your
user
> page (~~~).

Of course, my point was that people apparently have a need to create a few
pages for themselves. These pages should be in the user namespace. So it
should be tempting and obvious for the user to create his or her pages there
when on it's own home page.

The usual semantics of namespaces is that by default, i.e., when no
namespace is indicated, links stay within that namespace. Note how nicely
that fits in, for example, with special frozen articles that, by default,
link to other frozen articles. (But with some redirection of course, if
there is no such frozen version at the end of the link.) That is actually
what you would like for the user namespace, because then (when you are on
your home page) it would take some effort to create a page that is outside
that name space. But this is a bit cumbersome because the home pages and the
other personal pages of all the users would be in the same name space. This
suggests to me that every user should have his or her own namespace.

-- Jan Hidders


subject: [Wikipedia-l] Search Engine bug report

in_reply_to: <wikipedia@epoptic.org>'s message of "Thu, 14 Feb 2002 09:48:20 -0800"

body:
<wikipedia at epoptic.org> writes:

> Once the search engine updates, we'll see if it fixes the problem there too.

Whilst this may well fix it, I think the search engine needs to watch for
these things in future.  The indentation in my mail was added by me, the real
search engine page didn't have much whitespace.
--
Gareth Owen
"Wikipedia does rock.  By the count on the "brilliant prose" page, there
 are 14 not-bad articles so far" -- Larry Sanger (12 Jan 2001)


from_name: Axel Boldt

body:
With my new checkin permission, I went to work happy as a clam.

I added a README file and the GPL license in COPYING.

The file wikiTextEn.php uses the $THESCRIPT variable, and it therefore
has to be included at the end of wikiSettings.php, after
wikiLocalSettings.php has been read. I made a new variable
$wikiLanguage which can be set in wikiLocalSettings and which
determines which wikiText file is included later.

Axel


subject: [Wikitech-l] Problems with saving changes?

body:
I am running the current CVS version locally and updated my database
scheme. Search, RecentChanges, edit and preview works fine, but if I
try to save my changes, they are all lost and the file is back to the
previous version. It doesn't happen for all files, and I don't know
what the difference is (two example problem files are the main page
and "British Queen"). It's not a browser cache issue, since when I try
to edit the page again, the wiki text is also reverted to the old
version. I can write to the database, since the page counters change
correctly. Does anybody see that problem?

Axel


from_name: Simon Kissane

subject: [Wikipedia-l] In case anyone has noticed I haven't been around much

body:
For the last month or more I have not had the time to
do anything to do with Wikipedia due to other
commitments. I probably won't for some time yet
either. (I'm going on a holiday to Thailand, then uni
starts and I better get busy with that.) I'm not
"dropping out" though -- I will come back latter when
I have more time on my hands.

Just in case anyone was wondering, and good luck in my
absence,
Simon.

__________________________________________________
Do You Yahoo!?
Got something to say? Say it better with Yahoo! Video Mail
http://mail.yahoo.com


subject: [Wikitech-l] Sysop status

body:
For a few glorious days, I had sysop status.  I no longer do.  Magnus
knows at least a little about the problem, but if any of you know as well
and can fix the problem, that would be great.

The problem might be related to a login problem we still have.  When I go
to any page and press "Log out" and then "Log in", I come to a page that
says: "User Larry_Sanger1, you are already logged in!"

If I then attempt to log in as "Larry Sanger", which I *suspect* (don't
know) has sysop status (whereas "Larry_Sanger" does not), it tells me:
"Larry Sanger, you are logged in!"  Moreover, the link in the upper right
hand corner is to "Larry Sanger"'s homepage.  Hopefully, I then go back to
the Main Page only to find that I am once again "Larry_Sanger" (so says
the upper right corner link) and do not have sysop privileges.


subject: [Wikipedia-l] Search Engine bug report

body:
From: "Gareth Owen" <wiki at gwowen.freeserve.co.uk>

> <wikipedia at epoptic.org> writes:
>
> > Once the search engine updates, we'll see if it fixes the problem there
too.
>
> Whilst this may well fix it, I think the search engine needs to watch for
> these things in future.

Now it does. All allowed HTML is neutralized. Fix is in CVS.

-- Jan Hidders


from_name: Vicki Rosenzweig

subject: [Wikipedia-l] The "most wanted" page

body:
Is someone working on making this useful again. I decided to try refreshing
the page, despite
the warning at the top, and got the following message:

(The page was last refreshed just -154694 minutes ago; please wait another
154699 minutes and try again.)

That was in italics; the parentheses are from the original. Requiring five
minutes between refreshes
seems entirely reasonable, even generous--except that its clock is clearly off.
--
Vicki Rosenzweig
vr at redbird.org
http://www.redbird.org


subject: [Wikipedia-l] The "most wanted" page

body:
On dim, 2002-02-17 at 02:10, Jan Hidders wrote:
> From: "Vicki Rosenzweig" <vr at redbird.org>
> > Is someone working on making this useful again. I decided to try
> > refreshing the page,
>
> Yes, I am, but right now I first have to finish the search page.

I've already fixed it, but Jimbo hasn't installed the updated file yet.

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] update: boolean search is in CVS

body:
Wow, thanks Jan!

Larry

On Sun, 17 Feb 2002, Jan Hidders wrote:

> Dear fellow programmers,
>
> Boolean search has been implemented and committed to CVS. You can now use
> the keywords "and",  "or" and "not" in your queries such as "Harry and not
> Potter". The and operator is implicit so you also omit it as in "Harry not
> Potter". So the query "group theory" is interpreted as "group AND theory".
> The priority of the "not" is the highest, then the "and" and finally the
> "or". You can use brackets to alter this, for example as in "Harry and (
> Potter or Sally )".
>
> A few problems remain:
> 1. The index indexes only words with more than three characters. So searches
> for small words will always give empty result. This means that a query like
> "World war" which is interpreted as "World AND war" give always an empty
> result. I've made this a syntax error to warn the innocent user. However,
> the minimun index word size can be changed to 2, but we would have to
> recompile MySQL for that and rebuild the indexes.
> 2. The index has a fixed list of stopwords that also always give an empty
> query result. This can also be changed in the source code. The MySQL
> matching algorithm also takes into account that certain words are more
> critical than others. While this more a feauture than a bug, it also means
> that searches for very common words actually result in a very small number
> of pages, or even none at all. Als this behavior can be changed by setting
> some constants in the source and recompiling.
> 3. If you have a complex boolean query the search box seems a bit small. As
> this is a lay-out matter I leave this to Magnus to change. :-)
>
> Finally, I noticed that the search performs better if the title contains no
> "_"s because the index considers "Larry_Sanger" as one word that does not
> match very well with either "Larry" or "Sanger". To remedy this I have
> change the code a bit so that new titles are always stored in the indexed
> column without underscore's. However, for the old pages you still need to
> update this by hand. I have included the statement in 'updSchema.sql'.
>
> Enjoy,
>
> -- Jan Hidders
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at ross.bomis.com
> http://ross.bomis.com/mailman/listinfo/wikitech-l
>


subject: [Wikipedia-l] The "most wanted" page

body:
From: "Vicki Rosenzweig" <vr at redbird.org>

> Is someone working on making this useful again. I decided to try
refreshing
> the page,

Yes, I am, but right now I first have to finish the search page.

-- Jan Hidders


subject: [Wikipedia-l] The "most wanted" page

body:
From: "Brion Vibber" <brion at pobox.com>

> On dim, 2002-02-17 at 02:10, Jan Hidders wrote:
>
> I've already fixed it, but Jimbo hasn't installed the updated file yet.

I know. I meant giving it a decent response time. :-)

-- Jan Hidders


from_name: Michel Clasquin

subject: [Wikipedia-l] The "most wanted" page

body:
On Sunday 17 February 2002 02:01, Vicki Rosenzweig wrote:
> Is someone working on making this useful again. I decided to try
> refreshing the page, despite
> the warning at the top, and got the following message:
>
> (The page was last refreshed just -154694 minutes ago; please wait
> another 154699 minutes and try again.)

So what's the problem? That's only 107.4 days to wait ... <g>

--
Michel Clasquin, D Litt et Phil (Unisa)
clasqm at mweb.co.za/unisa.ac.za   http://www.geocities.com/clasqm
This message was posted from a Microsoft-free PC

...hey, at least a wild goose chase gives you some exercise-Neal Stephenson


body:
Dear fellow programmers,

Boolean search has been implemented and committed to CVS. You can now use
the keywords "and",  "or" and "not" in your queries such as "Harry and not
Potter". The and operator is implicit so you also omit it as in "Harry not
Potter". So the query "group theory" is interpreted as "group AND theory".
The priority of the "not" is the highest, then the "and" and finally the
"or". You can use brackets to alter this, for example as in "Harry and (
Potter or Sally )".

A few problems remain:
1. The index indexes only words with more than three characters. So searches
for small words will always give empty result. This means that a query like
"World war" which is interpreted as "World AND war" give always an empty
result. I've made this a syntax error to warn the innocent user. However,
the minimun index word size can be changed to 2, but we would have to
recompile MySQL for that and rebuild the indexes.
2. The index has a fixed list of stopwords that also always give an empty
query result. This can also be changed in the source code. The MySQL
matching algorithm also takes into account that certain words are more
critical than others. While this more a feauture than a bug, it also means
that searches for very common words actually result in a very small number
of pages, or even none at all. Als this behavior can be changed by setting
some constants in the source and recompiling.
3. If you have a complex boolean query the search box seems a bit small. As
this is a lay-out matter I leave this to Magnus to change. :-)

Finally, I noticed that the search performs better if the title contains no
"_"s because the index considers "Larry_Sanger" as one word that does not
match very well with either "Larry" or "Sanger". To remedy this I have
change the code a bit so that new titles are always stored in the indexed
column without underscore's. However, for the old pages you still need to
update this by hand. I have included the statement in 'updSchema.sql'.

Enjoy,

-- Jan Hidders


subject: [Wikitech-l] repeated mysql_select_db() calls

body:
Hi,

I started my mysqld with --log and found that we are constantly and
pointlessly selecting the wiki database. I combined all those calls in
getDBconnection() and made sure that we only call mysql_select_db()
and mysql_pconnect() when we really need to (i.e. once), using a new
global $wikiDBconnection.

Axel


subject: [Wikitech-l] Versions of software

body:
What versions of apache, php and mysql is everybody using? I want to
include the most appropriate versions in the README.

Myself:
# ../bin/mysqld --version
../bin/mysqld  Ver 4.0.0-alpha for pc-linux-gnu on i686

http://127.0.0.1/server-status
Server Version: Apache/1.3.20 (Unix) PHP/4.0.6

I suspect that my intermittent database problems have to do with the
4.0.0 mysql and its handling of fulltext indices. I'll downgrade
shortly.

Axel


subject: [Wikitech-l] Versions of software

in_reply_to: <200202180451.FAA11481@linde.uni-paderborn.de>; from axel@uni-paderborn.de on Mon, Feb 18, 2002 at 05:51:03AM +0100

body:
Axel Boldt wrote:
> What versions of apache, php and mysql is everybody using? I want to
> include the most appropriate versions in the README.

The main server...

/usr/local/libexec/mysqld  Ver 3.23.37 for pc-linux-gnu on i686

Server version: Apache/1.3.23 (Unix)
Server built:   Feb  6 2002 21:27:43

(Also relevant is httpd -l:)

[root at ross /root]# /apache/bin/httpd -l
Compiled-in modules:
  http_core.c
  mod_env.c
  mod_log_config.c
  mod_mime.c
  mod_negotiation.c
  mod_status.c
  mod_include.c
  mod_autoindex.c
  mod_dir.c
  mod_cgi.c
  mod_asis.c
  mod_imap.c
  mod_actions.c
  mod_speling.c
  mod_userdir.c
  mod_alias.c
  mod_rewrite.c
  mod_access.c
  mod_auth.c
  mod_proxy.c
  mod_setenvif.c
  mod_fastcgi.c
  mod_php4.c
suexec: disabled; invalid wrapper /usr/local/apache/bin/suexec

And finally php

[root at ross /root]# php -v
4.0.4pl1

--Jimbo


subject: [Wikitech-l] Versions of software

body:
----- Original Message -----
From: "Axel Boldt" <axel at uni-paderborn.de>
To: <wikitech-l at nupedia.com>
Sent: Monday, February 18, 2002 5:51 AM
Subject: [Wikitech-l] Versions of software

> What versions of apache, php and mysql is everybody using? I want to
> include the most appropriate versions in the README.
>
> Myself:
> # ../bin/mysqld --version
> ../bin/mysqld  Ver 4.0.0-alpha for pc-linux-gnu on i686
>
> http://127.0.0.1/server-status
> Server Version: Apache/1.3.20 (Unix) PHP/4.0.6

MySQL: Ver 3.23.47-max-debug for Win95/Win98 on i32

Apache: Apache for Windows 95/NT, Apache Version Apache/1.3.20, Apache
Release 10320100, Apache API Version 19990320

PHP: PHP Version 4.1.1, System Windows 95/98 4.10, Build Date Dec 30 2001,
Server API Apache

-- Jan Hidders


subject: [Wikitech-l] update: two new extra tables with linking information

body:
Dear fellow programmers,

I have extended the database schema with two new tables: 'linked' and
'unlinked'. As usual the SQL for this addition can be found in
updSchema.sql. However, in this case the contents of the tables cannot be
generated by SQL alone, so there is an extra upLinks.php script in PHP that
contains the PHP code to do so. Read this file for further instructions.

The intention of these tables is to replace the 'cur_linked_links' and
'cur_unlinked_links' columns in the cur table. This will make it possible to
give the special pages that query linking information reasonable response
times, so they won't have to be cached. Right now I've only added the code
to keep these tables up-to-date and they are not used yet. From this moment
on the usage of the 'cur_linked_links' and 'cur_unlinked_links' columns is
depreciated, and I will remove them as soon as all the code that uses them
has been replaced by code that uses the new tables.

-- Jan Hidders


from_name: lcrocker at nupedia.com

subject: [Wikipedia-l] Great news for the public domain

body:
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://lists.wikimedia.org/pipermail/wikipedia-l/attachments/20020219/680f3c5c/attachment.diff


subject: [Wikitech-l] Moving intl wikis to the PHP script?

body:
Jimbo, is there any chance we can move the Esperanto wikipedia over to
the PHP script soon? I've been promising people we'd be upgrading to the
new software (which will fix a number of annoying bugs in the old) for a
while, and the natives are getting restless. :)

I'm going to check in a couple more character set and case-conversion
fixes tonight, after which we should be ready anytime. At this point any
additional problems are only going to be discovered by having real users
bang at the real site with real non-English non-ISO-8859-1 text...

There are a few switches at the top of convertWiki2SQL.php for selecting
language-specific processing options, and wikiLocalSettings.php needs to
select the proper $wikiLanguage and $wikiSQLServer, but that's about it.
Other than those two, the PHP source files can be shared 100% between
various language wikipedias. (I've already included the Esperanto
message-localization file in the CVS repository, and I assume others
will be added as they are converted & translated.)

-- brion vibber (brion @ pobox.com)


body:
Woo-hoo!

On Tue, 19 Feb 2002 lcrocker at nupedia.com wrote:

> I've been away for a while as I've been setting up my new personal
> server, but I had to interject a bit of very good news for those who
> may not have heard or may not have understood the whole
> ramificiations of this:
>
> The Supreme Court has granted certiorari in the case of "Eldred v.
> Ashcroft" (originally Eldred v. Reno), which seeks to challenge the
> Sonny Bono copyright term extension act (an act of Congress intended
> to encourage dead authors to write more :-) as unconstitutional, and
> indeed seeks to overturn _all_ such term extensions.  This is not a
> ruling--the court has merely agreed to hear the case; but that in
> itself is a big step.  The court hasn't heard a major copyright case
> since 1985, and the fact that they took this one on is evidence that
> they think it will set important precedents.  If they do rule
> favorably, _thousands_ of older out-of-print works from dead authors
> may become available to us  and to other Internet projects.  Cross
> your fingers.
> 0


subject: [Wikitech-l] Moving day

body:
Brion Vibber wrote:
> Jimbo, is there any chance we can move the Esperanto wikipedia over to
> the PHP script soon? I've been promising people we'd be upgrading to the
> new software (which will fix a number of annoying bugs in the old) for a
> while, and the natives are getting restless. :)

Yes!

> I'm going to check in a couple more character set and case-conversion
> fixes tonight, after which we should be ready anytime. At this point any
> additional problems are only going to be discovered by having real users
> bang at the real site with real non-English non-ISO-8859-1 text...

How about this -- tomorrow morning, I will install this.  Will you be around
(in email) tomorrow for questions?

Best thing to do -- send me simple step by step instructions,
including instructions about the conversion script.  I'll back
everything up, run the conversion, install the new software, and
it'll work perfectly the first try! (ha ha!)

If it isn't working perfectly right out of the box, then I'll back out
the change, revert to the Usemod script, and we'll do a dry run in a
"safer" way with test.wikipedia.com or whatever.

> There are a few switches at the top of convertWiki2SQL.php for selecting
> language-specific processing options, and wikiLocalSettings.php needs to
> select the proper $wikiLanguage and $wikiSQLServer, but that's about it.
> Other than those two, the PHP source files can be shared 100% between
> various language wikipedias. (I've already included the Esperanto
> message-localization file in the CVS repository, and I assume others
> will be added as they are converted & translated.)

Cool!  I don't know anything about the conversion script, though.  Jason did it
for the main site, and he's out of town.  Oh, wait, he'll be back tomorrow.  But
still, if you have information, let me know.  :-)

----- End forwarded message -----


subject: [Wikitech-l] Re: [Intlwiki-l] Moving intl wikis to the PHP script?

body:
On mar, 2002-02-19 at 16:26, Jimmy Wales wrote:
> Brion Vibber wrote:
> > Jimbo, is there any chance we can move the Esperanto wikipedia over to
> > the PHP script soon? I've been promising people we'd be upgrading to the
> > new software (which will fix a number of annoying bugs in the old) for a
> > while, and the natives are getting restless. :)
>
> Yes!
>
> > I'm going to check in a couple more character set and case-conversion
> > fixes tonight, after which we should be ready anytime. At this point any
> > additional problems are only going to be discovered by having real users
> > bang at the real site with real non-English non-ISO-8859-1 text...
>
> How about this -- tomorrow morning, I will install this.  Will you be around
> (in email) tomorrow for questions?

Sounds great! If I remember correctly, we are both on Pacific time
(UTC-8), yes?

> Best thing to do -- send me simple step by step instructions,
> including instructions about the conversion script.  I'll back
> everything up, run the conversion, install the new software, and
> it'll work perfectly the first try! (ha ha!)

It's relatively simple (famous last words). For everybody's reference:

1. Edit convertWiki2SQL.php to set some options there. Right now it's a
little rough, eventually it may or may not get smoother. Basically,
uncomment the special settings for the target language, and set the
$rootDir variable to point to the "page" subdirectory of the usemod db
that the data is being sucked from.

2. Run "php convertWiki2SQL.php" as a user with write permission to the
wiki directory. This should spit out a bunch of article titles and
create a big file called "newiki.sql" which contains the SQL commands to
insert everything into the database. Note that it's normal to see a few
errors about not being able to open a directory -- that just means
there's a letter of the alphabet that no article titles start with.

3. Create the database. I've been calling my test database "wikieo", but
whatever sounds appropriate should work just as well. Something like:
  mysql -e "create database wikieo;"
should do it if I recall correctly. You might also have to specify the
proper username and password, I don't know how you guys have mysql set
up.

4. Initialize the tables and enter the data. Something like:
  mysql wikieo < wikipedia.sql
  mysql wikieo < newiki.sql
(After this step "newiki.sql" shouldn't be necessary anymore.) I'm also
going to put together a file with SQL commands to fix some articles with
extra uppercase letters in the titles which you can run here:
  mysql wikieo < titlefix.sql

5. Edit wikiLocalSettings.php to set the language and database name.
Roughly:
  $wikiLanguage = "eo" ;
  $wikiSQLServer = "wikieo" ;
and if the hostname isn't being automatically picked up:
  $wikiCurrentServer = "http://eo.wikipedia.com" ;

In theory everything should automagically work after that... (Assuming
of course that the apache rewrite rules are set up properly, etc.)

> If it isn't working perfectly right out of the box, then I'll back out
> the change, revert to the Usemod script, and we'll do a dry run in a
> "safer" way with test.wikipedia.com or whatever.

Great, I'll warn the others. :)

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] update: the new MostWanted page has arrived

body:
Gentlemen,

I've just committed to CVS the new implementation of the MostWanted page
based on the new tables 'linked' and 'unlinked'. I've left caching out but
it still takes quite a while to compute. If this turns out to be a problem I
will program it back in.

-- Jan Hidders


body:
On mar, 2002-02-19 at 13:14, Jan Hidders wrote:
> Gentlemen,
>
> I've just committed to CVS the new implementation of the MostWanted page
> based on the new tables 'linked' and 'unlinked'. I've left caching out but
> it still takes quite a while to compute. If this turns out to be a problem I
> will program it back in.

Cool! I'll try hacking the new tables into the conversion script in the
morning when I'm awake again. In the meantime, the update script should
work on a fresh, newly converted database as well.

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] New case conversion functions

body:
I've noticed that the traditional locale-based case conversion functions
(ucfirst(), strtolower(), etc) aren't too reliable for anything but
English. Even when they do work, it's very dependant on the system
configuaration, and thus isn't really transparently portable.

So, I've added new case conversion functions ucfirstIntl(),
strtoupperIntl(), and strtolowerIntl() which can more or less properly
convert cases in a system-independent manner. For single-byte character
encodings this is very simple, based on the PHP strtr() function; just
define strings $wikiUpperChars containing all the uppercase characters
and $wikiLowerChars containing all the lowercase chars. (See example for
iso-8859-1 in wikiTextEn.php)

For multibyte character sets it's a little more complex, using the same
function in an array mode that associates byte sequences. Most multibyte
character sets are for Asian languages which don't have a case
distinction, so it's not likely to come up often except for those using
UTF-8. I've included conversion arrays for UTF-8 in utf8Case.php which
should cover just about everything, so any future 'pedias that may use
UTF-8 need just include that (as does wikiTextEo.php).

Also, it should be possible to extend ucfirstIntl() a bit to allow for
multiple-character first letter sequences (for instance treating ij->IJ
as one letter, which I believe is the officially correct behavior for
Dutch).

-- brion vibber (brion @ pobox.com)


subject: [Wikipedia-l] Plea for help

body:
This is an unusual plea for help.  It's possible I'm going out of my
mind--I mean, I guess I don't think so, but sometimes it seems like it.
:-)

Occasionally in the past, controversy has erupted that has occupied
unfortunately large amounts of my time.  Sometimes the controversy is
definitely worth the time, but just as often it seems it isn't, not at
all.  Most recently, in the wake of my announcement that I am no longer a
paid employee, that I will still be working though as a volunteer, and
that we might (if we can) start selling ads and soliciting donations to
help pay for me once again, apparently some people reacted not by saying,
very kindly, "Oh, poor Larry, whatever will we do without him?" but
instead, "Huh?  There was a paid employee?  And they might sell ads?!
Someone was making money?!  This must be a greedy capitalist exploitative
project.  This shouldn't be!  We've got to do something about it!"

Well, I and sometimes Jimbo then spend a lot of time putting out fires.
In the present case, we inform the newcomers that, yes, I did do a few
useful things for the projects oh maybe occasionally; yes, the idea of
fees for viewing content is absolutely ludicrous and has never been
considered; yes, we *really, really are* committed to making and keeping
the content always completely free forever; and yes, we have been
considering making a nonprofit foundation for Wikipedia and Nupedia for a
long time now; it's mainly a matter of getting enough time to actually
start us formally down the road.  (We can start with a "OK now, we're
really going to do it, it's official, we're asking for pro bono legal help
to set it up" kind of announcement, but we don't want to do that until we
have the time to follow through properly.  "We" in this case mainly means
Jimbo.)

I think some people just sit back and enjoy the show, and figure that I
can hold my own well enough.

Well, er, not exactly.  I can, usually, if I spend the time.  But I don't
have the time anymore.  I literally *don't have the time* for this kind of
nonsense.  I've spent, I don't know, three hours so far today working on
various Wikipedia stuff, and it's not even 1 PM yet.  I should instead
have been, er, looking for a job, or a lot of other things my wife wants
me to do.  :-)

So, please, critics, have mercy--be gentle--and, non-critics, if you
support my cause and think you can supply the correct response to an
e-mail that, you can predict, I will want to have answered, please do
that.  Don't hold back waiting for me to do it.  I'll be very grateful.

Larry


body:
On mer, 2002-02-20 at 11:33, lcrocker at nupedia.com wrote:
> No! No!  The text stored in the database is _always_ single-byte
> ISO-8859-1, no exceptions, even for the foreign wikis.  Some of
> those ISO-8859-1 characters may spell out HTML entity references
> to Unicode characters outside the set, but the database should not
> know or care about that.

I'm sorry you feel that way, but that is in fact NOT TRUE. Please take a
look at the non-English non-ISO-8859-1 wikipedias sometime.

Hundreds of pages, with correct charset headers:
  ISO-8859-2:
    http://pl.wikipedia.com/
  UTF-8 with a custom conversion function for certain character
  sequences:
    http://eo.wikipedia.com/

Stubs:
  CP-1251
    http://ru.wikipedia.com/
  Shift-JIS
    http://ja.wikipedia.com/
  GB-2312 with a few character references thrown in:
    http://zh.wikipedia.com/
  Not sure which encodings, but certainly not ISO-8859-1:
    http://ar.wikipedia.com/
    http://he.wikipedia.com/

Now, if you honestly think that people are going to edit text that
consists *entirely* of HTML character entity references, you're
obviously not concerned about anything like "ease of use".

On top of which, the consensus seems to be to not allow &s (and thus
character entities) into page titles, which would effectively require
all page titles to be in ASCIIized roman characters. Can you imagine
this being acceptable on, say, the Chinese wiki if anyone actually used
it?

Gee, maybe someone *would* use it if they could use an appropriate
character set for their language!

> This policy might have to be changed for the Asian wikis if something
> like shift-JIS is universal enough and dealing with HTML entities
> problematic enough to make working with it difficult,

The mind boggles that you might imagine the situation to be otherwise.

> but in that
> case we'll still standardize on one and only one internal character
> representation for that particular wiki.  For all others, that
> internal representation (and also the encoding which is served via
> HTTP) is ISO-8859-1.

Bullshit. Ask the Poles if they'd like to convert their wikipedia to
ISO-8859-1 with HTML character entities.

> If you need to "uppercase" words in titles (as our consensus on
> canonization of titles specifies), go ahead and hard-code the
> function to deal with ISO-8859-1.

Gee, that would be great if such a function would do anything at all for
anything other than ISO-8859-1 characters. But, somehow I can't quite
see a function hardcoded to deal with ISO-8859-1 being the slightest bit
useful for anything else.

-- brion vibber (brion @ pobox.com)

> You Wrote:
> >I've noticed that the traditional locale-based case conversion
> functions
> >(ucfirst(), strtolower(), etc) aren't too reliable for anything but
> >English. Even when they do work, it's very dependant on the system
> >configuaration, and thus isn't really transparently portable.
> >
> >So, I've added new case conversion functions ucfirstIntl(),
> >strtoupperIntl(), and strtolowerIntl() which can more or less
> properly
> >convert cases in a system-independent manner. For single-byte
> character
> >encodings this is very simple, based on the PHP strtr() function;
> just
> >define strings $wikiUpperChars containing all the uppercase
> characters
> >and $wikiLowerChars containing all the lowercase chars. (See example
> for
> >iso-8859-1 in wikiTextEn.php)
> >
> >For multibyte character sets it's a little more complex, using the
> same
> >function in an array mode that associates byte sequences. Most
> multibyte
> >character sets are for Asian languages which don't have a case
> >distinction, so it's not likely to come up often except for those
> using
> >UTF-8. I've included conversion arrays for UTF-8 in utf8Case.php
> which
> >should cover just about everything, so any future 'pedias that may
> use
> >UTF-8 need just include that (as does wikiTextEo.php).
> >
> >Also, it should be possible to extend ucfirstIntl() a bit to allow
> for
> >multiple-character first letter sequences (for instance treating ij-
> >IJ
> >as one letter, which I believe is the officially correct behavior for
> >Dutch).
> >
> >-- brion vibber (brion @ pobox.com)
> >
> >_______________________________________________
> >Wikitech-l mailing list
> >Wikitech-l at ross.bomis.com
> >http://ross.bomis.com/mailman/listinfo/wikitech-l
> >0


subject: [Wikitech-l] Re: [Intlwiki-l] The Distributed Wikipedia

in_reply_to: <20020220211726.GA9768@tavaiah>; from taw@users.sourceforge.net on Wed, Feb 20, 2002 at 10:17:26PM +0100

body:
(I set the reply-to for wikitech-l at nupedia.com, since that's the appropriate
place for technical discussions.)

Tomasz Wegrzanowski wrote:
> Easiest distributed editing architecture:
> There is main server and other servers
>
> Every server handle read request itself.
> On all servers 'edit this page' links point to main server.
> Main server sends all changes to all subscribed servers,
> so they are always up to date.
>
> This won't require many changes in design, while
> may allow reasonable distribution of load.

But, there's no problem with load right now, and I stand ready to
supply whatever hardware we need here.  In this way, we don't have to
deal with complex distribution schemes.

--Jimbo


body:
I'm reposting this to wikitech-l so that discussion doesn't get lost.

On mer, 2002-02-20 at 15:20, lcrocker at nupedia.com wrote:
> You Wrote:
> > Please take a
> > look at the non-English non-ISO-8859-1 wikipedias sometime.
> >
> >Hundreds of pages, with correct charset headers:
> >  ISO-8859-2:
> >    http://pl.wikipedia.com/
> >  UTF-8 with a custom conversion function for certain character
> >  sequences:
> >    http://eo.wikipedia.com/
>
>
> You're right. Last time I looked at these, the test pages I retrieved
> gave 404s, and the 404 page is still served as ISO-8859-1, but the
> headers of contentful pages are indeed as you say: 8859-2 for "pl"
> and UTF-8 for "eo", etc.
>
> OK, then, I guess we do have to wade into the morass of national
> character sets.

Unless you want to switch to UTF-8, that is a given.

>  I have little or no experience using actual foreign-
> made computers; but I /do/ have extensive knowledge about character
> sets and communication protocols, so I'm just trying to make sure we
> don't make the same mistakes hundreds of others have made in the past
> by not getting this stuff right up front, but just diving headlong
> into coding without stepping back a moment to design something that
> will be usable and maintainable in the future.
>
> The way it is now, for example, we won't be able to cut-and-paste
> between wikis if, say, I wanted to include a quote from some Polish
> leader or something.

Sad but true.

>  Maybe that's a reasonable sacrifice for ease of
> editing on those wikis.

Lee, let me put it this way. Imagine, if you will, that history had gone
somewhat differently. Let's say that the first computers had been
developed in a politically free, economically strong, highly
industrialized Russia and the standard computer character set around the
world had been based on the Cyrillic alphabet.

In our hypothetical world, there's a Russian version of what we would
have called Wikipedia. They set up some subsites in other languages, one
of which is English, which uses the Latin alphabet.

Now, you want to add some articles to the English site, but the site
administrators have declared that only the standard cyrillic character
set is to be used, with special markup to allow other characters through
the use of numerical codes. This means:
  * Pages display fine for viewing, but when you edit, you see nothing
but numeric escape codes.
  * You can't type *a single letter of English text* without using a
special numeric escape code.
  * All page titles have to be transliterated into Cyrillic, because the
escape codes aren't allowed in titles.

Now, can you honestly tell me that you expect the average
English-speaking wiki contributor to edit a page that looks something
like this:
[[?????????:??????? ?????????|Welcome]]
to
[[?????????|Wikipedia]],
a
collaborative project to produce a complete encyclopedia from scratch.
We started in
January 2001 and
already have
over '''23,000
articles'''.
?

I can't imagine that you would expect that to be acceptable to anyone
else! You'll notice that the two non-ISO-8859-1-language 'pedias that
have actual content (Polish and Esperanto) both use the Latin alphabet
with a few diacritics. So theoretically, they would be the *most*
amenable to using HTML entities -- you can almost read text in the edit
box that way -- yet users of both wikipedias took the effort to tweak
the program to make their customary character encodings work so that
they could actually find people who would be willing to edit pages.

HTML entities are great for tossing in an occasional foreign letter or
word, but at the user level they are poor for regularly used diacritics
and utterly useless for text in other alphabets.

>  We could, alternatively, serve UTF-8 on all
> of them, but that would risk breaking older browsers.  There are side
> issues of what is stored in the database, and what is allowable in
> titles/URLs, etc.

Another alternative is to use the entities internally in the database,
but work some mojo to make them appear as normal characters in the edit
box. Which means you get zero advantage over simply using the national
character set -- you still have to send a character set header, you have
to know which Unicode characters can be passed through safely and which
need to be escaped, the search engine still breaks words, you still
can't capitalize non-ISO8859-1 titles, you still can't cut-n-paste, etc
etc etc. All of the pain, none of the gain.

> We really need to sit down and spec this out before we get too far
> down the road.  That's one reason why I posted the proposed policy on
> foreign characters for the English Wiki; it is explicitly for the
> English one only, but we need something equivalent for the other
> ones.
>
> We had a lot of discussion about these topics in the early months of
> the project: I don't want us to ignore everything we learned back
> then just because the folks working on the code now weren't around
> back then.
> 0

Indeed. What were the conclusions of these discussions, and the
reasoning behind them?

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] Blocking IPs

body:
Dear sirs :)

I just committed a change enabling sysops to block IPs for edit. This was a
request from Larry.

Sysops (and only those!) will have a "Block this IP" link next to each IP on
the history page of an article. That seemed like the logical place to put
it, since IP blocking will result from some form of editing.
NOTE: Currently, there is no way to block a logged-in user that way; not a
pressing issue at the time, though.

That blocked IP will go to "Wikipedia:Blocked IPs".
NOTE: That page should be protected, otherwise a troll could just go there
and remove his IP from the list!
NOTE: Currently, the only way to set an IP free again is to manually delete
the line on "Wikipedia:Blocked IPs". There's a timestamp so we can implement
time-based removal later.

Even blocked IPs can go to the edit page, but on pressing "Save", blocked
IPs will get a message ("Your IP has been blocked..."). This will even work
if the troll signs up with a user name after getting the block as an IP.
Sneaky, eh? ;)

Let's get Larry his sysop rights back ASAP!

Magnus


from_name: Julie Hofmann Kemp

subject: [Wikipedia-l] RE: Plea for help

body:
Larry, I have to apologize --

I've been so wrapped up trying to keep my course on track that I didn't
respond at all to your letter.  I was frankly just glad you were still
going to be around!  I'll help any way I can.

Also, (since I'm hoping lots of people are still reading this) we still
have NO ACCEPTABLE INFORMATION  on how some of the programming changes
affect day to day use.  I've offered to write something up -- given an
explanation, but have been repeatedly ignored or blown off (partially
because I seem to have offended some delicate programmer sensibilities
by saying that they were acting exactly like programmers -- good at what
they do, but less interested in making it clear to the user through
documentation).

Whether or not people want to hear it, the new site is not all that
user-friendly as far as explaining subpages and namespaces goes (or how
to report bugs, for that matter -- why isn't there a bug report link?).
People are still trying to create /Talk pages and /whatever subpages.
All we need is a carefully placed announcement or two -- I suggest
adding a special page (changes to the system) and links on the home
page, on how to edit a page, and on recent changes.  I know that Lars
added some stuff on namespaces under the article, but what is needed is
a primer -- something that says...where one did  x in the old version,
please do y in the new.    I'd do this myself, but am pretty sure I
can't create a special page (guessing it's an admin thing) and I'm still
not clear on it.  It appears that, instead of subpages, we should be
creating link pages (maybe) and that each page is generated with an
associated talk page.

JHK -- feeling very resentful that no one actually seems to think this
is worth their time.

Message: 3
Date: Wed, 20 Feb 2002 09:45:37 -0800 (PST)
From: Larry Sanger <lsanger at nupedia.com>
To: <wikipedia-l at nupedia.com>
cc: <intlwiki-l at nupedia.com>
Subject: [Wikipedia-l] Plea for help
Reply-To: wikipedia-l at nupedia.com

This is an unusual plea for help.  It's possible I'm going out of my
mind--I mean, I guess I don't think so, but sometimes it seems like it.
:-)

Occasionally in the past, controversy has erupted that has occupied
unfortunately large amounts of my time.  Sometimes the controversy is
definitely worth the time, but just as often it seems it isn't, not at
all.  Most recently, in the wake of my announcement that I am no longer
a paid employee, that I will still be working though as a volunteer, and
that we might (if we can) start selling ads and soliciting donations to
help pay for me once again, apparently some people reacted not by
saying, very kindly, "Oh, poor Larry, whatever will we do without him?"
but instead, "Huh?  There was a paid employee?  And they might sell
ads?! Someone was making money?!  This must be a greedy capitalist
exploitative project.  This shouldn't be!  We've got to do something
about it!"

Well, I and sometimes Jimbo then spend a lot of time putting out fires.
In the present case, we inform the newcomers that, yes, I did do a few
useful things for the projects oh maybe occasionally; yes, the idea of
fees for viewing content is absolutely ludicrous and has never been
considered; yes, we *really, really are* committed to making and keeping
the content always completely free forever; and yes, we have been
considering making a nonprofit foundation for Wikipedia and Nupedia for
a long time now; it's mainly a matter of getting enough time to actually
start us formally down the road.  (We can start with a "OK now, we're
really going to do it, it's official, we're asking for pro bono legal
help to set it up" kind of announcement, but we don't want to do that
until we have the time to follow through properly.  "We" in this case
mainly means
Jimbo.)

I think some people just sit back and enjoy the show, and figure that I
can hold my own well enough.

Well, er, not exactly.  I can, usually, if I spend the time.  But I
don't have the time anymore.  I literally *don't have the time* for this
kind of nonsense.  I've spent, I don't know, three hours so far today
working on various Wikipedia stuff, and it's not even 1 PM yet.  I
should instead have been, er, looking for a job, or a lot of other
things my wife wants me to do.  :-)

So, please, critics, have mercy--be gentle--and, non-critics, if you
support my cause and think you can supply the correct response to an
e-mail that, you can predict, I will want to have answered, please do
that.  Don't hold back waiting for me to do it.  I'll be very grateful.

Larry

--__--__--

_______________________________________________
Wikipedia-l mailing list
Wikipedia-l at nupedia.com
http://www.nupedia.com/mailman/listinfo/wikipedia-l

End of Wikipedia-l Digest

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com