body:
On Thu, May 01, 2003 at 08:10:38AM -0700, Brion Vibber wrote:
> On Wed, 2003-04-30 at 21:48, Tim Starling wrote:
> > Someone needs to run this:
> >
> > UPDATE cur SET cur_random=RAND()
>
> Ok, it'll be a bit before it's done. Hooray for big slow databases. :D
>
> -- brion vibber (brion @ pobox.com)

Off the top of my head, I can't think of any simple mathematical way to
do want you want to do.  (that being making articles selected randomly
less likely to be selected randomly over time)  Even if every cur_random
is totally re-randomized, it still isn't going to be fair.

Seems to me that the best way to do this is either:
1) Be truly random.  In PostgreSQL, this would be something like:
     SELECT cur_id from cur LIMIT 1 OFFSET random(SELECT count(cur_id) FROM cur)
   (this function should be really fast, not sure if faster than:)
     SELECT cur_id from cur LIMIT 1 OFFSET random(SELECT count(*) FROM cur)
   (but you get the idea, may need other constraints)
2) Keep a timestamp instead of a random number.  That way, whenever an
   article is "randomly" selected, it gets its timestamp updated to the
   current time.  Always select the oldest article for a "random" page.
   New articles always get a current timestamp here.

--
Nick Reinking -- eschewing obfuscation since 1981 -- Minneapolis, MN


list: wikitech-l

source: pipermail-mbox

from_name: Lee Daniel Crocker

subject: [Wikitech-l] Really good news!

body:
> (Nick Reinking <nick at twoevils.org>):
>
> However, is there a reason why we couldn't have ibiblio host
> the entire Wikipedia?  It seems that is what Brion was asking for.
> If they're willing to supply three, four (or more) servers to keep
> Wikipedia running, why isn't that a good thing?

Because they aren't us. We have a specific vision, and goals to
accomplish. Even if ibiblio totally supported those same goals
now, there's no guarantee they will in the future. Ultimately,
physical ownership of the servers is our guarantee that Wikipedia's
operation will serve our vision.

Also, my experience with ibiblio is that their servers tend to
run close to capacity anyway: try downloading Gentoo Linux, for
example.

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


from_name: Nick Reinking

subject: [Wikitech-l] Really good news!

body:
On Thu, May 01, 2003 at 11:39:19AM -0500, Lee Daniel Crocker wrote:
> > (Nick Reinking <nick at twoevils.org>):
> >
> > However, is there a reason why we couldn't have ibiblio host
> > the entire Wikipedia?  It seems that is what Brion was asking for.
> > If they're willing to supply three, four (or more) servers to keep
> > Wikipedia running, why isn't that a good thing?
>
> Because they aren't us. We have a specific vision, and goals to
> accomplish. Even if ibiblio totally supported those same goals
> now, there's no guarantee they will in the future. Ultimately,
> physical ownership of the servers is our guarantee that Wikipedia's
> operation will serve our vision.
>
> Also, my experience with ibiblio is that their servers tend to
> run close to capacity anyway: try downloading Gentoo Linux, for
> example.

Fair enough, was just wondering.  :)  Still, I think it would still be a
good idea for us to use them as a read-only mirror for when the database
is down (for server maintence, database maintenence, etc.)

--
Nick Reinking -- eschewing obfuscation since 1981 -- Minneapolis, MN


from_name: Jimmy Wales

subject: [Wikitech-l] Re: Chat about Wikipedia performance?

body:
David A. Wheeler wrote:
> If it's a choice between having a working wikipedia, and
> having the bells & whistles, I think working is the better plan.

I agree completely.

We've been advised by... I'm sorry, but I forgot who it was, but he's
the author of a well-known book on this sort of thing...  that
separating webserving and database should be a huge win.  If that's
right, then we should be good to go after the new server is installed
this weekend, and after some time spent getting it into service.

In general, I think that it is absolutely true that responsiveness is
more important than frills.  I have never thought of the feature of
links appearing differently depending on whether or not the article
exists as a frill, but I suppose it is.  We could conceivably abandon
that and any other feature that requires "on the fly" anything, and
make the site very fast.

But it's probably better, for some features, to throw hardware at it.

--Jimbo


list: wikien-l

from_name: Daniel Mayer

subject: [WikiEN-l] Conventions and movie vs. film

body:
The Cunctator wrote:
>We actually did have a big discussion about this
>and film ended up being nixed because of the problem
>of digital videa replacing actual film as the medium
>for "films". Thus, "movie".

That's one minor reason (but that convention was
written sometime in the UseMod days - well before
digital movies were widely available in movie
theaters). Another is that the word "film" is often
applied selectively (in at least the United States) to
movies that are considered to be of a higher caliber.
Thus, to many people, all films are movies but not all
movies are films.

However, it basically comes down to this - we had to
choose something to serve as a standard disambiguating
term just as we had to choose one format to have all
the day articles reside at.

Disambiguation is aimed at resolving ambiguities and
having both "film" and "movie" exist side by side is
inconsistent - creating ambiguity.

-- Daniel Mayer (aka mav)

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


body:
Daniel Mayer wrote:
> Disambiguation is aimed at resolving ambiguities and
> having both "film" and "movie" exist side by side is
> inconsistent - creating ambiguity.

I agree.  Having a standard is more important than which particular
standard.

Another aspect of this, apart from disambiguation, is that identifying
all motion pictures as (film) or (movie), even when there is no
ambiguity, is that this "meta-data" could be used later on to extract
a portion of our data for a movie history project or similar.

One reason we've been successful is that we didn't get bogged down in
a priori design before we just plunged in like maniacs and started
writing an encyclopedia.  But at the same time, looking forward, we
will likely someday want to come up with ways for people to usefully
(and SIMPLY!!!!) put some structure on the data.

It'd be nice to be able to do a query and pull out all biographies,
for example, or all countries, or all movies, or all dog breeds.  Not
so nice that we should get all bogged down in debates about the
perfect system, but nice enough that we should think about how
conventions can help us move in that direction gracefully.

--Jimbo


subject: [Wikitech-l] Really good news!

body:
> (Nick Reinking <nick at twoevils.org>):
>
> Fair enough, was just wondering.  :)  Still, I think it would still be a
> good idea for us to use them as a read-only mirror for when the database
> is down (for server maintence, database maintenence, etc.)

I agree that read-only mirrors are probably a good idea in our
future plans. It would require some software: namely, something
that ran maybe once or twice a day and generated a set of static
HTML pages from the "cur" table. This could be done incrementally
based on timestamps, so it wouldn't be such a burden). Things
like edit links would point to the live Wikipedia or to an
intermediate page that explained what was going on and that then
pointed to the live site. We could then have other sites rsync
those static pages. Ideally, the search links would point to
a search function on the other site as well, built by indexing
the static pages rather than going to the live database.

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


from_name: John Knouse

subject: [WikiEN-l] [Fwd: Re: Johnny Rebel]

body:
Message about the other Johnny Rebel, the one who's not the
racist:

-------- Original Message --------
Subject: Re: Johnny Rebel
Date: Thu, 1 May 2003 23:29:18 +1000
From: "John Castellain" <dbv at iprimus.com.au>
To: "John Knouse" <jaknouse at frognet.net>
References: <3EAE1A18.AD360007 at frognet.net>

Yes you are totally correct, This Johnny Rebel is a nickname
for a prominent
Australian artist who has released experimental music under
his lifelong
nickname.....and by the way he has worked with many black
artists and is
anti-racist!...thanks for your email
----- Original Message -----
From: "John Knouse" <jaknouse at frognet.net>
To: <dbv at sunet.com.au>
Sent: Tuesday, April 29, 2003 4:22 PM
Subject: Johnny Rebel

> I was just looking at info on Johnny Rebel.  There is a
> Johnny Rebel who is a white supremist.  I am inferring that
> this one and that one are entirely separate people.  Am I
> correct?
>
> --
> John Knouse
> jaknouse at frognet.net


from_name: The Cunctator

body:
On 5/1/03 12:10 PM, "Jimmy Wales" <jwales at bomis.com> wrote:

> One reason we should scrupulously stick to conventions is that in the
> event that we someday have a long discussion and decide that our old
> convention is undesirable, our sticking to the convention makes it a
> lot easier to change.
>
> That is, it would be pretty simple to write a script to move all pages
> from "Blah blah blah (movie)" to "Blah blah blah (film)", but a lot of
> work to fix "Blah blah blah (the movie)" and "Blah blah blah, a movie"
> and "The movie: Blah blah blah" and so on.

We actually did have a big discussion about this and film ended up being
nixed because of the problem of digital videa replacing actual film as the
medium for "films". Thus, "movie".

This was a *long* time ago so I have no memory where the record of that
discussion is.


from_name: Axel Boldt

subject: [WikiEN-l] wikipedia:copyright changes -- "special permission"

body:
I made a couple of changes to [[wikipedia:copyright]], explained on the
talk page. The most important one was to take out all references to
"special permission materials". Reading through the page, I found that
we actually had three categories of materials:
* GFDL
* fair use
* "special permission", which means that the copyright holder gives
permission to use the material *only for Wikipedia*.

The third category of materials was added by Lee on Aug 27, 2002. I
don't think it was properly discussed, and I disagree with it. The fair
use situation is already bad enough. Adding "special permission"
materials on top of that makes free reuse of Wikipedia next to
impossible.

If fair use cannot be justified and GFDL permission cannot be obtained,
then the material simply cannot be used in a project that claims to be
freely reusable.

There is also a legal issue: contributors license their article under
GFDL to the public. If somebody else adds material to that article,
then they are technically creating a derivative work, and are
*required* to license all of it under GFDL, i.e. they cannot insist on
a special permission only for Wikipedia.

Axel

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


from_name: Jason Williams

subject: [WikiEN-l] The "Young and Innocent" page

body:
On Thu, May 01, 2003 at 02:44:44PM +0200, K Forstner wrote:
> The reason is that Hitchcock's movie of the same name (which I'm going to
> rename "movie" instead of "film")

Erm. Why?

--

jason at jasonandali.org.uk    http://www.jasonandali.org.uk/jason/


body:
> (David A. Wheeler <david_a_wheeler at yahoo.com>):
>
>>> 1. Perhaps for simple reads of the current article (cur), you
>>> could completely skip using MySQL and use the filesystem instead.
>>
>> In other words, caching.
>
> Sorry, I wasn't clear.
> I wasn't thinking of caching - I was thinking of accessing the
> filesystem INSTEAD of MySQL when getting the current wikitext.

No, you were clear. I am using "caching" in the plain English
sense of the word. Using the file system as a cache in front of
the database is just one possible implementation of the idea.

>> It isn't. And there's no reason to expect flex to be any
>> faster than any other language.

> Actually, for some lexing applications flex can be MUCH faster.
> That's because it can pre-compile a large set of patterns
> into C, and compile the result.  Its "-C" option can, for
> some applications, result in blazingly fast operations.

I suppose that's true. I do want to formalize the wikitext
grammar at some point, and using something like Lex/Yacc
code compiled and linked into PHP as a module is certainly
a possibility.

> Oh, one note - if you want to simply store whether or not a
> given article entry exists or not, and quickly check it, one
> fancy way of doing this is by using a Bloom filter.
> You can hash the article title, and then using a fancy data
> structure can store its existance or non-existance.

Yes, that's a very good idea. I just recompiled the PHP on
the server to have the shared memory extensions, so putting
a Bloom filter into that memory is probably better than a
more typical hash table.

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


body:
On Thu, May 01, 2003 at 01:50:58PM +0100, Jason Williams wrote:
> On Thu, May 01, 2003 at 02:44:44PM +0200, K Forstner wrote:
> > The reason is that Hitchcock's movie of the same name (which I'm going to
> > rename "movie" instead of "film")
>
> Erm. Why?

Okay, so there's a naming convention on this at

 http://www.wikipedia.org/wiki/Wikipedia:Naming_conventions_(movies)

Why is it so? Why is the principle that American and British English
have equal status abandoned here? Why is the ugly "movie" explicitly
preferred over the perfectly good "film"?

--
jason at jasonandali.org.uk    http://www.jasonandali.org.uk/jason/


body:
On Thu, May 01, 2003 at 06:10:39AM -0700, Jimmy Wales wrote:
> Jason Williams wrote:
> > Why is it so? Why is the principle that American and British English
> > have equal status abandoned here? Why is the ugly "movie" explicitly
> > preferred over the perfectly good "film"?
>
> That "movie" is ugly is a pretty ad hoc aesthetic judgment, don't you
> think?

Of course; since it's quite obviously subjective I assumed people
would assume that I was stating it as such :-)  I wasn't trying
to argue for wholesale replacement of "movie" by "film", just the
same admirable agnosticism that is applied across the rest of
the wikipedia.

> At any rate, perhaps there was a discussion somewhere about this?

I expect so, but I couldn't find it by googling the mailing list
archives and there isn't a Talk page corresponding with
[[Wikipedia:Naming conventions (movies)]].

--
jason at jasonandali.org.uk    http://www.jasonandali.org.uk/jason/


from_name: K Forstner

body:
Could someone with sysop powers please permanently delete the "Young and
Innocent" page? It redirects to Ernest Hemingway, and there are no links to
that page either.

The reason is that Hitchcock's movie of the same name (which I'm going to
rename "movie" instead of "film") cannot be found otherwise ("database
error").

Thanks in advance,

KF


body:
> (Mark Christensen <mchristensen at humantech.com>):
> > A quick start might be to temporarily disable all checking
> > of links, and see if that helps much.
>
> This seems to be a helpful suggestion.  Without profiling, it's  hard to
> tell where the bottleneck is, but I think link checking is a good guess.
>
>
> And this fairly simple (now that Lee has created a functioning test
> suite) could probably tell us if this is a bottleneck.  If so, then at
> least we know where to focus our optimization efforts.
>
> If this is the problem, we are in luck because there have been a lot of
> good improvement suggestions.  But they all add complexity to the code
> (or database setup) and "premature optimization is the root of all kinds
> of evil," so if link checking isn't a bottleneck it would be
> counterproductive to spend a lot of time to try to optimize it.

I think it's a bit premature for that yet. I think the differently-
rendered missing links feature is pretty critical, not just a frill.
In the short term the hardware will bail us out until we can find a
solution.

Along those lines, I'd like to ask for some feedback: if I do the
Bloom filter in shared memory thing, I can choose parameters to
optimize things. So here's the first question: Bloom filters have no
false negatives (that is, there's no risk that they'll show an
existing page as non-existing), but there are false positives. What
is an acceptable false-positive rate? With a 16-bit filter, the
rate will be one in 65,000; I think that's a bit too high. With a
32-bit filter, it's one in 4 billion, which seems reasonable. 24 bits
is one in 16 million, which might also be OK.

A 32-bit filter for 200,000 pages will fit into 2Mb of memory, which
seems reasonable. It would require a 768-bit hash function. Anybody
have recommendations for a good 768-bit hash function for titles?

Alternatively, 2Mb is probably enough space for a traditional hash
table of titles and article IDs, provided the average length of a
title isn't too long, and there's some reasonably efficient
storage method for it.

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


from_name: Tim Starling

subject: [Wikitech-l] Please run this DB query to fix Special:Randompage

body:
Someone needs to run this:

UPDATE cur SET cur_random=RAND()

Here's a quote from Village Pump explaining why:

The pseudorandom number generator which backs the Special:Randompage link
appears to be not so good. I just hit it about two dozen times and had
several links come up multiple times. -???

Emphasis on pseudo. I got RNA transcription four times in about a dozen
clicks - twice in succession. Also, Zog's favorite song came up - but
redirected to Johnny Rebel. I thought redirects were excluded. Geoffrey

Here's what I found out with a few SQL queries. The cur_random field values
are strongly clustered up around the high end. In fact, there are no
articles at all between about 0.18 and 0.4, and only few below 0.18. A few
minutes of browsing through the old versions of SpecialRandompage.php shows
why. A previous version of the software selected the lowest-numbered
cur_random value, and set it to a random value. So here's why we now see
poor results: Most of the pages are clustered up above 0.9 or so, so when
you click Special:Randompage , there's a high chance of picking one of the
few low-numbered articles. The cur_random value is then reset, and there's
still a high chance of the new value being below 0.9. Hence, the few
priveleged low-numbered articles get selected far more often, and unless
someone re-randomizes cur_random column, it take a long time for the
high-numbered articles to diffuse back down. -- Tim Starling 04:25 May 1,
2003 (UTC)

--END QUOTE--

-- Tim Starling.

_________________________________________________________________
Hotmail now available on Australian mobile phones. Go to
http://ninemsn.com.au/mobilecentral/hotmail_mobile.asp


from_name: Mark Christensen

body:
> A quick start might be to temporarily disable all checking
> of links, and see if that helps much.

This seems to be a helpful suggestion.  Without profiling, it's  hard to
tell where the bottleneck is, but I think link checking is a good guess.

And this fairly simple (now that Lee has created a functioning test
suite) could probably tell us if this is a bottleneck.  If so, then at
least we know where to focus our optimization efforts.

If this is the problem, we are in luck because there have been a lot of
good improvement suggestions.  But they all add complexity to the code
(or database setup) and "premature optimization is the root of all kinds
of evil," so if link checking isn't a bottleneck it would be
counterproductive to spend a lot of time to try to optimize it.

--Mark


body:
----- Original Message -----
From: "Jason Williams" <jason at jasonandali.org.uk>
To: <wikien-l at wikipedia.org>
Sent: Thursday, May 01, 2003 2:50 PM
Subject: Re: [WikiEN-l] The "Young and Innocent" page

> On Thu, May 01, 2003 at 02:44:44PM +0200, K Forstner wrote:
> > The reason is that Hitchcock's movie of the same name (which I'm going
to
> > rename "movie" instead of "film")
>
> Erm. Why?

Why what? In case you want to know why I have moved the article from "Young
and Innocent (film)" to "Young and Innocent (movie)": Disambiguating film
titles is commonly done in Wikipedia by adding the word "(movie)" rather
than "film". There are plenty of examples (Notting Hill, Suspicion, etc.).

 In case you want to know why the "Young and Innocent" page should, IMHO, be
deleted: Because it is unnecessarily confusing and obsolete anyway.

Does this answer your question?

KF


body:
> jason at jasonandali.org.uk    wrote:

> Okay, so there's a naming convention on this at
>   http://www.wikipedia.org/wiki/Wikipedia:Naming_conventions_(movies)
>
> Why is it so? Why is the principle that American and British English
> have equal status abandoned here? Why is the ugly "movie" explicitly
> preferred over the perfectly good "film"?

No idea. I think someone (an American?) must have started using the word
"movie". Personally, I'd rather have an "ugly" word than one that might lead
to confusion.

KF


from_name: John R. Owens

body:
On Thu, 1 May 2003, Jimmy Wales wrote:

> Date: Thu, 1 May 2003 11:45:53 -0700
> From: Jimmy Wales <jwales at bomis.com>
> Subject: Re: [Wikitech-l] Re: Chat about Wikipedia performance?
>
> We've been advised by... I'm sorry, but I forgot who it was, but he's
> the author of a well-known book on this sort of thing...  that
> separating webserving and database should be a huge win.  If that's
> right, then we should be good to go after the new server is installed
> this weekend, and after some time spent getting it into service.

Well, to be fair, one of the very first questions I asked upon joining
wikien-l was whether the database and webserver were on one host or two,
with exactly that reasoning in mind. But I don't think I've written any
well-known books... unless the Wikipedia itself counts? ;)

--
John R. Owens http://www.ghiapet.homeip.net/
Something far worse than the Shadows: reporters.
				--Captain John Sheridan


from_name: David A. Wheeler

body:
> From: "Mark Christensen" <mchristensen at humantech.com>
> To: <wikitech-l at wikipedia.org>
> Reply-To: wikitech-l at wikipedia.org
>
> > A quick start might be to temporarily disable all checking
> > of links, and see if that helps much.
>
> This seems to be a helpful suggestion.  Without profiling, it's  hard
> to
> tell where the bottleneck is, but I think link checking is a good
> guess.

Thanks very much!!  I think measuring without link-checking would
great, it would certainly answer many questions. I don't have a
machine I can test on, sadly; does someone else?

It's worth noting that link-checking not only causes
additional processing - if link-checking is disabled, and user
formatting is limited, many OTHER optimizations become easy.
In particular, caching becomes really easy if article text
doesn't depend on other state (i.e., doesn't require link-checking and
processing to support fancy user options).  For example, without
link-checking, you don't have to follow lists to invalidate
"related" caches.  The most effective optimization is to do
nothing at all :-).  In-filesystem caches of HTML fragments would
make sense in such a situation, and Linux's sendfile()
could do a rather impressive job of improving performance
when sending cached article text.  Allowing users to select
which stylesheet to blast back at them would give users a limited
amount of control, but seriously improve performance.

Link-checking is that it's not as useful
as you'd wish, anyway.  After all, it only identifies
existence.  An article with only a tiny amount of content appears
"complete" to the link-check, but clearly you'd want people to
work on that article too! If disabling link-checking (and the
optimizations it makes complicated) turns out to seriously improve
performance, then I think it's an obvious capability to disable
(at least as a configurable option).

My two cents, hope they help.


body:
>> If this is the problem, we are in luck because there have been a lot
>> of good improvement suggestions.  But they all add complexity to the
>> code (or database setup) and "premature optimization is the root of
>> all kinds of evil," so if link checking isn't a bottleneck it would
be
>> counterproductive to spend a lot of time to try to optimize it.

> I think it's a bit premature for that yet. I think the differently-
> rendered missing links feature is pretty critical, not just a frill.
> In the short term the hardware will bail us out until we can find
> a solution.

What I was suggesting is just that we '''test''' the performance with
differently rendered links turned off, so that we know if this is a
place where optimization is required. I expect it is, but I think it
makes sense to check.

Even a simple hash in shared means that we are maintaining data in two
places all the time and thus and makes things somewhat more complex to
maintain.

I think this is a good idea -- if link checking is creating the current
bottleneck.  And that's why I voiced support turning off the links to
see the performance impact before anybody spends time coding an
optimized link checking routine.

--Mark Christensen


list: wikipedia-l

from_name: Jay Bowks

subject: [Wikipedia-l] [Wikitech-l] server names

body:
From: "Sean Barrett" <sean at epoptic.org>
> Isadore (of Seville).

Or "Isadoor" (of Seville) for the gateway ;-)


body:
> Well, to be fair, one of the very first questions
> I asked upon joining wikien-l was whether the
> database and webserver were on one host or two,
> with exactly that reasoning in mind. But I don't
> think I've written any well-known books...
> unless the Wikipedia itself counts? ;)

I believe Jimbo was talking about Ian Gilfillan, author of "Mastering
MySQL 4", who posted a couple of suggestions back in February.

--Mark


from_name: Brion Vibber

body:
On Thu, 2003-05-01 at 15:56, Lee Daniel Crocker wrote:
> Moving only the old data to the filesystem would also be less of a
> problem than moving the cur records, because we don't have the
> problem of losing MySQL's fulltext index, which we need to keep for
> the cur table to implement the search function.

Actually, that's not the case. We've always used a separate, munged text
field for the searchable text (stripping HTML markup, accounting for
things like changing "[[foobar]]s" to "foobar foobars", massaging
Unicode chars for the non-Latin wikis to make search work right in
Esperanto, Polish, Japanese, Chinese, etc).

Originally these were fields in cur, but now they're in a separate table
(searchindex) since cur is InnoDB and we can only do fulltext searches
on MyISAM tables.

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20030501/7a9976fa/attachment.pgp>


body:
I would have a lot to add to the Hemingway Votes for deletion and Young and
Innocent talk pages, but, again, all I get is error messages.

KF


body:
> > A quick start might be to temporarily disable all checking
> > of links, and see if that helps much.
>
> This seems to be a helpful suggestion. Without profiling, it's
> hard to tell where the bottleneck is, but I think link checking
> is a good guess.

It's more than a guess--one of the very first things we did
to test the new software is to run bots to fetch lots of pages,
which we then sorted by response time. The pages that sank to
the bottom of that list were special pages with complex queries
and long pages with lots of links. Long pages without many links
were not a problem.

I just did an ad-hoc benchmark on Piclab of the same installation
with and without link-checking, and on the limited set of pages
used by the test suite, the speedup was only about 3%. Of course
all benchmarks on single-servers may be less applicable to the
multiple-server installation we're going to have soon.

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


body:
On Thu, May 01, 2003 at 10:31:06PM +0200, Luc Van Oostenryck wrote:
> On Thu, May 01, 2003 at 02:47:35PM -0500, Lee Daniel Crocker wrote:
> > Along those lines, I'd like to ask for some feedback: if I do the
> > Bloom filter in shared memory thing, I can choose parameters to
> > optimize things. So here's the first question: Bloom filters have no
> > false negatives (that is, there's no risk that they'll show an
> > existing page as non-existing), but there are false positives. What
> > is an acceptable false-positive rate? With a 16-bit filter, the
> > rate will be one in 65,000; I think that's a bit too high. With a
> > 32-bit filter, it's one in 4 billion, which seems reasonable. 24 bits
> > is one in 16 million, which might also be OK.
>
> If all the content of CUR is moved to/duplicat as a filesystem,
> like David Wheeler suggest it (and a agree completly with him),
> you don't need the Bloom filter, the filesystem will do the work.

Actually, I think the best win would come from having OLD on the
filesystem.  CUR would be excellent, too.  As it is, OLD has complete
copies of every edit ever made, and it forces MySQL to consume an unholy
amount of memory.  :(

--
Nick Reinking -- eschewing obfuscation since 1981 -- Minneapolis, MN


from_name: Erik Moeller

body:
> Of course; since it's quite obviously subjective I assumed people
> would assume that I was stating it as such :-)  I wasn't trying
> to argue for wholesale replacement of "movie" by "film", just the
> same admirable agnosticism that is applied across the rest of
> the wikipedia.

I strongly object to this. Conventions should be maintained consistently
across Wikipedia. If you want to change the naming convention, propose a
vote. But there's nothing admirable about ignoring conventions. It's
simply unprofessional.

Regards,

Erik


body:
> Actually, I think the best win would come from having OLD on the
> filesystem.  CUR would be excellent, too.  As it is, OLD has complete
> copies of every edit ever made, and it forces MySQL to consume an
> unholy amount of memory.  :(

There's nothing magical about filesystems--the same work has to be
done whether the data is in a database or a filesystem. However it is
true that filesystems tend to be optimized for different kinds of
access patterns than databases, and those access patterns may be
better suited to big chunks of text.

Also, I don't see why the size of the old table has anything to do
with the amount of memory used by MySQL. There's almost no difference
in performance between a database with 10,000 entries and one with
100,000 entries. A more significant win is likely to be reducing the
size of the /individual records/, perhaps by putting the full text in
the filesystem and just having pointers and stats in the database.

Moving only the old data to the filesystem would also be less of a
problem than moving the cur records, because we don't have the
problem of losing MySQL's fulltext index, which we need to keep for
the cur table to implement the search function.

Another win for the two-server setup might be keeping old text in
the front-end file system rather than the database machine, to reduce
traffic over the wire.

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


from_name: Graham Burnett

subject: [WikiEN-l] Zog- sorry for repeating what had already been established

body:
> Message: 8
> From: "Graham Burnett" <grahamburnett at blueyonder.co.uk>
> To: <wikien-l at wikipedia.org>
> Date: Wed, 30 Apr 2003 18:19:59 +0100
> Subject: [WikiEN-l] Zog
> Reply-To: wikien-l at wikipedia.org
>
> I havn't seen any of 'Zogs' edits but picked up some concerns about this
> person from the mailing list- just in case anyone is unaware, 'Zog' is a
> term used by white supremacists and other nazis and is an abreviation of
> 'Zionist Occupational Governmnent', and is i believe a reference to that
> famous nazi publication/forgery 'The protocols of Zion' (ie, that the
world
> is run by a Jewish conspiracy). as such this username is far more
offensive
> than the likes of 'fuckyou' or 'i've got a big cock' or whatever. I
> seriously doubt any such user has any intentions of doing anything other
> than spreading race hate.
>
> Cheers graham (Quercus robur)
>

Appologies- i realised this HAD been discussed on the zog user pages which I
hadn't seen at the time, also my wikien digests arrived in a jumbled up
order yesterday, hence sorry for restating what everybody else already
knew...

Cheers Graham (Quercus robur)


from_name: Zoe

body:
Consistancy.

Zoe

--- Jason Williams <jason at jasonandali.org.uk> wrote:
> On Thu, May 01, 2003 at 02:44:44PM +0200, K Forstner
> wrote:
> > The reason is that Hitchcock's movie of the same
> name (which I'm going to
> > rename "movie" instead of "film")
>
> Erm. Why?
>
> --
>
> jason at jasonandali.org.uk
> http://www.jasonandali.org.uk/jason/
> _______________________________________________
> WikiEN-l mailing list
> WikiEN-l at wikipedia.org
> http://www.wikipedia.org/mailman/listinfo/wikien-l

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


body:
Films are what auteurs make.

Zoe

--- Jason Williams <jason at jasonandali.org.uk> wrote:
> On Thu, May 01, 2003 at 01:50:58PM +0100, Jason
> Williams wrote:
> > On Thu, May 01, 2003 at 02:44:44PM +0200, K
> Forstner wrote:
> > > The reason is that Hitchcock's movie of the same
> name (which I'm going to
> > > rename "movie" instead of "film")
> >
> > Erm. Why?
>
> Okay, so there's a naming convention on this at
>
>
>
http://www.wikipedia.org/wiki/Wikipedia:Naming_conventions_(movies)
>
> Why is it so? Why is the principle that American and
> British English
> have equal status abandoned here? Why is the ugly
> "movie" explicitly
> preferred over the perfectly good "film"?
>
>
> --
> jason at jasonandali.org.uk
> http://www.jasonandali.org.uk/jason/
> _______________________________________________
> WikiEN-l mailing list
> WikiEN-l at wikipedia.org
> http://www.wikipedia.org/mailman/listinfo/wikien-l

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


subject: [Wikitech-l] Really good news!

body:
Lee-
> Also, my experience with ibiblio is that their servers tend to
> run close to capacity anyway: try downloading Gentoo Linux, for
> example.

Correct. Ibiblio does a lot of good, but because they do so much, they're
not very fast. I think it's nice to have this option should Jimbo ever
decide to shut down Wikipedia because some vandal insulted his mother, but
right now we should concentrate working with the assets we have, and
expanding them where necessary. The Nupedia Foundation should help greatly
with that.

Regards,

Erik


from_name: Lee Daniel Crocker

body:
> (Brion Vibber <brion at pobox.com>):
>
> > Moving only the old data to the filesystem would also be less of a
> > problem than moving the cur records, because we don't have the
> > problem of losing MySQL's fulltext index, which we need to keep for
> > the cur table to implement the search function.
>
> Actually, that's not the case. We've always used a separate, munged text
> field for the searchable text (stripping HTML markup, accounting for
> things like changing "[[foobar]]s" to "foobar foobars", massaging
> Unicode chars for the non-Latin wikis to make search work right in
> Esperanto, Polish, Japanese, Chinese, etc).
>
> Originally these were fields in cur, but now they're in a separate table
> (searchindex) since cur is InnoDB and we can only do fulltext searches
> on MyISAM tables.

Ah, that's right. Funny how one can forget details about one's
own code (sort of; I wrote the text munging, you moved it to the
new table). At any rate, yes, we could store cur text in the
filesystem as well. I'll have to try that sometime.

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


references:
<Sea2-F118KZTNKnPgml0003ed8e@hotmail.com>
	 <oproisvhcczs1r4a@localhost>

body:
On Thu, 2003-05-01 at 17:55, Richard Grevers wrote:
> I thought the reason we were having this discussion was because articles
> were repeating too often, indicating some fault in the randomizer?

Well, two things:

First, the existing assigned indexes were very un-randomly distributed,
which would may have lead to a somewhat increased chance of duplicates.
I re-randomized the whole set this morning, and I just now ran 100
random pages and got no duplicates.

Second, I'm not 100% sure the randomizer is reliable under heavy load.
We're running MySQL 3.23.54; I notice a line in the changelog for
3.23.56 (released in mid-March, apparently, when I wasn't looking) which
may be relevent:
 * Better RAND() initialization for new connections.

Were we not about to replace it with a new machine and MySQL 4.0, I'd
recommend trying an upgrade. :D

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20030501/1d7a9275/attachment.pgp>


from_name: Nick Reinking

body:
> Also, I don't see why the size of the old table has anything to do
> with the amount of memory used by MySQL. There's almost no difference
> in performance between a database with 10,000 entries and one with
> 100,000 entries. A more significant win is likely to be reducing the
> size of the /individual records/, perhaps by putting the full text in
> the filesystem and just having pointers and stats in the database.

I didn't say having more entries will reduce performance.  A select on
one of the columns probably doesn't take a lot of time.  Just having a
lot of records in old isn't making MySQL slow.  But, it is trying to
keep each individual recard in memory.  Just look at the bz2 files
available for download.  The cur table is 73MB, and old is 740MB.  The
old table makes the database 10x as big.  Ouch!  Since it is that big,
the processes grow and grow, eventually causing the machine to swap like
mad.

Even with the new machine, I think it would be helpful to perhaps split
old into two databases.  The database we have now would contain
everything we have now, plus a key to articles on the other database.
That way, looking at comments and dates of old articles (ie, Page
History) would still be fast, and we could reduce the memory footprint
of the primary server.  If you wanted to retrieve an older article, it
would take that key, connect to the other database and pull it down.
That other database could have perhaps a lower priority, and a much
lower memory buffer.  Or, we could store it on the filesystem, whatever.

In the evening (1:00 AM GMT), I see:
MemTotal:      2059088 kB
MemFree:         93096 kB
MemShared:          40 kB
Buffers:         14892 kB
SwapTotal:     2047992 kB
SwapFree:      1289140 kB

That's pretty heavy swap usage (33%), and heavy memory usage.

--
Nick Reinking -- eschewing obfuscation since 1981 -- Minneapolis, MN


body:
On Thu, 2003-05-01 at 17:16, Tim Starling wrote:
> >From: Brion Vibber <brion at pobox.com>
> >The resetting of the random weight upon load is just meant to keep the
> >pot stirring, and "in theory" shouldn't have any effect at all on
> >randomness if the random indexes were random to begin with.
>
> Quite right. I would recommend only setting cur_random on article creation.

Eh, may as well. Done.

> The MySQL documentation suggests using "ORDER BY RAND() LIMIT 1", for
> versions later than 3.23.

That seems to end up being dreadfully slow... "where used; Using
temporary; Using filesort" Hmm, I wonder if it's changed.... Yeah, okay,
after a minute and a half of "Copying to tmp table" I decided to cancel
that one. Still sucks. :)

The original code used the ORDER BY RAND() trick to fill up a pool of
1000, then grabbed out of those 1000, which was faster than out of the
whole wiki. However when it came time to refill the queue, it was still
incredibly slow, so I replaced it with a nice friendly *indexable*
pre-chosen random number attached to every page.

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20030501/7107a2e3/attachment.pgp>


from_name: Tony Wilson

subject: [WikiEN-l] A plea for sanity in capitalisation from the coalface

body:
Ray wrote: "Although I am usually responsive to blaming the Americans
for everything, it needs to be remarked that the capitalist charge in
this matter has been led by an Australian" - which made me smile. The
irony is that I also tend to blame the Americans for everything - and
in this context, I blame them for *not* capitalising things when they
ought to!  So perhaps we had best categorise this particular Australian
as one of Little Johnny Howard's "American Deputy Sherrifs in
South-east Asia."  .... Err - sorry, Jimbo. I know - no politics on the
list. :)

Enough humour. When Ray wrote that "the problem ... operates on two
levels" and that "as long as it is only on a stylistic level ..." he
missed the point. It is *not* only stylistic, nor indeed even primarily
stylistic: it is a matter of clear, unabiguous, and accurate
communication. There are many long-tailed shrikes, but only one
Long-tailed Shrike. It is not a matter of 'preffered styles", but a
matter of a precise and correct name or a vague and incorrect name.

However, Ray's third point raises a significant issue. Creating
lower-case redirects to entries that are (quite properly) capitalised
involves a good deal of attention to detail - something that I can
speal with authority about, as I have spent 20 or 30 hours attending to
this stuff in the four days or so, It is tedius and a small number of
mistakes will undoubtedly have slipped through. A technical solution
would indeed be very welcome. The obvious starting point would be a
search function that works. I understand that that matter is high on
the developers' priority list, and wish them every success in the
quest. Pending that (or more comprehensive technical solutions), it
seems we will just have to rely on the trusty old Mark 1 Eyeball.

Tony Wilson
(Tannin)


from_name: Marco Krohn

subject: [WikiEN-l] Re:categorizing data / meta-data   was: Conventions and movie vs. film

body:
On Thursday 01 May 2003 20:54, Jimmy Wales wrote:

> Another aspect of this, apart from disambiguation, is that identifying
> all motion pictures as (film) or (movie), even when there is no
> ambiguity, is that this "meta-data" could be used later on to extract
> a portion of our data for a movie history project or similar.

I thought about this possibility too and think it has lots of potential.

[snip]
> It'd be nice to be able to do a query and pull out all biographies,
> for example, or all countries, or all movies, or all dog breeds.  Not
> so nice that we should get all bogged down in debates about the
> perfect system, but nice enough that we should think about how
> conventions can help us move in that direction gracefully.

I agree. Categorizing the data, could also be useful for automatically
generating overview lists like [[List of movies]], [[List of mathematicans]]
etc. which are often incomplete and hard to maintain.

Perhaps the "meta-data" could be added in a similar way as it is done for the
international webpages. For instance on the page for "Javan Tiger" one would
add [[category:tiger]] in the very beginning. This information could be
expanded in a "wordnet" like way, e.g. see

http://www.cogsci.princeton.edu/cgi-bin/webwn1.7.1?stage=2&word=tiger&posnumber=1&searchtypenumber=-2&senses=&showglosses=1

such that the "Javan Tiger" would automatically appear in the "list of
mammals".

For what it is worth,

  Marco

--
Marco Krohn
Theoretical Physics
University of Hannover


body:
On Thu, May 01, 2003 at 08:17:27PM -0700, Brion Vibber wrote:
> On Thu, 2003-05-01 at 17:16, Tim Starling wrote:
> > >From: Brion Vibber <brion at pobox.com>
> > >The resetting of the random weight upon load is just meant to keep the
> > >pot stirring, and "in theory" shouldn't have any effect at all on
> > >randomness if the random indexes were random to begin with.
> >
> > Quite right. I would recommend only setting cur_random on article creation.
>
> Eh, may as well. Done.
>
> > The MySQL documentation suggests using "ORDER BY RAND() LIMIT 1", for
> > versions later than 3.23.
>
> That seems to end up being dreadfully slow... "where used; Using
> temporary; Using filesort" Hmm, I wonder if it's changed.... Yeah, okay,
> after a minute and a half of "Copying to tmp table" I decided to cancel
> that one. Still sucks. :)
>
> The original code used the ORDER BY RAND() trick to fill up a pool of
> 1000, then grabbed out of those 1000, which was faster than out of the
> whole wiki. However when it came time to refill the queue, it was still
> incredibly slow, so I replaced it with a nice friendly *indexable*
> pre-chosen random number attached to every page.
>
> -- brion vibber (brion @ pobox.com)

If MySQL is like PostgreSQL, a query like that gets _all_ the rows
first, randomizes them, and then applies the limit.  That's why you have
to use count and offset, which makes things super-duper fast.  :)

--
Nick Reinking -- eschewing obfuscation since 1981 -- Minneapolis, MN


subject: [WikiEN-l] Zog is back as Anti-Zog

body:
Zog is back as "Anti-Zog". He's not doing an Adam trick and pretending he's
someone else, he's just trying to get the song lyrics back into [[Thats how
a nigger goes]]. Can someone please block him without delay?

-- Tim Starling.

_________________________________________________________________
MSN Instant Messenger now available on Australian mobile phones.?Go to
http://ninemsn.com.au/mobilecentral/hotmail_messenger.asp


from_name: Luc Van Oostenryck

body:
On Thu, May 01, 2003 at 02:47:35PM -0500, Lee Daniel Crocker wrote:
> Along those lines, I'd like to ask for some feedback: if I do the
> Bloom filter in shared memory thing, I can choose parameters to
> optimize things. So here's the first question: Bloom filters have no
> false negatives (that is, there's no risk that they'll show an
> existing page as non-existing), but there are false positives. What
> is an acceptable false-positive rate? With a 16-bit filter, the
> rate will be one in 65,000; I think that's a bit too high. With a
> 32-bit filter, it's one in 4 billion, which seems reasonable. 24 bits
> is one in 16 million, which might also be OK.

If all the content of CUR is moved to/duplicat as a filesystem,
like David Wheeler suggest it (and a agree completly with him),
you don't need the Bloom filter, the filesystem will do the work.

----
Luc Van Oostenryck aka User:Looxix


from_name: Jens Frank

body:
On Thu, May 01, 2003 at 04:10:27PM -0400, Mark Christensen wrote:
>
> >> If this is the problem, we are in luck because there have been a lot
> >> of good improvement suggestions.  But they all add complexity to the
> >> code (or database setup) and "premature optimization is the root of
> >> all kinds of evil," so if link checking isn't a bottleneck it would
> be
> >> counterproductive to spend a lot of time to try to optimize it.
>
> > I think it's a bit premature for that yet. I think the differently-
> > rendered missing links feature is pretty critical, not just a frill.
> > In the short term the hardware will bail us out until we can find
> > a solution.
>
> What I was suggesting is just that we '''test''' the performance with
> differently rendered links turned off, so that we know if this is a
> place where optimization is required. I expect it is, but I think it
> makes sense to check.
>

OK, I did a little testing. The backend used was wikipedia.org, so
there is a small background noise caused by other users in the data.
Tests were performed "logged in" as JeLuF and without being logged in.
Only the "Cookie:" request header was removed. Everything else (Accept: etc)
was the same.

The following User Preferences were common to all "logged in" tests:

Quickbar:	floating left
Rendering Math:	HTML if simple else PNG
Textbox:	20x60
Local time:	+2
[ ] Underline Links
[X] Format broken links like this
[ ] Justify paragraphs
[ ] Hide minor edits in RC
[X] Enhanced RC
[ ] Auto-number Headings
[X] Remember password across sessions
[X] Edit box has full width
[X] Edit on double click
[ ] Watch new and modified articles
[ ] Mark all edits minor
[ ] Show preview before edit box
Nickname:	empty
# titles in RC:	150
Search settings
 Hits p. page:	40
 Lines p. hit:	5
 character:	50

JeLuF is an administrator.

The page fetched was [[Pope]], it has a medium number of links to other pages.

The other parameters were altered:

anonymous:

  Average:	0,81 s
  Std dev:	0,65 s
  95% Conf Int: 0,65 - 0,98 s
  Samples:	60

Logged in
stub threshold 500
Nostalgica skin

  Average:      3,57 s
  Std dev:      1,94 s
  95% Conf Int: 2,97 - 4,17
  Samples:      40

Logged in
stub threshold 0
Nostalgica skin

  Average:      3,26 s
  Std dev:      2,29 s
  95% Conf Int: 2,68 - 3,83
  Samples:      60

Logged in
stub threshold 500
Standard skin

  Average:      2,05 s
  Std dev:      1,07 s
  95% Conf Int: 1,59 - 2,52
  Samples:      20

Logged in
stub threshold 0
Standard skin

  Average:      2,62 s
  Std dev:      1,39
  95% Conf Int: 2,19 - 3,05
  Samples:      40

Logged in
stub threshold 0
Standard skin
[ ] Show hover over wiki links  disabled

  Average:      1,98 s
  Std dev:      1,21 s
  95% Conf Int: 1,45 - 2,51 s
  Samples:      20

The tests were performed in chunks of 20. The anonymous tests were done at first,
at last and in the middle of the tests. All three resulted in similar datasets.

Conclusion:

Apparently, anonymous user get served much faster than logged in ones.
The stub threshold seems to be of minor effect. The 95% confidence intervals
with and without stub threshold covered each other pretty much.

The Standard skin seems to be faster than the Nostalgica skin.

----

Open questions:
Why are anonymous users so much faster than logged in ones? The existance of
link targets is checked for both anonymous and logged in users. This can't be
the difference. Are there any additional database requests? Obviously there are
- User profile
- Checks for stubs
only for logged in users. The later apparently has only minor influence.
But loading the user profile can't take that long I'm sure.

Regards,

	JeLuF


body:
>From: "K Forstner" <kurt.forstner at chello.at>
>
>Could someone with sysop powers please permanently delete the "Young and
>Innocent" page? It redirects to Ernest Hemingway, and there are no links to
>that page either.

This is obviously a discussion for [[Wikipedia:Votes for deletion]], where
issues like this are sorted out every day. Kurt, if you still wish to pursue
this, please make an entry there.

-- Tim Starling.

_________________________________________________________________
MSN Instant Messenger now available on Australian mobile phones.?Go to
http://ninemsn.com.au/mobilecentral/hotmail_messenger.asp


body:
Hi David,

nice to see you here -- I enjoyed reading your Linux/OSS-related papers. I
have to say that disabling link checking on the live Wikipedia, even for a
short time, is hardly acceptable. It is essential for both readers and
authors. For readers, having to check pages via "trial and error", esp. on
link heavy lists where 10 or 90% of articles may already be written, is
very painful, aside from the irritation factors in normal articles. For
authors, link checking is essential to tell them whether what they were
trying to do worked -- is there an article at [[J.K. Rowling]], or at [[J.
K. Rowling]], or at [[Joanne K. Rowling]] ...

Having a clear and simple distinction - red=exists,blue=doesn't exist - is
one of the trademark signs of Wikipedia, helps people to understand the
very idea of an open growing encyclopedia, and is essential to its
operation. It is, in fact, essential to all wikis.

However, the shared memory idea sounds very reasonable. It would be
tremendously cool to store more than just 1/0 for each of these links in
the hash, and to store the byte size instead (we have to update the hash
on save anyway). Then we could use the stub checking feature without a bad
conscience (we currently have it disabled by default for performance
reasons), which adds a third link state for very short articles.

Regards,

Erik


from_name: Jimmy Wales

body:
Axel Boldt wrote:
> * GFDL

These are golden.

> * fair use

These we should be cautious and skeptical about.

> * "special permission", which means that the copyright holder gives
> permission to use the material *only for Wikipedia*.

These are very bad, and I'm in favor of removing them completely.

> Adding "special permission" materials on top of that makes free
> reuse of Wikipedia next to impossible.

Agreed.

> If fair use cannot be justified and GFDL permission cannot be obtained,
> then the material simply cannot be used in a project that claims to be
> freely reusable.
>
> There is also a legal issue: contributors license their article under
> GFDL to the public. If somebody else adds material to that article,
> then they are technically creating a derivative work, and are
> *required* to license all of it under GFDL, i.e. they cannot insist on
> a special permission only for Wikipedia.

This problem does come up for texts, but doesn't necessarily come up
for images.  That is, someone could give us special permission to use
an image, and unless someone downloaded the image, edited, and
uploaded it, the problem you identify here wouldn't come up.

But I completely agree that special (non-GFDL) permission is a very
bad thing for us.

--Jmibo


from_name: Toby Bartels

body:
Jason Williams wrote:

>Of course; since it's quite obviously subjective I assumed people
>would assume that I was stating it as such :-)  I wasn't trying
>to argue for wholesale replacement of "movie" by "film", just the
>same admirable agnosticism that is applied across the rest of
>the wikipedia.

I generally agree with you that Wikipedia should be agnostic
about matters of style like whether things are "movies" or "films".
(That's why I'm horrified at the growth of the Manual of Style.)
But we have to draw the line when it comes to article *names*,
so that linking will go smoothly.  Naming conventions
often must make arbitrary stylistic choices, but that's OK,
since it's only for article names, not for every item of text.

BTW, this issue is moot as far as [[Young and Innocent]] is concerned,
since in fact it is *not* the title of a Hemingway novel after all;
thus, no disambiguation (nor deletion) is needed in this particular case.

-- Toby


in_reply_to: <8l4ejWBSpVB@erik_moeller>; from erik_moeller@gmx.de on Fri, May 02, 2003 at 01:18:00AM +0200

body:
Erik Moeller wrote:
> nice to see you here -- I enjoyed reading your Linux/OSS-related papers. I
> have to say that disabling link checking on the live Wikipedia, even for a
> short time, is hardly acceptable. It is essential for both readers and
> authors.

I know I expressed sympathy to disabling link checking yesterday or the
day before, but several developers have come out against it, and I'm now
swayed to the view that it's an essential, not a frill.

I now wonder about our usage patterns and whether it would be best to
update a cache at edit-time, rather than at read-time.

Changes in 'existence' (yes or no) come up infrequently.  When someone
creates a brand new article, all other cached articles that are
affected by the change could be updated at that time.  So a hundred
times a day (if that), we have to do a fancy cache update to change
affected other articles.  But most edits don't affect other articles,
because they are edits to already existing pages.

Just an idea...


body:
Toby Bartels wrote:
> I generally agree with you that Wikipedia should be agnostic
> about matters of style like whether things are "movies" or "films".
> (That's why I'm horrified at the growth of the Manual of Style.)
> But we have to draw the line when it comes to article *names*,
> so that linking will go smoothly.  Naming conventions
> often must make arbitrary stylistic choices, but that's OK,
> since it's only for article names, not for every item of text.

I think that's a very important insight.  We want to be careful about
imposing a lot of standards, and the more standards we have, the less
we should expect newcomers to follow them.  Nupedia died (or nearly
so, it still breathes on occassion!) under the weight of extreme a
priorism that made the whole process impossible for newcomers to get
invested in.

But the thing about standards for article titles is that they make
things easier for newcomers and oldtimers alike.

--Jimbo


subject: [Wikitech-l] Please kill processe

body:
There seems to be a runaway Apache process (8453) that has already
sucked up about 800 minutes of CPU time.  Can somebody with root kill
it off?

--
Nick Reinking -- eschewing obfuscation since 1981 -- Minneapolis, MN


subject: [WikiEN-l] Zog is back as Anti-Zog

body:
I think we need to revisit having the ability for sysops to ban logged
in users.  Maybe the wiki way of doing this is to put the ability into
the software, but all sysops must agree to use it *only* to ban
*certain* variants on a known banned troll.

In the current case, it seems clear to me that banning Zog, Anti-Zog,
Baboon Mouth, JamesERay, and so on, should be done virtually
instantly, so as to discourage the behavior.

--Jimbo


subject: [WikiEN-l] Zog is back as Anti-Zog: a compromise to Jimbo's suggestion

body:
daniwo59 at aol.com wrote:
> As much as I would like to see Zog banned, this could potentially lead to
> horrible flame wars. Maybe a compromise would be to give sysops the ability
> to ban a logged in user for a set period of time, say 12 hours, until a
> decision can be reached and a developer can put a longer term ban in place.

Sure, but we don't necessarily have to have fancy code to do that.  We
can just have that be the social custom, rigidly enforced.

--Jimbo


subject: [WikiEN-l] Zog is back as Anti-Zog

in_reply_to: <Sea2-F22x4BlLMokiAW000421ad@hotmail.com>; from ts4294967296@hotmail.com on Fri, May 02, 2003 at 09:55:24PM +1000

body:
Tim Starling wrote:
> I agree. But how do we implement it?

Nothing fancy, just implement it more or less the same way we have
the ability to ban non-logged-in users.

The sysop clicks on 'block', and then the system says "This is a
logged in user, so don't do this unless you're really prepared to take
the heat for it.  This has to be an emergency situation to ban someone
who is doing something really egregious right now, or to ban someone
who you are *certain* is one of our usual suspects.  Abuse this, and
you'll face the wrath of the community!  No cupcakes for you at the
annual convention!"

And then, conceivably, it also has a checkbox for: Block IP too?

At that point, both the username block and the ip block would show up
in the usual log, so that the action is open to public scrutiny.

Keeping things like this simple and not having the code enforce a
particular solution seems like a good idea to me.

This is the second time now that we've had an emergency where some
jerk logs in again and again.

--Jimbo


subject: [Wikitech-l] Really good news!

body:
Felt it would be fair to forward on the most recent message I've received from
ibiblio:

> Nick,
>
> We ran across the wikitech-l thread about ibiblio hosting and found it
> quite interesting.  I hope you don't read this as an intrusion, but I'd
> like to offer more details on the systems side of ibiblio.  I'm one of
> the systems engineers here and can give a techie perspective.
>
> Many large, independent sites have similar discussions when considering
> moving to ibiblio.  Some moved, some decided to stay on their own
> systems.  Either way is fine with us as long as everyone's happy.  Some
> sites move their web/db content and retain their own mail servers.  A
> drawback for some is that we manage the systems and don't provide co-lo
> services.
>
> Concerning performance of the ibiblio site, one person mentioned gentoo
> downloads.  Part of the challenge of having so much content available is
> the outgoing bandwidth.  We use traffic shaping software to keep it
> under control.  As a result, Linux distribution download traffic is
> given a lower priority than the main web site content.  When judging
> performance, http page load speed for www.ibiblio.org might be a better
> example.  db.etree.org is heavily mysql driven, so you could check that
> out, too.
>
> You might find this set of pages interesting:
> http://www.ibiblio.org/systems/
> The system status page has a real-time (almost) network traffic graph.
>
> Also, we're close to moving the site to an LVS cluster and upgrading our
> database systems.  That should greatly improve capacity.
>
> But you guys should certainly do what best meets your needs.  We're
> happy to help provide useful and interesting content in any capacity.
>
> Thanks,
>
> -jrr

--
Nick Reinking -- eschewing obfuscation since 1981 -- Minneapolis, MN


subject: [WikiEN-l] Zog is back as Anti-Zog

body:
On Fri, 2 May 2003 daniwo59 at aol.com wrote:

> Date: Fri, 2 May 2003 06:39:20 EDT
> From: daniwo59 at aol.com
> Subject: Re: [WikiEN-l] Zog is back as Anti-Zog
>
> He's back as Anti-Zog, Baboon Mouth, and JamesERay

Don't forget Niggardly and UncleT, though not much was done as them.

--
John R. Owens http://www.ghiapet.homeip.net/
In Valen's name.
				--Delenn


body:
erik_moeller at gmx.de (Erik Moeller) declared:
> Hi David,
>
> nice to see you here -- I enjoyed reading your Linux/OSS-related
> papers.
Thanks!!

> I have to say that disabling link checking on the live Wikipedia,
> even for a short time, is hardly acceptable.

Well, at least a few measurements to determine if it _IS_
a bottleneck would help.  If the Wikipedia can be made to run
efficiently with link checking, then link checking should be left in.

I'm not against link checking, per se.
It's just that link checking is less important than
having a running Wikipedia.  Faster hardware will
probably help, but software solutions are still worth pursuing.
A hardware speedup of 2x won't be enough if usage increases 20x.

Other alternatives include storing the "current" text - or even
ALL text - in the filesystem, using special filenames so that
MySQL doesn't need to be consulted (or only consulted a little)
for certain common queries. It could store
either the original wiki text, or the HTML'ized versions, or both.
MySQL is actually a pretty good SQL database.  However,
it's optimized for serving structured data.  The underlying OS
has received FAR more optimization work to find and retrieve
unstructured data, and it also has more information available to it
so it can swap/drop information more efficiently.
As was noted elsewhere, if old text is stored in the filesystem,
it would reduce the memory usage of MySQL substantially.
If all text were stored in the filesystem, MySQL would then
primarily be used for metadata storage and search index uses
(if search is on).

Anyway, I'm just typing ideas, hoping that some are useful.
My real goal is that I don't want the Wikipedia to be
a victim of its own success :-).  A solution that works is,
by definition, a good solution :-).


from_name: Fred Bauder

subject: [WikiEN-l] Zog is back as Anti-Zog

body:
> From: "Tim Starling" <ts4294967296 at hotmail.com>
> Reply-To: wikien-l at wikipedia.org
> Date: Fri, 02 May 2003 21:55:24 +1000
> To: wikien-l at wikipedia.org
> Subject: Re: [WikiEN-l] Zog is back as Anti-Zog
>
>
>> From: Jimmy Wales <jwales at bomis.com>
>> Reply-To: wikien-l at wikipedia.org
>> To: wikien-l at wikipedia.org
>> Subject: Re: [WikiEN-l] Zog is back as Anti-Zog
>> Date: Fri, 2 May 2003 04:33:20 -0700
>>
>> I think we need to revisit having the ability for sysops to ban logged
>> in users.  Maybe the wiki way of doing this is to put the ability into
>> the software, but all sysops must agree to use it *only* to ban
>> *certain* variants on a known banned troll.
>>
>> In the current case, it seems clear to me that banning Zog, Anti-Zog,
>> Baboon Mouth, JamesERay, and so on, should be done virtually
>> instantly, so as to discourage the behavior.
>
> I agree. But how do we implement it? We could create the concept of a
> "trusted user", perhaps defined in terms of number of edits and joining
> date. If a user is not "trusted", a sysop can ban him/her. But ideally the
> sysop should be able to also block the IP address. Here's an idea: maybe the
> IPs of users could be available to sysops, but in encrypted form. So instead
> of banning 152.163.189.233, we ban 675C4216 (possibly using slightly better
> encryption than that).
>
> -- Tim Starling.

I think we are talking about extreme cases here, even with Adam and his
clones it took a while to be sure we were dealing with one. This Zog is not
even posing as a legitimate user. (If I accurately understand the nature of
his edits).

Fred


subject: [Wikitech-l] Chat about Wikipedia performance?

body:
Lee Daniel Crocker <lee at piclab.com> said:
> I just did an ad-hoc benchmark on Piclab of the same installation
> with and without link-checking, and on the limited set of pages
> used by the test suite, the speedup was only about 3%. Of course
> all benchmarks on single-servers may be less applicable to the
> multiple-server installation we're going to have soon.

Was this a representative setup (e.g., with the full database)?

If so, that sounds like removing link-checks won't help
much - at least by itself and for single servers.

Thanks for the info!!
As I said, measuring is the only real answer, and I'm willing to
accept "nope, wrong guess" as the answer.

So turning to other ideas and performance measurements...

The swapping overhead does suggest that an excessive use of memory
by MySQL is resulting in the performance hit.
Perhaps reducing the amount of data
stored in MySQL (by moving the text of cur and old text into
the filesystem and OUT of MySQL entirely).
As another poster noted, the "old" text is
an especially large database, but presumably only a few rare
articles are actually read from "old".
Obviously reading an article from the filesystem will read the
data into memory too, but the underlying OS doesn't try to preload
the entire filesystem into memory, which MySQL appears to be
trying to do.  Doing this will mean that archiving the encyclopedia
would have to archive both the MySQL metadata and the
article filesystem, but archiving a filesystem is a rather
well-understood problem :-).


from_name: daniwo59 at aol.com

subject: [WikiEN-l] Zog is back as Anti-Zog

body:
He's back as Anti-Zog, Baboon Mouth, and JamesERay

Danny
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.wikimedia.org/pipermail/wikien-l/attachments/20030502/ecc5a494/attachment.htm


from_name: Thomas Corell

body:
Nick Reinking wrote:
>
> I didn't say having more entries will reduce performance.  A select on
> one of the columns probably doesn't take a lot of time.  Just having a
> lot of records in old isn't making MySQL slow.  But, it is trying to
> keep each individual recard in memory.  Just look at the bz2 files
> available for download.  The cur table is 73MB, and old is 740MB.  The
> old table makes the database 10x as big.  Ouch!  Since it is that big,
> the processes grow and grow, eventually causing the machine to swap like
> mad.

The database have to handle that. And I think the database is used because
of the fact that a lot of data is involved. Possibly you can setup the
database with appropriate memory usage options, but I think thats done.

>
> Even with the new machine, I think it would be helpful to perhaps split
> old into two databases.  The database we have now would contain
> everything we have now, plus a key to articles on the other database.
> That way, looking at comments and dates of old articles (ie, Page
> History) would still be fast, and we could reduce the memory footprint
> of the primary server.  If you wanted to retrieve an older article, it
> would take that key, connect to the other database and pull it down.
> That other database could have perhaps a lower priority, and a much
> lower memory buffer.  Or, we could store it on the filesystem, whatever.
>
> In the evening (1:00 AM GMT), I see:
> MemTotal:      2059088 kB
> MemFree:         93096 kB
> MemShared:          40 kB
> Buffers:         14892 kB
> SwapTotal:     2047992 kB
> SwapFree:      1289140 kB
>
> That's pretty heavy swap usage (33%), and heavy memory usage.

I hope you know that this values are very useless. You have to know the
ratio of pagefaults to pagehits. Linux often has a lot of nearly never used
things in swapspace.

Smurf
--
Schlumpf at AdamAnt.mud.de
-------------------------  Anthill inside! ---------------------------


subject: [WikiEN-l] Zog is back as Anti-Zog

body:
In a message dated 5/2/2003 6:00:37 AM Eastern Standard Time,
jowens.wiki at ghiapet.homeip.net writes:

> >He's back as Anti-Zog, Baboon Mouth, and JamesERay
>
> Don't forget Niggardly and UncleT, though not much was done as them.
>
>

And I really wish Martin would stop reverting the pages into tributes to
white racism with a Confederate war flag, a picture of HItler, and links to
the lyrics of "Nigger Hating Me". If he was banned, why are we giving him a
forum?

Danny
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.wikimedia.org/pipermail/wikien-l/attachments/20030502/030fbaa3/attachment.htm


body:
In a message dated 5/2/2003 6:36:26 AM Eastern Standard Time,
jwales at bomis.com writes:

> I think we need to revisit having the ability for sysops to ban logged
> in users.  Maybe the wiki way of doing this is to put the ability into
> the software, but all sysops must agree to use it *only* to ban
> *certain* variants on a known banned troll.
>

As much as I would like to see Zog banned, this could potentially lead to
horrible flame wars. Maybe a compromise would be to give sysops the ability
to ban a logged in user for a set period of time, say 12 hours, until a
decision can be reached and a developer can put a longer term ban in place.

Just a suggestion.

Danny
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.wikimedia.org/pipermail/wikien-l/attachments/20030502/8bf8a763/attachment.htm


subject: [WikiEN-l] Zog is back as Anti-Zog

in_reply_to: <1051888132.25586.6.camel@venus.mit.edu>; from cunctator@kband.com on Fri, May 02, 2003 at 11:08:51AM -0400

body:
The Cunctator wrote:
> I don't understand what the problem is. Aren't people just deleting the
> content when it comes on line?

Yes, but it takes a lot of time.  If we're going to delete everything
he writes (and we are), then it makes sense to raise the cost of
persistence to him, and lower the cost of persistence to us.  Username
and ip bans are very effective for that, forcing him to dialup over
and over, and login again and again.

--Jimbo


from_name: David A. Wheeler

body:
Quoting various people:

> >Changes in 'existence' (yes or no) come up infrequently.  When
> someone
> >creates a brand new article, all other cached articles that are
> >affected by the change could be updated at that time. ...
> >
> More than that, you don't have to re-generate the cached pages, you
> only have to invalidate them.
> Thus, when a page is created, you have to "touch" all pages with a
> link to that not-yet-existing article,
> and when a page is deleted, you do the same to all pages that link to

> that article.

Yes.  If stored in a filesystem, the caches of generated HTML for
the linking articles can be simply removed when the existence state
of an article is changed.  This would be relatively rare; looking
at the "recent changes" log shows clearly that most edits are
of EXISTING articles.  Editing a previous existing article shouldn't
modify any caches except of the article being edited.

...
> By not updating articles until they are accessed, you can defer a lot
> of
> work that would otherwise bog the system down at update time.

Yes.  It also means that if someone creates a number of
related "previously non-existing" articles, cached HTML files are
only created when they're needed.

> Lazy
> evaluation is much nicer: when the page is demanded, the code should
> first look for a cached page, and generate it if necessary, before
> using
> that data to generate the article output.

Agreed.

> Potential downsides:
> * some articles are linked by 1000s+ of pages (like the pages linked
> by
> auto-generated articles). Hitting these could cause a significant
> pause
> as the cache is invalidated -- or will it?

No, it shouldn't be a problem in normal circumstances.
An HTML cache should only be changed if
the EXISTENCE of a cached page changes. If an article is
linked to by 1000s+ of pages, it almost certainly already exists,
so no cache (other than the one edited article) would be invalidated.

If an article is created (and didn't exist before), or completely
deleted from the database, I think the usual case is that it would
have have relatively few links (say, 0-10).
Thus, you'd only have a few cache invalidations.

The NON-normal case would actually be interesting.
If an article was widely linked - but didn't exist - then it's
likely that someone finally created "the article everybody wanted".
If that's so, a small hick-up to store a widely-desired article would
be reasonable - people would be glad for the article!
In practice, I doubt there'd be many - the people who monitor the
"most requested" article lists will create article before too many
people link to the non-existent article.
Altenernatively, perhaps all those links are a creation from a
vandalizing bot - in which case, you'd like to know about it.
If an article is deleted, and MANY pages refer to it, I'd
worry - that could signal serious vandalism.

Just to be clear: I'm specifically talking about
caches of stored generated HTML in this email.
This could be done by "front end" web servers inside their
filesystem, without touching MySQL, as long as the web servers
were told when to invalidate their caches.
Say, via a separate process that simply get told via broadcast to
"invalidate cache of article X" - it then removes the
corresponding file (there's a risk of getting "old" articles in
some circumstances - whether or not that's a problem is worth
discussion). You can even imagine, say, 4-5 front-end
webservers, with caches of article to serve read requests,
and only talking to the database when updates occur.

A separate issue is the possibility of storing _all_ of the article
text in the filesystem, instead of MySQL.
If that's done, a cache may or may not be useful; it depends on
how much it takes to render the HTML from the wikitext.
If the rendering to HTML is sufficiently fast, it's possible that
the memory pressure from having both cached & original text will
cause the system to be less efficient than if it regenerated the
text each time.  I suspect caching would still be useful, even
if the original text is in the filesystem; sendfile() and friends
are blazingly fast, while PHP working to render the HTML
simply can't be.

It'd be possible to store both the original wikitext and the
rendered HTML in the filesystem.  If you did that, perhaps they
should be in separate directories,
to simplify moving to multiple front-ends if that's desired later.


from_name: The Cunctator

subject: [Wikipedia-l] Gibson props

body:
Cyberpunk impresario William Gibson gives props to Wiktionary:

So is Google officially a verb now?
When I wrote Pattern Recognition, it occurred to me that I could use it as a
verb and it also occurred to me that someone might already have done so. I
thought it didn't matter too much. If I'm first that's great, but if I'm
not, then it's just good reportage in a way. Sites like Wiktionary track new
usages and neologisms. The page on Google as a verb went back almost two
years!

http://www.guardian.co.uk/online/story/0,3605,946503,00.html


body:
Tomasz Wegrzanowski queried:
>What about storing all markup and all rendered (or semi-rendered)
>html files on disk, under names being md5 hashes of them,
>and having database store only pointers and metadata ?

Storing all markup & rendered (cached) files on disk is one of
the options I've been raising as a possible performance enhancement.

However, I don't see there being a real performance advantage to
storing by hash value, instead of simply using an encoded article name
as the filename.  And, if you use the article name as the filename,
the system's underpinnings become MUCH clearer (making it easier to
debug, etc.).

You probably don't want to just store
"all the files" in one big directory anyway.
Most filesystems do not handle large directories efficiently -
in fact, some are implemented as a linear search, which would make
finding the hashed file a real problem.  This is easily handled
by using the first few characters of the article name as a hashing
function, e.g., Europe is stored in "E/u/r/Europe.wk".
The hash used here is "imperfect", but as a programmer I'd be
grateful for such a simple system when things go wrong.
And it makes lots of processing easier ("process articles in
ASCIIbetical order" is trivial).

Some filesystems _DO_ handle massive directories efficiently.
Reiser does well, for example.  However, but a design that works well
on arbitrary filesystems would have the advantage of letting you
switch to more filesystems, depending on other factors.
If you're willing to accept implementations that limit the
code utility to specific filesystems (like Reiser),
I still don't see the advantage of hashed names - the
underlying filesystem will use its own hashing system anyway,
so you may as well use reasonable names.

However, maybe there's something I've missed.
If there is, please let me know!

Thanks...


from_name: Axel Boldt

body:
--- Erik Moeller <erik_moeller at gmx.de> wrote:

> If you want to change the naming convention,
> propose a  vote.

Since the naming conventions were not brought about by a vote, but by
discussion and consensus, their change now does not require a vote, but
discussion and consensus.

Axel

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


body:
I stated:
>No, it shouldn't be a problem in normal circumstances.
>An HTML cache should only be changed if
>the EXISTENCE of a cached page changes. If an article is
>linked to by 1000s+ of pages, it almost certainly already exists,
>so no cache (other than the one edited article) would be invalidated.

Neil Harris said:

>By "hitting" in this case, I meant "deleting or creating".
>Sorry for the ambiguity.
>Take a look at something like [[census]], which is linked by about
>36.000 articles.

It sounds like we're in agreement.  [[census]] already
exists, and LOTS of articles link to it.  If someone completely
DELETED [[census]] (and didn't just edit it), then
that would cause a lot of caches to be affected.  But that should
be a warning signal - perhaps [[census]] should be edited, but
it should probably still exist as an article (and NOT deleted).

>... but there are lots of
>data integrity and race condition / transaction issues to be thought
>about before any of this can be implemented.

Yes, anytime there are multiple front-ends, there are
potentials for race conditions.  I heartily agree.

>Let's finish splitting the
>system into two machines, DB and WWW, before any re-architecture is
>performed.

Fair enough.

However, if all wikitext is removed from MySQL
and placed in the filesystem, splitting out the MySQL database on
a separate machine may not buy much.  Then, most of the work would be
then done by the filesystem, with only housekeeping metainformation
being accessed through MySQL, and only when editing or accessing
special pages.

Unfortunately, we can't really separate processor architecture from
software architecture - the goal is to maximize performance while
minimizing hardware & development-time cost.  Hopefully, this
kind of free-flowing discussion of alternatives will yield
that perfect combination - or at least a good one.


from_name: Tim Starling

body:
>From: Brion Vibber <brion at pobox.com>
>
>On Thu, 2003-05-01 at 08:53, Nick Reinking wrote:
> > Off the top of my head, I can't think of any simple mathematical way to
> > do want you want to do.  (that being making articles selected randomly
> > less likely to be selected randomly over time)
>
>Not sure who wants to do that... For me it's quite sufficient to make
>the probability of selection random, and let the size of the selection
>field make multiple selections _reasonably_ rare.

With over 100,000 articles, it's unlikely the user will ever see the same
article come up twice. A previous flawed algorithm had the property of
making a few articles much more likely to be selected.

>The resetting of the random weight upon load is just meant to keep the
>pot stirring, and "in theory" shouldn't have any effect at all on
>randomness if the random indexes were random to begin with.

Quite right. I would recommend only setting cur_random on article creation.
It would save that little tiny bit of database load. There's no reason to
think this "pot stirring" will make the selection more random, now that the
pot has been pre-stirred.

> > Seems to me that the best way to do this is either:
> > 1) Be truly random.  In PostgreSQL, this would be something like:
> >      SELECT cur_id from cur LIMIT 1 OFFSET random(SELECT count(cur_id)
>FROM cur)
> >    (this function should be really fast, not sure if faster than:)
>
>Mysql doesn't seem to let you use a non-constant expression for
>'LIMIT offset,count' (and besides, still no subqueries), but we can pick
>a random number easily enough in PHP. The problem is that we aren't sure
>what range to use; hence a random index attached to each article whose
>range is known (between 0 and 1).

The MySQL documentation suggests using "ORDER BY RAND() LIMIT 1", for
versions later than 3.23. But I think the current algorithm is quite clever.
Its only drawback is the overhead associated with maintaining the extra
index, and initialising the field on article creation.

>We don't want to select non-article pages or redirects, but we don't
>currently have a count of exactly that. There's the "good articles"
>count, but it has other criteria, so if we used it we'd always crop off
>relatively recent articles. If we inflate it based on an estimate we'd
>get overemphasis on the _most_ recent article if we estimate too high.
>
>Sure, there's count(*), but that takes time to run... might be faster
>with an index specifically on cur_namespace,cur_is_redirect. Or we could
>maintain another column in the stats table which counts non-redirect
>article pages with _no_ other criteria (such as size or comma/bracket
>count). Rebuilding indexes on cur is _very_ slow, so if we need to
>change the indexes, best to plan ahead.

Maybe we could have a "good article" flag, stored in cur on edit.

-- Tim Starling.

_________________________________________________________________
MSN Instant Messenger now available on Australian mobile phones.?Go to
http://ninemsn.com.au/mobilecentral/hotmail_messenger.asp


from_name: Ray Saintonge

body:
Zoe wrote:

>Films are what auteurs make.
>
Starring Pepe le-Pew :-)

I'm sorry but I can't help it


subject: [WikiEN-l] Zog is back as Anti-Zog

body:
On Fri, 2003-05-02 at 10:10, Tony Wilson wrote:
> Jimbo is right. There are users who make a complete joke of our
> "enforcement" procedures. Right now, Zog has headed one of his user
> talk pages with a picture of a Klan ritual, complete with text: "Look
> out niggers the Klan is getting bigger!!!"
>
> Either we figure out a way to deal promptly with scum like this, or we
> quit pretending that we are trying.
> Tony Wilson
> (Tannin)
>
I don't understand what the problem is. Aren't people just deleting the
content when it comes on line?


from_name: Jimmy O'Regan

subject: [WikiEN-l] Zog is back as Anti-Zog

body:
Tim Starling wrote:

> Zog is back as "Anti-Zog". He's not doing an Adam trick and pretending
> he's someone else, he's just trying to get the song lyrics back into
> [[Thats how a nigger goes]]. Can someone please block him without delay?
>
> -- Tim Starling.
>
Now he's JamesERay.


body:
Lee Daniel Crocker wrote:
> Hey Jason, now that I bring it up, what's the chance that the new
> machine can have a /var partition that we can reformat to Reiserfs?

Perhaps this is just superstition, but I once had a huge headache with
Reiserfs.  Is it sold enough for production nowadays?  Is it likely to
be sufficiently superior to more standard alternatives to risk using
it?

At some point, I fear that too-exotic efforts to squeeze every last
ounce of performance out of the hardware will be counter-productive.

Maybe.

I'm just saying...

--Jimbo