body:
On Tue, 29 Apr 2003, Nick Reinking wrote:

> Date: Tue, 29 Apr 2003 11:44:45 -0500
> From: Nick Reinking <nick at twoevils.org>
> Subject: Re: [Wikitech-l] Chat about Wikipedia performance?
>
> On Mon, Apr 28, 2003 at 05:45:18PM -0500, Lee Daniel Crocker wrote:
> >
> > The code issue is mostly a matter of focus: one or two developers
> > is probably sufficient to keep the codebase up to date, but neither
> > Brion nor I are focused on that right now.
> >
> > So while Brion has issued a call for coders, that could be answered
> > in other ways: for example, if a good admin stepped up to take some
> > of admin tasks Brion is currently swamped with, he might be more free
> > to code (assuming he's interested, which is not a given either).
> > I've chosen to focus more on long-term goals because like Brion I was
> > expecting hardware to bail us out in the short term. If that's going
> > to be delayed, then I can put off things like testing file systems
> > and focus on caching and tuning.
>
> I'm certainly willing to help out here.  I'm not in SoCal, but I should
> be able to help out with most administrivial tasks.  I'm going to be
> able to help out a much more with tuning at a file system/OS/Apache
> level than I will be at a PHP/SQL level.

Since I've just joined on to the tech list, might as well introduce myself
in the tech context. My specialty is administration of routers and WANs,
and along the way I've come to know general Linux quite well, HTML & HTTP,
Apache, Perl & CGI, & MySQL. A few other things that I don't think would
be very relevant, but you never know, would be DNS (Bind, of course),
Sendmail, and TCP/IP details. The only thing holding me back from being a
useful coder so far seems to be that I don't know beans about PHP, but I
could certainly be similarly helpful in that "relief pitcher" kind of way.

--
John R. Owens http://www.ghiapet.homeip.net/
Sleep is a common substitute for adequate caffeine intake.
				--John Owens


list: wikitech-l

source: pipermail-mbox

from_name: Nick Reinking

subject: [Wikitech-l] Possible performance issue?

body:
On Tue, Apr 29, 2003 at 05:07:24PM -0500, Lee Daniel Crocker wrote:
> > (Nick Reinking <nick at twoevils.org>):
> > I notice that /usr (/dev/sda2) is at 96%...
> > Perhaps somebody can remove a bunch of the packages that
> > are installed that we don't use?
>
> The server is very clean in terms of software. The big culprits
> for disk usage are MySQL's Innodb transaction data (currently a
> single 10Gb file!), and logfiles from MySQL and Apache.
>
> I don't know enough about MySQL to know how to limit that data
> and what the impact might be. I do think we could lighten the
> load on Apache log files considerably now, to save both disk
> space and gain some performance. For instance, we logged user
> agents and referrers to get some stats, but I don't think we
> really need that anymore.

Yeah, that's what I was thinking.  Putting the Apache logs on another
drive, or turning off the access_logs altogether would probably be quite
helpful; at the cost of statistics (since we don't have a second drive.

Of course, it doesn't really matter since Apache will be on a different
machine soon enough - but we should still get /usr utilization down,
else we're going to fragment the heck out of that partition.

--
Nick Reinking -- eschewing obfuscation since 1981 -- Minneapolis, MN


list: wikien-l

from_name: Zoe

subject: [WikiEN-l] John Stewart

body:
I'm really beginning to wonder if [[User: John
Stewart]] isn't another troll.

Zoe

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


from_name: sannse

subject: [WikiEN-l] Lir is back

body:
Zoe wrote

> No, Michael and Lir are different people.
>

I'm sorry, I was unclear.  I know Lir and Michael are different people, and
I agree that Ril is almost certainly Lir.

I forget sometimes how difficult it is to express tone in text.

Regards,

sannse

(I also see that the link I gave in my last was broken in two - appologies
for that too)


from_name: Brion Vibber

subject: [WikiEN-l] Zog again

body:
On Tue, 2003-04-29 at 18:55, a.crossman at blueyonder.co.uk wrote:
> Zog still hasn't been banned, and is now uploading pictures of
> primates and linking to them from civil rights activists.

That ain't cool, man.

>  Can a ban be done as a matter of urgency please?

Username 'Zog' and IP address 131.247.157.27 (from University of South
Florida, where we've had troublemakers in the past) are now banned.

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.wikimedia.org/pipermail/wikien-l/attachments/20030429/110e11e2/attachment.pgp


from_name: David A. Wheeler

subject: [Wikitech-l] Chat about Wikipedia performance?

body:
Hi - clearly, it'd be great if Wikipedia had better performance.

I looked at some of the "Database benchmarks" postings,
but I don't see any analysis of what's causing the ACTUAL bottlenecks
on the real system (with many users & full database).
Has someone done that analysis?

I suspect you guys have considered far more options, but as a
newcomer who's just read the source code documentation, maybe
some of these ideas will be helpful:

1. Perhaps for simple reads of the current article (cur),
you could completely skip using MySQL and use the filesystem instead.
Simple encyclopedia articles could be simply stored in the
filesystem, one article per file.  To avoid the huge directory problem
(which many filesystems don't handle well, though Reiser does),
you could use the terminfo trick.. create subdirectories for the
first, second, and maybe even the third characters. E.G., "Europe"
is in "wiki/E/u/r/Europe.text".  The existence of a file can be used as
the link test.  This may or may not be faster than MySQL, but
it's probably faster: the OS developers have been optimizing
file access for a very long time, and instead of having
userspace<->kernel<->userspace interaction, it's
userspace<->kernel interaction.  You also completely avoid
locking and other joyless issues.

2. The generation of HTML from the Wiki format could be cached,
as has been discussed.  It could also be sped up, e.g., by
rewriting it in flex.  I suspect it'd be easy to rewrite the
translation of Wiki to HTML in flex and produce something quite fast.
My "html2wikipedia" is written in flex - it's really fast and didn't
take long to write.  The real problem is, I suspect that
isn't the bottleneck.

3. You could start sending out text ASAP, instead of batching it.
Many browsers start displaying text as it's available, so to
users it might _feel_ faster.  Also, holding text in-memory
may create memory pressure that forces more useful stuff out of
memory.

Anyway, I don't know if these ideas are all that helpful,
but I hope they are.


list: wikipedia-l

from_name: Sean Barrett

subject: [Wikipedia-l] [Wikitech-l] server names

body:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jay Bowks wrote:
| ----- Original Message ----- > > From: Lee Daniel Crocker <lee at piclab.com>
|
|>>>Pliny, Vincent of Beavais, Denis Diderot, Jimbo? (ha ha)
|>>
|>>...don't forget Mortimer Adler. Besides, having a server named
|>>mortimer is about as nerdy as it's possible to get.
|
|
| Pliney, Quintillion, Bevis, DiderDot, pSeudas,
| Gaylen, Harrass (Harris), Rondda(lembert),
| Allstud (Alsted), Morerror, Bail, Huffman,
| ColonelLee, A2Zedler, ZSwinger,
| Mor(e)timer
|
| :-)
|
| [[User:ILVI]]

Isadore (of Seville).

Rue Britannica.

- --
~ Sean Barrett     | For a mountain, I am very young.
~ sean at epoptic.com | For a man, I am just right. --Chiun

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD4DBQE+r0guv/8xpnvE6M8RAnuqAJYznzN4m2fQsT4k3pCBCf3ToZO2AKCUeTFP
gDTks318vNaeNVRSGkMx2A==
=w++B
-----END PGP SIGNATURE-----


from_name: Tomasz Wegrzanowski

subject: [Wikitech-l] server names

body:
On Tue, Apr 29, 2003 at 11:36:27AM -0700, Jimmy Wales wrote:
> Jason Richey wrote:
> > I can barely stand to refer to the current wiki server as "the current
> > wiki server", and I don't even know how I'll refer to the new server.
> > I'd like to give the servers official names, just so I can refer to
> > them in a single word.
>
> I propose db.wikipedia.org for the backend and some system like
> www1.wikipedia.org for the frontend (web) server.  These are internal
> names, not for the public, because of course the public doesn't care
> about that.
>
> In this way, when we add more frontend machines (inevitable) and a
> loadbalancing system, we will have a good naming scheme, www2, www3
> and so on.

But these names are sooo boring ;-)
And they don't follow RFC 1178.


from_name: Jay Bowks

subject: [Wikipedia-l] [Wikitech-l] server names

body:
----- Original Message ----- > > From: Lee Daniel Crocker <lee at piclab.com>
> > > Pliny, Vincent of Beavais, Denis Diderot, Jimbo? (ha ha)
> > ...don't forget Mortimer Adler. Besides, having a server named
> > mortimer is about as nerdy as it's possible to get.

Pliney, Quintillion, Bevis, DiderDot, pSeudas,
Gaylen, Harrass (Harris), Rondda(lembert),
Allstud (Alsted), Morerror, Bail, Huffman,
ColonelLee, A2Zedler, ZSwinger,
Mor(e)timer

:-)

[[User:ILVI]]


subject: [Wikitech-l] server names

body:
On Tue, Apr 29, 2003 at 12:14:35PM -0700, Jimmy Wales wrote:
> Tomasz Wegrzanowski wrote:
> > But these names are sooo boring ;-)
> > And they don't follow RFC 1178.
>
> You're right, and I retract my suggestion completely if someone comes
> up with something more fun and standards-compliant!  :-)
>
> Famous encyclopedists?
>
> Pliny, Vincent of Beavais, Denis Diderot, Jimbo? (ha ha)
>
> Or, how about, for an infinite supply of names, we use various
> incarnations of Lir
>
> Lir, Adam, Bridget, Vera Cruz, Ril, etc., etc., etc., etc.!!!

"helga" would be a wonderful name, don't you think ? ;)


body:
On Tue, 2003-04-29 at 23:33, Lee Daniel Crocker wrote:
> > (David A. Wheeler <dwheeler at dwheeler.com>):
> > 1. Perhaps for simple reads of the current article (cur), you
> > could completely skip using MySQL and use the filesystem instead.
>
> In other words, caching.

Not necessarily; it would also be possible to keep the wiki text in
files. But I'm not sure what great benefit this would have, as you still
have to go looking up various information to render it.

>  Yes, various versions of that have been
> tried and proposed, and more will be. The major hassles are (1) links,
> which are displayed differently when they point to existing pages, so
> a page may appear differently from one view to the next depending on
> the existence of other pages,

That's not a problem; one simply invalidates the caches of all linking
pages when creating/deleting.

This is already done in order to handle browser-side caching; each
page's cur_touched timestamp is updated whenever a linked page is
created or deleted. Simply regenerate the page if cur_touched is more
recent than the cached HTML.

> > 3. You could start sending out text ASAP, instead of batching it.
> > Many browsers start displaying text as it's available, so to
> > users it might _feel_ faster.

A few things (like language links) currently require parsing the entire
wikitext before we output the topbar. Hypothetically we could output the
topbar after the text and let CSS take care of its location as we do for
the sidebar, but this may be problematic (ie in case of varying vertical
size due to word wrap) and would leave users navigationally stranded
while loading.

> >  Also, holding text in-memory
> > may create memory pressure that forces more useful stuff out of
> > memory.
>
> Not an issue. HTML is sent out immediately after it's rendered.

Well... many passes of processing are done over the wikitext on its way
to HTML, then the whole bunch is dumped out in a chunk.

> Things like database updates are deferred until after sending;

I'm not 100% sure how safe this is; if the user closes the connection
from their browser deliberately (after all, the page _seems_ to be done
loading, why is the icon still spinning?) or due to an automatic
timeout, does the script keep running through the end or is it halted in
between queries?

> One things that would be nice is if the HTTP connection could be
> dropped immediately after sending and before those database updates.
> That's easy to do with threads in Java Servlets, but I haven't
> found any way to do it with Apache/PHP.

For some things (search index updates) we use INSERT/REPLACE DELAYED
queries, whose actual action will happen at some point in the future,
taken care of for us by the database. There doesn't seem to be an
equivalent for UPDATE queries.

Hypothetically we could have an entirely separate process to perform
asynchronous updates and just shove commands at it via a pipe or shared
memory, but that's probably more trouble than it's worth.

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20030430/b32259e6/attachment.pgp>


subject: [WikiEN-l] Sysop limits

body:
On Wed, 2003-04-30 at 00:04, John R. Owens wrote:
> I just wanted to check, if somewhat belatedly, that this wouldn't be
> considered an abuse of sysop abilities, if I protect one of my own
> subpages. Taken strictly, it would break the rules, but I don't think it
> would break the spirit of them at all.

The obvious question is: why would you want to do that? What benefit
will it bring you to prevent your neighbors from editing that page? Is
there some reason that other people possibly editing this mysterious
unnamed page would be harmful?

Protecting a page without a pressing technical or 'wikipedia national
security' need is Seriously Un-Wiki Bad Mojo; expect it to get
un-protected and to receive dirty looks from passers-by if anyone
notices you've done it. ;)

It very very much violates the spirit of the rules; locking of pages is
a technical remedy meant for very rare emergency cases; it isn't
something you do because they're "your" pages and you don't want other
people to touch them. Subpages of your user page (like your user page
itself) are "socially" in your space, but it's a friendly, neighborly,
open space into which you invite your neighbors to play in if they're
interested. There's no picket fence with a locked gateway, and certainly
no razor wire and guard dogs or "no tresspassing" sign.

Of course, if a persistent vandal has been tromping all over one of
"your" subpages, then it probably makes sense to protect it. (But better
if you don't keep editing it during that period.)

> Also, while I'm at it, I wasn't sure about an edit I did earlier on
> [[Johnny Rebel]] while it was protected. It was just a simple typo
> correction, "Moustrap" -> "Mousetrap". Is that kind of controversy-free
> thing OK while it's protected, or should it be completely avoided to
> prevent even the appearance of possible abuse?

I've got no real objection to fixing typos I guess, unless typos are the
subject of controversy. ;)

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.wikimedia.org/pipermail/wikien-l/attachments/20030430/7bddff3b/attachment.pgp


subject: [WikiEN-l] [Fwd: [ wikipedia-Feature Requests-730012 ] scientific names]

body:
And people ask us why we never seem to respond to feature requests... ;)

-- brion vibber (brion @ pobox.com)

----------------------------------------------------------------------

Category: Page editing
Group: None
Status: Open
Priority: 5
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Lee Daniel Crocker (lcrocker)
Summary: scientific names

Initial Comment:
to do separate the scientific names from the common
languages normally use italics

But I can not do make headword by
italics.

Of this little cause, Wikipedia do never beeen a
serious encyclopedia

----------------------------------------------------------------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.wikimedia.org/pipermail/wikien-l/attachments/20030430/fcb39880/attachment.pgp


from_name: Lee Daniel Crocker

body:
> (David A. Wheeler <dwheeler at dwheeler.com>):
>
> 1. Perhaps for simple reads of the current article (cur), you
> could completely skip using MySQL and use the filesystem instead.

In other words, caching. Yes, various versions of that have been
tried and proposed, and more will be. The major hassles are (1) links,
which are displayed differently when they point to existing pages, so
a page may appear differently from one view to the next depending on
the existence of other pages, and (2) user settings, which will cause
a page to appear differentlt for different users. But caching is
still possible within limits, and using the filesystem rather than
the database to store cached page info is certainly one possible
implementation to be tried.

> [Rendering] could also be sped up, e.g., by rewriting it in flex.
> My "html2wikipedia" is written in flex - it's really fast and didn't
> take long to write.  The real problem is, I suspect that
> isn't the bottleneck.

It isn't. And there's no reason to expect flex to be any faster
than any other language.

> 3. You could start sending out text ASAP, instead of batching it.
> Many browsers start displaying text as it's available, so to
> users it might _feel_ faster.  Also, holding text in-memory
> may create memory pressure that forces more useful stuff out of
> memory.

Not an issue. HTML is sent out immediately after it's rendered.
Things like database updates are deferred until after sending;
the only time taken before that is spent in rendering, and as I
said, that's not a bottleneck.

One things that would be nice is if the HTTP connection could be
dropped immediately after sending and before those database updates.
That's easy to do with threads in Java Servlets, but I haven't
found any way to do it with Apache/PHP.

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


from_name: John R. Owens

subject: [WikiEN-l] Sysop limits

body:
I just wanted to check, if somewhat belatedly, that this wouldn't be
considered an abuse of sysop abilities, if I protect one of my own
subpages. Taken strictly, it would break the rules, but I don't think it
would break the spirit of them at all.

Also, while I'm at it, I wasn't sure about an edit I did earlier on
[[Johnny Rebel]] while it was protected. It was just a simple typo
correction, "Moustrap" -> "Mousetrap". Is that kind of controversy-free
thing OK while it's protected, or should it be completely avoided to
prevent even the appearance of possible abuse? I just can't stand seeing
misspellings left alone, though. :(
Side note: It's so bad, since I started doing Wikipedia fairly regularly,
I find myself frequently looking for an "Edit this page" link while I'm on
other, non-wiki sites and I see an error. ;)

--
John R. Owens http://www.ghiapet.homeip.net/
Only one human captain has ever survived battle with the Minbari fleet.
He is behind me.  You are in front of me.  If you value your lives, be
somewhere else.  		--Delenn


from_name: a.crossman at blueyonder.co.uk

subject: [WikiEN-l] Zog again

body:
Hi,

Zog still hasn't been banned, and is now uploading pictures of
primates and linking to them from civil rights activists. Can a ban
be done as a matter of urgency please?

Evercat

--
Allan Crossman
a.crossman at blueyonder.co.uk
http://dogma.pwp.blueyonder.co.uk


subject: [Wikitech-l] Apache updates

body:
At Nick's suggestion, I recompiled Apache with mod_mmap_static (and
upgraded PHP to 4.3 while I was at it). It tested well on Piclab, so
I installed it on the server and made the logo, icon, stylesheets,
and robots.txt memory mapped.

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


from_name: Jimmy Wales

subject: [WikiEN-l] Zog again

in_reply_to: <1051672787.32061.1.camel@frank.vibber.org>; from brion@pobox.com on Tue, Apr 29, 2003 at 08:19:48PM -0700

body:
Brion Vibber wrote:
> Username 'Zog' and IP address 131.247.157.27 (from University of South
> Florida, where we've had troublemakers in the past) are now banned.

Gee, that's right here where I live.

--Jimbo


subject: [WikiEN-l] Sysop limits

body:
John R. Owens wrote:
> I just wanted to check, if somewhat belatedly, that this wouldn't be
> considered an abuse of sysop abilities, if I protect one of my own
> subpages. Taken strictly, it would break the rules, but I don't think it
> would break the spirit of them at all.

It seems like a very minor issue to me, but I'm curious as to why you
want to do it.  Is there a problem with someone trying to vandalize
the page or something?  If not, the wiki way is to just relax and
trust people not to screw with your page.  :-)

> Also, while I'm at it, I wasn't sure about an edit I did earlier on
> [[Johnny Rebel]] while it was protected. It was just a simple typo
> correction, "Moustrap" -> "Mousetrap". Is that kind of controversy-free
> thing OK while it's protected, or should it be completely avoided to
> prevent even the appearance of possible abuse? I just can't stand seeing
> misspellings left alone, though. :(

I have no strong opinion about this.  The pro's and con's a pretty
obvious, and you've spelled them out pretty well.

--Jimbo


from_name: Anthere

subject: [Wikipedia-l] Developper limits

body:
Hum, I am a bit embarrassed here. But well...

We have a bottle neck problem I fear.

Utilisateur:Alvaro asked to be a sysop on the french
wiki quite a while ago, and nobody spoke against
(indeed, several spoke for him). Then, since this is
asked on our pump, he was first forgot for a while.
Had to kindly ask again.
So I put a message asking for him to be made sysop in
the database on your page Brion, on the english
wikipedia, as well as the metapedia if I remember
well.

He still is not sysop. It is clear it is a pain for
you developpers to do this type of chore; Ed
mentionned it several times. So, we wait, and ask
again, and again.

But, then, isnot there a way for us to make people on
the international wikipedia ourselves ?

Could there be something like a query, which could
allow french sysop to make french user (just an
example of course) a sysop automatically ? Maybe a
list of user could be displayed, and one sysop could
click on the name of one user to make him sysop ? Of
course, it would be absolutely required then than a
log is clearly visible to everyone, to avoid abuse.

Is something like this possible or not ?

Meanwhile, could someone please make Alvaro sysop ?

Also, as Aoineko mentionned previously, the mispelling
page is quite broken, and has never been working well
since we are phase III (31th of october). It was
already reported.

I know you are overbooked. Is there something we can
do (ok, not me, but there are some developpers among
us) to make that work ? Do the other international
wikipedias using accentuated letters do have the
problem as well ?

Anthere

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


subject: [Wikitech-l] Developper limits


subject: [WikiEN-l] Sysop limits

body:
On 30 Apr 2003, Brion Vibber wrote:

> Date: 30 Apr 2003 01:14:20 -0700
> From: Brion Vibber <brion at pobox.com>
> Subject: Re: [WikiEN-l] Sysop limits
>
> On Wed, 2003-04-30 at 00:04, John R. Owens wrote:
> > I just wanted to check, if somewhat belatedly, that this wouldn't be
> > considered an abuse of sysop abilities, if I protect one of my own
> > subpages. Taken strictly, it would break the rules, but I don't think it
> > would break the spirit of them at all.
>
> The obvious question is: why would you want to do that? What benefit
> will it bring you to prevent your neighbors from editing that page? Is
> there some reason that other people possibly editing this mysterious
> unnamed page would be harmful?
>
> Protecting a page without a pressing technical or 'wikipedia national
> security' need is Seriously Un-Wiki Bad Mojo; expect it to get
> un-protected and to receive dirty looks from passers-by if anyone
> notices you've done it. ;)
>
> It very very much violates the spirit of the rules; locking of pages is
> a technical remedy meant for very rare emergency cases; it isn't
> something you do because they're "your" pages and you don't want other
> people to touch them. Subpages of your user page (like your user page
> itself) are "socially" in your space, but it's a friendly, neighborly,
> open space into which you invite your neighbors to play in if they're
> interested. There's no picket fence with a locked gateway, and certainly
> no razor wire and guard dogs or "no tresspassing" sign.
>
> Of course, if a persistent vandal has been tromping all over one of
> "your" subpages, then it probably makes sense to protect it. (But better
> if you don't keep editing it during that period.)

Hmm, definite grey area. It's a list of links to persistent vandal and
potential vandal User:, User_talk:, & contributions pages that I was
referring to. So while it hasn't come up on any vandal's radar screen yet,
I would like to know that none of them have been altering the contents
without needing to check the history each time, or anything like that. And
yes, of course I've already added it to my watchlist.

--
John R. Owens http://www.ghiapet.homeip.net/
Londo Mollari: It is good to have friends, is it not, Mr. Garibaldi?  Even
	if maybe only for a little while.
Michael Garibaldi: Even if only for a little while.


from_name: Jason Richey

subject: [Wikitech-l] Developper limits

body:
Done...  This is the first I recal hearing of this.  Was it posted to
this list originally?

jason

Anthere wrote:

>
> Hum, I am a bit embarrassed here. But well...
>
> We have a bottle neck problem I fear.
>
> Utilisateur:Alvaro asked to be a sysop on the french
> wiki quite a while ago, and nobody spoke against
> (indeed, several spoke for him). Then, since this is
> asked on our pump, he was first forgot for a while.
> Had to kindly ask again.
> So I put a message asking for him to be made sysop in
> the database on your page Brion, on the english
> wikipedia, as well as the metapedia if I remember
> well.
>
> He still is not sysop. It is clear it is a pain for
> you developpers to do this type of chore; Ed
> mentionned it several times. So, we wait, and ask
> again, and again.
>
> But, then, isnot there a way for us to make people on
> the international wikipedia ourselves ?
>
> Could there be something like a query, which could
> allow french sysop to make french user (just an
> example of course) a sysop automatically ? Maybe a
> list of user could be displayed, and one sysop could
> click on the name of one user to make him sysop ? Of
> course, it would be absolutely required then than a
> log is clearly visible to everyone, to avoid abuse.
>
> Is something like this possible or not ?
>
> Meanwhile, could someone please make Alvaro sysop ?
>
> Also, as Aoineko mentionned previously, the mispelling
> page is quite broken, and has never been working well
> since we are phase III (31th of october). It was
> already reported.
>
> I know you are overbooked. Is there something we can
> do (ok, not me, but there are some developpers among
> us) to make that work ? Do the other international
> wikipedias using accentuated letters do have the
> problem as well ?
>
> Anthere
>
> __________________________________
> Do you Yahoo!?
> The New Yahoo! Search - Faster. Easier. Bingo.
> http://search.yahoo.com
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at wikipedia.org
> http://www.wikipedia.org/mailman/listinfo/wikitech-l

--
"Jason C. Richey" <jasonr at bomis.com>


subject: [Wikitech-l] Developper limits

body:
> Done, Alvaro is now a sysop.

thanks for him

> > Could there be something like a query, which could
> > allow french sysop to make french user (just an
> > example of course) a sysop automatically ?
>
> I believe the last thing that was suggested was to
> require some minimum
> time to pass before it took effect, and to allow
> other people to cancel
> the request during that time. See:
>
http://meta.wikipedia.org/wiki/User_details_special_page

when are we started this debate again ?

> > Also, as Aoineko mentionned previously, the
> mispelling
> > page is quite broken...
>
> If I understand it correctly, those using latin1
> will likely have the
> same problem, while those using unicode should _not_
> have that problem.
> On the other hand, they might not work at all. I'm
> not very familiar
> with the workings of the mispeelings page... Making
> it both work right
> _and_ not bog down the server may be tricky.

Then maybe I should say there is no problem, before
any one suggest we should upgrade to
unicode...oups...I said nothing...everything is
working perfectly :-)))

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


subject: [Wikitech-l] Developper limits

body:
--- Jason Richey <jasonr at bomis.com> wrote:
> Done...  This is the first I recal hearing of this.
> Was it posted to
> this list originally?
>
> jason
>
no, we usually ask Brion directly (which I did on meta
and english). Or post it at the french list, which
Brion read. But, when I saw he was obviously not
around, I thought useless to post it on our list,
where no other developper would see it. I don't any
other developper did upgrade anyone sysop for us. So,
we entirely rely on Brion. I don't think it is the
best option to entirely rely only on one person.

the option indicated on meta would be a good one

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


subject: [Wikitech-l] Developper limits

body:
On Wed, 2003-04-30 at 04:07, Anthere wrote:
> Utilisateur:Alvaro asked to be a sysop on the french
> wiki quite a while ago...
> So I put a message asking for him to be made sysop in
> the database on your page Brion, on the english
> wikipedia, as well as the metapedia if I remember
> well.

So you did. I was out of town last week and got behind on various
messages, sorry!!

Done, Alvaro is now a sysop.

> Could there be something like a query, which could
> allow french sysop to make french user (just an
> example of course) a sysop automatically ?

I believe the last thing that was suggested was to require some minimum
time to pass before it took effect, and to allow other people to cancel
the request during that time. See:
http://meta.wikipedia.org/wiki/User_details_special_page

> Also, as Aoineko mentionned previously, the mispelling
> page is quite broken...

If I understand it correctly, those using latin1 will likely have the
same problem, while those using unicode should _not_ have that problem.
On the other hand, they might not work at all. I'm not very familiar
with the workings of the mispeelings page... Making it both work right
_and_ not bog down the server may be tricky.

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20030430/14e5f802/attachment.pgp>


subject: [WikiEN-l] Zog again

body:
--- Jimmy Wales <jwales at bomis.com> wrote:
> Brion Vibber wrote:
> > Username 'Zog' and IP address 131.247.157.27 (from
> University of South
> > Florida, where we've had troublemakers in the
> past) are now banned.
>
> Gee, that's right here where I live.
>
> --Jimbo

Did you check you could still edit ?

Concerned Anthere

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


from_name: Daniel Mayer

subject: [Wikitech-l] Re: server names

body:
How about Yin and Yang? Or chaos and opportunity?

-- mav


subject: [WikiEN-l] Zog again

body:
Oliver Pereira wrote:
> On Wed, 30 Apr 2003, Jimmy Wales wrote:
> > Gee, that's right here where I live.
>
> Aha! Now we have a prime suspect...

Evidence:
http://www.bomis.com/hawaii/

And check this picture from Larry's wedding:
http://www.timshell.com/pics/lasvegas/JimmyKira4.jpg

Note the monkey.

--Jimbo


subject: [WikiEN-l] Re: Sysop limits

body:
John R. Owens wrote:
> I just wanted to check, if somewhat belatedly, that this wouldn't be
> considered an abuse of sysop abilities, if I protect one of my own
> subpages. Taken strictly, it would break the rules, but I don't think it
> would break the spirit of them at all.

Hm. Why would you want to do that? I guess it wouldn't be abuse, but odd.

> Also, while I'm at it, I wasn't sure about an edit I did earlier on
> [[Johnny Rebel]] while it was protected. It was just a simple typo
> correction, "Moustrap" -> "Mousetrap". Is that kind of controversy-free
> thing OK while it's protected, or should it be completely avoided to
> prevent even the appearance of possible abuse? I just can't stand seeing
> misspellings left alone, though. :(

Some people jump all over any appearance of abuse (as a few users did after
they saw a few Admins edit the page while it was protected). So yes it is not
a great idea to edit a page at all when it has been protected to stop an edit
war.

> Side note: It's so bad, since I started doing Wikipedia fairly regularly,
> I find myself frequently looking for an "Edit this page" link while I'm on
> other, non-wiki sites and I see an error. ;)

What? There is an Internet outside of Wikipedia? :-)

-- Daniel Mayer (aka mav)


from_name: Krzysztof Kowalczyk

subject: [Wikitech-l] Question about accessing HTTP variables in Wikipedia code

body:
Hi,

I have a question about Wikipedia code. I noticed that the way it
accesses GET/POST variables from URL is by using global variables. There
are two problems with that:
- it doesn't work if register_globals options is off (which is a default
in newer versions of PHP)
- it is considered to be a security risk
(http://www.php.net/manual/en/configuration.directives.php#ini.register-globals,
http://www.php.net/manual/en/security.registerglobals.php)

The fix for those problems is very simple: for each variable passed
through GET/POST add the code like this:
$title = $HTTP_GET_VARS['title'];

My questions:
a) is there any special reason it's being done this way in Wikipedia?
b) any chance it can be changed? If yes, what can I do to help make this
happen (I can write the code, test it and submit a patch)

Thanks,

Krzysztof Kowalczyk


subject: [Wikitech-l] Ad-hoc changes

body:
On Tue, 29 Apr 2003, Mark Christensen wrote:
> The only issue I see would be if database threads crowd out apache on
> the old server, and all the foreign wiki's go down, while the English
> wiki stays up.  This could create new tensions with the foreign wiki's.

And this potential temporary situation would be different from the
present how?

-- brion vibber (brion @ pobox.com)


from_name: cprompt

subject: [WikiEN-l] Zog again

body:
Maybe you should go and have a talk with them. =P

Jimmy Wales wrote:

>Brion Vibber wrote:
>
>
>>Username 'Zog' and IP address 131.247.157.27 (from University of South
>>Florida, where we've had troublemakers in the past) are now banned.
>>
>>
>
>Gee, that's right here where I live.
>
>--Jimbo
>
>


subject:
[Wikitech-l] Question about accessing HTTP variables in Wikipedia
 code

body:
On Wed, 30 Apr 2003, Krzysztof Kowalczyk wrote:
> I have a question about Wikipedia code. I noticed that the way it
> accesses GET/POST variables from URL is by using global variables. There
> are two problems with that:
> - it doesn't work if register_globals options is off (which is a default
> in newer versions of PHP)

The wiki uses a number of non-standard options...

> - it is considered to be a security risk

Sure, if you use *uninitialized* global variables and assume they can only
have trusted values. Don't do that. :)

> My questions:
> a) is there any special reason it's being done this way in Wikipedia?

Force of habit.

> b) any chance it can be changed? If yes, what can I do to help make this
> happen (I can write the code, test it and submit a patch)

Sure, please send patches. $_GET / $_POST are ugly as heck, but it's
theoretically a better coding practise.

Keep in mind that a few things might work by either GET or POST
(searches; some legit bots).

-- brion vibber (brion @ pobox.com)


body:
On Wed, 30 Apr 2003, Lee Daniel Crocker wrote:
> > The wiki uses a number of non-standard options...
>
> Actually, register_globals is the only thing you have to change
> in php.ini to get the wiki running.

You also need iconv support compiled in, although for a latin-1-only wiki
that doesn't need to interact with incoming and outgoing links in UTF-8 it
_probably_ won't get triggered.

> > Sure, if you use *uninitialized* global variables and assume they
> > can only have trusted values. Don't do that. :)
>
> Hopefully. I'm not that confident that either we don't do that, or
> that future coders won't do that, so I think avoiding the problem by
> coding so that register_globals isn't needed is a good idea.

Yup. Like overflowing your buffers: nobody does it on _purpose_. :)

> In SearchEngine.php, I used $_REQUEST[], because I don't really
> care whether the variables come from a GET or a POST.

Oh hey, I learn something new every day. :)

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] Developper limits

body:
> (Anthere <anthere6 at yahoo.com>):
>
> no, we usually ask Brion directly (which I did on meta
> and english). Or post it at the french list, which
> Brion read. But, when I saw he was obviously not
> around, I thought useless to post it on our list,
> where no other developper would see it. I don't any
> other developper did upgrade anyone sysop for us. So,
> we entirely rely on Brion. I don't think it is the
> best option to entirely rely only on one person.

This mailing list is the place to put sysop requests.
That's what the Wikipedia:Administrators page says to do,
and if that page isn't yet translated onto the French wiki,
it should be. ALL of the developers read this list, and
any of us can respond to a simple sysop request.

In the long run, we'll have a process that doesn't require
developer intervention, but we've got more pressing matters
to deal with at the moment.

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


body:
> (Tim Starling <ts4294967296 at hotmail.com>):
> [Notes on caching]

When I recompiled Apache to add mod_mmap_static, I also updated
PHP to 4.3, and compiled the latter with the shared memory functions.
After we get things stable again, the first optimization I plan to
try is to have a shared-memory cache of existing page titles, so we
don't have to go to the database dozens of times per page render.
I suspect that will gain us more than caching whole pages.

But for now, we're in rescue mode.

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


body:
> [register_globals problem]
> My questions:
> a) is there any special reason it's being done this way in Wikipedia?
> b) any chance it can be changed? If yes, what can I do to help make this
> happen (I can write the code, test it and submit a patch)

No, and yes. You'll notice I already started doing that for
SearchEngine.php.  If you want to help me out with the others,
go for it. Please see that as a model.

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


subject: [Wikitech-l] Ad-hoc changes

body:
--- Mark Christensen <mchristensen at humantech.com>
wrote:
> The only issue I see would be if database threads
> crowd out apache on
> the old server, and all the foreign wiki's go down,
> while the English
> wiki stays up.  This could create new tensions with
> the foreign wiki's.
>
>
> -----Original Message-----
> From: Lee Daniel Crocker [mailto:lee at piclab.com]
> Sent: Tuesday, April 29, 2003 5:13 PM
> To: wikitech-l at wikipedia.org
> Subject: Re: [Wikitech-l] Ad-hoc changes
>
>
> > (Brion Vibber <vibber at aludra.usc.edu>):
> > [Configuration info]
>
> Here's a thought: when we get the new server up,
> let's install the
> English wiki cleanly from the distribution, make
> sure it's all happy,
> then do the switch, _leaving the foreign wikis on
> the old server_ for a
> while, then do the same thing for each of the
> foreign ones in turn,
> updating them to the newest software, and making
> everything in sync,
> before we shut down apache on the old server.
>
> Any problems with that?

It depends on the following points

1) what does it imply that english is on one server,
and the international wikis on another ?

Does that impact only on the interwiki links ? If so,
we can very well put a message on the main page, on
the recent change and on the pump to warn users of the
event. (Were would the links go to meanwhile ?)

Are there other impacts as well ?

2) what does that mean "for a while" ? Depends on the
length of the while. A month would be an delay
difficult to accept I guess. A couple of days probably
would not.

Fact is, when an update is done on the en server, the
base is locked for quite a long time. Our updates are
of course done much more quickly. I think if you tell
internationals "everyone is down for 2 days - then all
will be set (with luck)" versus "we take care of the
english first - meanwhile your wikis are up - and we
will take of you one after the others within 5 days",
the second option could maybe be preferred.

Of course, I have no idea of what delay means :-)

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


subject: [Wikitech-l] Ad-hoc changes

body:
--- Brion Vibber <vibber at aludra.usc.edu> wrote:
> On Tue, 29 Apr 2003, Mark Christensen wrote:
> > The only issue I see would be if database threads
> crowd out apache on
> > the old server, and all the foreign wiki's go
> down, while the English
> > wiki stays up.  This could create new tensions
> with the foreign wiki's.
>
> And this potential temporary situation would be
> different from the
> present how?
>
> -- brion vibber (brion @ pobox.com)

This is not a very nice comment Brion. I think people
are very tired of the server being very very slow, and
will be quite tolerant

At least, I hope so.

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


from_name: Thomas Corell

subject: [Wikitech-l] server names

body:
Jimmy Wales wrote:

> In this way, when we add more frontend machines (inevitable) and a
> loadbalancing system, we will have a good naming scheme, www2, www3
> and so on.

And for the mailserver (mailinglists) e.g. mail.

Have you enought IP - Addresses to give the mail-service it's own?
Then you can add an additional IP to one of the servers. If somethink goes
wrong, only move the IP Address to get the service running on the other
machine. You can also have a mailinglist service preinstalled on the second
machine.

Smurf
--
-------------------------  Anthill inside! ---------------------------


subject: [Wikitech-l] Developper limits

body:
--- Lee Daniel Crocker <lee at piclab.com> wrote:
> This mailing list is the place to put sysop
> requests.
> That's what the Wikipedia:Administrators page says
> to do,
> and if that page isn't yet translated onto the
> French wiki,
> it should be. ALL of the developers read this list,
> and
> any of us can respond to a simple sysop request.
>
> In the long run, we'll have a process that doesn't
> require
> developer intervention, but we've got more pressing
> matters
> to deal with at the moment.

Okay. Will add it to our admin page.

No, all of our meta pages are not entirely translated
from the english wiki.
It is not especially fun to do, not many people do
care of these matters, so many things are not done
yet.
Besides, some of our meta decisions are different from
the english wiki, so some of our pages are different.

But, I agree on this matter, this list is the right
place to ask, so from now on, I will ask here.

I just feel a bit "cheated". As already said, Brion
has been the only one to ever help us on these
matters, you specifically mentionned many months ago
you did not intend to take care of international
matters, and Ed sounds like he is bored about it.
Hence, the habit of asking Brion directly. I am sorry
that was a wrong move. Apologies. Anthere

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


from_name: Ray Saintonge

subject: [WikiEN-l] RE: A plea for sanity in capitalisation from the coalface

body:
Krzysztof P. Jasiutowicz wrote:

>Another area of possible problematic capitalization are names of diseases and syndromes. I feel especially Americans tend to excessively capitalize every word in these names. Diabetes Mellitus, Chronic Fatigue Syndrome, etc etc
>
>Can this be clarified and the proper practice added to naming conventions
>please ?
>
This is an excellent point.  In a general work there needs to be a
generalized approach to this matter.  If we agree to a special rule to
please the birds we can expect that the whole argument will break out
again at some later time.  It could be with diseases, but it could also
be in any of a large number of specialized areas.

(BTW.  Although I am usually responsive to blaming the Americans for
everything, it needs to be remarked that the capitalist charge in this
matter has been led by an Australian.) ;-)

I surely think we need a technical review of how we capitalize things.
 Unless there really are two separate articles for [[common crow]] and
[[Common Crow]] the software should bring us to the one that exists.
 Options for dealing with ambiguities could be developed.

Linked to this is how we handle accented characters so that "e" and "?"
don't need to be distinguished unless there is an ambiguity.  It also
ties in with the issue of enforced capitals in the first letter of an
article title.  That has proven problematical in Wiktionary.

The problem of capitalized bird names operates on two levels.  As long
as it is only on a stylistic level many of us would probably not be so
concerned about it that we would change every occurrence of a bird name
in the middke of an article to our preferred styles.  When it starts to
affect the titles of articles and how those titles are linked it is a
more serious issue because it relates to the functionality of Wikipedia.

Eclecticology


subject: [Wikitech-l] Re: Developper limits

body:
--- Giskart <giskart at wikipedia.be> wrote:
> Anthere <anthere6 at yahoo.com> wrote in
>
news:20030430110711.39550.qmail at web41708.mail.yahoo.com:
> >
> > But, then, isnot there a way for us to make people
> on
> > the international wikipedia ourselves ?
> >
> > Could there be something like a query, which could
> > allow french sysop to make french user (just an
> > example of course) a sysop automatically ? Maybe a
> > list of user could be displayed, and one sysop
> could
> > click on the name of one user to make him sysop ?
> Of
>
> I must say that until now the responds time about
> serious problems or
> requests to make a user a sysop is not to bad if you
> take in account how
> much work Brion has.

Absolutely ! But given the amount of important things
the developpers have to do right now, to solve
performance issue, I think we are somehow making poor
use of their time having this kind of request. I am
just trying to have a more efficient process here.

er....delegation of authority ? handing over chores to
others ? unballasting a ship ? :-)

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


from_name: Oliver Pereira

subject: [WikiEN-l] Zog again

body:
> Brion Vibber wrote:
> > Username 'Zog' and IP address 131.247.157.27 (from University of South
> > Florida, where we've had troublemakers in the past) are now banned.

On Wed, 30 Apr 2003, Jimmy Wales wrote:
> Gee, that's right here where I live.

Aha! Now we have a prime suspect...

;)

Oliver

+-------------------------------------------+
| Oliver Pereira                            |
| Dept. of Electronics and Computer Science |
| University of Southampton                 |
| omp199 at ecs.soton.ac.uk                    |
+-------------------------------------------+


body:
>> - it doesn't work if register_globals options is off (which is
>> a default in newer versions of PHP)
>
> The wiki uses a number of non-standard options...

Actually, register_globals is the only thing you have to change
in php.ini to get the wiki running.

> > - it is considered to be a security risk
>
> Sure, if you use *uninitialized* global variables and assume they
> can only have trusted values. Don't do that. :)

Hopefully. I'm not that confident that either we don't do that, or
that future coders won't do that, so I think avoiding the problem by
coding so that register_globals isn't needed is a good idea.

> > My questions:
> > a) is there any special reason it's being done this way in Wikipedia?
>
> Force of habit.

Don't forget laziness. :-)

> Sure, please send patches. $_GET / $_POST are ugly as heck, but
> it's theoretically a better coding practise.

In SearchEngine.php, I used $_REQUEST[], because I don't really
care whether the variables come from a GET or a POST.

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


from_name: Geoffrey Thomas

subject: [Wikitech-l] Caching

body:
A suggestion for caching: the home page, [[Main
Page]], never should include links to nonexistent
pages. Could that be cached (in standard stylesheet),
e.g., as /index.html? I would think that page is
accessed very frequently and that caching it would
save some database work for other pages. When the page
is edited (only by a sysop), he/she would render the
page to HTML, possibly by explicitly accessing
/wiki/Main_Page, and save it as /index.html. Would
this help some server strain?

Could we also cache other pages known to link only to
existing pages, such as this week in dates ([[April
30]], etc.), [[Current events]], and [[Recent
deaths]]? Or are these pages updated too often to be
useful? Could we also cache the protected pages?

-[[User:Geoffrey]]

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


subject: [Wikitech-l] Developper limits

body:
--- Anthere <anthere6 at yahoo.com> wrote:
>
> --- Lee Daniel Crocker <lee at piclab.com> wrote:
> > This mailing list is the place to put sysop
> > requests.
> > That's what the Wikipedia:Administrators page says
> > to do,

Btw, the en:Wikipedia:Administrators page absolutely
do not indicate this mailing list as being the right
place to ask for sysop rights. It mentions it *only*
for developper status.

The page recommand to ask sysop status on the
en.mailing list. Then I guess someone agrees for the
new sysop, then by some magic, the user is named
sysop.

Clearly, international wikipedias cannot follow that
pattern. No international user will come either to the
english list, nor to the tech list for sysophood, but
likely to the specific language list, or the admin
page, or pump. Hence, the procedure given for
english-speaking people cannot be identically followed
on international wikipedia. It requires one more step,
as not only other users have to agree, but someone has
to ask and check until it is done.

Regards

Anthere

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


from_name: Brion Vibber

subject: [Wikitech-l] Re: Possible performance issue?

body:
On Wed, 30 Apr 2003, Giskart wrote:
> I like also stats. If possibel, i would like more stats. Now there are only
> Top 30 of 10972 Total Referrers. I would like to have the top 250. Same
> whit Top 30 of 12904 Total URLs. I like the top 500.

Well, at the least we should merge the fifty variants of google. :)

> Now there is no registration of the Top of Total Countries.

Yeah, I had to disable that because it took ALL DAY to do the reverse DNS
lookups. Hypothetically we could have Apache do lookups as connections
happen and store the hostnames in the log, but that would lead to a
performance hit during busy times...

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] Caching

body:
On Wed, 30 Apr 2003, Geoffrey Thomas wrote:
> A suggestion for caching: the home page, [[Main
> Page]], never should include links to nonexistent
> pages. Could that be cached (in standard stylesheet),
> e.g., as /index.html?

I did this manually during the slashdotting a couple months back. It was
probably a bigger help then, as a LARGE chunk of our hits came from one
direct link to http://www.wikipedia.org/ from the /. article.

Currently, the main page accounts for 1.2% of hits. Not zero by any means,
but not _that_ huge. Compare with 2.71% in January, where the whole
month's average was shot up by the /. story spike.

It's not really an issue of nonexistent links, though, but of general
things that change: news items are added daily, the date counter changes
on its own daily, the article count changes constantly. And of course the
login issue.

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] XHTML, CSS...

body:
Just for information...

One of our user, Oliezekat has decided that it would
be nice for the french wikipedia to provide
user-modifiable main page; with possibly blocs of
information movable, and colors choosable at user
wish.

For this, he mentionned some stuff I didnot understand
very well, css external files for colors, some static
links or something...

He put his proposition of page on the french main
page. I must mention this page requires XHTML
politically correct browser.

Attached what I see of this proposition of him of our
main page.

To compare with http://fr.wikipedia.org/wiki/Accueil

If XHTML is accepted on the main page, there is no
reason it won't be afterwards on other pages. Of
course, that implies that wikipedia becomes
non-editable non readable to non-XHTML friendly
browsers.

I am a bit embarrassed, and would be happy to have any
advice on this.

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Main page.jpg
Type: image/jpeg
Size: 113051 bytes
Desc: Main page.jpg
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20030430/3b344385/attachment.jpg>


subject: [Wikitech-l] Developper limits

body:
--- Lee Daniel Crocker <lee at piclab.com> wrote:
> > (Anthere <anthere6 at yahoo.com>):
> >
> > Btw, the en:Wikipedia:Administrators page
> absolutely
> > do not indicate this mailing list as being the
> right
> > place to ask for sysop rights. It mentions it
> *only*
> > for developper status.
> >
> > The page recommand to ask sysop status on the
> > en.mailing list. Then I guess someone agrees for
> the
> > new sysop, then by some magic, the user is named
> > sysop.
>
> Sorry, I assumed this was posted to wikipedia-l, not
> wikitech-l, since it's not a technical issue. Yes,
> wikipedia-l is the right place to ask.

ok again. Changing the page.

> And the magic is "update user set
> user_rights='sysop'
> where user_name='newsysopname';", which can be done
> from a MySQL prompt on the server, or with the SQL
> query page by any developer.

I know so little of database handling that I would
never dare doing it myself :-)))

> Until a few more French and German an other foreign
> language-speaking developers step up to help manage
> things, you won't get quick response on lists the
> current developers don't read.

So true. Sigh. But I can't resist a user making droopy
eyes because he asked to be sysop nearly a month ago
and still is not :-(((

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


subject: [WikiEN-l] Sysop limits

body:
On Thu, 1 May 2003, Tim Starling wrote:

> Date: Thu, 01 May 2003 00:45:48 +1000
> From: Tim Starling <ts4294967296 at hotmail.com>
> Subject: Re: [WikiEN-l] Sysop limits
>
> >Side note: It's so bad, since I started doing Wikipedia fairly regularly,
> >I find myself frequently looking for an "Edit this page" link while I'm on
> >other, non-wiki sites and I see an error. ;)
>
> I believe that's a known clinical symptom of Wikipediholism. Yes,
> [[Wikipedia:Are You a Wikipediholic Test]] number 42. Of course, I never do
> anything silly like that.

But as I make clear on my User: page, I don't dare go anywhere near that
test. So of course I wouldn't know anything about that. See also
[[Wikipedia:Wikipediholics in denial]].

--
John R. Owens http://www.ghiapet.homeip.net/
Marcus, this is the kind of conversation that can only end in a gunshot.
				--Dr. Stephen Franklin


from_name: Jason Williams

subject: [WikiEN-l] Sysop limits

body:
On Thu, May 01, 2003 at 12:45:48AM +1000, Tim Starling wrote:
> >Side note: It's so bad, since I started doing Wikipedia fairly regularly,
> >I find myself frequently looking for an "Edit this page" link while I'm on
> >other, non-wiki sites and I see an error. ;)
>
> I believe that's a known clinical symptom of Wikipediholism. Yes,
> [[Wikipedia:Are You a Wikipediholic Test]] number 42. Of course, I never do
> anything silly like that.

Nah, you've not got it bad until you're reading a (paper) newspaper and
want to click on the Talk page to complain about POV!

--

jason at jasonandali.org.uk    http://www.jasonandali.org.uk/jason/


subject: [Wikitech-l] Patch for LanguageDe.php

body:
Some typos, as usually.

Smurf
--
-------------------------  Anthill inside! ---------------------------
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: LanguageDe.diff
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20030430/eecab390/attachment.ksh>


subject: [Wikitech-l] Caching

body:
> (Geoffrey Thomas <geoffreyerffoeg at yahoo.com>):
> A suggestion for caching: the home page, [[Main
> Page]], never should include links to nonexistent
> pages. Could that be cached (in standard stylesheet),
> e.g., as /index.html? I would think that page is
> accessed very frequently and that caching it would
> save some database work for other pages. When the page
> is edited (only by a sysop), he/she would render the
> page to HTML, possibly by explicitly accessing
> /wiki/Main_Page, and save it as /index.html. Would
> this help some server strain?
>
> Could we also cache other pages known to link only to
> existing pages, such as this week in dates ([[April
> 30]], etc.), [[Current events]], and [[Recent
> deaths]]? Or are these pages updated too often to be
> useful? Could we also cache the protected pages?

It might be possible to either (1) have Apache look for our
cookies, and if it determines that this is a non-logged-in
user, serve a static front page instead of going through
the script; or (2) have the "/" url itself be a small and
fast script that checks the cookies and redirects to either
the large script or the static page.

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


subject: [Wikitech-l] Caching

body:
On Wed, 30 Apr 2003, Geoffrey Thomas wrote:

> Date: Wed, 30 Apr 2003 14:47:20 -0700 (PDT)
> From: Geoffrey Thomas <geoffreyerffoeg at yahoo.com>
> Subject: [Wikitech-l] Caching
>
> A suggestion for caching: the home page, [[Main
> Page]], never should include links to nonexistent
> pages. Could that be cached (in standard stylesheet),
> e.g., as /index.html? I would think that page is
> accessed very frequently and that caching it would
> save some database work for other pages.

Actually, it seems to happen rather often that someone or something
formerly rather obscure suddenly hits the headlines, or dies, and suddenly
appears on the [[Main Page]]. Of course, usually at least a stub is made
very soon after, but I don't think it qualifies as a "never". And it's
good to be able to see whether or not the article does exist yet when you
see the Main Page, of course.
For instance, right now, the first two "Recent deaths" on the English
Wikipedia, Bernard Katz & Nina Simone, both didn't have any page at all
until they died.

--
John R. Owens http://www.ghiapet.homeip.net/
You know what I wish?  I wish all the scum of the Earth had one throat
and I had my hands about it.
		-- Rorschach, "Watchmen"


from_name: Nick Reinking

body:
On Wed, Apr 30, 2003 at 10:01:33PM +0000, Giskart wrote:
> Brion Vibber <vibber at aludra.usc.edu> wrote in
> news:Pine.GSO.4.33.0304291740490.16252-100000 at aludra.usc.edu:
>
> > Oh, I think it's quite useful to get that information, otherwise I
> > wouldn't know about *($%@^&*$%@# Grub.
> >
> > Anyway, I cleaned out a few things and moved some of the older
> > tarballs over to the archives in the home partition, and we're down to
> > 85% usage on /usr.
> >
> > -- brion vibber (brion @ pobox.com)
>
> I like also stats. If possibel, i would like more stats. Now there are only
> Top 30 of 10972 Total Referrers. I would like to have the top 250. Same
> whit Top 30 of 12904 Total URLs. I like the top 500.
>
> Now there is no registration of the Top of Total Countries.

I imagine it will matter much less with the new server, but it would
still be a good idea to add a second drive that is only used for logs
and database dumps.  That would help quite a bit with the high I/O
levels on the primary disk.

--
Nick Reinking -- eschewing obfuscation since 1981 -- Minneapolis, MN


subject: [Wikitech-l] Developper limits

body:
> (Anthere <anthere6 at yahoo.com>):
>
> Btw, the en:Wikipedia:Administrators page absolutely
> do not indicate this mailing list as being the right
> place to ask for sysop rights. It mentions it *only*
> for developper status.
>
> The page recommand to ask sysop status on the
> en.mailing list. Then I guess someone agrees for the
> new sysop, then by some magic, the user is named
> sysop.

Sorry, I assumed this was posted to wikipedia-l, not
wikitech-l, since it's not a technical issue. Yes,
wikipedia-l is the right place to ask.

And the magic is "update user set user_rights='sysop'
where user_name='newsysopname';", which can be done
from a MySQL prompt on the server, or with the SQL
query page by any developer.

> Clearly, international wikipedias cannot follow that
> pattern. No international user will come either to the
> english list, nor to the tech list for sysophood, but
> likely to the specific language list, or the admin
> page, or pump. Hence, the procedure given for
> english-speaking people cannot be identically followed
> on international wikipedia. It requires one more step,
> as not only other users have to agree, but someone has
> to ask and check until it is done.

Until a few more French and German an other foreign
language-speaking developers step up to help manage
things, you won't get quick response on lists the
current developers don't read.

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


from_name: Graham Burnett

subject: [WikiEN-l] Zog

body:
I havn't seen any of 'Zogs' edits but picked up some concerns about this
person from the mailing list- just in case anyone is unaware, 'Zog' is a
term used by white supremacists and other nazis and is an abreviation of
'Zionist Occupational Governmnent', and is i believe a reference to that
famous nazi publication/forgery 'The protocols of Zion' (ie, that the world
is run by a Jewish conspiracy). as such this username is far more offensive
than the likes of 'fuckyou' or 'i've got a big cock' or whatever. I
seriously doubt any such user has any intentions of doing anything other
than spreading race hate.

Cheers graham (Quercus robur)


from_name: Giskart

subject: [Wikipedia-l] Re: Developper limits

body:
Anthere <anthere6 at yahoo.com> wrote in
news:20030430110711.39550.qmail at web41708.mail.yahoo.com:
>
> But, then, isnot there a way for us to make people on
> the international wikipedia ourselves ?
>
> Could there be something like a query, which could
> allow french sysop to make french user (just an
> example of course) a sysop automatically ? Maybe a
> list of user could be displayed, and one sysop could
> click on the name of one user to make him sysop ? Of

I must say that until now the responds time about serious problems or
requests to make a user a sysop is not to bad if you take in account how
much work Brion has.

So far I know if a user has "developer" status you can run a query to make
user sysop or remove sysop status.

But I suppose you can do a lot of damage to a wikipedia if you run a bad
query when you have developer status. Something to be carefull whit.

--
Contact: giskart AT wikipedia.be
Ook een artikeltje schrijven? WikipediaNL, de vrije GNU/FDL encyclopedie
http://www.wikipedia.be


subject: [Wikitech-l] Re: Developper limits


subject: [Wikitech-l] Re: Re: Developper limits

body:
Tomasz Wegrzanowski
<taw at users.sourceforge.net> wrote in
news:20030430192437.GA6461 at tavaiah:
> Here's standard one:
> update user set user_rights = 'sysop' where user_name =
> 'NameOfNewAdmin';
>
> -- Taw, developer

Yes, i know. I have put it on the list of Database queries of my wikipedia
a while ago. Not that you can do a lot whit it.

--
Contact: giskart AT wikipedia.be
Ook een artikeltje schrijven? WikipediaNL, de vrije GNU/FDL encyclopedie
http://www.wikipedia.be


from_name: tarquin

subject: [Wikipedia-l] [Wikitech-l] server names

body:
The Cunctator wrote:

>I like the idea
>of naming the servers after famous encyclopedists.
>
>
Nice idea :-)
let's not argue too much about *which* two names to pick -- I predict we
will need many more servers in years to come .... ;-)


from_name: Tomasz Wegrzanowski

subject: [Wikitech-l] Re: Developper limits

body:
On Wed, Apr 30, 2003 at 07:11:27PM +0000, Giskart wrote:
> I must say that until now the responds time about serious problems or
> requests to make a user a sysop is not to bad if you take in account how
> much work Brion has.
>
> So far I know if a user has "developer" status you can run a query to make
> user sysop or remove sysop status.
>
> But I suppose you can do a lot of damage to a wikipedia if you run a bad
> query when you have developer status. Something to be carefull whit.

Here's standard one:
update user set user_rights = 'sysop' where user_name = 'NameOfNewAdmin';

-- Taw, developer


subject: [Wikitech-l] Really good news!

body:
One fine day, Brion Vibber said:
> Do feel free to ask other free projects and universities if they'd be
> interested in supporting the project...

I thought it might be a good idea to ask, so I sent out an email to ibiblio:
> Greetings - my name is Nick, and I help out with a site called Wikipedia.
> (http://www.wikipedia.org).  It is a free, multi-lingual project to
> create a complete, accurate, and more importantly open content
> encyclopedia.  All of the content is licensed under the GNU FDL (GFDL),
> meaning that anybody has the freedom to copy and redistribute it, with
> or without modifications, either commercially or non-commercially -
> although they may not put in place technical measures to conceal the
> content.
>
> The English language Wikipedia has over 117 thousand articles already,
> and by our calculations, we have about half the content of the
> Encyclopedia Britannica.  We have many other languages, which are also
> quickly growing in size and diversity.
>
> However, we are currently facing both a budget and capacity crunch.
> In short, we have neither.  Since Wikipedia is a volunteer based
> program, it is hard for us to raise the funds to purchase additional
> hardware.  Right now, we are running off of a dual Athlon 1800+ server
> with 2GB of RAM and 36GB of SCSI storage.  Unfortunately, the system is
> being pushed to its absolute limits, with little relief in sight.  We
> are installing a second system this weekend as a front end, but even
> with that, we are not sure how long we can hold out.  The dual Athlon
> system runs at a typical load of 15-20 during normal US working hours.
>
> Since our mission seems to be very much inline with ibiblio's mission, I was
> wondering if there was any way that Wikipedia could be hosted by ibiblio?
> It would be a great help to our project and the community.
>
> Thanks!

Much to my surprise, they replied:
>   hi nick,
>
>   we would LOVE to host wikipedia.org.  our only concern is with the
> additional load wikipedia might put on our mysql server.  BUT... if you
> could possibly hold off moving the site for two weeks or so john and
> fred will have us up on our new hardware - we're moving to a web cluster
> and will have a much more powerful database machine.
>
>   if this is acceptable to you, please check out
> http://www.ibiblio.org/faq/ for more information about our setup, and
> just drop me a list of what accounts, dbs, unix groups, web directories,
> etc and we'll go from there.  just let me know?
>
>   thanks,
>   donald                                       www.ibiblio.org
>                                          formerly known as SunSITE
>   919.843.8215                                    and stoof.

Seems like a good deal to me.  We should probably tell them that they should
keep our database on a seperate MySQL server, as it will absolutely demolish
just about anything they make available.

Anyways, if somebody who knows the server requirements, layout, and other
what-not wants to let Donald and/or the list know, that would be awesome
(assuming that Brion is still interested in having someone else host the
site).  Having ibiblio make all the outlays seems like a good deal to me
(plus, they're a non-profit org, so you could likely make tax-deductable
donations to them).

Anyways, that's that.  :)
--
Nick Reinking -- eschewing obfuscation since 1981 -- Minneapolis, MN


body:
Brion Vibber <vibber at aludra.usc.edu> wrote in
news:Pine.GSO.4.33.0304291740490.16252-100000 at aludra.usc.edu:

> Oh, I think it's quite useful to get that information, otherwise I
> wouldn't know about *($%@^&*$%@# Grub.
>
> Anyway, I cleaned out a few things and moved some of the older
> tarballs over to the archives in the home partition, and we're down to
> 85% usage on /usr.
>
> -- brion vibber (brion @ pobox.com)

I like also stats. If possibel, i would like more stats. Now there are only
Top 30 of 10972 Total Referrers. I would like to have the top 250. Same
whit Top 30 of 12904 Total URLs. I like the top 500.

Now there is no registration of the Top of Total Countries.

--
Contact: giskart AT wikipedia.be
Ook een artikeltje schrijven? WikipediaNL, de vrije GNU/FDL encyclopedie
http://www.wikipedia.be


from_name: Guillaume Blanchard

subject: [Wikitech-l] Re: server names

body:
I propose "Thot" the egyptian god of knowledge or "Ptah"  the god of
artisans.

Aoineko


from_name: Jens Frank

subject: [Wikitech-l] Ad-hoc changes

body:
On Tue, Apr 29, 2003 at 04:13:26PM -0500, Lee Daniel Crocker wrote:
> > (Brion Vibber <vibber at aludra.usc.edu>):
> > [Configuration info]
>
> Here's a thought: when we get the new server up, let's install the
> English wiki cleanly from the distribution, make sure it's all happy,
> then do the switch, _leaving the foreign wikis on the old server_ for
> a while, then do the same thing for each of the foreign ones in turn,
> updating them to the newest software, and making everything in sync,
> before we shut down apache on the old server.
>
> Any problems with that?

Will this conflict with the image upload area? I remember some
languages using english images and thereby creating "pseudo-orphans".

Regards,

JeLuF


from_name: Tim Starling

body:
>From: Lee Daniel Crocker <lee at piclab.com>
>One things that would be nice is if the HTTP connection could be
>dropped immediately after sending and before those database updates.
>That's easy to do with threads in Java Servlets, but I haven't
>found any way to do it with Apache/PHP.

:P  No, I looked into exactly this problem in connection with my own little
project (improved Special:Movepage). PHP and threads don't mix. As far as I
could see, the PHP subprocess has to exit (taking all threads with it)
before Apache will drop the connection. Like Brion said, you'd have to set
up another process, and use PHP's poorly documented IPC functions. As for
what improvement it would achieve: it wouldn't reduce database load per
view, it would just allow users to hit more pages sooner.

I think caching HTML is the way to go, in the short term. If people don't
want to code something complicated, you could ignore user preferences for
now and only cache pages for "anonymous" users. The cached version could
leave little notes in the HTML like

Isaac Newton was a <<WIKILINK[[physics|physicist]]>> born
in...

and maybe

<<USERIP>> (<a href ="http://www.wikipedia.org/wiki/User_talk:<<USERIP>>"
class='internal' title="User talk:<<USERIP>>">Talk

Then a cache processing script would look up the link table and replace the
links with real HTML. I imagine looking up the link table is much, much
faster than looking up cur_text. Plus the cached text would be stored on the
web server, thereby distributing disk load more evenly.

As for invalidation, the easiest, and possibly ugliest way I can think of is
implementing it in wfQuery() *cringe*. That's a very simple function with
very diverse uses, but every single update query passes through that point.
Just use a hash table (always in RAM) to store the article name of every
cache entry, and remove the rows when they're invalidated.

There'd also have to be a check for an altered user talk page. This could be
handled with another of my <<TAGS>>.

This idea is likely to be met with apathy. I'd like to code it myself, but I
don't have Linux on my PC, or a broadband connection, or much free hard
drive space, or... time. So there you have it: my two cents, backed up by
hot air.

-- Tim Starling.

_________________________________________________________________
Hotmail now available on Australian mobile phones. Go to
http://ninemsn.com.au/mobilecentral/hotmail_mobile.asp


subject: [WikiEN-l] Sysop limits

body:
>Side note: It's so bad, since I started doing Wikipedia fairly regularly,
>I find myself frequently looking for an "Edit this page" link while I'm on
>other, non-wiki sites and I see an error. ;)

I believe that's a known clinical symptom of Wikipediholism. Yes,
[[Wikipedia:Are You a Wikipediholic Test]] number 42. Of course, I never do
anything silly like that.

-- ~~~~

_________________________________________________________________
MSN Instant Messenger now available on Australian mobile phones.?Go to
http://ninemsn.com.au/mobilecentral/hotmail_messenger.asp


from_name: Lee Daniel Crocker

subject: [Wikitech-l] Really good news!

body:
>>  [ibiblio] would LOVE to host wikipedia.org...

I don't think that's an option, but certainly they might be
useful as a mirror, or storage for backups, etc.

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


from_name: Jimmy Wales

subject: [Wikitech-l] Really good news!

body:
I'm not the least bit interested in moving wikipedia off my servers.

The claims of budget crisis are wildly exagerrated.  I'm donating one
and possibly two more servers to the project (one for a www frontend
and one for a mailserver and auxiliary front end).


subject: [Wikitech-l] Really good news!

body:
Nick Reinking wrote:
> However, is there a reason why we couldn't have ibiblio host the entire
> Wikipedia?  It seems that is what Brion was asking for.  If they're
> willing to supply three, four (or more) servers to keep Wikipedia
> running, why isn't that a good thing?

There are a lot of reasons, but the main reason is that we shouldn't
make ourselves dependent on anyone else.

--Jimbo


subject: [WikiEN-l] The "Young and Innocent" page

in_reply_to: <20030501130103.GD75102@mrtall.compsoc.man.ac.uk>; from jason@jasonandali.org.uk on Thu, May 01, 2003 at 02:01:04PM +0100

body:
Jason Williams wrote:
> Why is it so? Why is the principle that American and British English
> have equal status abandoned here? Why is the ugly "movie" explicitly
> preferred over the perfectly good "film"?

That "movie" is ugly is a pretty ad hoc aesthetic judgment, don't you
think?

At any rate, perhaps there was a discussion somewhere about this?

--Jimbo


from_name: JFrost8401 at aol.com

subject: [WikiEN-l] A plea for sanity in capilaisation

body:
I also blame the Americans for everything, but in this case I must  plead
guilty (and guilt by association with Australians)

Jim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.wikimedia.org/pipermail/wikien-l/attachments/20030501/11da2738/attachment.htm


from_name: David A. Wheeler

subject: [Wikitech-l] Re: Chat about Wikipedia performance?

body:
Lee Daniel Crocker <lee at piclab.com> said:
> (David A. Wheeler <dwheeler at dwheeler.com>) said:
> >
> > 1. Perhaps for simple reads of the current article (cur), you
> > could completely skip using MySQL and use the filesystem instead.
>
> In other words, caching.

Sorry, I wasn't clear.
I wasn't thinking of caching - I was thinking of accessing the
filesystem INSTEAD of MySQL when getting the current wikitext.

Why?  Well, I suspect that accessing the filesystem directly
is much faster than accessing the data via MySQL - if most
accesses are simple reads, then you can access it without
user-level locks, etc., etc.  Even more importantly,
checking for existence is a simple filesystem check - which
is likely to be much faster than the MySQL request.

Would it be faster?  I don't know; the only _real_ way to find out
is to benchmark it.

Of course, if wikipedia is near the breaking point for performance,
another approach would be to change the design so that reading
only requires one lookup (for the data itself).
You noted the two big problems, and I agree that they're the
sticking points.
You could abandon many user settings, except ones that the user
can supply themselves to select between different stylesheets, and
abandon displaying links differently depending on whether or not
they're there.  Less desirable, but you've already abandoned supporting
search!  Then you can cache the generated HTML as well.

If it's a choice between having a working wikipedia, and
having the bells & whistles, I think working is the better plan.
You can always include them as settable options, to be returned once
the system doesn't have performance problems.

Although databases are more flexible for storing structured
data, for simple unstructured data, a simple filesystem-based
approach might be more suitable.  This also lets you use other
existing tools (like the many tools that let you store
indexes for later rapid searching based on files).

A quick start might be to temporarily disable all checking
of links, and see if that helps much.

> > [Rendering] could also be sped up, e.g., by rewriting it in flex.
> > My "html2wikipedia" is written in flex - it's really fast and
> didn't
> > take long to write.  The real problem is, I suspect that
> > isn't the bottleneck.
>
> It isn't. And there's no reason to expect flex to be any faster
> than any other language.

Actually, for some lexing applications flex can be MUCH faster.
That's because it can pre-compile a large set of patterns
into C, and compile the result.  Its "-C" option can, for
some applications, result in blazingly fast operations.
You CAN do the same thing by hand, but it takes a long time to
hand-optimize that kind of code.

However, there's no point in rewriting what is not the bottleneck.
Which I why I was hoping to hear if someone has done measurements
to identify the real bottlenecks, e.g., "50% of the system
time is spent doing X".  If most time is spent rendering
articles for display (without editing), then it's worth examining
what's taking the time.  If the time is spent on checking if
links exist, then clearly that's worth examining.

Oh, one note - if you want to simply store whether or not a
given article entry exists or not, and quickly check it, one
fancy way of doing this is by using a Bloom filter.
You can hash the article title, and then using a fancy data
structure can store its existance or non-existance.
More info, and MIT-licensed code, for a completely different
application are at:
 http://www.ir.bbn.com/projects/SPIE
(there, they hash packets so that later queries can ask
"did you see this packet"?).  Given the relatively small
size of article text, it's not clear you need this
(you can store all the titles in memory), but I just thought
I'd mention it.

Anyway, thanks for listening.  My hope is that the Wikipedia
doesn't become a victim of its own success :-).


subject: [Wikitech-l] Really good news!

body:
On Thu, May 01, 2003 at 01:01:06AM -0500, Lee Daniel Crocker wrote:
> >>  [ibiblio] would LOVE to host wikipedia.org...
>
> I don't think that's an option, but certainly they might be
> useful as a mirror, or storage for backups, etc.

That's something I was thinking - perhaps, for example, the database goes
down.  At that point, we make a quick Apache change, and all requests
URLs are re-written to point at a read-only copy of Wikipedia @ ibiblio.

However, is there a reason why we couldn't have ibiblio host the entire
Wikipedia?  It seems that is what Brion was asking for.  If they're
willing to supply three, four (or more) servers to keep Wikipedia
running, why isn't that a good thing?

--
Nick Reinking -- eschewing obfuscation since 1981 -- Minneapolis, MN


subject: [Wikitech-l] Please run this DB query to fix Special:Randompage

body:
On Wed, 2003-04-30 at 21:48, Tim Starling wrote:
> Someone needs to run this:
>
> UPDATE cur SET cur_random=RAND()

Ok, it'll be a bit before it's done. Hooray for big slow databases. :D

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20030501/f2c2706d/attachment.pgp>


body:
On Thu, 2003-05-01 at 06:10, Jimmy Wales wrote:
> Jason Williams wrote:
> > Why is it so? Why is the principle that American and British English
> > have equal status abandoned here? Why is the ugly "movie" explicitly
> > preferred over the perfectly good "film"?
>
> That "movie" is ugly is a pretty ad hoc aesthetic judgment, don't you
> think?
>
> At any rate, perhaps there was a discussion somewhere about this?

Should be somewhere in the mailing list archives. Basically, there were
many variants in use:

  Bla bla (movie)
  Bla bla (1967 movie)
  Bla bla (film)
  Bla bla (1967 film)
  Bla bla (motion picture)
  Bla bla (Soderbergh film)
  Bla bla (Soderbergh movie)
  Bla bla (2002 Soderbergh movie)
and whatnot.

"Bla bla (movie)" seemed to have the largest 'installed base', as it
were, so when it was decided to establish some consistency, "(movie)"
won out over the equally-simple but thus far less used (though more
aesthetic, at least for us film folks) "(film)".

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.wikimedia.org/pipermail/wikien-l/attachments/20030501/0ef1f0f0/attachment.pgp


from_name: Richard Grevers

subject: [Wikitech-l] Re: server names

body:
On Wed, 30 Apr 2003 22:07:10 +0900, Guillaume Blanchard
<gblanchard at arcsy.co.jp> gave utterance to the following:

> I propose "Thot" the egyptian god of knowledge or "Ptah"  the god of
> artisans.
>
Or if the second server is really really really big it could be "Deep Thot"
:-) (SCNR)
--
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/


subject: [WikiEN-l] Conventions and movie vs. film

body:
One reason we should scrupulously stick to conventions is that in the
event that we someday have a long discussion and decide that our old
convention is undesirable, our sticking to the convention makes it a
lot easier to change.

That is, it would be pretty simple to write a script to move all pages
from "Blah blah blah (movie)" to "Blah blah blah (film)", but a lot of
work to fix "Blah blah blah (the movie)" and "Blah blah blah, a movie"
and "The movie: Blah blah blah" and so on.


from_name: Ray Saintonge

body:
Jason Williams wrote:

>On Thu, May 01, 2003 at 06:10:39AM -0700, Jimmy Wales wrote:
>
>>Jason Williams wrote:
>>
>>>Why is it so? Why is the principle that American and British English
>>>have equal status abandoned here? Why is the ugly "movie" explicitly
>>>preferred over the perfectly good "film"?
>>>
>>That "movie" is ugly is a pretty ad hoc aesthetic judgment, don't you
>>think?
>>
>
>Of course; since it's quite obviously subjective I assumed people
>would assume that I was stating it as such :-)  I wasn't trying
>to argue for wholesale replacement of "movie" by "film", just the
>same admirable agnosticism that is applied across the rest of
>the wikipedia.
>
>>At any rate, perhaps there was a discussion somewhere about this?
>>
>
>I expect so, but I couldn't find it by googling the mailing list
>archives and there isn't a Talk page corresponding with
>[[Wikipedia:Naming conventions (movies)]].
>
I was an active participant in that discussion in March 2002.  Its
apparently factored remnants are at [[Talk:Film]], but it seems to me
that it also went on across several talk pages.  I've never considered
this debate as an American vs. British English sort of issue.  As a
supporter of "movie", then as now, I obviously don't consider "movie" to
be ugly.  I am of course pleased that (movie) has become our standard
disambiguating term, but even here I recognize that most Wikipedia rules
are subject to change.  As things stand "movie" has developed
considerable inertia, and making the change over such a large array of
articles could be very confusing.  Making the disambiguating term
optional doesn't seem very attractive.

Eclecticology


subject: [WikiEN-l] [Fwd: Re: Johnny Rebel]

in_reply_to: <3EB1759B.4D726419@frognet.net>; from jaknouse@frognet.net on Thu, May 01, 2003 at 12:29:31PM -0700

body:
John Knouse wrote:
> Message about the other Johnny Rebel, the one who's not the
> racist:

Clearly we need some disambiguation.

--Jimbo


body:
On Thu, 2003-05-01 at 08:53, Nick Reinking wrote:
> Off the top of my head, I can't think of any simple mathematical way to
> do want you want to do.  (that being making articles selected randomly
> less likely to be selected randomly over time)

Not sure who wants to do that... For me it's quite sufficient to make
the probability of selection random, and let the size of the selection
field make multiple selections _reasonably_ rare.

The resetting of the random weight upon load is just meant to keep the
pot stirring, and "in theory" shouldn't have any effect at all on
randomness if the random indexes were random to begin with.

> Seems to me that the best way to do this is either:
> 1) Be truly random.  In PostgreSQL, this would be something like:
>      SELECT cur_id from cur LIMIT 1 OFFSET random(SELECT count(cur_id) FROM cur)
>    (this function should be really fast, not sure if faster than:)

Mysql doesn't seem to let you use a non-constant expression for
'LIMIT offset,count' (and besides, still no subqueries), but we can pick
a random number easily enough in PHP. The problem is that we aren't sure
what range to use; hence a random index attached to each article whose
range is known (between 0 and 1).

We don't want to select non-article pages or redirects, but we don't
currently have a count of exactly that. There's the "good articles"
count, but it has other criteria, so if we used it we'd always crop off
relatively recent articles. If we inflate it based on an estimate we'd
get overemphasis on the _most_ recent article if we estimate too high.

Sure, there's count(*), but that takes time to run... might be faster
with an index specifically on cur_namespace,cur_is_redirect. Or we could
maintain another column in the stats table which counts non-redirect
article pages with _no_ other criteria (such as size or comma/bracket
count). Rebuilding indexes on cur is _very_ slow, so if we need to
change the indexes, best to plan ahead.

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20030501/acba33c8/attachment.pgp>


from_name: Daniel Mayer

subject: [Wikitech-l] Re: Really good news! (Lee Daniel Crocker)

body:
LDC wrote:
>>  [ibiblio] would LOVE to host wikipedia.org...

>I don't think that's an option, but certainly
>they might be useful as a mirror, or storage
>for backups, etc.

My thoughts exactly LDC. If they go for it, we would
have a fantastic mirror AND we still would still have
direct control over our own servers.

--Daniel Mayer (aka mav)

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


subject: [WikiEN-l] A plea for sanity in capitalisation from the coalface

body:
Tony Wilson wrote:

>Ray wrote: "Although I am usually responsive to blaming the Americans
>for everything, it needs to be remarked that the capitalist charge in
>this matter has been led by an Australian" - which made me smile. The
>irony is that I also tend to blame the Americans for everything - and
>in this context, I blame them for *not* capitalising things when they
>ought to!  So perhaps we had best categorise this particular Australian
>as one of Little Johnny Howard's "American Deputy Sherrifs in
>South-east Asia."  .... Err - sorry, Jimbo. I know - no politics on the
>list. :)
>
I can never pass up a good opportunity for a bad pun.  This was a chance
to look at capitalism with an entirely fresh meaning.   :-)

>Enough humour. When Ray wrote that "the problem ... operates on two
>levels" and that "as long as it is only on a stylistic level ..." he
>missed the point. It is *not* only stylistic, nor indeed even primarily
>stylistic: it is a matter of clear, unabiguous, and accurate
>communication. There are many long-tailed shrikes, but only one
>Long-tailed Shrike. It is not a matter of 'preffered styles", but a
>matter of a precise and correct name or a vague and incorrect name.
>
The problem here is in insisting that there is only one way to do
things.  My personal preference would be to put all these animal
articles under the Latin binomial.  That option would be the least vague
and least ambiguous of all, but I haven't had much of a following in
this.  As I said in a previous post your position tends to be most
adamantly held by birders than by any other students of life forms; the
opposite point of view tends to be favoured by grammarians.  It is once
again a matter of POV's.  Certainly the issue does not carry the heavy
emotional baggage of many of the political issues that have flared up,
but it is a POV issue nevertheless.  How do you choose in a battle
between lexicologists and ornithologists?

>However, Ray's third point raises a significant issue. Creating
>lower-case redirects to entries that are (quite properly) capitalised
>involves a good deal of attention to detail - something that I can
>speal with authority about, as I have spent 20 or 30 hours attending to
>this stuff in the four days or so, It is tedius and a small number of
>mistakes will undoubtedly have slipped through. A technical solution
>would indeed be very welcome. The obvious starting point would be a
>search function that works. I understand that that matter is high on
>the developers' priority list, and wish them every success in the
>quest.
>
This was indeed my most important point since it affects such a wide
range of articles well beyond avian capitalism (or Avian Capitalism).
 Using Tony's distinction, very few of the terms that he would put with
lower case letters would lead to articles anyway so why not use
re-directs?  It's as effective a temporary measure as any.

Unfortunately, even though I can envision how I would like to see this
sort of thing work I have to admit to a complete lack of technical
expertise to apply this in out multilingual environment.

Eclecticology


subject: [WikiEN-l] A plea for sanity in list communication

body:
Jim wrote:
>I also blame the Americans for everything,
>but in this case I must plead guilty (and
>guilt by association with Australians)

Funny, I blame the French for everything... Oh wait -
that's politics and is also fairly insulting to a
large group of Wikipedians so that type of thing
shouldn't be stated on the list. Silly me.

It should also be noted that an American, me, was the
main person arguing for the down style and I still
maintain that this is the most grammatically correct
way to express these names in a general context (that
is, outside of the bird articles). Thus redirects are
absolutely necessary.

-- Daniel Mayer (aka mav)

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com


subject: [Wikitech-l] Caching

body:
On Wed, 30 Apr 2003 17:02:31 -0500 (CDT), John R. Owens
<jowens.wiki at ghiapet.homeip.net> gave utterance to the following:

> On Wed, 30 Apr 2003, Geoffrey Thomas wrote:
>
>> Date: Wed, 30 Apr 2003 14:47:20 -0700 (PDT)
>> From: Geoffrey Thomas <geoffreyerffoeg at yahoo.com>
>> Subject: [Wikitech-l] Caching
>>
>> A suggestion for caching: the home page, [[Main
>> Page]], never should include links to nonexistent
>> pages. Could that be cached (in standard stylesheet),
>> e.g., as /index.html? I would think that page is
>> accessed very frequently and that caching it would
>> save some database work for other pages.
>
> Actually, it seems to happen rather often that someone or something
> formerly rather obscure suddenly hits the headlines, or dies, and
> suddenly appears on the [[Main Page]]. Of course, usually at least a stub
> is made very soon after, but I don't think it qualifies as a "never". And
> it's good to be able to see whether or not the article does exist yet
> when you see the Main Page, of course.
> For instance, right now, the first two "Recent deaths" on the English
> Wikipedia, Bernard Katz & Nina Simone, both didn't have any page at all
> until they died.
>
I presume that it is one of the admin jobs to select and update the items
that appear on the "In the news" "Recent deaths" etc. link lists. That is
likely to happen twice a day at most, probably once. What would happen
under a caching system is that the interface to gegenerate those components
simply rewrites the cached version of the page.
What we would lose with caching, however, is indicators such as the
asterisk showing that someone has edited your talk page if you are logged
in. Perhaps (if people don't scream blue murder about popups), logged in
users could have a little popup console where these dynamic things happen.
For Opera/Mozilla it could be written as a panel or sidebar.

--
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/


from_name: Brion Vibber

body:
On Thu, 2003-05-01 at 09:35, The Cunctator wrote:
> We actually did have a big discussion about this and film ended up being
> nixed because of the problem of digital videa replacing actual film as the
> medium for "films". Thus, "movie".

Wow, I sure don't remember *that* discussion, and I find that argument
fundamentally flawed to the point of being comical. Might there have
been two such discussions, I wonder?

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.wikimedia.org/pipermail/wikien-l/attachments/20030501/b1d0daf3/attachment.pgp