from_name: Clifford Adams

subject: [Wikitech-l] Slow response due to hit tracking?

body:
[Neil wrote regarding MySQL performance:]

> In the 10% write case, however, Postgres kept going beyond the
> client-concurrency load where MySQL falls over.

    Personally, I suspect the hit-tracking feature is responsible for much of
the slowdown.  Currently *every* normal page load causes a write to update the
number of accesses ("This page has been accessed 281 times.")

    I recommend making the hit-tracking feature optional in the code, then
turning it off for a few days.

    In the long run, this kind of per-page tracking could be done more
efficiently using the webserver's logs, perhaps updating the access count once
per day using a separate script.

            --Cliff


list: wikitech-l

source: pipermail-mbox

from_name: Axel Boldt

subject: [Wikitech-l] Re: Is the database logging?

body:
> It is logging slow requests...  320 Meg since April 19.

Assuming an average log message length of 100 bytes, this amounts to
between two and three logs per second, so this could conceivably be
the bottle neck. Could you chop off the last 10 megs or so, make them
available on www.wikipedia.com/tarballs, and switch off the logging?

Axel


subject: [Wikitech-l] Re: Slow Response

body:
>I'm getting incredibly slow responses from the main Wikipedia now and
>in the last few days.  Why is this?  Who is working to fix it?  How
>can I help?

I don't think anybody is actively working on it right now. Jimbo has
said that he will soon move some other web sites off the wikipedia
server and will then give some of us user accounts on that machine, so
that we can better diagnose the problems. Right now, it is not clear
what we can do. For you, maybe the best would be to download the
current code from cvs, get it to work and familiarize yourself with
it; once we have access to the server we should be able to figure out
what's going on pretty quickly.

Axel


from_name: Jason Richey

subject: [Wikitech-l] Re: Slow Response

in_reply_to: <200205041955.VAA02025@linde.uni-paderborn.de>; from axel@uni-paderborn.de on Sat, May 04, 2002 at 09:55:15PM +0200

body:
What about making some of our update queries 'delayed'?

--
"Jason C. Richey" <jasonr at bomis.com>


list: wikipedia-l

from_name: Jimmy Wales

subject: [Wikipedia-l] Collection copyrights

in_reply_to: <2100F5D84712D411A00E00B0D0221D1A972C82@kodiak.htec.com>; from mchristensen@htec.com on Wed, May 01, 2002 at 10:16:54AM -0400

body:
Right.  (I'm back, by the way, but will be deeply behind in email for a few
days.)

The only real reason to consider what kind of copyright Bomis might
have in the collection of articles is to consider whether we have
standing to sue people who violate the GFDL.  My position is that the
individual articles are released GFDL by the author(s), and the
collection is released GFDL by Bomis.  (And the software is all GPL.)

Let's imagine a scenario.  Imagine that a paper-book publisher
downloads our tarball and polishes it into multi-volume book form.
That's great!  But let's imagine further that they publish it merely
with a copyright notice, i.e. they don't relicense it.  Imagine still
further that they start sending someone (or me!) cease-and-desist
letters for copying and modifying "their" version.

I'd like to serve as a convenient focal point for a lawsuit against
them.

However, it's probably a minor point either way.  If someone did
something that annoyed the community, I have little doubt that we
could quickly work together to establish ownership of a significant
portion.  At that time, I could offer to buy individual copyrights
from you all, i.e. we could go through the copyright assignment
process.

I think that in reality, all of this is fairly pedantic, though.  If
someone tried to violate the GFDL, there would be such a huge outcry
from the wider free software community, that almost anyone would have
to back down in the face of the negative publicity.

Mark Christensen wrote:

> This raises an important question in my mind.  The Wikipedia FAQ says that
> "the articles" are licensed under the FDL, but says nothing about the
> collection.  Is it, like all of the articles, licensed under the FDL?
>
> It has been my understanding that Bomis has released the collection under
> the terms of the Free Document License, but, as I did  a quick search of the
> list archive and on the wikipedia I find that it shows no specific text
> stating that this is the case.
>
> I did find an e-mail from Larry Sanger saying that he thought everybody
> assigned copyright to Bomis, and then Bomis which licensed everything thing
> under the GNU/FDL.  I think the part about copyright assignment has been
> hashed out sufficiently on this list, but I would like clarification on the
> issue of the license status of the collection.
>
> My best guess is that it is. I'm fairly certain that Jimbo and Larry
> intended for the entirety of the "free encyclopedia" to be licensed under
> the FDL.  If I am correct in assuming that the FDL also applies to the
> collection copyright, then I'd amend Lee's analysis to mention that: though
> copyright law gives Bomins the right to control the use of the wikipedia
> collection, the FDL explicitly allows republication and modification.
> Therefore anyone who downloads and republishes the wikipedia would not be in
> violation of copyright unless they violate the terms of the FDL.  Moreover,
> since it is only the FDL that grants anybody the right to reproduce
> Wikipedia articles at all, it is not particularly burdensome to be required
> to abide by the FDL if somebody publishes enough of it to violate Bomis's
> collection copyright.
>
> -----Original Message-----
> From: lcrocker at nupedia.com [mailto:lcrocker at nupedia.com]
> Sent: Monday, April 29, 2002 2:23 PM
> To: wikipedia-l at nupedia.com
> Subject: Re: [Wikipedia-l] Collection copyrights
>
>
> >This means that Bomis' collection copyright would be violated if
> >somebody were to copy the website wholesale. If on the other hand the
> >wiki sources of the articles are downloaded one by one, and a new web
> >site created out of those, then Bomis' collection copyright won't be
> >violated. If this new web site doesn't offer the articles under GFDL,
> >then the individual article authors could sue of course.
>
> There's a little more to it than that: copyrights apply to "creative
> expression", and "selection" of what to present is a creative act
> specifically recognized (see Feist v. Rural).  So it doesn't matter
> how they acquire the whole collection, whether they copy the website
> whole or download the articles one by one; if they present the same
> collection of articles that we do, they have violated our copyright
> on that creative choice of which articles to present.
>
> > This is similar to a Linux distribution. If you create a
> > distribution, you can claim collection copyright, and somebody
> > who copies the CDROM image without permission is in violation.
>
> ...that's true...
>
> > Everybody can however create their own distribution out of the
> > exact same free software components that you used, without
> > violating your collection copyright.
>
> ...but that's not true.  You /can/ be sued for violating a
> collection copyright for publishing the same collection of free
> components as someone else, regardless of how you acquired them,
> and even if you package them differently.  Choice of what to
> present is a creative act, and is protected.
> 0
> [Wikipedia-l]
> To manage your subscription to this list, please go here:
> http://www.nupedia.com/mailman/listinfo/wikipedia-l


from_name: Brion L. VIBBER

subject: [Wikitech-l] response times

body:
Just a couple quick notes for now...

On dim, 2002-05-05 at 13:21, Lars Aronsson wrote:
> The Esperanto Wikipedia has a small number of samples above 15
> seconds, which is sad but perhaps not alarming.  The "recent changes"
> has 56 % of its samples in the high 5-15 seconds response time
> interval, which is a little high.

Note also that the perl-based Esperanto wiki filters output through a
character set conversion and doesn't do any page caching (since caching
didn't interact well with the conversion), which is bound to slow it
down a little bit.

>  Perhaps this could be fixed by
> setting the default "recent changes" list from 7 or 3 days.  Who can
> change this setting?  Tell me when you change, and I will report how
> the response time changed.  Almost all other samples are in the 0-2
> and 2-5 categories, which is very good.

Have you tried running your tests on, say,
http://eo.wikipedia.com/wiki.cgi?action=rc&days=3 ?

-- brion vibber (brion @ pobox.com)


from_name: Magnus Manske

subject: [Wikitech-l] Skin classes

body:
I took LDC's advice and did most of the copy-and-past work for the PHP skin
classes (as I was the one responsible for the mess in the first place;) and
added the following files to the CVS:

* wikiSkins.php (has a dummy class and includes all other skins)
* wikiSkinStandard (the good ol' boring one)
* wikiSkinNostalgy (for those Perl fans)
* wikiSkinStarTrek (not really working right now!)
* wikiSkinCologneBlue

For Standard and CologneBlue, there are stylesheet functions in place. The
stylesheet function in wikiUser.php is hereby obsolete (I didn't remove it
yet, though).

Otherwise, the classes contain functions for header, footer, middle and
QuickBar.

I noticed that the namespace backgrounds are not working anymore after the
change. I couldn't figure out why. Someone please have a look. And if you're
at it, try to fix the Star Trek skin as well ;)

In time, I'll get the script just big enough to halt the server
altogether...

Magnus


from_name: Lars Aronsson

subject: [Wikitech-l] response times

body:
Source code and discussion follow below.

Over the last 24 hours I've polled some URLs each 10 minutes and
measured the response time.  This is a statistics report:

 min    avg    max   (count)  <2  2-5  5-15 >15  URL - response times in seconds
  0.75   2.00  11.79 (  144)  68%  27%   4%   0% http://pl.wikipedia.com/wiki.cgi?Szwecja
  0.56   2.70  89.78 (  144)  95%   0%   0%   3% http://www.wikipedia.com/wiki.png
  1.47   2.90   8.63 (  144)  27%  64%   7%   0% http://pl.wikipedia.com/wiki.cgi?Ostatnie_zmiany
  1.36   2.98  28.68 (  144)  30%  62%   6%   0% http://pl.wikipedia.com/
  0.83   3.21  91.86 (  144)  94%   0%   0%   4% http://eo.wikipedia.com/vikio.png
  1.00   4.45 140.53 (  144)  63%  29%   2%   4% http://eo.wikipedia.com/wiki/Svedio
  1.08   5.70 137.43 (  144)  53%  34%   7%   4% http://eo.wikipedia.com/
  3.40   7.09  68.23 (  144)   0%  38%  56%   4% http://eo.wikipedia.com/wiki/Lastaj_Sxangxoj
  3.35  13.59 203.39 (  144)   0%  18%  63%  18% http://www.wikipedia.com/wiki/special:RecentChanges
  1.61  28.38 411.43 (  144)   4%  23%  33%  38% http://www.wikipedia.com/wiki/Sweden
  3.86  45.16 359.17 (  144)   0%   2%  38%  58% http://www.wikipedia.com/

The first three columns present the minimum, average, and maximum
response times.  The rows are sorted on the average column.  As you
can see the minimums are very good: 0.56 seconds roundtrip from Sweden
to San Diego is excellent.  Even the English Wikipedia start page
(with the worst average in the list) has been served on 3.86 seconds,
which is quite good (this happened 6:10 am GMT).  However, the
striking numbers are the maximum response times of several minutes.

The fourth column is the number of samples, which is 144 in 24 hours.

The following four columns present the statistical distribution of
samples in four categories: The percentage of samples that were less
than 2 seconds, those between 2 and 5 seconds, those between 5 and 15
seconds, and those in excess of 15 seconds.  I thing usability gurus
like Jakob Nielsen has declared that 5 seconds is an acceptable
maximum for normal pages and most people can accept 15 seconds
response time for special functions such as searches and the recent
changes list.

The Polish Wikipedia has not a single sample above 15 seconds in these
24 hours.  The majority of samples is in the lower two categories,
which is very good.

The Esperanto Wikipedia has a small number of samples above 15
seconds, which is sad but perhaps not alarming.  The "recent changes"
has 56 % of its samples in the high 5-15 seconds response time
interval, which is a little high.  Perhaps this could be fixed by
setting the default "recent changes" list from 7 or 3 days.  Who can
change this setting?  Tell me when you change, and I will report how
the response time changed.  Almost all other samples are in the 0-2
and 2-5 categories, which is very good.

For the English Wikipedia, the static logotype image is served in less
than 2 seconds in 95 % of the samples.  For this URL, there are no
samples in the 2-5 or 5-15 intervals, but a few samples have very long
response times.  Perhaps the entire server was put on hold by some
other event?  The last three lines of the report are depressing.
Almost none of the samples fall in the 0-2 or 2-5 categories.  This
has to be analysed further by instrumenting the source code to report
where the delay is introduced.

I now have a running copy of the Wikipedia on my computer and have
started to experiment with this instrumentation.  It's really straight
forward.  In version 1.14 of wiki.phtml, Magnus Manske introduced the
function getmicrotime(), but the call to the function is commented
out.  Just after getmicrotime(), I introduce a new function:

    function trace($text){
      global $startTime, $traceText;
      $now = getmicrotime();
      $elapsed = $now - $startTime;
      if ($elapsed > 3.0) {
        $traceText = "$traceText\nAfter $elapsed seconds: $text";
        $startTime = $now;
      }
    }

Then at the beginning of the "main" program (where Magnus left a
commented-out first call to getmicrotime), I declare:

    global $startTime, $traceText;
    $startTime = getmicrotime();
    $traceText = "";

Then at various points throughout the code, just after function calls
that I suspect are time bandits, I insert calls to my trace function:

    trace("Just after updating the database");

These informative texts will accumulate in $traceText if the elapsed
time since the start is more than 3 seconds.

At the end of the "main" program comes the question, what should be
done with this $traceText?  Should it be inserted into a new database
table?  Or appended to the end of a text log file?  Or inserted as an
HTML comment into the generated web page?  Where can we best use this
information?  The easiest but perhaps least useful is this:

    trace("bottom of wiki.phtml main");
    if ($traceText != "")
      $out = "<!-- traceText:$traceText -->\n$out";

Who can implement this in the real source code?  I'm not in the gang.

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


subject: [Wikitech-l] Re: Slow Response

body:
Ok, Jason switched the logging off; it doesn't seem to have made much
of a difference. A log of the latest long queries can be downloaded
from our sourceforge website:

http://wikipedia.sourceforge.net/long_queries.gz

Regarding the "INSERT DELAYED" idea: I'm not sure that it will help
much: the manual states that it's not that useful for the MyISAM
tables we are using, since those tables allow asynchronous updates
anyway. It's worth a try though.

Another idea is to switch off the query cache (query_cache_size=0):
our tables change so frequently that a caching of queries seems to be
a big loss.

Axel


subject: [Wikitech-l] Wikipedia.sourceforge.net

body:
I have put some links together on http://wikipedia.sourceforge.net,
the home page of our Sourceforge project. All members of the
Sourceforge Wikipedia group can add more stuff, as explained on the
page itself.

Axel


subject: [Wikitech-l] response times

body:
Brion L. VIBBER wrote:
> Have you tried running your tests on, say,
> http://eo.wikipedia.com/wiki.cgi?action=rc&days=3 ?

I did a few samples now, but couldn't see any difference.  I'll let it
run for the next 24 hours, though.

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


subject: [Wikitech-l] eo-php wiki

body:
On lun, 2002-05-06 at 15:21, Jason Richey wrote:
> Brion L. VIBBER wrote:
> > Could you double-check that there was '$wikiLanguage = "eo";' set in
> > wikiSettings.php before the conversion?
>
> It was set in wikiLocalSettings...  That should be equivalent (or
> better), right?

Er, yes, that's what I meant. But clearly something didn't go right...

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] response times

body:
When I started to get extremely long response times (> 300 seconds), I
thought perhaps these are tiemouts or other kinds of accidents, so I
started to log HTTP status codes and the length of the returned
document as well.  But the status code is 200 (OK) and the full
document is returned, some 400-1800 seconds after the request was
sent.

Is anybody able to use the English Wikipedia now?  These are the last
two hours:

 min    avg    max   (count)  <2  2-5  5-15 >15  URL - response times in seconds
  0.72   0.79   1.54 (   15) 100%   0%   0%   0% http://www.wikipedia.com/wiki.png
 10.02 100.24 920.39 (   13)   0%   0%  30%  69% http://www.wikipedia.com/wiki/special:RecentChanges
 17.83 281.00 1840.27 (   12)   0%   0%   0% 100% http://www.wikipedia.com/wiki/Sweden
 22.14 333.33 1599.76 (   10)   0%   0%   0% 100% http://www.wikipedia.com/

--
  Lars Aronsson
  <lars at aronsson.se>
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


from_name: Lee Daniel Crocker

subject: [Wikitech-l] Spanish text in new software

body:
I'm curious: why is the Spanish text in wikiTextEs.php encoded
in utf-8?  ISO-8859-1 has all the needed characters for Spanish;
The German wikiTextDe.php uses plain ISO, and the current
es.wikipedia.com is in ISO.

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


from_name: Tomasz Wegrzanowski

body:
On Tue, May 07, 2002 at 01:47:57AM -0500, Lee Daniel Crocker wrote:
> I'm curious: why is the Spanish text in wikiTextEs.php encoded
> in utf-8?  ISO-8859-1 has all the needed characters for Spanish;
> The German wikiTextDe.php uses plain ISO, and the current
> es.wikipedia.com is in ISO.

All wikipedias are switching to utf-8.
Polish (normally latin2) and Esperanto (normally latin3) both decided
to do that.

I'm only wondering why English didn't do it *yet*
and I why Germans didn't switched.

I hope that both will switch.

Reasons include:

1.
We can't use latinX, we can either use utf8 or latinX +
&lot_of_silly_numeric_entities;

2.
UTF-8 allows you to insert all diactrics and other characters that
happen from time to time in proper names etc. Encoding them in html
codes is huge pita. There is no software that facilitate it.

3.
&silly_numeric_entities; are completely unreadable, and wikipedia
markup language should be easy to read and write.

4.
Interwiki links 100% require utf-8. Making interwiki links using
%-codes is completely out of the question. You can't even use
&silly_entities; in such links, as software don't know in what
encoding should it convert non-ascii characters to %-codes.

5.
General interoperability needs that. You wouldn't even be able to copy
from one wikipedia to another, if they used different charsets,
as software won't know in should convert diactrics to &entities;


subject: [Wikitech-l] Optimization

body:
I've been studying performance today.  I'm looking for the
bottlenecks.  The total volume of traffic on the site is not so high
as to cause me to reasonably suspect that hardware is the problem.  So
I'm looking for things that people do on the site that tend to be
expensive, so that we can eliminate features for awhile (if we must)
or find ways to optimize/cache them.

1.  I notice that many of the 'special' pages are quite popular.  I
suspect that these are our best bets.

2.  I propose that we test the site without the "this page has been
accessed N times" feature.  Doesn't it seem excessive to have a
database write for every single pageview, when we are trying to
increase performance?  A nearly equivalent feature could be provided
with a daily update from the access logs.

--Jimbo


subject: [Wikitech-l] Security issue--new software

body:
I finally came to the realization that the time I've spent getting
to know the software enough to make the few changes I wanted would
be wasted if I didn't do a good reorganization that the code badly
needs.  Hopefully that will also help us diagnose some of its
performance and scalability problems (for example, I've already
found that the code as it is now does 14 database queries to display
the front page!)

So, I'm refactoring the code and documenting as I go, but I want
to get an opinion from not only the tech folks but the list at
large about what the login/logout behavior should be.

Question 1: What is the behavior of the "remember password" user
option supposed to be?  Please don't describe code details--I need
to know WHAT it's supposed to do, not HOW (because I'm changing
the latter).  We already save the user's ID and some settings in
a cookie (we have to to handle prefs), and we delete the cookie
when the user explicitly logs out (which is good for security),
requiring him to log in and specify his password again.  So what's
the difference between a user with this setting and a user without
it?  Do we want to retain the user's password in a cookie even when
he logs out?  We don't do that currently, so I'm just not clear
what the option is supposed to do.

Question 2: What's a reasonable cookie expiration?  It's currently
a year, but I'm thinking a week or a month might be better for
security.

Question 3: Do we "trust" cookies for such issues as sysop access?
Some anonymous user could, for example, mockup the cookies of a
logged-in sysop and start deleting things.  Should we take measures
to secure against that, or just be more relaxed and rely on making
frequent backups?

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


from_name: Pierre Abbat

body:
On Tuesday 07 May 2002 20:20, Lee Daniel Crocker wrote:
> I finally came to the realization that the time I've spent getting
> to know the software enough to make the few changes I wanted would
> be wasted if I didn't do a good reorganization that the code badly
> needs.  Hopefully that will also help us diagnose some of its
> performance and scalability problems (for example, I've already
> found that the code as it is now does 14 database queries to display
> the front page!)
>
> So, I'm refactoring the code and documenting as I go, but I want
> to get an opinion from not only the tech folks but the list at
> large about what the login/logout behavior should be.
>
> Question 1: What is the behavior of the "remember password" user
> option supposed to be?  Please don't describe code details--I need
> to know WHAT it's supposed to do, not HOW (because I'm changing
> the latter).  We already save the user's ID and some settings in
> a cookie (we have to to handle prefs), and we delete the cookie
> when the user explicitly logs out (which is good for security),
> requiring him to log in and specify his password again.  So what's
> the difference between a user with this setting and a user without
> it?  Do we want to retain the user's password in a cookie even when
> he logs out?  We don't do that currently, so I'm just not clear
> what the option is supposed to do.

I'm working on a database with a web front end, and the way I handle logins
is that I set a cookie containing a random string and store the random
string, along with an expiration time and the user who is logged in, in a
table in the database. When a user logs in, all expired cookies are deleted.

> Question 2: What's a reasonable cookie expiration?  It's currently
> a year, but I'm thinking a week or a month might be better for
> security.
>
> Question 3: Do we "trust" cookies for such issues as sysop access?
> Some anonymous user could, for example, mockup the cookies of a
> logged-in sysop and start deleting things.  Should we take measures
> to secure against that, or just be more relaxed and rely on making
> frequent backups?

I would make the sysop's cookie expire faster than a regular user's cookie.
We might require that a sysop use HTTPS.

phma


subject: [Wikitech-l] Unusual behavior this morning...

body:
On mer, 2002-05-08 at 00:30, Neil Harris wrote:
> The English Wikipedia is doing two unusual things this morning:
>
> 1 It's generating error messages on 'ordinary' non-special pages, as
> follows:
>
> *   Warning*: Supplied argument is not a valid MySQL result resource in
> */home/wiki-newest/work-http/wikiPage.php* on line *86*
>
> *   Warning*: Supplied argument is not a valid MySQL result resource in
> */home/wiki-newest/work-http/wikiPage.php* on line *88*

See note in http://www.wikipedia.com/wiki/wikipedia:Bug+reports/Parser
(though it doesn't actually look parser-related, so feel free to move
it). Unless somebody's updated wikiPage.php since April 2 or so, the
errors seem to be in the load() method, when processing the results of
querying the unlinked table.

Someone with access to the server logs and database may be able to get a
better handle on what exactly is going on.

> 2 And... it seems much faster!
>
> Could these be related?

If so, vivent les erreur-messages! :P

-- brion vibber (brion @ pob.xocm)


from_name: Neil Harris

body:
The English Wikipedia is doing two unusual things this morning:

1 It's generating error messages on 'ordinary' non-special pages, as
follows:

*   Warning*: Supplied argument is not a valid MySQL result resource in
*/home/wiki-newest/work-http/wikiPage.php* on line *86*

*   Warning*: Supplied argument is not a valid MySQL result resource in
*/home/wiki-newest/work-http/wikiPage.php* on line *88*

2 And... it seems much faster!

Could these be related?

-- Neil


subject: [Wikitech-l] Problem

body:
http://www.wikipedia.com/wiki/Geography

Warning: Supplied argument is not a valid MySQL result resource in /home/wiki-newest/work-http/wikiPage.php on line 86

Warning: Supplied argument is not a valid MySQL result resource in /home/wiki-newest/work-http/wikiPage.php on line 88

----

I didn't do anything to cause this.  I am looking into it now.


subject: [Wikitech-l] Problem

body:
On mer, 2002-05-08 at 09:57, Jimmy Wales wrote:
> http://www.wikipedia.com/wiki/Geography
>
> Warning: Supplied argument is not a valid MySQL result resource in /home/wiki-newest/work-http/wikiPage.php on line 86
>
> Warning: Supplied argument is not a valid MySQL result resource in /home/wiki-newest/work-http/wikiPage.php on line 88
>
> ----
>
> I didn't do anything to cause this.  I am looking into it now.

Thanks for upgrading the program to the current version! Now that the
AskSQL feature is back (for sysops only), I tried the query:

SELECT DISTINCT unlinked_to FROM unlinked WHERE unlinked_from = "Geography"

and got this error message:
  Can't open file: 'unlinked.MYD'. (errno: 145)

Suspicious, no?

-- brion vibber (brion @ pobox.com)


subject: [Wikipedia-l] Re: [Wikitech-l] two suggestions

in_reply_to: <20020508205758.9997.qmail@www.llamacom.com>; from kband@www.llamacom.com on Wed, May 08, 2002 at 03:57:58PM -0500

body:
kband at www.llamacom.com wrote:
> Removing Recent Changes functionality removes most of the
> functionality for me as an editor...taking it down would be almost as
> bad as the current molasses we're in.

Yeah, totally.  Recent Changes is an absolute necessity.

But if we find out that some of the other "neat tricks" are weighing us down, we
could cut them out temporarily.

> A much better suggestion would be to fix the code. My completely
> unhelpful suspicion is that it's PHP's fault. But I'm a perl bigot, so
> take it for what it's worth. I'm just not surprised we're having
> problems.

I'm a perl fanatic, but not a perl bigot.  I haven't really considered
that PhP would be the culprit.


from_name: Chuck Smith

subject: [Wikitech-l] two suggestions

body:
Improving performance on English server:  when the
server reaches a certain threshhold of activity could
we replace the text of all the Special pages with:

"This page is currently inaccessible because of the
immensive amount of traffic we're getting.  Wikipedia
is just too popular."

Also, has the Lastaj Sxangxoj page on the Esperanto
Wikipedia page been defaulted to 3 days yet?

Thanks,
Chuck

=====
Come to my homepage!  Venu al mia hejmpagxo!
http://amuzulo.babil.komputilo.org/
====
Venu al la senpaga, libera enciklopedio
esperanta reta! http://eo.wikipedia.com/

_________________________________________________________
Do You Yahoo!?
Informaci?n de Estados Unidos y Am?rica Latina, en Yahoo! Noticias.
Vis?tanos en http://noticias.espanol.yahoo.com


subject: [Wikitech-l] two suggestions

body:
On mer, 2002-05-08 at 12:41, Chuck Smith wrote:
> Improving performance on English server:  when the
> server reaches a certain threshhold of activity could
> we replace the text of all the Special pages with:
>
> "This page is currently inaccessible because of the
> immensive amount of traffic we're getting.  Wikipedia
> is just too popular."

As others have said, I suspect that's not such a great idea -- it would
make editing the 'pedia virtually impossible instead of the realm of the
very very patient. :)

> Also, has the Lastaj Sxangxoj page on the Esperanto
> Wikipedia page been defaulted to 3 days yet?

Apparently not. I've attached a diff to the config file that does that,
and since while I was in there I noticed another setting, now tells it
to NOT delete old page versions after 14 days. Grrrggghg!!!!! I think
that's still going on on most of the non-English wikis -- that is a VERY
BAD THING that should be changed as soon as possible!

While we're mentioning applying patches etc; I've added a couple little
bug fixes to wikiTextEn.php (a couple missing/wrong entries in the intl
wiki language name table), special_recentchangeslayout.php (watchlist
entries are now boldened, but a bug was preventing those with
spaces/underscores in the names from working), and special_watchlist.php
(a little hack to cut down on the pain to the database -- every entry is
checked with a separate database query, and the new accept-any-namespace
version made it so slow as to time out on my long watchlist). Jimbo or
Jason, if you could update those three files from CVS, that'd be great.

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: eo-config-tweaks.diff
Type: text/x-diff
Size: 893 bytes
Desc: not available
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20020508/e9289a86/attachment.diff>


from_name: lcrocker at nupedia.com

subject: [Wikipedia-l]  [Wikitech-l] two suggestions

body:
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://lists.wikimedia.org/pipermail/wikipedia-l/attachments/20020508/1571624c/attachment.diff


from_name: kband at www.llamacom.com

body:
=?iso-8859-1?q?Chuck=20Smith?= wrote:
>
> Improving performance on English server:  when the
> server reaches a certain threshhold of activity could
> we replace the text of all the Special pages with:
>
> "This page is currently inaccessible because of the
> immensive amount of traffic we're getting.  Wikipedia
> is just too popular."

Removing Recent Changes functionality removes most of the
functionality for me as an editor...taking it down would be almost as
bad as the current molasses we're in.

A much better suggestion would be to fix the code. My completely
unhelpful suspicion is that it's PHP's fault. But I'm a perl bigot, so
take it for what it's worth. I'm just not surprised we're having
problems.

--tc


body:
Ian wrote:
> There needs to a series of mirrors to show static pages, and wikipedia.com
> only used for Recent Changes and editing. Their would still be a lot of
> load on wikipedia.com but not as much, and if you just have to look
> something up it would be easier. The mirrors could update about once or
> twice a day.
>
> School ends on the 24th, hopefully I'll be able to implement it this
> summer. It would just need something to automatically produce static pages
> which the mirrors would somehow be informed of what to download or it
> could all be bziped up, whatever. I'll try to make it in as seperate files
> as possible so it doesn't have to actually be implemented until its ready
> and wanted.

While I'm sympathetic to this in the long run, I'm not convinced that
this is the real problem.  We're not seeing anything like the levels
of traffic that should be difficult for a single server to handle.  We're
just not using our resources wisely.

--Jimbo


subject: [Wikitech-l] Most Wanted vanished

body:
On mer, 2002-05-08 at 15:37, Magnus Manske wrote:
> I can't get the Most Wanted list anymore. Not sure if this is caused by the
> apparent trouble with the database, or by my recent attempt to clean up the
> skins.

Warning: Supplied argument is not a valid MySQL result resource in
/home/wiki-newest/work-http/special_wantedpages.php on line 22

12  $connection = getDBconnection () ;
13  $sql = "SELECT unlinked_to, COUNT( DISTINCT unlinked_from ) AS ulf
14          FROM unlinked
15          GROUP BY unlinked_to
16          ORDER BY ulf DESC
17          LIMIT 50 " ;
18
19  $result = mysql_query ( $sql , $connection ) ;
20
21  $ti = new wikiTitle ;
22  while ( $s = mysql_fetch_object ( $result ) ) {

Sure sounds like the database trouble.

-- brion vibber (brion @ pobox.com)


from_name: Ian

body:
There needs to a series of mirrors to show static pages, and wikipedia.com
only used for Recent Changes and editing. Their would still be a lot of
load on wikipedia.com but not as much, and if you just have to look
something up it would be easier. The mirrors could update about once or
twice a day.

School ends on the 24th, hopefully I'll be able to implement it this
summer. It would just need something to automatically produce static pages
which the mirrors would somehow be informed of what to download or it
could all be bziped up, whatever. I'll try to make it in as seperate files
as possible so it doesn't have to actually be implemented until its ready
and wanted.

Ian Monroe
http://ian.webhop.org

On Wed, 8 May 2002, kband at www.llamacom.com XXXXXXXXXXXXXXXXXXXXX wrote:

> =?iso-8859-1?q?Chuck=20Smith?= wrote:
> >
> > Improving performance on English server:  when the
> > server reaches a certain threshhold of activity could
> > we replace the text of all the Special pages with:
> >
> > "This page is currently inaccessible because of the
> > immensive amount of traffic we're getting.  Wikipedia
> > is just too popular."
>
> Removing Recent Changes functionality removes most of the
> functionality for me as an editor...taking it down would be almost as
> bad as the current molasses we're in.
>
> A much better suggestion would be to fix the code. My completely
> unhelpful suspicion is that it's PHP's fault. But I'm a perl bigot, so
> take it for what it's worth. I'm just not surprised we're having
> problems.
>
> --tc
> [Wikipedia-l]
> To manage your subscription to this list, please go here:
> http://www.nupedia.com/mailman/listinfo/wikipedia-l
>


subject: [Wikitech-l] two suggestions

body:
On mer, 2002-05-08 at 18:50, Chuck Smith wrote:
>  --- "Brion L. VIBBER" <brion at pobox.com> escribi?: >
> On mer, 2002-05-08 at 12:41, Chuck Smith wrote:
> > > Improving performance on English server:  when the
> > > server reaches a certain threshhold of activity could
> > > we replace the text of all the Special pages with:
> > > "This page is currently inaccessible because of the
> > > immensive amount of traffic we're getting.  Wikipedia
> > > is just too popular."
> >
> > As others have said, I suspect that's not such a great idea -- it would
> > make editing the 'pedia virtually impossible instead of the realm of the
> > very very patient. :)
>
> You saw that I said _Special_ pages, right?

Yes, of which special:RecentChanges is one. I don't know about you, but
my Wikipedia regimen centers around the recent changes list: without it,
there's no good way to see what other people are adding to the wiki that
may need to be edited/corrected/expanded/commented on.

-- brion vibber (brion @ pobox.com)


from_name: Karen AKA Kajikit

subject: [Wikipedia-l] eeps! What's happened to the Wikipedia????

body:
I just tried to open it up and got an error message - 'supplied argument
is not a valid MySQL result' etc etc...

Who broke the pedia?

--

Karen AKA Kajikit

Come and visit my part of the web:
Kajikit's Corner: http://Kajikit.netfirms.com/
Aussie Support Mailing List: http://groups.yahoo.com/group/AussieSupport
Allergyfree Eating Recipe Swap:
http://groups.yahoo.com/group/Allergyfree_Eating

Love and huggles to all!


subject: [Wikitech-l] Internationalization issues

body:
On mer, 2002-05-08 at 20:33, Lee Daniel Crocker wrote:
> A lot of the character-encoding stuff in the present code is a mess
> too.

I appreciate your polite understatement.

>  I understand well and can handle all the details between server
> and browser, but two things I don't know all the quirks of are PHP and
> MySQL, so this is my attempt to pick the brains of those who have
> already found those problems:
>
> (1) Is MySQL 8-bit clean?  If I store a chunk of 8-bit bytes in a
> text field, will I get them back unmolested, or will MySQL try to be
> "helpful" and fuck them up?  If the latter, what are the limitations
> of what can be stored in a text field and where is that documented?

As far as I know, yes (the former). MySQL has no direct support for
UTF-8, but it does have explicit support for a number of single and
multibyte encodings, so one would expect it to be generally 8-bit clean,
and implementations of sticking UTF-8 strings into MySQL using the
default ISO-8859-1 setting abound. However there are limitations --
MySQL doesn't know about proper case matching or accent folding, for
instance, which would be nice for searching. (This is get-aroundable if
we do our own case/accent-folding when saving a page and storing it in a
separate field just for searches, which has been discussed from time to
time.)

> (2) Are PHP strings 8-bit clean?  I'd be amazed if they weren't,
> considering how much of PHP is modelled on Perl.

Claims to be.

> (3) Is the PHP on wikipedia.com compiled with the "iconv" library
> (an optional thing),

I don't believe it is. It really should be.

> and does PHP use it as documented?

That's a good question!

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] two suggestions

body:
 --- "Brion L. VIBBER" <brion at pobox.com> escribi?: >
On mer, 2002-05-08 at 12:41, Chuck Smith wrote:
> > Improving performance on English server:  when the
> > server reaches a certain threshhold of activity
> could
> > we replace the text of all the Special pages with:
> >
> > "This page is currently inaccessible because of
> the
> > immensive amount of traffic we're getting.
> Wikipedia
> > is just too popular."
>
> As others have said, I suspect that's not such a
> great idea -- it would
> make editing the 'pedia virtually impossible instead
> of the realm of the
> very very patient. :)

You saw that I said _Special_ pages, right?

=====
Come to my homepage!  Venu al mia hejmpagxo!
http://amuzulo.babil.komputilo.org/
====
Venu al la senpaga, libera enciklopedio
esperanta reta! http://eo.wikipedia.com/

_________________________________________________________
Do You Yahoo!?
Informaci?n de Estados Unidos y Am?rica Latina, en Yahoo! Noticias.
Vis?tanos en http://noticias.espanol.yahoo.com


subject: [Wikitech-l] two suggestions

body:
 --- "Brion L. VIBBER" <brion at pobox.com> escribi?: >
On mer, 2002-05-08 at 18:50, Chuck Smith wrote:
> >  --- "Brion L. VIBBER" <brion at pobox.com>
> escribi??: >
> > On mer, 2002-05-08 at 12:41, Chuck Smith wrote:
> > > > Improving performance on English server:  when
> the
> > > > server reaches a certain threshhold of
> activity could
> > > > we replace the text of all the Special pages
> with:
> > > > "This page is currently inaccessible because
> of the
> > > > immensive amount of traffic we're getting.
> Wikipedia
> > > > is just too popular."
> > >
> > > As others have said, I suspect that's not such a
> great idea -- it would
> > > make editing the 'pedia virtually impossible
> instead of the realm of the
> > > very very patient. :)
> >
> > You saw that I said _Special_ pages, right?
>
> Yes, of which special:RecentChanges is one. I don't
> know about you, but
> my Wikipedia regimen centers around the recent
> changes list: without it,
> there's no good way to see what other people are
> adding to the wiki that
> may need to be edited/corrected/expanded/commented
> on.

Ah, I meant all the Special pages *but* Recent
Changes.  Read what I mean, not what I write!  :)  Now
how does the idea sound...?

Chuck

=====
Come to my homepage!  Venu al mia hejmpagxo!
http://amuzulo.babil.komputilo.org/
====
Venu al la senpaga, libera enciklopedio
esperanta reta! http://eo.wikipedia.com/

_________________________________________________________
Do You Yahoo!?
Informaci?n de Estados Unidos y Am?rica Latina, en Yahoo! Noticias.
Vis?tanos en http://noticias.espanol.yahoo.com


subject: [Wikitech-l] Speed

body:
I'll go through the source over the weekend and try to find and fix some
slow parts.

For now, it might improve the speed if the MySQL server would be located on
a different machine than the Apache. I think that could be done without too
much trouble, and without security problems for Bomis.

Magnus


body:
A lot of the character-encoding stuff in the present code is a mess
too.  I understand well and can handle all the details between server
and browser, but two things I don't know all the quirks of are PHP and
MySQL, so this is my attempt to pick the brains of those who have
already found those problems:

(1) Is MySQL 8-bit clean?  If I store a chunk of 8-bit bytes in a
text field, will I get them back unmolested, or will MySQL try to be
"helpful" and fuck them up?  If the latter, what are the limitations
of what can be stored in a text field and where is that documented?

(2) Are PHP strings 8-bit clean?  I'd be amazed if they weren't,
considering how much of PHP is modelled on Perl.

(3) Is the PHP on wikipedia.com compiled with the "iconv" library
(an optional thing), and does PHP use it as documented?

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


subject: [Wikitech-l] Most Wanted vanished

body:
I can't get the Most Wanted list anymore. Not sure if this is caused by the
apparent trouble with the database, or by my recent attempt to clean up the
skins.

Magnus


subject: [Wikitech-l] Optimization

body:
Jimmy Wales wrote:
> 2.  I propose that we test the site without the "this page has been
> accessed N times" feature.

This is my best guess too.  However, it is possible to get from
guesses to knowledge by checking getmicrotime() before and after the
call and log the elapsed time (to stderr?).  Wrap each database
transaction in this, and get some statistics on which calls are
worst, rather than removing calls by guessing.

Not knowing more about this specific system, my general experience is
that database locks can eat a lot of wallclock time without consuming
any CPU time.  During the delay, the memory allocated by each execution
thread can eat so much memory that a machine starts to swap.  I have
no idea if this happens in Wikipedia, but you can monitor disk I/O by
simple programs like vmstat (or iostat on some systems).  If this is
the case, adding RAM to avoid swapping is *not* a solution.

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


subject: [Wikitech-l] two suggestions

body:
> Why are we discussing ideas on wikitech-l, when we
> should discuss
> technology?  Guessing belongs in metaphysics,
> engineering should be
> about knowing.  This is not Big Brother where we
> vote on what to
> exclude.  We should go out and find numbers from
> which we can make
> informed decisions on how to improve the
> performance, not sacrifice
> functionality.  I'm suggesting ways to find such
> numbers, but I don't
> have access to update the source code.

As far as I know it, in the world of IT sometimes you
have to use work-arounds instead of finding the
problem *temporarily* to make sure systems don't fall
below a reasonable performance level.  It's not the
best idea, and it's not pretty, but it keeps the
systems running.  This new functionality is great, but
if it's hurting the project (I think we're probably
losing about 100-200 new articles/day), then it needs
to be disabled until a fix can be found so we can at
least stay as productive as before the upgrade.

Should I be posting to Wikipedia-L instead?

My thoughts,
Chuck

=====
Come to my homepage!  Venu al mia hejmpagxo!
http://amuzulo.babil.komputilo.org/
====
Venu al la senpaga, libera enciklopedio
esperanta reta! http://eo.wikipedia.com/

_________________________________________________________
Do You Yahoo!?
Informaci?n de Estados Unidos y Am?rica Latina, en Yahoo! Noticias.
Vis?tanos en http://noticias.espanol.yahoo.com


subject: [Wikipedia-l] Bugs? or Features?

body:
On ?a?, 2002-05-09 at 03:53, Karen AKA Kajikit wrote:
> I had my settings set to show up unmade links in red which made it easy
> to notice them and to see what needs making, and now they're back to
> blue with a question mark even though I have 'red links' selected in my
> options. Has that bit of scripting been disabled/removed or is something
> weird going on?
>
> The 'talk' option seems to show up as a blue link on all pages now
> whether ther's anything on the page or not.

Magnus recently rearranged the style/skin code, and it looks like some
of the bits that control link color got accidentally broken in the
process. Hopefully should be fixed soon.

> Also, what does the new menu option in the sidebar mean? 'Vote for this
> page'. Vote for this page for what?

I wish I knew, too. :) Magnus?

> Finally, a new option has been added to the edit window 'automatically
> wikify' but when I tried it to see what it did it didn't seem to do
> anything... is it enabled? And what is it MEANT to do in an ideal world?

I believe it's supposed to tidy up some of the mark-up. But exactly what
it's intended to touch, and whether it's safe to use at this point, I'm
not quite sure about.

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] The Need for Speed

in_reply_to: <200205091838.g49IcCB05946@levi.aronsson.se>; from lars@aronsson.se on Thu, May 09, 2002 at 08:38:12PM +0200

body:
I, too, can vouch for FastCGI.  We run tons of stuff through it on
Bomis.  It's the bomb.  Of course, changing Wikipedia from php/mysql
to perl fastcgi/mysql would be a major undertaking at this point, and
reports from around the web suggest strongly that php is fast,
particularly when persistency is used (as we do).

--Jimbo


subject: [Wikitech-l] two suggestions

body:
Chuck Smith wrote:
> Ah, I meant all the Special pages *but* Recent
> Changes.  Read what I mean, not what I write!  :)  Now
> how does the idea sound...?

Why are we discussing ideas on wikitech-l, when we should discuss
technology?  Guessing belongs in metaphysics, engineering should be
about knowing.  This is not Big Brother where we vote on what to
exclude.  We should go out and find numbers from which we can make
informed decisions on how to improve the performance, not sacrifice
functionality.  I'm suggesting ways to find such numbers, but I don't
have access to update the source code.

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


subject: [Wikitech-l] Wikipedia in crisis

body:
Well, for the past 2 hours, the load average has been hovering around
100.

So I'm preparing to do some (temporary) drastic measures to change
things around.  I'm not sure what, though.

--Jimbo


subject: [Wikipedia-l] have we lost three months?

body:
I just turned on page caching.  Maybe it has bugs?

Vicki Rosenzweig wrote:

> I just went into the "Talk" section of the main page to suggest it was time
> to remove
> the Winter Olympics from the main page, before noticing this at the bottom
> of the
> main page:
>
> Today is Tuesday, February 12, 2002, server time (U.S. Pacific Time).
>
> What's going on here?
> --
> Vicki Rosenzweig
> vr at redbird.org
> http://www.redbird.org
>
> [Wikipedia-l]
> To manage your subscription to this list, please go here:
> http://www.nupedia.com/mailman/listinfo/wikipedia-l


subject: [Wikitech-l] Re: two suggestions

body:
> I'm suggesting ways to find such numbers, but I don't
> have access to update the source code.

I agree that it would be useful to get numbers. Why don't you prepare
and test your patch and post it here (or get commit access from
Magnus).

I think it would be best to simply report the times in an HTML comment
attached to every served file, as you suggested earlier. This way we
can all perform our own tests from home and no artificial bottleneck
slowdowns are introduced (which could occur if the time data were logged in
a central file).

Axel


subject: [Wikitech-l] Slow parts

in_reply_to: <175rJx-0QqT2GC@fwd02.sul.t-online.com>; from Magnus.Manske@t-online.de on Thu, May 09, 2002 at 07:03:49PM +0200

body:
I just now made the function look like this:

    function getOtherNamespaces () {
        $a = array () ;
        # modification by Jimbo
        return $a;

I assume that forces it to return an empty array and not run the rest of the code.

The site didn't die, so I'll assume that's o.k. as an emergency measure.

Also, I turned on pagecaching.  This seems to have helped somewhat.

Magnus Manske wrote:

> I identified the "getOtherNamespaces" function in wikiPage.php as a major bad
> guy. It is used to link to the appropriate Talk namespace in the sidebar and
> the footer. On my test installation, returning no other namespaces (an empty
> array) cuts the time to generate a page in half! (Tested with the "Biology"
> article using ab -n 20 -c 4)
>
> I don't have time right now to fix this, someone please!!!!!!! help :)
>
> Magnus
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at ross.bomis.com
> http://ross.bomis.com/mailman/listinfo/wikitech-l


subject: [Wikitech-l] Wikipedia much better now!

body:
It's like a miracle cure.

  5:43pm  up 37 days,  8:13,  1 user,  load average: 0.76, 1.95, 6.84

Compare this to hovering around 100 earlier today.

I did two things -- turned on page caching, and forced
getOtherNamespaces to return an empty array.

The site feels usably fast now, and the load average is a lot lower.

--Jimbo


body:
Whatever the case, the fact is Wikipedia is hard to use. This one
relatively simple solution.

Ian Monroe
http://ian.webhop.org

On Wed, 8 May 2002, Jimmy Wales jwales at bomis.com XXXXXXXXXXXXXXXXXXXXX wrote:

> Ian wrote:
> > There needs to a series of mirrors to show static pages, and wikipedia.com
> > only used for Recent Changes and editing.
> > [snip]
> While I'm sympathetic to this in the long run, I'm not convinced that
> this is the real problem.  We're not seeing anything like the levels
> of traffic that should be difficult for a single server to handle.  We're
> just not using our resources wisely.
>
> --Jimbo
> [Wikipedia-l]
> To manage your subscription to this list, please go here:
> http://www.nupedia.com/mailman/listinfo/wikipedia-l
>


body:
On ?a?, 2002-05-09 at 17:08, Jimmy Wales wrote:
> I just turned on page caching.  Maybe it has bugs?

With caching disabled, the page cache hasn't been updated in three
months. :)

Just clear all the cached pages:

  UPDATE cur SET cur_cache=""

should do it. (Double-check that before you run it, obviously.)

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] Slow parts

body:
I identified the "getOtherNamespaces" function in wikiPage.php as a major bad
guy. It is used to link to the appropriate Talk namespace in the sidebar and
the footer. On my test installation, returning no other namespaces (an empty
array) cuts the time to generate a page in half! (Tested with the "Biology"
article using ab -n 20 -c 4)

I don't have time right now to fix this, someone please!!!!!!! help :)

Magnus


subject: [Wikitech-l] Wikipedia much better now!

body:
On ?a?, 2002-05-09 at 17:20, Jimmy Wales wrote:
> It's like a miracle cure.
>
>   5:43pm  up 37 days,  8:13,  1 user,  load average: 0.76, 1.95, 6.84
>
> Compare this to hovering around 100 earlier today.
>
> I did two things -- turned on page caching, and forced
> getOtherNamespaces to return an empty array.
>
> The site feels usably fast now, and the load average is a lot lower.

Huzzah, thanks!

Also, could you update special_watchlist.php (temporary work-around for
amazingly performance-killing database abuse that makes the watchlist
completely unusable if you've any serious number of items in it),
wikiTextEn.php (language link bugs), and special_recentchangeslayout.php
(cosmetic bug related to the watchlist) from CVS?

-- brion vibber (brion @ pobox.com)


from_name: Vicki Rosenzweig

body:
I just went into the "Talk" section of the main page to suggest it was time
to remove
the Winter Olympics from the main page, before noticing this at the bottom
of the
main page:

Today is Tuesday, February 12, 2002, server time (U.S. Pacific Time).

What's going on here?
--
Vicki Rosenzweig
vr at redbird.org
http://www.redbird.org


subject: [Wikitech-l] The Need for Speed

body:
FYI, I just changed http://susning.nu/ from plain old CGI to FastCGI
and it really makes a change.  This is a UseModWiki, just like the old
pre-PHP Wikipedias.  Plain wiki pages used to load in 2.5 seconds on
average.  Now they load in less than 0.8 seconds (searches and recent
changes take longer).

Q: Why don't I change to the Wikipedia PHP software?
A: I've invested too much of my own hacking into the UseModWiki script
   to abandon it quite yet.  And converting to FastCGI turned out to
   be a piece of cake.

Q: Why don't I run mod_perl instead of stinking old FastCGI technology?
A: I use a web hotel with Linux shell login accounts and they don't do
   mod_perl.  I think the reason is that scripts run under mod_perl
   cannot change user ID to each customer.

Q: Does the site get any traffic?
A: It had 180,000 page views in April from 40,000 unique IP addresses,
   growing at 40 % per month.  Many hits are referrals from Google.

--
  Lars Aronsson
  <lars at aronsson.se>
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


subject: [Wikipedia-l] Bugs? or Features?

body:
I've noticed a few things changing in the last few days, the error
messages for one. :( Maybe someone here can tell me something about
them?

I had my settings set to show up unmade links in red which made it easy
to notice them and to see what needs making, and now they're back to
blue with a question mark even though I have 'red links' selected in my
options. Has that bit of scripting been disabled/removed or is something
weird going on?

Also, what does the new menu option in the sidebar mean? 'Vote for this
page'. Vote for this page for what?

The 'talk' option seems to show up as a blue link on all pages now
whether ther's anything on the page or not. Clicking a few links the
entry consisted of the standard 'describe the new page here' only it's
been saved as page content instead of just appearing in the edit window.

Finally, a new option has been added to the edit window 'automatically
wikify' but when I tried it to see what it did it didn't seem to do
anything... is it enabled? And what is it MEANT to do in an ideal world?

--

Karen AKA Kajikit

Come and visit my part of the web:
Kajikit's Corner: http://Kajikit.netfirms.com/
Aussie Support Mailing List: http://groups.yahoo.com/group/AussieSupport
Allergyfree Eating Recipe Swap:
http://groups.yahoo.com/group/Allergyfree_Eating

Love and huggles to all!


from_name: koyaanisqatsi at nupedia.com

subject: [Wikipedia-l] missing Talk!

body:
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://lists.wikimedia.org/pipermail/wikipedia-l/attachments/20020509/d1b14bc6/attachment.diff


subject: Urgent correction! (Re: [Wikipedia-l] have we lost three months?)

body:
On ?a?, 2002-05-09 at 18:27, Brion L. VIBBER wrote:
> On ?a?, 2002-05-09 at 17:08, Jimmy Wales wrote:
> > I just turned on page caching.  Maybe it has bugs?
>
> With caching disabled, the page cache hasn't been updated in three
> months. :)
>
> Just clear all the cached pages:
>
>   UPDATE cur SET cur_cache=""
>
> should do it. (Double-check that before you run it, obviously.)

CORRECTION: DO NOT PEFORM THE ABOVE QUERY!!!!!!!!!! It would reset the
last edited time on every page to the current date!

Instead try:

  UPDATE cur SET cur_cache="",cur_timestamp=cur_timestamp

I'm beginning to think those auto-updating timestamp fields aren't such
a good idea.

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] Urgent correction! (Re: [Wikipedia-l] have we lost three months?)


subject: [Wikitech-l] Slow parts

body:
On ?a?, 2002-05-09 at 17:12, Jimmy Wales wrote:
>
> I just now made the function look like this:
>
>     function getOtherNamespaces () {
>         $a = array () ;
>         # modification by Jimbo
>         return $a;
>
> I assume that forces it to return an empty array and not run the rest of the code.
>
> The site didn't die, so I'll assume that's o.k. as an emergency measure.

As Chuck has noted, this has the effect of removing all the Talk links!

I suspect the problem may be this query:
  SELECT cur_title FROM cur WHERE cur_title LIKE "%:$n"

I put a similar query into the Watchlist code so that watchlisting a
title would list updates to pages with that title in all namespaces
(mainly for talk pages), and it slowed to a completely unusable crawl:
my (more than ample) watchlist eventually times out before giving any
output. (Note that I've already put a workaround for that into CVS to
explicitly look for talk pages.)

It may be useful to split the title into separate namespace and
(sub)title fields so that slow substring matches don't need to be used.
Alternatively, a more efficient way to search the existing field this
way would be welcome.

I've attached a patch for a workaround that replaces the query with just
the talk page link; give it a try and see if that's still helping.

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: namespace-hack.diff
Type: text/x-diff
Size: 1748 bytes
Desc: not available
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20020509/59602480/attachment.diff>


subject: [Wikipedia-l] cleaning up...

body:
On ?a?, 2002-05-09 at 22:17, The Cunctator wrote:
> I'm very excited that Wikipedia is back. I haven't been able to edit for
> about a month, and a lot's been going on in the world...
>
> Other things:
>
> 1) At some point links to nonexistent pages were changed from
> [New Page]?
>
> to
>
> [New
> Page]?
>
> Please change it back. It's much harder to differentiate nonexistent pages
> now.

That's a bug, some of the style sheet code was shuffled around and the
bit that made those links regular-text style somehow dropped out. I'll
fix it shortly.

-- brion vibber (brion @ pobox.com)


subject: [Wikipedia-l] missing Talk!

body:
On every article I've looked at on the English
Wikipedia, there is now NO link to the Talk page of
articles.  I'm using the standard skin.  Please fix
this SOON!

Thank you,
Chuck

=====
Come to my homepage!  Venu al mia hejmpagxo!
http://amuzulo.babil.komputilo.org/
====
Venu al la senpaga, libera enciklopedio
esperanta reta! http://eo.wikipedia.com/

_________________________________________________________
Do You Yahoo!?
Informaci?n de Estados Unidos y Am?rica Latina, en Yahoo! Noticias.
Vis?tanos en http://noticias.espanol.yahoo.com


subject: [Wikitech-l] Skin classes

body:
On dim, 2002-05-05 at 09:30, Magnus Manske wrote:
> I took LDC's advice and did most of the copy-and-past work for the PHP skin
> classes (as I was the one responsible for the mess in the first place;) and
> added the following files to the CVS:
>
> * wikiSkins.php (has a dummy class and includes all other skins)
> * wikiSkinStandard (the good ol' boring one)
[..]
> I noticed that the namespace backgrounds are not working anymore after the
> change. I couldn't figure out why. Someone please have a look.

Upon inspection, it turns out that setting the value of $stylesheet does
not change $styleSheet. Ahhh, case-sensitivity. Suddenly, BASIC and
Pascal seem so warm and cuddly in my memory...

Fix to wikiSkinStandard.php is in CVS. (This also fixes the edit-link
style problem that the Cunctator and others have noted.)

> And if you're
> at it, try to fix the Star Trek skin as well ;)

I'll be happier if that one stays dead! ;)

-- brion vibber (brion @ pobox.com)


from_name: The Cunctator

subject: [Wikipedia-l] cleaning up...

body:
On 5/9/02 11:39 PM, "Chuck Smith" <msochuck at yahoo.com> wrote:

> On every article I've looked at on the English
> Wikipedia, there is now NO link to the Talk page of
> articles.  I'm using the standard skin.  Please fix
> this SOON!

I'm very excited that Wikipedia is back. I haven't been able to edit for
about a month, and a lot's been going on in the world...

Other things:

1) At some point links to nonexistent pages were changed from
[New Page]?

to

[New
Page]?

Please change it back. It's much harder to differentiate nonexistent pages
now.

2) Please change the daily date link that's on every page. The current
implementation is a lot more crippled than (and redundant to) the Current
Events page.

Ways to change:

 a) simply drop the link. Featuritis can be cured.
 b) change the link to point to Current Events, while keeping the date
 c) change the link to Current Events, with that name
 d) implement complex templating/reference system that makes individual date
    pages a useful resource

I recommend a) or b).

Or just give me trunk access and I'll implement the changes myself, since
the above changes were added without any discussion.


subject: [Wikitech-l] speed at last

body:
My log says it happened at about 00:40 GMT, almost three hours ago.
It is wonderful.  Thanks!

                      Resp.
                      time
             GMT      secs      URL                       Length
             -------  -----     ----------                -----
20020509 Thu 23:50:00 28.89 200 http://www.wikipedia.com/ 18547
20020510 Fri 00:00:00 13.15 200 http://www.wikipedia.com/ 18547
20020510 Fri 00:10:00  9.34 200 http://www.wikipedia.com/ 18547
20020510 Fri 00:20:00 22.59 200 http://www.wikipedia.com/ 18547
20020510 Fri 00:30:00  3.43 200 http://www.wikipedia.com/ 18547
20020510 Fri 00:40:00  1.18 200 http://www.wikipedia.com/ 18441
20020510 Fri 00:50:00  1.18 200 http://www.wikipedia.com/ 18441
20020510 Fri 01:00:00  1.19 200 http://www.wikipedia.com/ 18630
20020510 Fri 01:10:00  1.21 200 http://www.wikipedia.com/ 18630

--
  Lars Aronsson
  <lars at aronsson.se>
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


subject: [Wikitech-l] The Need for Speed

body:
Jimmy Wales wrote:

> I, too, can vouch for FastCGI.  We run tons of stuff through it on
> Bomis.  It's the bomb.  Of course, changing Wikipedia from php/mysql
> to perl fastcgi/mysql would be a major undertaking at this point, and
> reports from around the web suggest strongly that php is fast,
> particularly when persistency is used (as we do).

FastCGI would be a step back for Wikipedia.  PHP is better.
FastCGI was an improvement for me, because I was running old CGI.

Plain old CGI is 1994.
FastCGI is 1996/1998.
PHP3 is 1999.
PHP4 is 2000.
 - or something like that.

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


subject: [Wikipedia-l] Bugs? or Features?

body:
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://lists.wikimedia.org/pipermail/wikipedia-l/attachments/20020510/a9bd3064/attachment.diff


subject: [Wikitech-l] Re: two suggestions

in_reply_to: <200205091509.RAA01890@linde.uni-paderborn.de>; from axel@uni-paderborn.de on Thu, May 09, 2002 at 05:09:52PM +0200

body:
I like Axel's idea.  It is a slight pain to parse the comments to gather
statistics, but the benefits of not having a central bottleneck are enormous.

Axel Boldt wrote:

> > I'm suggesting ways to find such numbers, but I don't
> > have access to update the source code.
>
> I agree that it would be useful to get numbers. Why don't you prepare
> and test your patch and post it here (or get commit access from
> Magnus).
>
> I think it would be best to simply report the times in an HTML comment
> attached to every served file, as you suggested earlier. This way we
> can all perform our own tests from home and no artificial bottleneck
> slowdowns are introduced (which could occur if the time data were logged in
> a central file).
>
> Axel
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at ross.bomis.com
> http://ross.bomis.com/mailman/listinfo/wikitech-l


subject: [Wikitech-l] [tshell@bomis.com: Wikipedia talk link]

body:
I assume this is because I'm returning an empty array from the getOtherNamespaces
function?

----- Forwarded message from tshell at bomis.com -----

From: tshell at bomis.com
Date: Thu, 9 May 2002 18:33:04 -0700 (PDT)
To: jwales <jwales at bomis.com>
Subject: Wikipedia talk link

Do you know what happened to the Talk link on Wikipedia article pages?
Now there is a link to today's date, instead of a link to the Talk page.

----- End forwarded message -----


subject: [Wikipedia-l] Bugs? or Features?

body:
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://lists.wikimedia.org/pipermail/wikipedia-l/attachments/20020510/7aa41d7c/attachment.diff


from_name: Tei

subject: [Wikitech-l] Wikipedia in crisis

body:
temporaly disable searcher?

At 04:02 PM 5/9/02 -0700, you wrote:
>Well, for the past 2 hours, the load average has been hovering around
>100.
>
>So I'm preparing to do some (temporary) drastic measures to change
>things around.  I'm not sure what, though.
>
>--Jimbo
>
>
>_______________________________________________
>Wikitech-l mailing list
>Wikitech-l at ross.bomis.com
>http://ross.bomis.com/mailman/listinfo/wikitech-l
>
>


subject: [Wikipedia-l] Bugs? or Features?

body:
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://lists.wikimedia.org/pipermail/wikipedia-l/attachments/20020510/77327a0a/attachment.diff


subject: [Wikipedia-l] Auto-wikification

body:
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://lists.wikimedia.org/pipermail/wikipedia-l/attachments/20020510/725d83d7/attachment.diff


subject: [Wikipedia-l] Bugs? or Features?

body:
On 5/10/02 8:27 AM, "Karen AKA Kajikit" <kaji at labyrinth.net.au> wrote:

> The coding for the pedia seems to have suddenly gotten extremely buggy
> this week - maybe people should be requested to stop playing and adding
> info until some of these bugs can be fixed? It can't be easy to work on
> the coding when it keeps getting changed...

Speaking of which, can we get rid of the "auto-wikification" button?

It's definitely featuritis: very prominent, highly confusing, and of minimal
utility.

If we want to bring back the functionality, we can discuss a better
interface first, then implement.

Anyone agree?


subject: [Wikipedia-l] Serious script or database bustage...

body:
Something is seriously broken at the moment -- articles are appearing
and disappearing, seemingly at random, without any record in Recent
Changes. I suggest taking a backup now, in case the site collapses.

On the other hand, speed is /excellent/ at the moment, but there's not
much point in contributing, given that
a) any edits might be lost, and
b) making edits may destabilise the system further....

Regards,

Neil


from_name: Axel Boldt

subject: [Wikitech-l] Re: Slow parts

body:
>I suspect the problem may be this query:
>  SELECT cur_title FROM cur WHERE cur_title LIKE "%:$n"

Wow! This is a pig. Linear search (with regular expression) through the
whole database for every single page served :-)

There should not be any LIKE operators in the code whatsoever, since
they cannot use the index. The quick and dirty fix would be to
explicitly search for the page in other namespaces, like:

SELECT cur_title FROM cur
  WHERE cur_title = "wikipedia:$n" OR
        cur_title="user:$n" OR
          ...

this uses the index on cur_title and is quick. A much
cleaner solution would be to have an additional field "namespace" in
the database. Encoding namespaces in the article title is a
kludge.

Axel


subject: [Wikipedia-l] Auto-wikification

body:
>Speaking of which, can we get rid of the "auto-wikification" button?

>It's definitely featuritis: very prominent, highly confusing, and of
>minimal utility.

...and undocumented. I agree, this button should go. At best, it could
be a user preference option (default: off).

Axel


subject: [Wikitech-l] response times

body:
Today Friday, the front page of the English Wikipedia has been fast
all day.

Another page (I monitor http://www.wikipedia.com/wiki/Sweden) was slow
for one period of 30 minutes (09:30-10:00 am GMT) and another period
of two hours (11:40-13:50 GMT).  Some other URLs on the international
Wikipedias were also affected at the same time.  This might be due to
maintenance or work being done on the scripts.

Subtract 7 hours from GMT to get the server's local time zone
(PDT = GMT -0700).

Apart from these two limited intervals, every URL that I monitor have
been fast all day, including the recent changes pages.

I'm very happy with this, and hope Brion and Jimmy (and who else?)
will soon get the talk namespace links back without hurting
performance.  (But hey, never make big fixes five minutes before you
leave for the weekend!  Better just leave it as is if you have to go.)

And now for some more relaxed Friday reading, actually related to
performance problems.  (The following analysis might be politically
slanted.  Don't take it too seriously.) The Swedish parliament
elections are coming up in September, so the political parties are
starting up their campaigns.  The problem is there are no big issues
to fight about.  The four non-socialist parties have unusually boring
candidates (Dukakis style), and everybody expects the current
social-democratic government to win.  The single issue that seems to
be coming up is the national sick leave insurance, which is paid by
tax money, and far over budget.  This is linked to the fact that
"burn-out" is now an accepted medical diagnosis for which you are
allowed to take a long sick leave on the tax payers' expense.  You
would expect such welfare excesses to be on the social democrat
agenda, and that non-socialists would urge for tax cuts and a balanced
budget.  However, the current s-d govt has been doing a great job
balancing the budget, and they will now have to deal with cutting back
this overgenerous sick leave compensation without hurting their
voters' feelings.  Tough job.  The Christian-democratic party's
candidate has already hurt a lot of feelings by claiming that "some"
of those receiving compensation are "cheating the system".  That might
be true, but accusing "some" (who?  me?) is obviously not the way to
attract voters.  This issue now has media attention and some
interesting example cases are reported.

Like this one: Attorneys in Swedish district courts have been
right-sized in the past years, as part of balancing the budget.  This
means that as soon as one gets sick, the rest get too much to do,
leading to stress and burn-out, which leads to more sick leaves.

Think of the court cases as HTTP requests arriving to Wikipedia.
There are some processes/attorneys there to handle the cases, but for
some reason one process gets blocked and cannot work.  This leaves
more work for the remaining workers, but they are probably waiting for
the first process to get finished and unlock the resources (database
records?) that it is using.  If processes are allowed to go to sleep
waiting for each other, the work will pile up.  It will never end.

So, what is the solution?  Throwing more attorneys at the problem?
Maybe, but more likely the work processes should be redesigned and
simplified.  That allows the available attorneys to finish up a case
and take on the next one.  Some of their tasks are more important than
others, but the performance or throughput of the system depends on
cutting away or redesigning the most time-consuming tasks.  The high
degree of sick-leave is an indicator of system design flaws (albeit an
one), and thus not altogether bad.

In the same way, a high "load average" (as reported by the "uptime" or
"top" commands) is one indicator that the Wikipedia system is flawed.
The load average in a UNIX system is the number of processes that are
ready to run, waiting for the CPU to become available.  Unfortunately,
most of them are just waiting to see if their wanted resource has
become available.  If this is not the case (e.g. database record still
locked), they will go back to the end of the line, waiting again.  Do
you remember those bread shop waiting lines in Soviet Russia?

Training new attorneys is in itself a time-consuming task, which
should be avoided if possible.  Instead of paying sick leave (for how
long?) to the already trained attorneys, a "cure" for "burn-out"
should be found that can bring them back to work, thus relieving the
overload from their colleagues and saving tax payers' money at the
same time.

I have no idea how a "cure" for burn-out can be found, but I think it
is a necessary political trick, and thus will happen.  It will not
hurt voters' feelings, and it is my guess that the people who can
achieve this will work for the winners of the election.

This might be the weakest analogy in history, but I think we should
treat the Wikipedia processes with the same dignity and respect that
the Swedish voters would expect.  After all, they're supposed to work
for us.  The processes feel self-fulfillment when they can finish
their job on time, and get distressed when they get locked up.  Any
uncalled for delay will only result in more work piling up.  That is a
flaw in the system design that has to be fixed, and we cannot go
around claiming that "some" of the workers are trying to cheat the
system.  That will only lead to us losing their confidence.

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


subject: [Wikipedia-l] Bugs? or Features?

body:
Another grrr...

We seem to have lost the 'talk' page links from the pedia entirely, but
I'm sure everyone's noticed it by now!

And I noticed some requests on the recent changes pages for people NOT
to make 'fill in text here' the contents of a page... I just did it by
accident and then I couldn't get the page to load to re-edit it.
Apparantly when you create a new page by redirecting it creates the
page, but instead of sending you to the edit screen it dumped me into
the pageview.

The coding for the pedia seems to have suddenly gotten extremely buggy
this week - maybe people should be requested to stop playing and adding
info until some of these bugs can be fixed? It can't be easy to work on
the coding when it keeps getting changed...

--

Karen AKA Kajikit

Come and visit my part of the web:
Kajikit's Corner: http://Kajikit.netfirms.com/
Aussie Support Mailing List: http://groups.yahoo.com/group/AussieSupport
Allergyfree Eating Recipe Swap:
http://groups.yahoo.com/group/Allergyfree_Eating

Love and huggles to all!


from_name: Robert Graham Merkel

subject: [Wikipedia-l] Main page broken

body:
I don't know how, I don't know why, but the main page is
giving me "describe the new page here" at the moment.

Doesn't appear to be in the Recent Changes, so I assume
there's some other breakage gone on.

--
------------------------------------------------------------
Robert Merkel	                           rgmerk at mira.net

Go You Big Red Fire Engine
-- Unknown Audience Member at Adam Hills standup gig
------------------------------------------------------------


from_name: Brion L. VIBBER

subject: [Wikitech-l] blank Main Page

body:
On sab, 2002-05-11 at 08:38, Lars Aronsson wrote:
> Why is the Main Page sometimes blank?
>
> This has been the case on Friday 5:00 am - 6:50 am PDT
> and on Saturday 6:20 am - 8:30 am PDT.
> Then at 8:36 am PDT it is back in full!
> (But right now the Sweden page is blank instead!)
>
> The server has also been unavailable for some periods.  Is there a
> natural explanation to this, or does it need to be investigated?

No idea. Maybe it's the same reason that THERE IS A TABLE MISSING FROM
THE DATABASE? The "unlinked" table is still inaccessable... "Can't open
file: 'unlinked.MYD'. (errno: 145)"

I most sincerely hope that there's a backup of the database more recent
than March 21.

-- brion vibber (brion @ pobox.com)


from_name: Magnus Manske

subject: [Wikitech-l] Fixes

body:
As promised, I took some time today to engage in bugfixing.

* Talk page links are back, and fast
* The auto-wikification button is now a user option, turned off by default

I also went through the code, looking for SQL queries containing "LIKE", and
deactivated one more. On my local machine, response times are now less than
half of what they were on Thursday morning, before the "Need for Speed";)

The only working queries containing "LIKE" are now in the statistics page,
the orphans page, the main page (through the variable that counts the
articles), and the search page (though I don't remember if these queries are
deactivated; I think so).

Jimbo, please install the current version to get the 'pedia back to full
power. And you others, don't get a speeding ticket when browsing this
version ;)

Magnus


from_name: wojtek pobratyn

subject: [Wikipedia-l] Bugs? or Features?

body:
----- Original Message -----
From: <koyaanisqatsi at nupedia.com>
To: <wikipedia-l at nupedia.com>
Sent: Friday, May 10, 2002 4:40 PM
Subject: Re: [Wikipedia-l] Bugs? or Features?

| We need an upload space where people who care & have the time can test new features on an active (but unpublicized) site before
the features go live.  |Wikipedia is crippled at the moment, and has been almost every time I've visited it in the last several
months.  When I go to a site on my broadband line at work |and it takes 5 minutes to bring up a page, that is IMO a crippled site.
We can't expect the casual user to have any patience for it, and also can't expect new users |to want to come back.

same here, the 'pedia is dead. Funnily enough when I try to ping the server I get an 80% packet loss.

regards,
WojPob

| kq
|
|
| You Wrote:
| >Another grrr...
| >
| >We seem to have lost the 'talk' page links from the pedia entirely, but
| >I'm sure everyone's noticed it by now!
| >
| >And I noticed some requests on the recent changes pages for people NOT
| >to make 'fill in text here' the contents of a page... I just did it by
| >accident and then I couldn't get the page to load to re-edit it.
| >Apparantly when you create a new page by redirecting it creates the
| >page, but instead of sending you to the edit screen it dumped me into
| >the pageview.
| >
| >The coding for the pedia seems to have suddenly gotten extremely buggy
| >this week - maybe people should be requested to stop playing and adding
| >info until some of these bugs can be fixed? It can't be easy to work on
| >the coding when it keeps getting changed...
| >
| >--
| >
| >Karen AKA Kajikit
| >
| >Come and visit my part of the web:
| >Kajikit's Corner: http://Kajikit.netfirms.com/
| >Aussie Support Mailing List: http://groups.yahoo.com/group/AussieSupport
| >Allergyfree Eating Recipe Swap:
| >http://groups.yahoo.com/group/Allergyfree_Eating
| >
| >Love and huggles to all!
| >[Wikipedia-l]
| >To manage your subscription to this list, please go here:
| >http://www.nupedia.com/mailman/listinfo/wikipedia-l
| >0


from_name: Lars Aronsson

subject: [Wikipedia-l] Bugs? or Features?

body:
wojtek pobratyn wrote (on wikipedia-l):
> same here, the 'pedia is dead. Funnily enough when I try to ping the
> server I get an 80% packet loss.

The technical discussion is taking place on wikitech-l at nupedia.com.
This is a short summary and unofficial status report.

In the last few weeks, the Wikipedia webserver has been very slow,
depending on new functions being added without enough concern about
their performance under high load.

At 5 pm on Thursday May 9, Jimmy disabled some of the functions that
consumed too much time, and there was an instant improvement in
response time.  The removed functionality include the "talk" links.
The implementation is studied right now, and an improved version will
probably be installed in the next few days.  At this point, I don't
think more programmers are needed, but if you want to stay informed,
you can join the wikitech-l to follow the reports and discussions.

It is good to see that so many are concerned to have Wikipedia back in
full operation, with all the necessary functionality.  However, it
does surprise me that so few were complaining about the absurdly slow
response times in the week before.

There are still periods when the server seems totally unavailable, and
the last hour seems to be one such case.  In the preceeding 24 hours,
the response time for the front page ranged from 0.95 to 12.31 seconds,
with an average of 2.18 seconds.  This could still be improved, but it
is actually very fast for Wikipedia.

The problems are with the English Wikipedia, which receives most
of the traffic.  The international versions run on the same server,
and are slowed down by the poor overall performance of the system.

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


subject: [Wikitech-l] Re: [Wikipedia-l] Bugs? or Features?


subject: [Wikitech-l] Weird screw-up of diffs

body:
On sab, 2002-05-11 at 15:03, Axel Boldt wrote:
> Have a look at the history of [[Pim Fortuyn]]
>
>  http://www.wikipedia.com/wiki/Pim_Fortuyn&action=history
>
> All versions are properly listed and stored correctly as far as I can tell.
> However, starting with the version of Tuesday May 7, 8:59, the diffs
> are screwed up. They do not compare to the next version, but to very
> old versions.
>
> I noticed that on the Talk page, Tim Shell explained one of his
> edits to the main article early Tuesday morning, and I remember seeing
> that edit (taking out the label "right wing"). This change however
> does not show up on the main page's history page.
>
> Does anybody have an idea what is going on here?

Short answer: there seems to be a break in the linked list of stored
edits.

The history list mostly ignores the linking and just orders by
timestamp, but when doing a diff on the most recent version, the
cur_old_version field is used to select which entry from the old table
to compare from. If that field is pointing to the wrong place, it'll
show the wrong version.

(The $64,000 question is: why is it wrong? I don't know offhand, but I'd
check the code for saving an article. As I recall, the bit that checks
which old_version ID to insert seemed a bit fragile last I looked at
it.)

Here're the item IDs for Pim_Fortuyn; old_id is a unique identifier, and
(cur|old)_old_version is supposed to reference the id of the previous
edit:

old_timestamp	old_id	old_old_version
20020324153744	36848	0
20020324160320	37456	36848
20020325143818	38206	37456
20020326133630	38299	38206
20020326153001	38302	38299
20020326154453	38305	38302
20020326154558	38307	38305
20020326155234	48454	38307
20020408152422	63034	48454
20020506110024	63063	63034
20020506130808	63123	63063
20020506162701	63126	63123
20020506164202	63129	63126
20020506164649	63130	63129
20020506165926	63284	63130 ^- Good
20020507065958	63367	36848 v- Bad
20020507104949	63372	37456
20020507114506	63426	38206
20020507154732	63428	38299
20020507155020	63429	38302
20020507155211	63431	38305
20020507161527	63453	38307
20020507174244	63519	48454
20020507201257	63775	63034
20020508092908	64614	63063
20020510210808	64721	63123
20020511045521	64847	63126
20020511140710	64855	63129

cur_timestamp		cur_old_version
20020511143100		63130

Things seem to have gone awry in the case of this article the morning of
May 7.

> Are similar things
> happening on other pages?

I wouldn't doubt it.

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] blank Main Page

body:
Why is the Main Page sometimes blank?

This has been the case on Friday 5:00 am - 6:50 am PDT
and on Saturday 6:20 am - 8:30 am PDT.
Then at 8:36 am PDT it is back in full!
(But right now the Sweden page is blank instead!)

The server has also been unavailable for some periods.  Is there a
natural explanation to this, or does it need to be investigated?

--
  Lars Aronsson
  <lars at aronsson.se>
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


subject: [Wikipedia-l] Bugs? or Features?

body:
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://lists.wikimedia.org/pipermail/wikipedia-l/attachments/20020511/efcf853b/attachment.diff


subject: [Wikitech-l] Weird screw-up of diffs

body:
Have a look at the history of [[Pim Fortuyn]]

 http://www.wikipedia.com/wiki/Pim_Fortuyn&action=history

All versions are properly listed and stored correctly as far as I can tell.
However, starting with the version of Tuesday May 7, 8:59, the diffs
are screwed up. They do not compare to the next version, but to very
old versions.

I noticed that on the Talk page, Tim Shell explained one of his
edits to the main article early Tuesday morning, and I remember seeing
that edit (taking out the label "right wing"). This change however
does not show up on the main page's history page.

Does anybody have an idea what is going on here? Are similar things
happening on other pages?

Axel


subject: [Wikitech-l] Database extension

body:
On dim, 2002-05-12 at 02:29, Magnus Manske wrote:
> I suggest to add two new fields to the database (cur table)
> * cur_namespace (varchar) so articles can be distinguished by namespace
> without using the "LIKE" MySQL query (this was mentioned before)

Should there be a separate title-without-namespace field as well, or
would cur_title now be that?

We want to be able to group things by namespace (ie, count the talk
pages) but also group things by title ignoring namespace (ie, watchlist
includes an article and any talk/whatever pages).

> * cur_is_redirect (boolean) so #REDIRECT pages can be excluded likewise

Not a bad idea! Might it be even easier to make it a string field which
contains the name of the page being redirected to? This would save
parsing the #REDIRECT [[x]] line or relying on the link tables when
doing mass checks in, eg, "links to this page".

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] Database extension

body:
On dim, 2002-05-12 at 08:17, Magnus Manske wrote:
> > > I suggest to add two new fields to the database (cur table)
> > > * cur_namespace (varchar) so articles can be distinguished by namespace
> > > without using the "LIKE" MySQL query (this was mentioned before)
> >
> > Should there be a separate title-without-namespace field as well, or
> > would cur_title now be that?
>
> I think a namespace field would do fine for now, as most of the slow queries
> look for the namespace.

Well, the statistics checks do I suppose, but the watchlist and
list-all-namespaces functions check for everything with the same
non-namespace portion. No?

> > > * cur_is_redirect (boolean) so #REDIRECT pages can be excluded likewise
> >
> > Not a bad idea! Might it be even easier to make it a string field which
> > contains the name of the page being redirected to? This would save
> > parsing the #REDIRECT [[x]] line or relying on the link tables when
> > doing mass checks in, eg, "links to this page".
>
> I agree. A boolean field might be faster, but not by much.
>
>
> Concerning the {{NUMBEROFARTICLES}} function, we should implement a
> cur_has_commas (boolean) field while we're at it. It seems to be the best
> way of telling articles apart from stubs. Or, we should name it
> cur_is_article, in case we come up with a better method some day.

Or, we could just update the value once a day. :)

Actually, that reminds me: should we add a timestamp field to indicate
when cur_cache was last filled? A simple check of that against
cur_timestamp would have prevented our present cache-of-ancient-
version-is-displayed-instead-of-recent-one troubles.

It would then be possible to expire caches on certain pages, such as
those containing {{variables}}.

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] blank Main Page

body:
On dim, 2002-05-12 at 08:32, Jason Richey wrote:
> Brion L. VIBBER wrote:
> > No idea. Maybe it's the same reason that THERE IS A TABLE MISSING FROM
> > THE DATABASE? The "unlinked" table is still inaccessable... "Can't open
> > file: 'unlinked.MYD'. (errno: 145)"
> >
> > I most sincerely hope that there's a backup of the database more recent
> > than March 21.
>
> The file is, in fact, still there.  I don't know how much that means
> if it is unreadable...

I don't know, but it still doesn't work. :(

The error message on every page load seems to have been repressed for
now, but:

* Explicit SQL queries to the unlinked table give the already mentioned
error, "Can't open file: 'unlinked.MYD'. (errno: 145)".

* special:WantedPages comes up with nothing, and notes PHP errors on the
lines that deal with the query to the unlinked table.

* Doing a "Pages that links here" gives PHP errors on the lines that
deal with the request to the unlinked table, and thus no longer can give
results for pages that don't exist yet.

* Saving a new page may also have some problems in this vein, since it
should be checking the unlinked table for pages that linked to the newly
created page and thus need their caches cleared to mark the link as
existing. (in wikiPage::ensureExistence())

-- brion vibber (brion @ pobox.com)

(PS: I've noticed that replies to my messages lately seem to be coming
directly to me rather than to the list. Do I have a freak Reply-To
header that I'm not aware of or something?)


subject: [Wikitech-l] Weird screw-up of diffs

body:
Even with the code in perfect shape (yeah, sure;) I doubt the system
(especially the database) would stay in flawless order for an unlimited time.
So, as suggested before, there should be a "maintnance" script that takes
care of fixing bugs that are known to occur (think of the linked/unlinked
entries). It could be run once a week, or on special invocation of a sysop,
if needed.

Magnus

Am Sonntag, 12. Mai 2002 01:57 schrieben Sie:
> On sab, 2002-05-11 at 15:03, Axel Boldt wrote:
> > Have a look at the history of [[Pim Fortuyn]]
> >
> >  http://www.wikipedia.com/wiki/Pim_Fortuyn&action=history
> >
> > All versions are properly listed and stored correctly as far as I can
> > tell. However, starting with the version of Tuesday May 7, 8:59, the
> > diffs are screwed up. They do not compare to the next version, but to
> > very old versions.
> >
> > I noticed that on the Talk page, Tim Shell explained one of his
> > edits to the main article early Tuesday morning, and I remember seeing
> > that edit (taking out the label "right wing"). This change however
> > does not show up on the main page's history page.
> >
> > Does anybody have an idea what is going on here?
>
> Short answer: there seems to be a break in the linked list of stored
> edits.
>
> The history list mostly ignores the linking and just orders by
> timestamp, but when doing a diff on the most recent version, the
> cur_old_version field is used to select which entry from the old table
> to compare from. If that field is pointing to the wrong place, it'll
> show the wrong version.
>
> (The $64,000 question is: why is it wrong? I don't know offhand, but I'd
> check the code for saving an article. As I recall, the bit that checks
> which old_version ID to insert seemed a bit fragile last I looked at
> it.)
>
> Here're the item IDs for Pim_Fortuyn; old_id is a unique identifier, and
> (cur|old)_old_version is supposed to reference the id of the previous
> edit:
>
> old_timestamp	old_id	old_old_version
> 20020324153744	36848	0
> 20020324160320	37456	36848
> 20020325143818	38206	37456
> 20020326133630	38299	38206
> 20020326153001	38302	38299
> 20020326154453	38305	38302
> 20020326154558	38307	38305
> 20020326155234	48454	38307
> 20020408152422	63034	48454
> 20020506110024	63063	63034
> 20020506130808	63123	63063
> 20020506162701	63126	63123
> 20020506164202	63129	63126
> 20020506164649	63130	63129
> 20020506165926	63284	63130 ^- Good
> 20020507065958	63367	36848 v- Bad
> 20020507104949	63372	37456
> 20020507114506	63426	38206
> 20020507154732	63428	38299
> 20020507155020	63429	38302
> 20020507155211	63431	38305
> 20020507161527	63453	38307
> 20020507174244	63519	48454
> 20020507201257	63775	63034
> 20020508092908	64614	63063
> 20020510210808	64721	63123
> 20020511045521	64847	63126
> 20020511140710	64855	63129
>
> cur_timestamp		cur_old_version
> 20020511143100		63130
>
> Things seem to have gone awry in the case of this article the morning of
> May 7.
>
> > Are similar things
> > happening on other pages?
>
> I wouldn't doubt it.
>
> -- brion vibber (brion @ pobox.com)
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at ross.bomis.com
> http://ross.bomis.com/mailman/listinfo/wikitech-l


from_name: Larry Sanger

subject: [Wikipedia-l] What happened to...?

body:
I know wikitech-l is probably working on this, but I want to express my
dismay at the fact that the most recently saved version of my user page
(and lots of other page) is something from back in February.  That's kind
of scary.

Larry


subject: [Wikitech-l] What happened to...?


from_name: Jimmy Wales

subject: [Wikitech-l] Speedup

in_reply_to: <MLEPJFEEOFMFHGIHGJPIOENMCAAA.magnus.manske@epost.de>; from magnus.manske@epost.de on Sun, May 12, 2002 at 04:45:45PM +0200

body:
Magnus Manske wrote:
> The real time killer is the {{NUMBEROFARTICLES}} variable on the main page.
> Its use increases the time per page by a factor of about 10! Only a change
> of the database can help here.

Of course, this is a fairly useless feature anyway.  I think it should
be removed!  Speed is far more important than cute features.  We could
have a cron job calculate the number of articles every night, or every
week, and post it on a plain old text file publicly accessible, and
people who want to do it could go there to look up the latest number
and update the homepage from time to time.

> Another thing that takes a lot of time is the headed or the Recent Changes
> page. I guess since all the links there have to be checked every time the
> page is displayed, it increases the load on the database quite a bit. On my
> (short) local Recent Changes page, the header text alone is responsible for
> about 20% of the total rendering time. We should probably force the Recent
> Changes page *not* to look for existing articles in the header text.

If a page appears on RecentChanges, doesn't that automatically imply
that the page exists?  So why do we need to check all the links there?
If we in some rare cases show a page there as if it exists, when it
does not, what's the harm?

If I don't hear about any showstoppers from the other developers, I'll upgrade
to this version later tonight or tomorrow morning.

--Jimbo


subject: [Wikitech-l] Weird screw-up of diffs

in_reply_to: <176pDw-1UWPFwC@fwd03.sul.t-online.com>; from Magnus.Manske@t-online.de on Sun, May 12, 2002 at 10:19:17AM +0200

body:
I will quickly and urgently run a command line script to recreate the missing/broken
tables, if someone sends it to me.

Wouldn't it be essentially the same as the script I ran to create those tables in the
first place?

--Jimbo


from_name: Jason Richey

subject: [Wikipedia-l] Re: [Wikitech-l] What happened to...?

body:
I assume this is because we recently turned on caching for
performance's sake...

Jason

Larry Sanger wrote:

> I know wikitech-l is probably working on this, but I want to express my
> dismay at the fact that the most recently saved version of my user page
> (and lots of other page) is something from back in February.  That's kind
> of scary.
>
> Larry
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at ross.bomis.com
> http://ross.bomis.com/mailman/listinfo/wikitech-l

--
"Jason C. Richey" <jasonr at bomis.com>


subject: [Wikitech-l] What happened to...?


subject: [Wikitech-l] Database extension

body:
I suggest to add two new fields to the database (cur table)
* cur_namespace (varchar) so articles can be distinguished by namespace
without using the "LIKE" MySQL query (this was mentioned before)
* cur_is_redirect (boolean) so #REDIRECT pages can be excluded likewise

As you can see, I now try to discuss things instead of just doing them ;)

Magnus