body:
On Sat, Nov 16, 2002 at 01:28:31AM -0800, Brion Vibber wrote:
>> 5) Why is a broken link table necessary?  Couldn't it be sufficient to
>> do a query to find out which links are broken?  Or is this another "we
>> have no subqueries" issue?
>
>I believe the primary use of the brokenlinks table is for generating the
>'most wanted' list.
>
>All articles have ID numbers; that's cur_id in the cur table. bl_from
>and l_to reference cur_id numbers. Articles that don't exist don't have
>cur_id numbers of course, so cannot be referenced by id, but only by
>title.

What if there were no "broken links", only articles that had no text?
ie, the text field would be NULL?  And we could do with a single "links"
table by collapsing article and image links together.  And one could
find the "dangling links" simply by searching for those articles with
the text property set to NULL.  The following query would do the trick:

    SELECT * from cur where cur_text = NULL;

>The current solution is more or less like that for the articles tables.
>There's 'image' ~= 'cur' and 'oldimage' ~= 'old'. Uploading a file with
>the same name as an existing one is considered creating a new revision
>of the same file; the old one is pushed back into the old list, and the
>new one put in the current place.

Hm. Maybe images could be collapsed into the "cur" table too.

>The primary use of the links table is to generate the 'Whatlinkshere'
>listing. No subqueries, and we need the titles to list the links, not
>the id numbers.

I think we could reduce to the following tables:

current_articles, previous_articles, deleted_articles, links

We just add a "type" field to the articles tables to indicate "wiki
text" or "image"; this gives us flexibility to support other types of
object too.

>Stores ~1000 randomly selected page names. When someone clicks 'Random
>page', one of those is selected rather than one of the whole bunch;
>every once in a while the list is regenerated. Allegedly this improves
>performance.

That makes sense, and it very well may improve performance.

>> 18) What is ss_row_id and ss_good_articles?  How does this table work?
>> Does it contain stats for each article?  Shouldn't that go in the table
>> that already contains the info for an article?  That would lessen the
>> number of SQL updates, collapsing two UPDATE statements into one.
>
>This contains a single row for the whole database. The good articles
>count is generated by faulty math anyway, though.

What is a "good article"?  What uses that table?  What would happen if
it went away?

>No. user_rights is a comma-separated list of priveledges. Currently the
>only ones defined are 'sysop' and 'developer', so possible values are:
>  '' (normal users)
>  'sysop' ("sysops" or "administrators")
>  'sysop,developer' ("developers")

Ah. Oddly enough, I would recommend splitting that out into two tables,
each containing just user ids; one called "developers", and the other
"sysops".

The newpassword feature sounds nifty. I like it.

Jonathan

--
                     Geek House Productions, Ltd.

  Providing Unix & Internet Contracting and Consulting,
  QA Testing, Technical Documentation, Systems Design & Implementation,
  General Programming, E-commerce, Web & Mail Services since 1998

Phone:   604-435-1205
Email:   djw at reactor-core.org
Webpage: http://reactor-core.org
Address: 2459 E 41st Ave, Vancouver, BC  V5R2W2
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 307 bytes
Desc: not available
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20021116/14d9b0c7/attachment.pgp>


list: wikitech-l

source: pipermail-mbox

from_name: Brion Vibber

subject: [Wikitech-l] Image archive download?

body:
On Fri, 2002-11-15 at 03:19, Erik Moeller wrote:
> is there a way to download the image directory of the en.wikipedia?
> There are the database tarballs, but those do not include the images.

Here ya go:
http://www.wikipedia.org/tarballs/image.tar.bz2
circa 125MB

This does *not* include old revisions of images that have been
re-uploaded, or the image/oldimage table entries in the db.

-- brion vibber (brion @ pobox.com)


from_name: Jonathan Walther

subject: [Wikitech-l] questions about the SQL structure

body:
On Sat, Nov 16, 2002 at 11:22:47AM +0100, Magnus Manske wrote:
>No, "broken links" are links to articles that don't exist at all, not to
>"blank" articles.

Correct. I am proposing that there should be no broken links because all
articles are created, even if only blank articles.  Periodically a
process could sweep blank articles with nothing linking to them from the
database.  For instance,

REMOVE FROM current_articles WHERE ...

A single SQL statement could do it; Ed, can you back me up on this?  The
SQL statement would do a complement of the union of article id's in the
link table and the current_articles table to find articles not linked,
and then further narrow it down to articles with NULL text properties.

>Then we could drop deleted_articles as well, just set some "cur_deleted"
>tag in "current_articles".
>But, you'll be the one to change every line of source code for that, I
>guess...

Tonight, I got a vision dancing through my head of an Apache module,
called mod_wiki.  I'm investigating it's possibilities.

Jonathan

--
                     Geek House Productions, Ltd.

  Providing Unix & Internet Contracting and Consulting,
  QA Testing, Technical Documentation, Systems Design & Implementation,
  General Programming, E-commerce, Web & Mail Services since 1998

Phone:   604-435-1205
Email:   djw at reactor-core.org
Webpage: http://reactor-core.org
Address: 2459 E 41st Ave, Vancouver, BC  V5R2W2
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 307 bytes
Desc: not available
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20021116/b11413b7/attachment.pgp>


body:
On Sat, Nov 16, 2002 at 02:35:01AM -0800, Jonathan Walther wrote:
>On Sat, Nov 16, 2002 at 11:22:47AM +0100, Magnus Manske wrote:
>>No, "broken links" are links to articles that don't exist at all, not to
>>"blank" articles.
>
>Correct. I am proposing that there should be no broken links because all
>articles are created, even if only blank articles.  Periodically a
>process could sweep blank articles with nothing linking to them from the
>database.  For instance,
>
>REMOVE FROM current_articles WHERE ...

Ok, here is what the SQL query would look like:

REMOVE FROM current_articles
WHERE
  text=NULL
  AND NOT
    article_id IN
      ( SELECT destination FROM links DISTINCT );

You see?  No need for a separate table of dangling links, if you make sure
that every article comes into existance.  Subqueries are really nice.

Cheers!

<Location /wiki>
  SetHandler mod-wikipedia
</Location>

Jonathan

--
                     Geek House Productions, Ltd.

  Providing Unix & Internet Contracting and Consulting,
  QA Testing, Technical Documentation, Systems Design & Implementation,
  General Programming, E-commerce, Web & Mail Services since 1998

Phone:   604-435-1205
Email:   djw at reactor-core.org
Webpage: http://reactor-core.org
Address: 2459 E 41st Ave, Vancouver, BC  V5R2W2
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 307 bytes
Desc: not available
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20021116/4c878a28/attachment.pgp>


list: wikipedia-l

from_name: Anthere

subject: [Wikipedia-l] Vandalisme en

body:
Some help against vandals is - I believe - needed on
the en.wiki

__________________________________________________
Do you Yahoo!?
Yahoo! Web Hosting - Let the expert host your site
http://webhosting.yahoo.com


subject: [Wikitech-l] Vandalisme en


from_name: The Cunctator

body:
On 11/16/02 4:44 AM, "Jonathan Walther" <krooger at debian.org> wrote:

> On Sat, Nov 16, 2002 at 01:28:31AM -0800, Brion Vibber wrote:
>>> 5) Why is a broken link table necessary?  Couldn't it be sufficient to
>>> do a query to find out which links are broken?  Or is this another "we
>>> have no subqueries" issue?
>>
>> I believe the primary use of the brokenlinks table is for generating the
>> 'most wanted' list.
>>
>> All articles have ID numbers; that's cur_id in the cur table. bl_from
>> and l_to reference cur_id numbers. Articles that don't exist don't have
>> cur_id numbers of course, so cannot be referenced by id, but only by
>> title.
>
> What if there were no "broken links", only articles that had no text?
> ie, the text field would be NULL?  And we could do with a single "links"
> table by collapsing article and image links together.  And one could
> find the "dangling links" simply by searching for those articles with
> the text property set to NULL.  The following query would do the trick:

We do distinguish between articles that have been never created and articles
that are currently blank, and we need to.

Not that this is necessarily a bad idea, but I'd want to be very clear
if/how this would alter current functionality.


from_name: Pierre Abbat

subject: [Wikipedia-l] Uploaded cracking attempt

body:
LearII uploaded the following, which appears to be an attempt to read from
one port on one box and write to a port on another box. He also uploaded
another PHP file, which was an attempt to list the files on the server. PHP
doesn't run in the upload directory, and he left out the semicolons, but
would someone please block him?

phma
---
<html>
<body>
<?php

?arg1=a&arg2=b&arg3=c&arg4=d

$fp2 = fsockopen($arg1, $arg2)
$fp1 = fsockopen($arg3, $arg4)
set_socket_blocking($fp1, false)
set_socket_blocking($fp2, false)
while (1) {

	$recvbuf = fgets( $fp1, 512)
	fwrite($fp2,$recvbuf,strlen($recvbuf))

	$recvbuf2 = fgets( $fp2, 512)
	fwrite($fp1,$recvbuf2,strlen($recvbuf2))

}

fclose($fp1);
fclose($fp2);

}

?>

</body>
</html>


from_name: Magnus Manske

body:
Jonathan Walther wrote:

> CREATE TABLE archive (
>  ar_namespace tinyint(2) unsigned NOT NULL default '0',
>  ar_title varchar(255) binary NOT NULL default '',
>  ar_text mediumtext NOT NULL,
>  ar_comment tinyblob NOT NULL,
>  ar_user int(5) unsigned NOT NULL default '0',
>  ar_user_text varchar(255) binary NOT NULL default '',
>  ar_timestamp varchar(14) binary NOT NULL default '',
>  ar_minor_edit tinyint(1) NOT NULL default '0',
>  ar_flags tinyblob NOT NULL
> ) TYPE=MyISAM PACK_KEYS=1;
>
> 1) What is the "archive" table for? Is that where articles are stored?

That's where "deleted" articles are stored, so they can be "undeleted".
That table will have to be flushed from time to time in the future.
The current articles are in the "cur" table, the old revisions in "old".

> 2) Why is there a separate ar_user_text property?  Is it because MySQL
> doesn't support subqueries?
>
> 3) What is ar_flags used for?  And why is it a tinyblob?

I'm not sure myself about these, as I didn't really work with that table
before. "tinyblob" is the same as "tinytext", but with case-sensitive
search (basically).

> 4) Could we save a lot of disk space by replacing ar_title with
> article_id, and having all the article titles and their corresponding
> ids in a separate table?

No.

> CREATE TABLE brokenlinks (
>  bl_from int(8) unsigned NOT NULL default '0',
>  bl_to varchar(255) binary NOT NULL default '',
>  KEY bl_from (bl_from),
>  KEY bl_to (bl_to(10))
> ) TYPE=MyISAM;
>
> 5) Why is a broken link table necessary?  Couldn't it be sufficient to
> do a query to find out which links are broken?  Or is this another "we
> have no subqueries" issue?

It is very useful for the "most wanted" page, among other things.

> 6) I notice bl_from is an integer, while bl_to is a string; why the
> difference?  I thought articles didn't have identifying id numbers; if
> so, what is bl_from then?

Say, article A links to non-existing article B. bl_from is the cur_id of
A, and bl_to is the name of the non-existent article B.

> CREATE TABLE cur (
>  cur_id int(8) unsigned NOT NULL auto_increment,
>  cur_namespace tinyint(2) unsigned NOT NULL default '0',
>  cur_title varchar(255) binary NOT NULL default '',
>  cur_text mediumtext NOT NULL,
>  cur_comment tinyblob NOT NULL,
>  cur_user int(5) unsigned NOT NULL default '0',
>  cur_user_text varchar(255) binary NOT NULL default '',
>  cur_timestamp varchar(14) binary NOT NULL default '',
>  cur_restrictions tinyblob NOT NULL,
>  cur_counter bigint(20) unsigned NOT NULL default '0',
>  cur_ind_title varchar(255) NOT NULL default '',
>  cur_ind_text mediumtext NOT NULL,
>  cur_is_redirect tinyint(1) unsigned NOT NULL default '0',
>  cur_minor_edit tinyint(1) unsigned NOT NULL default '0',
>  cur_is_new tinyint(1) unsigned NOT NULL default '0',
>  UNIQUE KEY cur_id (cur_id),
>  KEY cur_namespace (cur_namespace),
>  KEY cur_title (cur_title(20)),
>  KEY cur_timestamp (cur_timestamp),
>  FULLTEXT KEY cur_ind_title (cur_ind_title),
>  FULLTEXT KEY cur_ind_text (cur_ind_text)
> ) TYPE=MyISAM PACK_KEYS=1;
>
> 7) What is the table cur for?  I find myself understanding nothing about
> it; I even looked at the 3 default entries that are inserted, and they
> didn't clue me in.

Current article versions.

> CREATE TABLE image (
>  img_name varchar(255) binary NOT NULL default '',
>  img_size int(8) unsigned NOT NULL default '0',
>  img_description tinyblob NOT NULL,
>  img_user int(5) unsigned NOT NULL default '0',
>  img_user_text varchar(255) binary NOT NULL default '',
>  img_timestamp varchar(14) binary NOT NULL default '',
>  KEY img_name (img_name(10)),
>  KEY img_size (img_size),
>  KEY img_timestamp (img_timestamp)
> ) TYPE=MyISAM PACK_KEYS=1;
>
> 8) Is img_user_text present for the "MySQL has no subquery
> functionality" reason?

That table is Lee's invention. img_user_text contains thename of the
user who uploaded it.

> 9) What format are timestamps in?

UNIX, I think.

> 10) Is there special PHP code that ensures an image with the same name,
> but differing descriptions or img_user properties are present in the
> table?  Does MySQL not have the "UNIQUE" keyword which could eliminate
> all need for that code, or is it wanted to have a record of all changes
> to the description of an image?  If so, I would recommend a slightly
> different solution, similar to that for the articles table.

I'm not sure I understand that question. If there's an image "A", and
you upload a new image "A", you'll do the same as saving a new version
of an article.

> CREATE TABLE imagelinks (
>  il_from varchar(255) binary NOT NULL default '',
>  il_to varchar(255) binary NOT NULL default '',
>  KEY il_from (il_from(10)),
>  KEY il_to (il_to(10))
> ) TYPE=MyISAM;
>
> 11) Surely we could save space by using integer article ids?

I guess so. But, the totally saved space would probably be less than a
single "old" revision of a long article. Not worth the bother, I'd say.

> CREATE TABLE ipblocks (
>  ipb_address varchar(40) binary NOT NULL default '',
>  ipb_user int(8) unsigned NOT NULL default '0',
>  ipb_by int(8) unsigned NOT NULL default '0',
>  ipb_reason tinyblob NOT NULL,
>  ipb_timestamp varchar(14) binary NOT NULL default '',
>  KEY ipb_address (ipb_address),
>  KEY ipb_user (ipb_user)
> ) TYPE=MyISAM PACK_KEYS=1;
>
> 12) No questions, but why is the KEY keyword used? Is this a MySQL
> trigger for making indexes?  Also, theres a bunch of things that could
> use the UNIQUE keyword to save php code.

You're probably right on both.

> CREATE TABLE links (
>  l_from varchar(255) binary NOT NULL default '',
>  l_to int(8) unsigned NOT NULL default '0',
>  KEY l_from (l_from(10)),
>  KEY l_to (l_to)
> ) TYPE=MyISAM;
>
> 13) Similar to the question about the orphanedlinks table, why is l_from
> a string, while l_to is an integer?

I think l_from could be an integer as well. Would probably increase
lookup speed, and save memory. Some things might get more complicated,
though, like "what links here".

> CREATE TABLE old (
>  old_id int(8) unsigned NOT NULL auto_increment,
>  old_namespace tinyint(2) unsigned NOT NULL default '0',
>  old_title varchar(255) binary NOT NULL default '',
>  old_text mediumtext NOT NULL,
>  old_comment tinyblob NOT NULL,
>  old_user int(5) unsigned NOT NULL default '0',
>  old_user_text varchar(255) binary NOT NULL default '',
>  old_timestamp varchar(14) binary NOT NULL default '',
>  old_minor_edit tinyint(1) NOT NULL default '0',
>  old_flags tinyblob NOT NULL,
>  UNIQUE KEY old_id (old_id),
>  KEY old_title (old_title(20)),
>  KEY old_timestamp (old_timestamp)
> ) TYPE=MyISAM PACK_KEYS=1;
>
> 14) What is table "old" used for? Deleted articles?  Versions of
> articles that are no longer "the most current"?

The latter. Deleted articles go into "archive".

> CREATE TABLE oldimage (
>  oi_name varchar(255) binary NOT NULL default '',
>  oi_archive_name varchar(255) binary NOT NULL default '',
>  oi_size int(8) unsigned NOT NULL default '0',
>  oi_description tinyblob NOT NULL,
>  oi_user int(5) unsigned NOT NULL default '0',
>  oi_user_text varchar(255) binary NOT NULL default '',
>  oi_timestamp varchar(14) binary NOT NULL default '',
>  KEY oi_name (oi_name(10))
> ) TYPE=MyISAM PACK_KEYS=1;
>
> 15) So, history IS stored for each image?  Then I recommend the scheme
> of having a separate table for unique image names, and their
> corresponding "image id"

That's what "Image" does...

> CREATE TABLE random (
>  ra_current tinyint(1) unsigned NOT NULL default '0',
>  ra_title varchar(255) binary NOT NULL default ''
> ) TYPE=MyISAM PACK_KEYS=1;
>
> 16) What is this "random" table for?

You got me there. Could be a relict. Lee once tried some stunts to get a
faster "random page". If that's where it comes from, it's obsolete now.

> CREATE TABLE recentchanges (
>  rc_timestamp varchar(14) binary NOT NULL default '',
>  rc_cur_time varchar(14) binary NOT NULL default '',
>  rc_user int(10) unsigned NOT NULL default '0',
>  rc_user_text varchar(255) binary NOT NULL default '',
>  rc_namespace tinyint(3) unsigned NOT NULL default '0',
>  rc_title varchar(255) binary NOT NULL default '',
>  rc_comment varchar(255) binary NOT NULL default '',
>  rc_minor tinyint(3) unsigned NOT NULL default '0',
>  rc_new tinyint(3) unsigned NOT NULL default '0',
>  rc_cur_id int(10) unsigned NOT NULL default '0',
>  rc_this_oldid int(10) unsigned NOT NULL default '0',
>  rc_last_oldid int(10) unsigned NOT NULL default '0'
> ) TYPE=MyISAM PACK_KEYS=1;
>
> 17) Could the recent changes table be replaced with an appropriately
> defined "view"?  Or are views something MySQL doesn't do very well?

That table is pretty new. I think Lee's still working on it, probably
Brion as well.

> CREATE TABLE site_stats (
>  ss_row_id int(8) unsigned NOT NULL default '0',
>  ss_total_views bigint(20) unsigned default '0',
>  ss_total_edits bigint(20) unsigned default '0',
>  ss_good_articles bigint(20) unsigned default '0',
>  UNIQUE KEY ss_row_id (ss_row_id)
> ) TYPE=MyISAM;
>
> 18) What is ss_row_id and ss_good_articles?  How does this table work?
> Does it contain stats for each article?  Shouldn't that go in the table
> that already contains the info for an article?  That would lessen the
> number of SQL updates, collapsing two UPDATE statements into one.

That's where the "overall statistics" are parked. ss_good_articles is
the number of articles you see on the Main Page.

> CREATE TABLE user (
>  user_id int(5) unsigned NOT NULL auto_increment,
>  user_name varchar(255) binary NOT NULL default '',
>  user_rights tinyblob NOT NULL,
>  user_password tinyblob NOT NULL,
>  user_newpassword tinyblob NOT NULL,
>  user_email tinytext NOT NULL,
>  user_options blob NOT NULL,
>  user_watch mediumblob NOT NULL,
>  user_newtalk tinyint(1) unsigned NOT NULL default '1',
>  UNIQUE KEY user_id (user_id),
>  KEY user_name (user_name(10))
> ) TYPE=MyISAM PACK_KEYS=1;
>
> 19) Does user_rights correspond to ar_flags?

May be. I'm not that familiar with the archive table.

> 20) What is "newpassword" for?

When you let the system mail you a new password, you can log in with
both the old and the new one, until you use the new one, which then
becomes the "standard".

> 21) Are all the "options" really necessary?

Yes!

> 22) It might be the SQL way to make a "watchlist" table, where each row
> contains a user id, and an article id.  Or maybe this would be useless
> in a database that doesn't have subqueries.

You're right on this one. Guess noone really paid attention to that
before...

> 23) What is the user_newtalk field?

When your talk page was edited, that flag is set, so it displays a "*"
next to your talk link when you're logged in.

Hope that helps,

Magnus


body:
Jonathan Walther wrote:

> What if there were no "broken links", only articles that had no text?
> ie, the text field would be NULL?

No, "broken links" are links to articles that don't exist at all, not to
"blank" articles.

> Hm. Maybe images could be collapsed into the "cur" table too.

But they have special properties, which would have to be blank in every
"normal" article. Also, searches etc. would be slower.

> I think we could reduce to the following tables:
>
> current_articles, previous_articles, deleted_articles, links

Then we could drop deleted_articles as well, just set some "cur_deleted"
tag in "current_articles".
But, you'll be the one to change every line of source code for that, I
guess...

> We just add a "type" field to the articles tables to indicate "wiki
> text" or "image"; this gives us flexibility to support other types of
> object too.

"image" is already used for other files (e.g., ogg).

> What is a "good article"?  What uses that table?  What would happen if
> it went away?

A "good article" contains at least a comma...

> Ah. Oddly enough, I would recommend splitting that out into two tables,
> each containing just user ids; one called "developers", and the other
> "sysops".

OK, but why bother? Seems to work, and saves some queries.

Magnus


from_name: Daniel Mayer

subject: [Wikipedia-l] Re: Erik's team certification proposal

body:
I'm still seeing way too many similarities between having multiple themed
Sifter groups and Erik's team certification proposal. What is the putpose of
Erik's idea? From what I gather it is to create selections of articles that
have been reviewed by different groups. Each group would have its own rules
on certification and on who they decide should be group members. These
members would "certify" certain articles based upon those criteria and then a
list of articles certified by any particular group would be automatically
generated. Heck, this can be done right now without any change to the
software so long as group members manually added articles to a list.

Sifter will be doing the /exact/ same thing except instead of compiling a
list, particular article versions are automatically copied to a separate
static website. There is no reason why a log of articles that have been
uploaded by any particular Sifter group couldn't be generated and available
at Wikipedia.

One of the great things, it was thought, about Sifter was that /no/ changes
would have to be made to PediaWiki for it to work. But Erik wants to change
the software anyway to do a very similar thing from within Wikipedia. So why
not marry the two ideas and get the best of both worlds? Then when somebody
visits a Wikipedia article that has been reviewed they will be able to see
that at least a previous version had been reviewed. They would also be
presented a link to a static copy of the certified version (with maybe a diff
link so that the person can easily compare it with the current Wikipedia
version). I don't think this is possible with the "hands-off Wikipedia"
version of Sifter.

The trouble with Erik's certification idea is that people still can't rely on
certified articles because somebody could completely replace a certified
article with another version right after it is certified. Sifter versions are
stable and therefore something that can be trusted (so long as you trust the
quality and reputation of the Sifter member that certified the article and
Sifter group that the member belongs to).

And as soon as there are a certain number of certified articles for any
particular themed Sifter group, then a CD can be made of that selection -- no
need to recheck each article for vandalism or rubbish that has been added
since certification.

-- Daniel Mayer (aka mav)

PS I probably missed some points or even misrepresented Erik's certification
idea. If this is the case, I apologize in advance and please do correct me
Erik (I can't find your original email on this).


subject: [Wikitech-l] userprefs redesign

body:
On Sat, 2002-11-16 at 11:37, elian wrote:
> I've never worked on an open-source project, so if there are any
> conventions to follow, please tell me.
> I did some redesign of the user-prefs form (tested on my MacOS X browser
> zoo and Linux, seems to work still ;-). Shall I commit such stuff directly
> to CVS or post it here so that everyone can test it first?

It looks pleasant enough, I say go ahead and drop it in.

The important thing is to make sure the code runs before committing
stuff, and if it's a major change, be sure to tell everybody!

I put in translation file updates all the time without mentioning it, as
nobody else here will care about them. ;)

> Question to all: I got wikipedia running (macos 10.1.5, mysql 3.23, apache
> 1.3.22), but one problem remains, I get
> the following error when trying to upload a file:
>
> UPDATE cur SET cur_timestamp='20021116204723', cur_user=3, cur_user_text='Elian', cur_text='
>
>         * 20:47, 16. Nov 2002 [[Homepage:Elian|Elian]] \"[[:Bild:Mjwinde.png|Mjwinde.png]]\" hochgeladen (test)
>           ', cur_comment='\"Mjwinde.png\" hochgeladen: test' WHERE cur_id=
> aus der Funktion "wfRecordUpload". MySQL meldete den Fehler "1064: You have an error in your SQL syntax near '' at line 1".
>
> Any ideas? The files are uploaded, though.

That looks like what used to happen if you didn't create the upload log
page manually. It should now work around that, though, and if the log
page doesn't exist it creates it with an INSERT instead of the UPDATE.
Check wfRecordUpload() in GlobalFunctions.php...

-- brion vibber (brion @ pobox.com)


from_name: tarquin

subject: [Wikipedia-l] vandalism spree

body:
seems we have multiple vandals:

* TMC (though whether it's the same as the TMC we know of old, who knows)
* Wikipedia moderator
* Vandalbot

Recent Changes appears to be swamped with them.

I think the time has come for sysops to be able to ban usernamed users.


list: wikien-l

from_name: Toby Bartels

subject: [WikiEN-l] Re: WikiEN-L

body:
elian wrote:

>Toby Bartels wrote:

>>Or not have moderation.
>>Other than approving posts by nonmembers, why do we need it?
>>When was this decided?

>Never. The question was about administration, not moderation. Choosing
>taglines, defining welcome greetings and all the technical stuff.

Well, I certainly trust Ed to do *that*.

-- Toby


subject: [Wikitech-l] Re: WikiEN-L


subject: [Wikipedia-l] goatse

body:
Ortolan88 wrote:

>Anthere wrote:

>>Ed Poor wrote:

>>>We have a [[goatse.cx]] article, and I don't like it. But anyone who's
>>>foolish enough to click on the link for it will be able to read it. And
>>>anyone who's foolhardy enough to click on the external image link will see
>>>it.

Meaning "anyone who's foolish enough to click on the link for it
when they know that they're the type to be offended or disturbed
by an image such as the one described on the page", of course.

>>You might - perhaps - avoid saying people who have different sensibilities
>>than yours, are fools.

>You don't have to be a fool to be foolhardy "delighting in needless
>risks".  Have you followed that link?  Do you ever want to see it
>again?

Making assumptions?
I thought that the picture was rather interesting when I first saw it.
Now I'm too familiar with it to care; ho hum, says I, it's goatse.cx again.

-- Toby


subject: [Wikitech-l] "View Article"

body:
Could somebody change the displayed name for links to subject pages from "View
Article" to "View Subject Page" (or something similar). Per our own
definition, my user page is not an article and neither are any
Wikipedia:namespaced pages.

Is sourceforge a better place for this? Better question: is anybody reviewing
sourceforge bug reports and feature requests anymore?

-- Daniel Mayer (aka mav)


body:
Jonathan Walther wrote:

>> Correct. I am proposing that there should be no broken links because all
>> articles are created, even if only blank articles.  Periodically a
>> process could sweep blank articles with nothing linking to them from the
>> database.  For instance,
>> REMOVE FROM current_articles WHERE ...
>
So, you'll replace the broken_links table by more entries in the article
table. Granted, we'd have one table less, but more articles instead. And
these dummy articles would have to be distinguished from the "real" ones
in *every* query regarding existing/broken articles. More storage (for
the dummy articles), more code, for one table less.

Sure, it can be done, but I don't see the advantage. Maybe it's just me.

Magnus


subject: [Wikipedia-l] vandalism spree

body:
tarquin wrote:

> seems we have multiple vandals:
>
> * TMC (though whether it's the same as the TMC we know of old, who knows)
> * Wikipedia moderator
> * Vandalbot
>
> Recent Changes appears to be swamped with them.
>
> I think the time has come for sysops to be able to ban usernamed users.

I have removed two of these usernames, hoping it would break any
bot/script running, at least for a while.

Magnus


body:
On Sat, Nov 16, 2002 at 09:26:49AM -0500, The Cunctator wrote:
>We do distinguish between articles that have been never created and articles
>that are currently blank, and we need to.

Quite right; so I can extend the query to see if the article is in the
previous_articles table as well.

So, like this:

REMOVE FROM current_articles
WHERE
  text=NULL
  AND NOT article_id IN
    ( SELECT destination FROM links DISTINCT )
  AND NOT article_id IN
    ( SELECT article_id FROM previous_articles DISTINCT ) ;

As you can see, we doing this entirely with SQL; no need for extra
parsing code on the PHP side at all.

>Not that this is necessarily a bad idea, but I'd want to be very clear
>if/how this would alter current functionality.

That's what I'm hoping someone on this list will answer.  I see no
reason it should alter current functionality at ALL, although it should
reduce the amount of PHP code needed.

Jonathan

--
                     Geek House Productions, Ltd.

  Providing Unix & Internet Contracting and Consulting,
  QA Testing, Technical Documentation, Systems Design & Implementation,
  General Programming, E-commerce, Web & Mail Services since 1998

Phone:   604-435-1205
Email:   djw at reactor-core.org
Webpage: http://reactor-core.org
Address: 2459 E 41st Ave, Vancouver, BC  V5R2W2
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 307 bytes
Desc: not available
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20021116/1d3f45ca/attachment.pgp>


body:
On Sat, Nov 16, 2002 at 06:08:00PM +0100, Erik Moeller wrote:
>> You see?  No need for a separate table of dangling links, if you make sure
>> that every article comes into existance.  Subqueries are really nice.
>
>That's a bad idea, because often you will have articles with many links to
>non-existent pages, which would all have to be created. This would blow
>the CUR table out of proportion, with no real advantage.

I don't think so.  With the right data definitions, the number of
entries in the cur table would no more than double, and the amount of
space taken up would definately NOT double; I doubt if it would add even
10 megs to the table size.

>Note that my impression is that the main reason the site is so slow is
>simply that MySQL doesn't handle very large tables very well (maybe
>subqueries would help here, I don't know). That's why we should try to
>avoid making them bigger while we still use it.

Postgres on the other hand, does have mechanisms for dealing with large
tables.  First, you create an index.  Or several indexes.  Then you run
the vacuum optimizer program every night.  It just gets faster and
faster.  This is the advantage of using a real database.

If we port this to Postgres, and then expand yet again, we'll be in good
shape if some kind soul wants to donate an Oracle license, because we
won't have to worry about MySQLisms or PostgreSQLisms in doing the port
to Oracle.

Jonathan

--
                     Geek House Productions, Ltd.

  Providing Unix & Internet Contracting and Consulting,
  QA Testing, Technical Documentation, Systems Design & Implementation,
  General Programming, E-commerce, Web & Mail Services since 1998

Phone:   604-435-1205
Email:   djw at reactor-core.org
Webpage: http://reactor-core.org
Address: 2459 E 41st Ave, Vancouver, BC  V5R2W2
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 307 bytes
Desc: not available
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20021116/c8279242/attachment.pgp>


subject: [Wikitech-l] user deletion and "top" user changes

body:
Regarding the latest vandalism:

Two things that could be achieved with minimum effort are:
1. On the user contributions list, mark those changes that are in the
"cur" table (call it "top" changes, as they are on top of the article
stack). For a vandal user, that means they have not been fixed yet.
2. On the user list, for sysops, have a "delete this user" link after
each user.

The latter could be combined with a "delete top edits", automatically
reverting all "top" changes by this user prior to deletion. That way, we
wouldn't have to clean up everything manually.

I might do that later, if noone beats me to it...

Magnus


from_name: Erik Zachte

subject: [Wikitech-l] vandalism

body:
Maybe this has been asked before, anyway: couldn't a script be writtten
for administrators to roll back all changes made by a certain user or ip
address after a certain date/time, provided noone else has updated
since.

Links to articles that have been updated by someone else than the vandal
could be stored in a special Vandalism/Userxxxx/Versionyyy article for
easy reference.

Erik Zachte


subject: [Wikitech-l] Postgres and large tables


subject: [Wikipedia-l] Re: [Wikitech-l] vandalism

body:
Erik Zachte wrote:

>Maybe this has been asked before, anyway: couldn't a script be writtten
>for administrators to roll back all changes made by a certain user or ip
>address after a certain date/time, provided noone else has updated
>since.
>
>
I wrote about just that 10 minutes ago; didn't my mail make it through?

Anyway, on user contributions, all edits of a user that are still
current revision are now marked with "(top)". Makes it easier to see
what still needs reverting. Online for en and de.

Magnus


subject: [Wikitech-l] vandalism


subject: [Wikitech-l] Postgres and large tables

body:
On Sat, Nov 16, 2002 at 11:48:00PM +0100, Erik Moeller wrote:
>can you try importing the current CUR table into PgSQL (download link from
>wikipedia.sf.net) and do some typical queries also found in the PHP
>scripts to test the performance?

Sure. Can you give me some example queries I can cut'n'paste from email?
Then I'll run 'em 10 or 100 times in a row to get a really good
performance estimate.

I do have to figure out the differences in the data types though. I
don't think PgSQL has tinyblob, and I still have no idea with
type=MyISAM is.

Jonathan

--
                     Geek House Productions, Ltd.

  Providing Unix & Internet Contracting and Consulting,
  QA Testing, Technical Documentation, Systems Design & Implementation,
  General Programming, E-commerce, Web & Mail Services since 1998

Phone:   604-435-1205
Email:   djw at reactor-core.org
Webpage: http://reactor-core.org
Address: 2459 E 41st Ave, Vancouver, BC  V5R2W2
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 307 bytes
Desc: not available
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20021116/dd7bf615/attachment.pgp>


subject: [Wikitech-l] Postgres and large tables

body:
On Sat, Nov 16, 2002 at 02:42:15PM -0800, Jonathan Walther wrote:
>I do have to figure out the differences in the data types though. I
>don't think PgSQL has tinyblob, and I still have no idea with
>type=MyISAM is.

I did some research; Postgres only has one table type, so the
type=MyISAM statements can all be removed.  In fact, I also found out
that the MyISAM table type is not "transaction safe".

Jonathan

--
                     Geek House Productions, Ltd.

  Providing Unix & Internet Contracting and Consulting,
  QA Testing, Technical Documentation, Systems Design & Implementation,
  General Programming, E-commerce, Web & Mail Services since 1998

Phone:   604-435-1205
Email:   djw at reactor-core.org
Webpage: http://reactor-core.org
Address: 2459 E 41st Ave, Vancouver, BC  V5R2W2
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 307 bytes
Desc: not available
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20021116/0b70b2d5/attachment.pgp>


body:
On Sun, Nov 17, 2002 at 12:06:00AM +0100, Erik Moeller wrote:
>Many Wikipedia users try to insert as many relevant links as possible,
>making the relationship between "broken" and real links often 10:1. I
>don't think it would make much sense to create blank rows for each one of
>them just to have simpler table definitions.

That doesn't sound right.  There are definately more "good" links in the
database than dangling ones. Maybe you could run a query and tell us the
results?

>First, we're not on Postgres yet. I appreciate that you're helping with
>the port, but let's not make any changes that would negatively affect the
>system we're using now. We're already slow enough as it is.

If you can send over some "typical queries" I'm about ready to do
testing.

Jonathan

--
                     Geek House Productions, Ltd.

  Providing Unix & Internet Contracting and Consulting,
  QA Testing, Technical Documentation, Systems Design & Implementation,
  General Programming, E-commerce, Web & Mail Services since 1998

Phone:   604-435-1205
Email:   djw at reactor-core.org
Webpage: http://reactor-core.org
Address: 2459 E 41st Ave, Vancouver, BC  V5R2W2
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 307 bytes
Desc: not available
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20021116/ff7a5bdb/attachment.pgp>


from_name: Lars Aronsson

subject:
[Wikipedia-l] Wikipedia moderators and moral authority (was Re:
 Repost: clear guidelines and the power to enforce)

body:
Poor, Edmund W wrote:
> What it comes down to is a tough choice between two choices:
> (1) We are building a free encyclopedia. Therefore, we use Wiki
> software.
> (2) We are maintaining a Wiki community. If we make some good
> encyclopedia articles, that's nice too.
>
> Which is it going to be, people?

Wikipedia is clearly of type 1.  I don't see how there can be any
doubt about that.  But it is still an interesting issue.

I think you can say that http://susning.nu/ (in Swedish) is of type 2,
which is one of several differences between my project and Wikipedia.
I consider individual self regulation (as opposed to enforcement of
rules) to be an important part of the wiki concept, but when I'm in
Wikipedia I follow Wikipedia's rules.  I feel no need to rebel.

I think there is still room to create another large, general purpose
wiki (type 2) in English, which doesn't have the ambition to create an
encyclopedia, and where people can go that don't like Wikipedia's
rules.  From Wikipedia's position, this would mean walking in the
opposite direction of Larry's sifter project.  Starting this would be
a big project, and I doubt that anybody has the energy to do it.  If
it was a GFDL project, it could start out with a copy of the current
Wikipedia.

Wikipedia is the combination of a free encyclopedia (an old idea, as
Axel Boldt can tell you) and the wiki concept.  This combination can
be problematic for those who don't already know both components, which
is the case in many non-English languages where the Wikipedia is the
first wiki that people see.  Maybe this is a problem for many English
Wikipedia newcomers as well.

--
  Lars Aronsson (lars at aronsson.se)
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


from_name: Christian List

subject: [Wikitech-l] Request for Danish mailing list (Was: Request for setting up of the Polish mailing list)

body:
> Anybody else need a list?

Could we please have a WikiDA-l at wikipedia.org for the danish Wikipedia.
I am willing to be the administrator.

Regards
Christian


from_name: Larry Sanger

subject: [Wikipedia-l] Encyclopedia meta-searcher up and running

body:
Hi all,

I took a some time out the past few days and used my developing Perl
skills to write an encyclopedia meta-search engine, which will at least
temporarily live at:

	http://www.seeatown.com/search/

The default settings search only four encyclopedias, but you can choose
from up to 18 (and counting).  You can change the encyclopedias searched
on the settings page.

I hope you find this useful!  Obviously, I think Wikipedia and Nupedia
should write articles better than all the articles you'll see via this
interface.  ;-)  But it should be a useful way to get a benchmark idea of
what we want out of a basic encyclopedia article on a topic.

I could really use your help in finding more encyclopedia-type websites to
include.  The way the script is written now, one requirement is that the
source website's search results page reproduce the search terms in the URL
(i.e., for those who know about HTML forms, the website should use "GET"
rather than "POST").

Feature requests are also welcome.

By the way, in coding and testing this website I discovered (at least to
my own satisfaction) that Wikipedia is far and away the most complete
*free* (gratis) resource online.

Larry
--
"We have now sunk to a depth at which the re-statement of the obvious is
the first duty of intelligent men." --George Orwell


subject: [Wikitech-l] Bug in editor

body:
On Sat, 2002-11-16 at 14:24, Pierre Abbat wrote:
> I just came upon an article [[Connected]] containing a definition of a
> connected graph. I added a definition of a connected set and got an edit
> conflict, containing lots of text about graphs. The web page title was "Edit
> conflict: Connected". I added my text to the bottom and hit the save button.
> I then found that someone had turned [[Connected]] into a redirect behind my
> back and my definition of a connected set is now at the bottom of the [[Graph
> theory]] page!
>
> AFAICT this is the same as bug 602823. The bug is 5/2 months old and has not
> been fixed.

Is it a bug or intentional behavior? (I don't know the answer, I didn't
write the redirect handling stuff.)

-- brion vibber (brion @ pobox.com)


body:
On Sat, 2002-11-16 at 15:09, Jonathan Walther wrote:
> On Sun, Nov 17, 2002 at 12:06:00AM +0100, Erik Moeller wrote:
> >Many Wikipedia users try to insert as many relevant links as possible,
> >making the relationship between "broken" and real links often 10:1. I
> >don't think it would make much sense to create blank rows for each one of
> >them just to have simpler table definitions.
>
> That doesn't sound right.  There are definately more "good" links in the
> database than dangling ones. Maybe you could run a query and tell us the
> results?

Assuming we're talking about pages that are linked to rather than the
raw total of links:

mysql> select count(distinct bl_to) from brokenlinks;
+-----------------------+
| count(distinct bl_to) |
+-----------------------+
|                188126 |
+-----------------------+

mysql> select count(distinct l_to) from links;
+----------------------+
| count(distinct l_to) |
+----------------------+
|               115708 |
+----------------------+

It's certainly not 10:1, but more like 5:3 in favor of not yet existing
pages on the English wiki.

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] Bug in editor

body:
Brion Vibber wrote:

>Pierre Abbat wrote:

>>I just came upon an article [[Connected]] containing a definition of a
>>connected graph. I added a definition of a connected set and got an edit
>>conflict, containing lots of text about graphs. The web page title was "Edit
>>conflict: Connected". I added my text to the bottom and hit the save button.
>>I then found that someone had turned [[Connected]] into a redirect behind my
>>back and my definition of a connected set is now at the bottom of the [[Graph
>>theory]] page!

>Is it a bug or intentional behavior? (I don't know the answer, I didn't
>write the redirect handling stuff.)

There may well be a reason that <wiki.phtml?title=Connected&action=edit>
should edit [[Graph theory]] if [[Connected]] redirects there,
but this is clearly not desirable behaviour in Pierre's situation.
The solution is to alter the URL that Pierre sees
when he begins to edit a page that is not a redirect,
to <wiki.phtml?title=Connected&action=redit&redirect=no>
(which is already what you get when you edit a redirect page
after viewing it with <&redirect=no>).

-- Toby


body:
On Sat, 2002-11-16 at 06:26, The Cunctator wrote:
> On 11/16/02 4:44 AM, "Jonathan Walther" <krooger at debian.org> wrote:
> > On Sat, Nov 16, 2002 at 01:28:31AM -0800, Brion Vibber wrote:
> >> All articles have ID numbers; that's cur_id in the cur table. bl_from
> >> and l_to reference cur_id numbers. Articles that don't exist don't have
> >> cur_id numbers of course, so cannot be referenced by id, but only by
> >> title.
> >
> > What if there were no "broken links", only articles that had no text?
> > ie, the text field would be NULL?  And we could do with a single "links"
> > table by collapsing article and image links together.  And one could
> > find the "dangling links" simply by searching for those articles with
> > the text property set to NULL.  The following query would do the trick:
>
> We do distinguish between articles that have been never created and articles
> that are currently blank, and we need to.

Hmm, if I understand correctly, a blank article will have cur_text='',
while a nonexistent article with a db entry would have cur_text=NULL.
This is quite distinguishable.

> Not that this is necessarily a bad idea, but I'd want to be very clear
> if/how this would alter current functionality.

In principle, it wouldn't affect functionality. On the backend, it would
enable the various X_title and X_namespace string fields in other tables
to be replaced with straightforward numeric references to the cur table.
Less string parsing should give better performance.

However, I don't think this can be done with MySQL 3.x as subqueries are
not available. Ergo, moving to postgresql or waiting for MySQL 4.1
(could be a while!) would be necessary to implement that.

While we're discussing future plans of totally restructuring the
database, may I also request comments on:
http://meta.wikipedia.org/wiki/Thoughts_on_language_integration

Putting data of all languages (and other subwikis such as meta and
sep11) into a single database could simplify inter-language and
inter-subwiki linking (including sharing of uploaded files).

Oh, a reminder before I forget -- the *_user_text fields store the IP
address for non-logged in users, "conversion script" for usemod->phase
II conversion step, or the old username of a usemod-era edit by a logged
in user, for all of whom the *_user id field is 0.

-- brion vibebr (brion @ pobox.com)


subject: [Wikitech-l] Bug in editor

body:
On Sat, 2002-11-16 at 17:04, Toby Bartels wrote:
> There may well be a reason that <wiki.phtml?title=Connected&action=edit>
> should edit [[Graph theory]] if [[Connected]] redirects there,
> but this is clearly not desirable behaviour in Pierre's situation.
> The solution is to alter the URL that Pierre sees
> when he begins to edit a page that is not a redirect,
> to <wiki.phtml?title=Connected&action=redit&redirect=no>
> (which is already what you get when you edit a redirect page
> after viewing it with <&redirect=no>).

I suspect then that &redirect=no is redundant in every single case
except for viewing the current revision of a page. If the desired
behavior is that diffs, edits, histories, and old revisions should not
follow redirects, the code should simply not attempt to do so.

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] Bug in editor

body:
I just came upon an article [[Connected]] containing a definition of a
connected graph. I added a definition of a connected set and got an edit
conflict, containing lots of text about graphs. The web page title was "Edit
conflict: Connected". I added my text to the bottom and hit the save button.
I then found that someone had turned [[Connected]] into a redirect behind my
back and my definition of a connected set is now at the bottom of the [[Graph
theory]] page!

AFAICT this is the same as bug 602823. The bug is 5/2 months old and has not
been fixed.

phma


subject: [Wikitech-l] Re: [Wikipedia-l] Uploaded cracking attempt

body:
On Sat, 2002-11-16 at 07:09, Pierre Abbat wrote:
> LearII uploaded the following, which appears to be an attempt to read from
> one port on one box and write to a port on another box. He also uploaded
> another PHP file, which was an attempt to list the files on the server. PHP
> doesn't run in the upload directory, and he left out the semicolons, but
> would someone please block him?

That reminds me -- although PHP scripts do not run in our upload
directory, HTML files containing JavaScript are possible. If no one
catches them, a wikipedia user could be tricked into viewing the file
and, if they have Javascript enabled, their login cookies could be
stolen, including the password hash if 'remember password across
sessions' is enabled.

Three ways around this come to mind:
1) A tighter cookie path could prevent untrusted files in /upload/ from
reading the cookie. But, we have three paths to worry about which need
to access the cookie:
 / (main page, raw host url)
 /wiki/ (normal pages)
 /w/ (directly linking the script for various options; edit etc)
Currently we can serve all three only by setting the cookie path to /,
which lets the whole site read them. Either we need to change the paths
(force / to redirect to /wiki/Main_Page and use /wiki/Foo?bar instead of
/w/wiki.phtml?title=Foo&bar) or think of something else.

2) Don't allow HTML files to be uploaded, or serve them with plain text
mimetype. This could be annoying.

3) Allow HTML files to be uploaded, but attempt to strip out javascript.
This could be annoying.

-- brion vibber (brion @ pobox.com)


from_name: Erik Moeller

body:
> You see?  No need for a separate table of dangling links, if you make sure
> that every article comes into existance.  Subqueries are really nice.

That's a bad idea, because often you will have articles with many links to
non-existent pages, which would all have to be created. This would blow
the CUR table out of proportion, with no real advantage.

Note that my impression is that the main reason the site is so slow is
simply that MySQL doesn't handle very large tables very well (maybe
subqueries would help here, I don't know). That's why we should try to
avoid making them bigger while we still use it.

Regards,

Erik


subject: [Wikitech-l] Image archive download?

body:
> Here ya go:
> http://www.wikipedia.org/tarballs/image.tar.bz2
> circa 125MB

Brion,

thanks a lot! Didn't think it would be so small. Maybe we need a Wikipedia
photo squad after all ..

Regards,

Erik


from_name: elian

subject: [Wikitech-l] userprefs redesign

body:
Hello,

I've never worked on an open-source project, so if there are any
conventions to follow, please tell me.
I did some redesign of the user-prefs form (tested on my MacOS X browser
zoo and Linux, seems to work still ;-). Shall I commit such stuff directly
to CVS or post it here so that everyone can test it first?

Note to Magnus:
Could you mail me your test.php for the portal page?

Question to all: I got wikipedia running (macos 10.1.5, mysql 3.23, apache
1.3.22), but one problem remains, I get
the following error when trying to upload a file:

UPDATE cur SET cur_timestamp='20021116204723', cur_user=3, cur_user_text='Elian', cur_text='

        * 20:47, 16. Nov 2002 [[Homepage:Elian|Elian]] \"[[:Bild:Mjwinde.png|Mjwinde.png]]\" hochgeladen (test)
          ', cur_comment='\"Mjwinde.png\" hochgeladen: test' WHERE cur_id=
aus der Funktion "wfRecordUpload". MySQL meldete den Fehler "1064: You have an error in your SQL syntax near '' at line 1".

Any ideas? The files are uploaded, though.

greetings,
elian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SpecialPreferences.php
Type: application/octet-stream
Size: 9536 bytes
Desc: not available
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20021116/a3698cd2/attachment.obj>
-------------- next part --------------
--
Microsoft is not the answer.
Microsoft is the question and the answer is NO.


from_name: Steve Rawlinson

subject: [Wikitech-l] Static HTML Wikipedia (CD-ROM)

body:
There seems to be some interest in creating a static HTML distribution
(dump) of Wikipedia, most notably it is requested on the
Wikipedia:Database_download page and in Feature Requests #596830 on
Sourceforge.  This would allow people to download the Wikipedia for use
offline, for example from a CDROM.

So, I have started work and made my initial version (English only)
available online for anyone on this list to evaluate and test.  I am
looking for feedback, suggestions, bug reports and general comments.

http://www.rawlinson.ca:8080/wikipedia/index.html

Please do not attempt to mirror the site as my server and bandwidth won't
be able to handle it.  The site is only intended for developers to try and
give feedback.  Once everyone is happy with it I will make .tar and .iso
packages available for distribution.

At the moment, the method I use to create the static HTML version is very
lengthy in terms of processing time and requires a number of manual step.
It takes my 1 GHz machine about 5 hours to generate all the pages.
Ideally, I'll have something more automated and efficient as time goes on.
My plan for how things would work is I will produce an updated static HTML
version every few months or significant milestones.

I'm not sure how to distribute this static HTML version when it's ready
for a public release.  Currently it's about 500 Meg in size (that includes
everything).  As I mentioned above I have limited server resources.  For
distribution maybe it could be put on the Sourceforge download page, or on
the Wikipedia.org server somewhere (/tarballs)?

Finally, since I am new to Wikipedia and this list, please excuse me while
I learn how things work around here.  I am open to criticism, suggestions
and discussion.  I am looking forward to working with everyone on
Wikipedia and contributing where I can.

Some Technical Details (for those interested):

- English only (currently)

- uses "printable" pages, no top or side navigation bars

- added links to home, back, copyright and Wikipedia.org to bottom of all
  pages (TODO: if a talk page exists a link should be added)

- pages are stored in directories based on first two characters of MD5
  hash, same as image storage scheme

- includes all namespaces (talk, users, users_talk, wikipedia_talk, etc.)

- created a list with links to all the items in each namespace to allow
  for basic searching of page titles

- redirects replaced with direct link to article

Regards,
Steve Rawlinson


body:
I realize I'm arriving late to the database discussion, but has the use of
an abstraction library been considered?  There is a PHP library called
ADODB (http://php.weblogs.com/ADODB/) that is used by PostNuke and others.
I've personally used it on smaller projects very successfully.  It's
modeled after Microsoft's [[ADO]] library that performs a similar function
(as I understand).  An abstraction layer like this gives you the
flexibility to use whatever database backend you want.

Description from Freshmeat.net:

"ADODB is a set of advanced PHP database abstraction classes. It supports
MySQL, PostgreSQL, Interbase/Firebird, Informix, Sybase SQL Anywhere,
Oracle, MS SQL 7, Sybase, DB2, FrontBase, Foxpro, Access, ADO, and generic
ODBC. A metatype system is built in, making it possible to figure out that
types such as CHAR, TEXT, and STRING are equivalent in different
databases. It also features an SQL to HTML popup menu and SQL to HTML
table support. It has code to support record paging and blob/clob
support."

Of course nothing is ever free, so I suspect there could be a slight price
to pay in terms of performance.  I don't know enough about ADODB to say
what the performance costs are, nor could I find any discussion on their
site.  However, I suspect that the cost of an extra PHP function call or
two is small compared to the overall cost of performing the SQL query.

Anyway, it's just a suggestion.  Others more knowledgeable about Wikipedia's
current implementation and requirements are probably better judges of
whether or not this could be useful.

Steve Rawlinson


subject: [Wikitech-l] "View Article"

body:
> Could somebody change the displayed name for links to subject pages from
> "View Article" to "View Subject Page" (or something similar). Per our own
> definition, my user page is not an article and neither are any
> Wikipedia:namespaced pages.

Hi,

"subject page" is not a good name because it's not really clear what it
means in any context. It's not a term we use anywhere else. Let's either
try to come up with something less ambiguous or change the text depending
on the namespace (e.g. "View user page" from User: pages).

Regards,

Erik


subject: [Wikitech-l] Postgres and large tables

body:
Jon,

can you try importing the current CUR table into PgSQL (download link from
wikipedia.sf.net) and do some typical queries also found in the PHP
scripts to test the performance?

Regards,

Erik


subject: [Wikitech-l] Too many darn queries!

body:
Simply viewing an article currently involves performing a lot of
queries. Since it's our most common event, we should perhaps be
directing optimizing effort in that direction.

(if logged in) check user info:
User::loadFromDatabase:SQL: SELECT user_name,user_password,
  user_newpassword,user_email,user_options,user_rights,user_newtalk
  FROM user WHERE user_id=1

Check if page exists:
LinkCache::addLink:SQL: SELECT cur_id FROM cur
  WHERE (cur_namespace=0 AND cur_title='Test:stuff')

Load the content:
Article::loadContent:SQL: SELECT cur_text,cur_timestamp,cur_user,
  cur_counter FROM cur WHERE cur_id=19

[These last two can be combined into one query.]

Now, separately for each and every linked page:
  Check if it exists:
  LinkCache::addLink:SQL: SELECT cur_id FROM cur
    WHERE (cur_namespace=0 AND cur_title='A_link')

  (if stub threshold set) check its size and redirect status:
  SQL: SELECT length(cur_text) AS x, cur_namespace, cur_is_redirect
    FROM cur WHERE cur_id='96'

[Checking the size can be done with checking existence, and it may be
better to chalk up a list of all linked pages during initial parsing,
then check them all in one big query.]

Check if current page is restricted for editing:
wfGetSQL:SQL: SELECT cur_restrictions FROM cur WHERE (cur_id=19)

[We should get cur_restrictions along with the rest in the initial
query.]

(if logged in) check if on watchlist:
wfGetSQL:SQL: SELECT user_watch FROM user WHERE (user_id=1)

[For users with big watchlists like me, it may be inefficient to load
and parse the entire list on every single page view. It should perhaps
either be grabbed along with the rest of the user info, or stored in a
separate table where the database can parse the gobs of strings for us.]

Check for existence of talk page (twice - sidebar and bottom bar):
LinkCache::addLink:SQL: SELECT cur_id FROM cur
  WHERE (cur_namespace=1 AND cur_title='Test:stuff')
LinkCache::addLink:SQL: SELECT cur_id FROM cur
  WHERE (cur_namespace=1 AND cur_title='Test:stuff')

[Should only need to do this once and store the result...]

...output the page...

Update counters:
ViewCountUpdate::doUpdate:SQL: UPDATE cur
  SET cur_counter=(1+cur_counter),cur_timestamp=cur_timestamp
  WHERE cur_id=19
SiteStatsUpdate::doUpdate:SQL: UPDATE site_stats
  SET ss_total_views=(ss_total_views+1),
  ss_total_edits=(ss_total_edits),
  ss_good_articles=(ss_good_articles) WHERE ss_row_id=1

[Note that that single row of site_stats is updated on every single view
of any page in the database.]

-- brion vibber (brion @ pobox.com)


body:
> I don't think so.  With the right data definitions, the number of
> entries in the cur table would no more than double, and the amount of
> space taken up would definately NOT double; I doubt if it would add even
> 10 megs to the table size.

Many Wikipedia users try to insert as many relevant links as possible,
making the relationship between "broken" and real links often 10:1. I
don't think it would make much sense to create blank rows for each one of
them just to have simpler table definitions.

> Postgres on the other hand, does have mechanisms for dealing with large
> tables.  First, you create an index.  Or several indexes.  Then you run
> the vacuum optimizer program every night.  It just gets faster and
> faster.  This is the advantage of using a real database.

First, we're not on Postgres yet. I appreciate that you're helping with
the port, but let's not make any changes that would negatively affect the
system we're using now. We're already slow enough as it is.

MySQL also has indexes :-). It just doesn't use them in some special
cases; it would probably already give us a significant performance boost
to figure out what these cases are and to fix the code accordingly.

Note that Yahoo! uses MySQL, so it's quite possible to run very large
sites with it.

Regards,

Erik


from_name: Tomasz Wegrzanowski

body:
On Sat, Nov 16, 2002 at 02:15:40PM -0800, Jonathan Walther wrote:
> If we port this to Postgres, and then expand yet again, we'll be in good
> shape if some kind soul wants to donate an Oracle license, because we
> won't have to worry about MySQLisms or PostgreSQLisms in doing the port
> to Oracle.

Don't even dare to suggest such thing. Should Wikipedia ever cease to be
runable on 100% Open Source system, it will cause immediate fork of
the project.


from_name: Brion Vibber

body:
On Sat, 2002-11-16 at 18:36, Steve Rawlinson wrote:
> There seems to be some interest in creating a static HTML distribution
> (dump) of Wikipedia, most notably it is requested on the
> Wikipedia:Database_download page and in Feature Requests #596830 on
> Sourceforge.  This would allow people to download the Wikipedia for use
> offline, for example from a CDROM.
>
> So, I have started work and made my initial version (English only)
> available online for anyone on this list to evaluate and test.  I am
> looking for feedback, suggestions, bug reports and general comments.
>
> http://www.rawlinson.ca:8080/wikipedia/index.html

A cool beginning, thanks! :)

> I'm not sure how to distribute this static HTML version when it's ready
> for a public release.  Currently it's about 500 Meg in size (that includes
> everything).  As I mentioned above I have limited server resources.  For
> distribution maybe it could be put on the Sourceforge download page, or on
> the Wikipedia.org server somewhere (/tarballs)?

I expect we could provide both a tarball and a static tree which could
be rsync'ed.

> Finally, since I am new to Wikipedia and this list, please excuse me while
> I learn how things work around here.  I am open to criticism, suggestions
> and discussion.  I am looking forward to working with everyone on
> Wikipedia and contributing where I can.
>
>
> Some Technical Details (for those interested):
>
> - English only (currently)

That will need to be fixed, of course! :)

> - uses "printable" pages, no top or side navigation bars

Could probably stand to be purtied up at least a little bit.

> - added links to home, back, copyright and Wikipedia.org to bottom of all
>   pages (TODO: if a talk page exists a link should be added)

A link to that particular page on the live server would be a *very* good
idea. The regular printable pages include this.

> - pages are stored in directories based on first two characters of MD5
>   hash, same as image storage scheme

Some things to think about as far as the actual filenames:
* Length. Wikipedia titles can I think get up to ~255 characters; this
may be too long for some systems.
* Acceptable characters. Colons, slashes, quotes, and various non-ascii
characters may appear in titles that cannot be reliably reproduced on
many filesystems. I notice that colons and commas at least are changed
to underscores, possibly some other characters too; conflicts may occur.
Non-ascii chars appear to be left intact; will this work consistently
across different filesystems which may be configured for different
character encodings?
* Case sensitivity. Many filesystems are not case sensitive; we may have
conflicts.

> - includes all namespaces (talk, users, users_talk, wikipedia_talk, etc.)

User and talk pages are probably not necessary; if you're looking to
discuss the page, you'll be doing it on the live site where you can edit
it (and see the last 6 months' worth of edits which aren't on your
CD-ROM). And, of course, they take up a large chunk of valuable CD real
estate better devoted to future articles.

Thoughts?

> - created a list with links to all the items in each namespace to allow
>   for basic searching of page titles

A simple JavaScript-based title search could probably be rigged up out
of that.

> - redirects replaced with direct link to article

Nice.

-- brion vibber (brion @ pobox.com)


from_name: Jonathan Walther

body:
On Sun, Nov 17, 2002 at 11:26:03AM +0100, Magnus Manske wrote:
>As long as we're talking about different open source databases, there's
>also SAP DB (http://www.sapdb.org/). This is a once-commercial database
>that is now under open source, similar to Netscape/Mozilla. As this once
>was commercial, running databases for the largest companies, it seems to
>outclass both MySQL and PostgreSQL in speed, reliability, and supported
>SQL language. The strange thing is that it is hardly known in the open
>source scene. I didn't try it myself yet, either.

There's a reason for that.  It's a bitch to get up and running, and
tuned to run anywhere near at a reasonable performance.  People that use
the SAP DB invariably end up paying SAP to set it up for them.

Oh, and benchmarks show it's not superior to, only comparable to,
PostgreSQL and Oracle.

Jonathan

--
                     Geek House Productions, Ltd.

  Providing Unix & Internet Contracting and Consulting,
  QA Testing, Technical Documentation, Systems Design & Implementation,
  General Programming, E-commerce, Web & Mail Services since 1998

Phone:   604-435-1205
Email:   djw at reactor-core.org
Webpage: http://reactor-core.org
Address: 2459 E 41st Ave, Vancouver, BC  V5R2W2
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 307 bytes
Desc: not available
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20021117/29558bdb/attachment.pgp>


from_name: Jens Frank

subject: [Wikitech-l] Too many darn queries!

body:
Hello Brion,

this is a very good overview of the SQL needed to show a page.
You suggest to merge several of these queries, which is an idea
to improve performance without to much changing of code.

But this idea could be moved even further: Most of our users not
only want to read one page, they read lots of them. It is possible
to make several queries only once in the lifetime of a user session
or even the lifetime of the server process.

On Sat, Nov 16, 2002 at 11:53:58PM -0800, Brion Vibber wrote:
> Simply viewing an article currently involves performing a lot of
> queries. Since it's our most common event, we should perhaps be
> directing optimizing effort in that direction.
>
> (if logged in) check user info:
> User::loadFromDatabase:SQL: SELECT user_name,user_password,
>   user_newpassword,user_email,user_options,user_rights,user_newtalk
>   FROM user WHERE user_id=1

This could be done once the user session starts and be kept in a session
variable. This would require us to keep a persistent session which would
imply use of additional cookies. Since we already use cookies this
should not cause any headache.

> Check if page exists:
> LinkCache::addLink:SQL: SELECT cur_id FROM cur
>   WHERE (cur_namespace=0 AND cur_title='Test:stuff')

Checking for a pages existance is a very common task. The information
needed is
- Does the page exist?
- Is it a stub?
- Is it locked?
This could be stored in shared memory. A rough estimate: there are
about 150,000 pages at the moment. If we store 64 bytes of information
per article (its name, size and lock. Something else?) this is an amount
of 9.6 MB.

> Load the content:
> Article::loadContent:SQL: SELECT cur_text,cur_timestamp,cur_user,
>   cur_counter FROM cur WHERE cur_id=19
>
> [These last two can be combined into one query.]
>
> Now, separately for each and every linked page:
>   Check if it exists:
>   LinkCache::addLink:SQL: SELECT cur_id FROM cur
>     WHERE (cur_namespace=0 AND cur_title='A_link')
>
>   (if stub threshold set) check its size and redirect status:
>   SQL: SELECT length(cur_text) AS x, cur_namespace, cur_is_redirect
>     FROM cur WHERE cur_id='96'
>
> [Checking the size can be done with checking existence, and it may be
> better to chalk up a list of all linked pages during initial parsing,
> then check them all in one big query.]

This information can be fetched from the shared memory table proposed
above, too.

> Check if current page is restricted for editing:
> wfGetSQL:SQL: SELECT cur_restrictions FROM cur WHERE (cur_id=19)
>
> [We should get cur_restrictions along with the rest in the initial
> query.]
>
> (if logged in) check if on watchlist:
> wfGetSQL:SQL: SELECT user_watch FROM user WHERE (user_id=1)
>
> [For users with big watchlists like me, it may be inefficient to load
> and parse the entire list on every single page view. It should perhaps
> either be grabbed along with the rest of the user info, or stored in a
> separate table where the database can parse the gobs of strings for us.]
>
> Check for existence of talk page (twice - sidebar and bottom bar):
> LinkCache::addLink:SQL: SELECT cur_id FROM cur
>   WHERE (cur_namespace=1 AND cur_title='Test:stuff')
> LinkCache::addLink:SQL: SELECT cur_id FROM cur
>   WHERE (cur_namespace=1 AND cur_title='Test:stuff')
>
> [Should only need to do this once and store the result...]

Could be fetched from the shared memory table proposed above, too.

The code for creating and changing an article would have to update the
shared memory segment. Care has to be taken that no two users change
the memory at the same time. Since the operations to perform on the
memory are not that time-intensive a global semaphore locking
write access should be sufficient.

	Regards,

		JeLuF


subject: [Wikitech-l] Too many darn queries!

body:
Brion Vibber wrote:

>Now, separately for each and every linked page:
>  Check if it exists:
>  LinkCache::addLink:SQL: SELECT cur_id FROM cur
>    WHERE (cur_namespace=0 AND cur_title='A_link')
>
I think that's one of the big evils. How many pages are linked from the
Main Page alone? 50? 100?
WIth stub detector and all, the other queries sum up to 10...

This is how I worked around that problem (well, sorta) in Phase II:

There's a field in the cur table with all "known as existing" links, a
single string with a separator ("\n") between the entries. It got loaded
together with the article from the database. Now, we'd only have to
check all the links that are *not* in that list, and eventually update
the field if new articles were created. This would be done on viewing.
The major drawback of that seemed to be the stunt I had to pull for
every page deletion, as you'd have to fulltext-search through all of
these fields.
As page deletion is a rather rare event compared to viewing, it might
pay off in the long run.
All there'd be to do is to add a field (tinyblob) to the cur table, then
add the checking/auto-update code on the view routine, and patch the
page deletion code.
Maybe cleaning of the field might be appropriate on occasion (when
moving an article, maybe).

Using the links/brokenlinks table instead will not do any good, IMHO;
which table we search doesn't really matter, and the links table is
longer...

Magnus


body:
Cool! I like it! Hope to see the CDs at Wal-Mart soon ;-)

A few minor thoughts:
* On every page, there should be a link to the online article, and to
the online edit.
* Images should link to the image text, like wikipedia does, not to the
image itself. One can save the image directly from the page.
* A nice CSS will make it look less "plain", without compromising
compatibility with older browsers. The CD could also have a recent
Mozilla installer on it ;-)
* There could be other index.html files with frames in them, where a
small frame on top, bottom, or side, could carry useful links (Main
Page, Copyrights, the "live" wikipedia, "live" search engine, etc.) and
maybe the logo.
* If you want statistics (date of last edit) at all, maybe add some more
("this was edited 123 times"). [Might need modification of the
"printable" function]
* Should non-existent links be displayed as links? I think so, because
people might feel tempted to write something ;-)  [Might need
modification of the "printable" function]

There should also be a "real" offline search function. Shouldn't be too
hard to hack some smallish programs for Windows, Linux, maybe Mac.
Anyone knows if there's something like that out there?

We might also think of adding special "view" functions to the software
(like the "printable version"), to aid projects like this, and maybe
Larry's wikipedia sifter.

Magnus

Steve Rawlinson wrote:

>There seems to be some interest in creating a static HTML distribution
>(dump) of Wikipedia, most notably it is requested on the
>Wikipedia:Database_download page and in Feature Requests #596830 on
>Sourceforge.  This would allow people to download the Wikipedia for use
>offline, for example from a CDROM.
>
>So, I have started work and made my initial version (English only)
>available online for anyone on this list to evaluate and test.  I am
>looking for feedback, suggestions, bug reports and general comments.
>
>http://www.rawlinson.ca:8080/wikipedia/index.html
>
>Please do not attempt to mirror the site as my server and bandwidth won't
>be able to handle it.  The site is only intended for developers to try and
>give feedback.  Once everyone is happy with it I will make .tar and .iso
>packages available for distribution.
>
>At the moment, the method I use to create the static HTML version is very
>lengthy in terms of processing time and requires a number of manual step.
>It takes my 1 GHz machine about 5 hours to generate all the pages.
>Ideally, I'll have something more automated and efficient as time goes on.
>My plan for how things would work is I will produce an updated static HTML
>version every few months or significant milestones.
>
>I'm not sure how to distribute this static HTML version when it's ready
>for a public release.  Currently it's about 500 Meg in size (that includes
>everything).  As I mentioned above I have limited server resources.  For
>distribution maybe it could be put on the Sourceforge download page, or on
>the Wikipedia.org server somewhere (/tarballs)?
>
>Finally, since I am new to Wikipedia and this list, please excuse me while
>I learn how things work around here.  I am open to criticism, suggestions
>and discussion.  I am looking forward to working with everyone on
>Wikipedia and contributing where I can.
>
>
>Some Technical Details (for those interested):
>
>- English only (currently)
>
>- uses "printable" pages, no top or side navigation bars
>
>- added links to home, back, copyright and Wikipedia.org to bottom of all
>  pages (TODO: if a talk page exists a link should be added)
>
>- pages are stored in directories based on first two characters of MD5
>  hash, same as image storage scheme
>
>- includes all namespaces (talk, users, users_talk, wikipedia_talk, etc.)
>
>- created a list with links to all the items in each namespace to allow
>  for basic searching of page titles
>
>- redirects replaced with direct link to article
>
>
>Regards,
>Steve Rawlinson
>
>_______________________________________________
>Wikitech-l mailing list
>Wikitech-l at wikipedia.org
>http://www.wikipedia.org/mailman/listinfo/wikitech-l
>
>
>


body:
As long as we're talking about different open source databases, there's
also SAP DB (http://www.sapdb.org/). This is a once-commercial database
that is now under open source, similar to Netscape/Mozilla. As this once
was commercial, running databases for the largest companies, it seems to
outclass both MySQL and PostgreSQL in speed, reliability, and supported
SQL language. The strange thing is that it is hardly known in the open
source scene. I didn't try it myself yet, either.

Anyone willing to give it a try? (I'm busy with something else right now...)

Magnus


list: sifter-l

source: wayback-mbox

from_name: Axel Boldt

subject: [Sifter-l] Lings to other Wikipedia articles

body:
I have a technical question: suppose I review an article and want to
import it into sifter, but it has lots of links and See alsos pointing
to other Wikipedia articles that haven't yet been imported. How should
we deal with those links?

One idea would be this: following such a link brings the reader to a
disclaimer page "You are now leaving the peer reviewed encyclopedia
project and entering Wikipedia, where anybody can edit any article.
Click here to proceed."

As soon as one of those articles is imported into sifter, the
disclaimer link would automatically change into a sifter-internal link.
This assumes that article titles in Wikipedia and sifter are identical,
which I think is a good thing anyway.

Other ideas?

Axel

P.S. Name proposal: ReviewedEncyclopedia.org

__________________________________________________
Do you Yahoo!?
Yahoo! Web Hosting - Let the expert host your site
http://webhosting.yahoo.com


from_name: Neil Harris

subject: [Wikipedia-l] Vandals,proxies, and so on:a proposal

body:
Many vandals hide behind shared HTTP proxies, which makes them difficult
to ban.

Here's an idea:

Add a new attribute to user accounts:
* "authenticated" users are users **who have supplied a non-throwaway
E-mail address**: authentication to be done by sending them an E-mail
which they have to reply to, in the same way as mailing list authentication.

We can then "greylist" IP addresses or ranges, so that only
''authenticated'' logged-in users  can post from behind these addresses.
 We can point out to new users from these ISPs that the reason why they
are being asked to authenticate is that other users from the same ISP
have acted as vandals.

The good bit:
* At the same time, non-greylisted IP addresses can still allow
anonymous or non-authenticated user account edits, so we stay "open" to
 >99.99% of all users.

We should greylist just the IP address for a proxy, or the whole /19
range for a user IP address: this is the minimum routable block on the
Internet, and will generally catch all users from a particular region.

This significantly increases the costs to vandals, and provides
traceability back to providers, or even real identities if necessary.
Vandals can go on making new accounts as many times as they like, but
they have to incur the costs of setting up new provider accounts every
time we ban their user account.  (I believe that ISPs share phone
numbers and credit card numbers of persistent abusers, so these people
will either end up without access, or using rogue providers, who we can
then blacklist. )

Then, we can reserve "blacklisting" only for IP addresses that are
beyond hope, such as individual users who are non-cooperative, or
providers without a workable anti-abuse policy. "Blacklisting" should
then ban all editing.
We can also refuse to accept authentication E-mails from E-mail
providers who do not have a good abuse policy.

Neil


subject: [Wikitech-l] Vandals,proxies, and so on:a proposal


from_name: Karen AKA Kajikit

subject: [Wikipedia-l] to put it politely, the EN wikipedia is f#ed...

body:
I wondered why there were about 60 articles with zero bytes waiting to
be deleted, and nearly as many with less than 50 (usually a sign that
they're either a typoed redirect, vandalism, or a mistake). But after
waiting more than five minutes for a single delete to be confirmed now I
know...

You can add as many bells and whistles to the wikipedia as you like, but
they do no good when the basic functioning doesn't.

--

Karen AKA Kajikit

To err is human... to really foul things up add kitten and stir.

Come and visit my part of the web:
Kajikit's Corner: http://Kajikit.netfirms.com/
Aussie Support Mailing List: http://groups.yahoo.com/group/AussieSupport
Allergyfree Eating Recipe Swap:
http://groups.yahoo.com/group/Allergyfree_Eating
Ample Aussies Mailing List: http://groups.yahoo.com/group/ampleaussies/


body:
On Sat, 2002-11-16 at 01:10, Magnus Manske wrote:
> Jonathan Walther wrote:
> > 13) Similar to the question about the orphanedlinks table, why is l_from
> > a string, while l_to is an integer?
>
> I think l_from could be an integer as well. Would probably increase
> lookup speed, and save memory. Some things might get more complicated,
> though, like "what links here".

On closer inspection, I think using an integer would make whatlinkshere
*simpler*. Currently, every linking page's title has to be parsed into
namespace/title and checked for redirect status:
SELECT DISTINCT l_from FROM links WHERE l_to={$lid}
(now loop over the results with:)
SELECT cur_id,cur_is_redirect FROM cur
  WHERE cur_namespace={$ns} AND cur_title='{$t}'

If we used an integer index, I think we could do this in one step:

SELECT cur_id,cur_namespace,cur_title,cur_is_redirect
  FROM cur,links WHERE l_from=cur_id AND l_to={$lid}

Of course, I'm the guy flipping through "Learn SQL in a weekend" as I
type, so take all my SQL advice with a suitably large grain of salt. ;)

> > 22) It might be the SQL way to make a "watchlist" table, where each row
> > contains a user id, and an article id.  Or maybe this would be useless
> > in a database that doesn't have subqueries.
>
> You're right on this one. Guess noone really paid attention to that
> before...

I think I'm converted to this idea; subqueries might help for listing
talk pages of watched pages, but I think we can work around it.

The most common case for the watchlist is checking whether the current
article is in the list; this can be done a lot faster by checking two
integers than by loading up a huge string and parsing it on every page
load.

Additionally, when a page is renamed, it should stay in your watchlist
under the new name. (I hate clearing out "X moved to Y" redirects from
my watchlist...)

-- brion vibber (brion @ pobox.com)


body:
Thanks for doing this!

Two little comments:

* The Back link on every page seems redundant: every browser has a back
button, but not every browser has javascript.
* The link to www.wikipedia.org on every page should be turned into a
link to [[Wikipedia]] where the offline user can find some information
about the project and the online user can find the URL.

Axel

__________________________________________________
Do you Yahoo!?
Yahoo! Web Hosting - Let the expert host your site
http://webhosting.yahoo.com


body:
Magnus and Brion:

[I've merged both of your comments into one email.  The message is a bit
long, but it keeps it all in one place.  Note the M> and B> refer to
Magnus and Brion respectively.]

M> Cool! I like it! Hope to see the CDs at Wal-Mart soon ;-)
B> A cool beginning, thanks! :)

Thanks, I'm glad to see the idea receive such a positive response :)

M> * On every page, there should be a link to the online article, and to
M> the online edit.
B> A link to that particular page on the live server would be a *very* good
B> idea. The regular printable pages include this.

I will add links to the current online article, online edit, and offline
talk.  The offline talk page will link to the online talk and online talk
edit.  I think that provides access to the online version of everything.

B> User and talk pages are probably not necessary; if you're looking to
B> discuss the page, you'll be doing it on the live site where you can edit
B> it (and see the last 6 months' worth of edits which aren't on your
B> CD-ROM). And, of course, they take up a large chunk of valuable CD real
B> estate better devoted to future articles.

I have debated whether to include the User and Talk pages myself.  As
space on the CD gets tight they'll probably be the first things to go.
However, I noticed that articles sometimes provide a link to the
corresponding Talk page.  If the talk page was somehow relevant to the
article (which it probably shouldn't be) I wanted to make sure people had
access to it offline.

As for the User pages, I thought it was a good way to give credit to the
Wikipedia contributors.  Also, many of the Wikipedia pages and Talk pages
link to User pages where people have made comments.

M> * Images should link to the image text, like wikipedia does, not to the
M> image itself. One can save the image directly from the page.

I'm not sure what happened there, I'll fix it so it behaves the same as
Wikipedia.

M> * A nice CSS will make it look less "plain", without compromising
M> compatibility with older browsers.
B> Could probably stand to be purtied up at least a little bit.

I agree, I'll try using a different CSS to improve things.  My intention
is to keep a simple look and feel, but also to stay true to the Wikipedia
style.

M> * The CD could also have a recent Mozilla installer on it ;-)

My assumption is that everyone has a HTML browser on their machine or can
easily get one.  Maybe I didn't get your joke?

M> * There could be other index.html files with frames in them, where a
M> small frame on top, bottom, or side, could carry useful links (Main
M> Page, Copyrights, the "live" wikipedia, "live" search engine, etc.) and
M> maybe the logo.

Good idea.  Thanks for the sample you emailed me personally.  I'll
create two interfaces, one frame based and the other with just the pages.

M> * If you want statistics (date of last edit) at all, maybe add some more
M> ("this was edited 123 times").

The number of edits could be useful to a person reading the article to
determine how "mature" the article is.  I'll look into how easy it would
be to implement.

M> * Should non-existent links be displayed as links? I think so, because
M> people might feel tempted to write something ;-)

I agree that links to non-existent articles would be a good way to
encourage people to contribute, but it's at the expense of linking online
which might not be available to them.  I have tried to keep the number of
links to online sources low so that offline users aren't frustrated by
numerous broken (in their mind) links.  Maybe using the stub marker (like
an ! or *) after the topic would be best (I've seen this preference
setting somewhere).  Using an "external" CSS tag with a different color,
as is done with the online version, would also help people distinguish.

B> Some things to think about as far as the actual filenames:

Some good observations about potential filename problems.  I suppose I
could use something other than the title, like the unique cur_id integer,
but it wouldn't be as meaningful to people.  Using eight characters (26
letters and 10 numbers) there are about 2.8 trillion possible filenames,
so each article could be mapped to a unique 8 character filename.  I think
this would avoid all the length and conflict issues you've raised.  Or
maybe 64 characters would be better since some meaning could be kept in
the name.  I'll think about these issues and see what I can come up with.

CD-ROM formats such as Rock Ridge Extension and Joliet support longer
filenames (at least 64 characters), but the ISO 9660 standard only
supports the 8+3 convention.  I read that Joliet also supports Unicode
characters for international support.  Perhaps it would be enough to
ensure the filenames conform to one of these CD-ROM standards and leave
the rest up to the OS?

The other thing to note about potential filename conflicts with my current
setup is that the MD5 hash directory structure provides some safety.
Even if two filenames conflicted (through removing invalid characters)
with the 256 subdirectories their is only a 0.4% chance they'll be put in
the same directory and actually conflict.  It's not perfect, but it's not
very likely to happen.

B> Non-ascii chars appear to be left intact; will this work consistently
B> across different filesystems which may be configured for different
B> character encodings?

I don't know much about different character encodings.  I'll have to do
some research on this unless anyone here knows the answer?

M> There should also be a "real" offline search function. Shouldn't be too
M> hard to hack some smallish programs for Windows, Linux, maybe Mac.
B> A simple JavaScript-based title search could probably be rigged up out
B> of that.

I agree, there needs to be a "real" way to search the articles.  My focus
at the moment is to get the HTML and layout correct.  The JavaScript-based
search is a nice idea to provide some basic functionality.

One of my goals is to try and keep everything OS and browser independent.
This may not be possible for extra features like a search function, but I
think it should be the goal.  It's also one of my goals to try and ensure
it works on the most basic setup that libraries, schools and third-world
countries might have.

M> We might also think of adding special "view" functions to the software
M> (like the "printable version"), to aid projects like this, and maybe
M> Larry's wikipedia sifter.

By software I assume you mean the Wikipedia code base.  Doesn't the
current Skin object provide something like this already?  If you want to
view things differently you just need to implement a new skin.

> > - English only (currently)
B> That will need to be fixed, of course! :)

It's on my todo list :)  Once the English version is working, I'll try
some of the other language Wikipedias.  I'm looking forward to learning
more about different character sets and internationalization.

B> > - redirects replaced with direct link to article
B>
B> Nice.

I was really fun to implement because I got to use recursion to follow
the redirects to the final page :)  Hopefully, Wikipedia never has any
infinite redirection loops.

Thank you both Brion and Magnus for your comments.  Anyone else with some
thoughts or ideas?  I'll post my next revision when it's ready.

Best Regards,
Steve Rawlinson


from_name: Throbbing Monster Cock

subject: [Wikipedia-l] Re: vandalism spree

body:
tarquin wrote:
> seems we have multiple vandals:
>
> * TMC (though whether it's the same as the
> TMC we know of old, who knows)

I had no involvement with any edits made as "TMC".
Whatever happened on Saturday (when the above was
sent), I missed it.

__________________________________________________
Do you Yahoo!?
Yahoo! Web Hosting - Let the expert host your site
http://webhosting.yahoo.com


subject: [Wikitech-l] how many different pages are there?

body:
I mean, types of pages?  Here are the ones I've noticed.

Regular pages include articles, talk pages, and user pages.

Here are the non-regular pages I've noticed:

Blocked IP's
Recent Changes
Watch List
User Preferences Page
History (now called Older Versions)
Related Changes
What Links Here (and maybe Where This Links To)

Which ones did I miss?  I'm doing this enumeration so I can look at the
database schema and see what information is actually needed, and what
isn't, and what would allow for optimal access and use of space.

For instance, right now, the "options" field could be a simple integer
index which refers to a specific CSS file; when a user changes their
options they would get linked to a different CSS file, that is all. I
don't know, maybe that is already what happens.

Jonathan

--
                     Geek House Productions, Ltd.

  Providing Unix & Internet Contracting and Consulting,
  QA Testing, Technical Documentation, Systems Design & Implementation,
  General Programming, E-commerce, Web & Mail Services since 1998

Phone:   604-435-1205
Email:   djw at reactor-core.org
Webpage: http://reactor-core.org
Address: 2459 E 41st Ave, Vancouver, BC  V5R2W2
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 307 bytes
Desc: not available
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20021117/91d65998/attachment.pgp>


body:
Axel, thanks for the comments.

> * The Back link on every page seems redundant: every browser has a back
> button, but not every browser has javascript.

Now that I think about it some more, you're right.  The back button is
redundant and the Javascript has the potential to cause problems (or not
work if Javascript has been disabled).  I'll remove it.

> * The link to www.wikipedia.org on every page should be turned into a
> link to [[Wikipedia]] where the offline user can find some information
> about the project and the online user can find the URL.

An excellent idea.  My original reason for the link was to give people a
way to find the online Wikipedia.  A link to [[Wikipedia]] will accomplish
this and provide more information about the project as you suggest.
Also, Magnus and Brion suggested adding a links to the current online
version of the article and a link to the online edit page, which should
also help people find the online version.

Thanks,
Steve


from_name: Bridget

from_email: Bridget

subject: [Wikipedia-l] anglicization is stupid

body:
we should refer to spacecraft from the USSR according to their Russian name-not according to the English name.

---------------------------------
Do you Yahoo!?
Yahoo! Web Hosting - Let the expert host your site
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.wikimedia.org/pipermail/wikipedia-l/attachments/20021117/106aedb5/attachment.htm


from_name: Giskart

body:
Steve Rawlinson heeft geschreven:
[cut]
> Thank you both Brion and Magnus for your comments.  Anyone else with some
> thoughts or ideas?  I'll post my next revision when it's ready.

Why should someone buy or use a encyclopedia on a CD-ROM and not use the
?normal way? online?

I suppose because the do not have internet or only a very slow and/or
expencive connection.

Mayby those CD-ROM readers would also like to write or inprove articles.
It would be nice if there is a way for off-line Wikipedians to work at
wikipedia.

Idea;
someone reads a article on the Wikipedia cd-rom and wants to change
something. He clicks on ?edit this page?. He gets a form that sends a
email to Wikipedia by use of his email program.

Something like: to: offlinewiki-request at wikipedia.org
	subject: request [[Babel fish]]

He gets back a email whit instructions how to import the attachment in
the off line wikipedia. After the update of the article he can modify it
like on the on line wikipedia. When he is ready whit the article he
select ?Send update to on line Wikipedia?. The update is send back to
Wikipedia by use of his email program. It also sends the revision number
of that article. If the on line article is not changed it gets directly
on line. If it has been changed it goes to a que were a on line
Wikipedian must check it.

Or something like that, you get the general idea.
--
Contact: giskart AT wikipedia.be
Ook een artikeltje schrijven? Wikipedia, de vrije GNU/FDL encyclopedie
http://www.wikipedia.be


from_name: Derek Ross

body:
----- Original Message -----
From: "Brion Vibber" <brion at pobox.com>
To: <wikitech-l at wikipedia.org>
Sent: Sunday, November 17, 2002 11:35 PM
Subject: Re: [Wikitech-l] questions about the SQL structure

> On Sat, 2002-11-16 at 01:10, Magnus Manske wrote:
> > Jonathan Walther wrote:
> > > 13) Similar to the question about the orphanedlinks table, why is
l_from
> > > a string, while l_to is an integer?
> >
> > I think l_from could be an integer as well. Would probably increase
> > lookup speed, and save memory. Some things might get more complicated,
> > though, like "what links here".
>
> On closer inspection, I think using an integer would make whatlinkshere
> *simpler*. Currently, every linking page's title has to be parsed into
> namespace/title and checked for redirect status:
> SELECT DISTINCT l_from FROM links WHERE l_to={$lid}
> (now loop over the results with:)
> SELECT cur_id,cur_is_redirect FROM cur
>   WHERE cur_namespace={$ns} AND cur_title='{$t}'
>
> If we used an integer index, I think we could do this in one step:
>
> SELECT cur_id,cur_namespace,cur_title,cur_is_redirect
>   FROM cur,links WHERE l_from=cur_id AND l_to={$lid}
>
> Of course, I'm the guy flipping through "Learn SQL in a weekend" as I
> type, so take all my SQL advice with a suitably large grain of salt. ;)

An even better move would be to use a 64-bit number for the index rather
than an integer.  If this were to be done it would become possible to assign
article id numbers even to articles which did not exist by using a
cryptographic checksum function to create an article ID directly from the
article title text.  This then gives the simpler query benefits that
Jonathan Walter was talking about earlier without the need to actually
create a whole load of empty articles some of which will  inevitably be
orphaned before they are actually used.  It is necessary to use at least a
64-bit number because anything smaller gives too much chance of a hash
collision occurring and two articles accidentally ending up with the same
id.

By the way an SQL clause like -- SELECT a FROM b WHERE c = NULL -- should
always select zero records because comparing anything to NULL always
produces FALSE as a result.  The correct way to do this in standard SQL
s  -- SELECT a FROM b WHERE c IS NULL.

Cheers

Derek