subject: [Foundation-l] complaining to ISPs

body:
Yes... stating on a page "VERIZON IS BANNED!" for example would definitely
make them scared and start taking actions against jerks.

Developers, are you listening? Jeff's got it right!

On 8/4/06, Jeff V. Merkey <jmerkey at wolfmountaingroup.com> wrote:
>
> James Hare wrote:
>
> >Jeff Merkey -- defender of American Indians, defeater of Asian ISPs.
> >
> >Wikipedia could technically set up such an extreme system, but it would
> >cause too many people to be blocked at once, and thus less people can
> edit
> >(Jimmy-Jimmy seems to like allowing everyone to edit). Also, such high
> >levels of blocking could give us a bad name.
> >
> >But I'd implement it.
> >
> >
>
> Well, the only way to get an ISPs attention is to it them in their
> pocketbooks. If their users cannot use
> the site and are "BANNED" on a public list -- you can very well bet they
> will jump to start swatting
> these gadfly vandals and pretty quick.
>
> Jeff
>
> >
> >On 8/4/06, Jeff V. Merkey <jmerkey at wolfmountaingroup.com> wrote:
> >
> >
> >>James Hare wrote:
> >>
> >>
> >>
> >>>Oooh! War of attrition!
> >>>
> >>>James approves.
> >>>
> >>>
> >>>
> >>>
> >>I had to do the same thing to chinanet and several other asian ISPs as
> >>well -- they remain blocked -- permanently.
> >>
> >>Jeff
> >>
> >>
> >>
> >>>On 8/4/06, Jeffrey V. Merkey <jmerkey at wolfmountaingroup.com> wrote:
> >>>
> >>>
> >>>
> >>>
> >>>>geni wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>On 8/4/06, Nathan Carter <cartmanau at gmail.com> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>Is there already a section within the foundation which can deal with
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>this?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>>If not is some form of "abuse reporting" procedure needed?
> >>>>>>I think the best way to do it is for the foundation itself to do the
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>dirty
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>>work so to speak, which should avoid any privacy issues.
> >>>>>>Cheers,
> >>>>>>Nathan Carter (Cartman02au)
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>While that may be an option in the short term in the long run this is
> >>>>>likely to become more common and thus it is probably best that there
> >>>>>to be a way to deal with this below foundation level.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>Most of the ISP's ignore these reports, including DCMA takedown
> notices
> >>>>(they just throw them in the trash). One very effective
> >>>>method that both works and gets their attention I have used at
> >>>>WikiGadugi dissuades 100% of vandals, gets the ISP's attention, and
> >>>>gets the problem fixed immediately. It is as follows:
> >>>>
> >>>>1. When I first setup WikiGadugi, I had the distinct pleasure of being
> >>>>visited by WillyonWheels from the UK.
> >>>>2. I tried the whole page move vandal, blocking, blah blah blah, it
> only
> >>>>made him more persistent.
> >>>>3. I wrote a shim program into IP tables as a MediaWiki addon that
> does
> >>>>WHOIS lookup everytime
> >>>>the IP address gets used to write a page and saves not only the IP
> >>>>address for the write, but the ISP range as well.
> >>>>4. The shim checks a file not visible where I record IP addresses for
> >>>>vandal addresses and autoblocks the entire ISP IP range AT THE
> FIREWALL
> >>>>blocking both reading and writing, shutting down all access for that
> >>>>
> >>>>
> >>ISP.
> >>
> >>
> >>>>I had someone from the UK email me (my email servers are on a separate
> >>>>network) from this ISP range. It was
> >>>>blueyonder.uk and apparently, other universities were studying the
> >>>>Cherokee translation and wondered why the site was down. It was not
> long
> >>>>before the ISP notified me a certain account was "suspended".
> >>>>
> >>>>I would suggest creating a banned ISP listing with ranges based on
> >>>>persistent vandals, and when they suspend the accounts, you will
> unblock
> >>>>them
> >>>>at the firewall. It fixed my vandalism problems. I have 0% vandalism
> at
> >>>>the WikiGadugi site and I stuck it to WillyOnWheels.
> >>>>
> >>>>Jeff
> >>>>_______________________________________________
> >>>>foundation-l mailing list
> >>>>foundation-l at wikimedia.org
> >>>>http://mail.wikipedia.org/mailman/listinfo/foundation-l
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>_______________________________________________
> >>>foundation-l mailing list
> >>>foundation-l at wikimedia.org
> >>>http://mail.wikipedia.org/mailman/listinfo/foundation-l
> >>>
> >>>
> >>>
> >>>
> >>>
> >>_______________________________________________
> >>foundation-l mailing list
> >>foundation-l at wikimedia.org
> >>http://mail.wikipedia.org/mailman/listinfo/foundation-l
> >>
> >>
> >>
> >_______________________________________________
> >foundation-l mailing list
> >foundation-l at wikimedia.org
> >http://mail.wikipedia.org/mailman/listinfo/foundation-l
> >
> >
> >
>
> _______________________________________________
> foundation-l mailing list
> foundation-l at wikimedia.org
> http://mail.wikipedia.org/mailman/listinfo/foundation-l
>


list: wikimedia-l

source: pipermail-mbox

from_name: James Hare

subject: [Foundation-l] complaining to ISPs


list: commons-l

from_name: Patrick-Emil Zörner

subject: [Commons-l] Editorial use vs commercial exploitation

body:
--- Fredrik Josefsson <fred_chessplayer at yahoo.se> wrote:

> There are some images that are protected from
> "commercial exploitation", although they are not
> restricted in commercial editorial use such as
> Wikipedia, newspapers, magazines and books on the
> subject.

So you give the answer yourself. They are not 100% commercial. What is
"commercial exploitation"? Sell things to get rich. Well that is why
you sell stuff.

> If I undestand it correctly, such images are for
> example protected from being: postcards , sold
> separately and used in advertisement.

It could mean anything. That is the reason why commons wants pictures
that where we can clearly say they are free (Even though we turn a
blind eye on the GFDL ;-)

> What is the view of the Wikimedia foundation about
> such images?

It is not only a matter of the foundation but also a matter of the
community. Because if these will be allowed on commons I will stop
doing my adminstrative work there and make use of my right to leave.

regards

Paddy

___________________________________________________________
Der fr?he Vogel f?ngt den Wurm. Hier gelangen Sie zum neuen Yahoo! Mail: http://mail.yahoo.de


list: wikitech-l

from_name: Julien Lemoine

subject: [Wikitech-l] Wikipedia Suggest

body:
* Steve Bennett <stevage at gmail.com> [2006-08-03 09:17:40 +0200]:
> > I will generates a new index with redirects, I will give you the size
> > before/after this evening.
>
> Redirects should be indicated with what they redirect to, like:
>
> ...
> Top-rope climbing -> top roping
> Top roping
> ...
>
> Or something - but maybe with the proviso that it doesn't show a
> redirect if the actual target is currently being displayed.

I uploaded the index with redirections (the automaton has now a size of
200M instead of 132M). But, with the query "a", I got 10 alias of
'United States' starting with "a". I will probably force the top 10
results to have unique articles. I will change this behavior this WE.

Best Regards.
Julien Lemoine


from_name: Simetrical

subject: [Wikitech-l] Smarter name matching for templates

body:
On 8/4/06, Jay R. Ashworth <jra at baylink.com> wrote:
> My instinct is to say that's a Bad Idea, and on reflection, I know why.
>
> I can't locate it just now, but there's an Eric Allman paper that ships
> with the sendmail distribution, that explains why mailbox-name-generation
> protocols are a bad idea: because (among other things) they can
> *reveal* collisions, making things that weren't broken break as the
> environment (invisible to a mail sender) change.
>
> Case folding is inherently a wart, because (unless you use it
> *everywhere*), it can make things break. That is, if you permit pages
> whose names differ only in case, then you cannot safely automatically
> fold case *anywhere else*, because you open the possibility that, for
> example, someone will pick a template that has been included in
> hundreds of pages -- with the wrong case -- and create the template to
> which that *actually* points.

I think the request was only for when one types
Template:Somethingorother into the search bar.  When it comes to
actual transclusions ? or even links, anything that sticks around ?
then yes, I would agree.  But more intelligent search heuristics are a
different matter.


subject: [Wikitech-l] Blocked titles

body:
On 8/4/06, Magnus Manske <magnus.manske at web.de> wrote:
> Many would prefer a clean technical soution. The simple way would be to
> add a new table to the database, listing the topics blocked from
> recreation.

Why not extend the protection mechanism to be applicable to deleted
pages?  That would be the most intuitive way, to my mind.  See also
http://bugs.wikimedia.org/show_bug.cgi?id=2919.


from_name: Jay R. Ashworth

body:
On Fri, Aug 04, 2006 at 02:03:32PM -0400, Simetrical wrote:
> On 8/4/06, Jay R. Ashworth <jra at baylink.com> wrote:
> > My instinct is to say that's a Bad Idea, and on reflection, I know why.
> >
> > I can't locate it just now, but there's an Eric Allman paper that ships
> > with the sendmail distribution, that explains why mailbox-name-generation
> > protocols are a bad idea: because (among other things) they can
> > *reveal* collisions, making things that weren't broken break as the
> > environment (invisible to a mail sender) change.
> >
> > Case folding is inherently a wart, because (unless you use it
> > *everywhere*), it can make things break. That is, if you permit pages
> > whose names differ only in case, then you cannot safely automatically
> > fold case *anywhere else*, because you open the possibility that, for
> > example, someone will pick a template that has been included in
> > hundreds of pages -- with the wrong case -- and create the template to
> > which that *actually* points.
>
> I think the request was only for when one types
> Template:Somethingorother into the search bar.  When it comes to
> actual transclusions ??? or even links, anything that sticks around ???
> then yes, I would agree.  But more intelligent search heuristics are a
> different matter.

I may well have misunderstood the OP, but it sure sounded to me like
they actually meant to casefold on a {{template call}}.

Cheers,
-- jra
--
Jay R. Ashworth                                                jra at baylink.com
Designer                          Baylink                             RFC 2100
Ashworth & Associates        The Things I Think                        '87 e24
St Petersburg FL USA      http://baylink.pitas.com             +1 727 647 1274

	The Internet: We paved paradise, and put up a snarking lot.


subject: [Wikitech-l] Wikipedia Suggest

body:
Hello Nick,

* Nick Jenkins <nickpj at gmail.com> [2006-08-04 16:54:20 +1000]:

> A few more suggestions / comments / ideas after compiling it:
>
> == Small addition for README file ==
>
> Perhaps add this to the end of the README file (about lib
> requirements) - (these worked for me) :
>
> # If you're on Debian/Ubuntu, you can probably install expat & icu with:
> apt-get install libicu34 libexpat1 libexpat1-dev libicu34-dev
>
> # The compile with:
> ./configure
> make

Thanks you for your contribution, I will add it in README file (you
will also need to install cppunit-dev package for make check).

> == Pull data from MySQL? ==
>
> Currently the data used by cmd/Analyzer appears to be generated from
> the XML data file (e.g. the 1.1 gigabyte XmlEnWikipedia.tar.bz2 file).
>
> Could it be possible instead to generate the required data from a
> MySQL database? That way the whole XML step could be avoided by just
> connecting directly to the database, and generating the required data
> directly from the database.
>
> Also this way, you _may_ be able to get away with reading far less
> data, and so it could potentially be quicker. Also it would be more
> up-to-date, because there would be intermediate step to add to the
> latency.
>
> For example, this way you could get the number of links to each page
> by doing something like this:
> select pl_title, count(*) from pagelinks group by pl_title;
>
> ... and you could get the valid page names by doing something like this:
> select page_title from page where page_namespace = 0;

It is definitively possible and it will probably be the best way to
integrate it in Wikipedia. I preferred working with XML files to produce
a proof of concept because I do not know details of wikipedia database
architecture (and I have others proof of concept in mind :))

> == Maybe include JS / HTML files ==
>
> Maybe bundle the JS / HTML / PHP / Perl files that you're using at
> http://suggest.speedblue.org/ with the wikipedia-suggest.tar.gz
> archive? Basically I'd like to try and reproduce what you've got
> happening on your website so that I can play with it too ;-)

No problem, I will put them in the tarball.

> == Documentation for Gumbies ==
>
> Potentially stupid question, but how do I use the resulting executable
> files in the 'cmd' directory? I couldn't see documentation on this in
> the wikipedia-suggest.tar.gz archive (but it is version 0.1, so it's
> not unexpected). Or to put it differently, what series of commands did
> you use to get your http://suggest.speedblue.org/ site working?

The tarball contains only the analyzer/query binaries, the analyzer need
to be called with two arguments. The first one is a file containing all
xml filename to analyze (one filename per line) and the second is the
path to the redirect.xml file (more details about how to obtains this
XML and redirect.xm files are given below).

The query and tcpquery binaries load output of analyzer to perform
queries.

>
> E.g.
> wget http://www2.speedblue.org/download/XmlEnWikipedia.tar.bz2
> # How did you create XmlEnWikipedia.tar.bz2 by the way? E.g. Did it
> come from doing something to
> http://download.wikimedia.org/enwiki/20060717/enwiki-20060717-pages-articles.xml.bz2
> ?

Hum, I did not know the availability of these xml archives. In fact, I
wrote scripts to grabb the whole content of wikipedia. You will have
details about grabbing in the history section (june 2006) :
http://suggest.speedblue.org/history.php

> cmd/Analyzer XmlEnWikipedia.tar.bz2 fsa.bin page.bin
> # (i.e. Not sure what creates fsa.bin & page.bin , or how to persuade
> it to do so )

fsa.bin and page.bin are created by analyzer.

> Then presumably either cmd/Query or cmd/TcpQuery is invoked on fsa.bin
> and page.bin, and connected to somehow to query for results.
> # What's the difference between these two versions? E.g. is one the
> DiskQuery implementation, and the other the MemoryQuery
> implementation? Or is it just how you connect to them (e.g. one via
> TCP/IP, the other via some other method?)

 * cmd/query is a command line version using DiskQuery
 * cmd/TcpQuery is a TCP/IP version using DiskQuery

 * DiskQuery does not load automaton and articles in memory, it keep
only a file descriptor and perform seek (I implemented it because I have few
RAM available on my server).
 * MemoryQuery load automaton and articles in memory, it is
multi-threaded safe and you can use it to perform a lot of query per seconds.

Sorry for the lack of documentation
Best Regards.
Julien Lemoine


list: foundation-l

from_name: Jeff V. Merkey

subject: [Foundation-l] complaining to ISPs

body:
James Hare wrote:

>Oooh! War of attrition!
>
>James approves.
>
>
I had to do the same thing to chinanet and several other asian ISPs as
well -- they remain blocked -- permanently.

Jeff

>
>On 8/4/06, Jeffrey V. Merkey <jmerkey at wolfmountaingroup.com> wrote:
>
>
>>geni wrote:
>>
>>
>>
>>>On 8/4/06, Nathan Carter <cartmanau at gmail.com> wrote:
>>>
>>>
>>>
>>>
>>>>Is there already a section within the foundation which can deal with
>>>>
>>>>
>>this?
>>
>>
>>>>If not is some form of "abuse reporting" procedure needed?
>>>>I think the best way to do it is for the foundation itself to do the
>>>>
>>>>
>>dirty
>>
>>
>>>>work so to speak, which should avoid any privacy issues.
>>>>Cheers,
>>>>Nathan Carter (Cartman02au)
>>>>
>>>>
>>>>
>>>>
>>>While that may be an option in the short term in the long run this is
>>>likely to become more common and thus it is probably best that there
>>>to be a way to deal with this below foundation level.
>>>
>>>
>>>
>>>
>>>
>>Most of the ISP's ignore these reports, including DCMA takedown notices
>>(they just throw them in the trash). One very effective
>>method that both works and gets their attention I have used at
>>WikiGadugi dissuades 100% of vandals, gets the ISP's attention, and
>>gets the problem fixed immediately. It is as follows:
>>
>>1. When I first setup WikiGadugi, I had the distinct pleasure of being
>>visited by WillyonWheels from the UK.
>>2. I tried the whole page move vandal, blocking, blah blah blah, it only
>>made him more persistent.
>>3. I wrote a shim program into IP tables as a MediaWiki addon that does
>>WHOIS lookup everytime
>>the IP address gets used to write a page and saves not only the IP
>>address for the write, but the ISP range as well.
>>4. The shim checks a file not visible where I record IP addresses for
>>vandal addresses and autoblocks the entire ISP IP range AT THE FIREWALL
>>blocking both reading and writing, shutting down all access for that ISP.
>>
>>I had someone from the UK email me (my email servers are on a separate
>>network) from this ISP range. It was
>>blueyonder.uk and apparently, other universities were studying the
>>Cherokee translation and wondered why the site was down. It was not long
>>before the ISP notified me a certain account was "suspended".
>>
>>I would suggest creating a banned ISP listing with ranges based on
>>persistent vandals, and when they suspend the accounts, you will unblock
>>them
>>at the firewall. It fixed my vandalism problems. I have 0% vandalism at
>>the WikiGadugi site and I stuck it to WillyOnWheels.
>>
>>Jeff
>>_______________________________________________
>>foundation-l mailing list
>>foundation-l at wikimedia.org
>>http://mail.wikipedia.org/mailman/listinfo/foundation-l
>>
>>
>>
>_______________________________________________
>foundation-l mailing list
>foundation-l at wikimedia.org
>http://mail.wikipedia.org/mailman/listinfo/foundation-l
>
>
>


subject: [Foundation-l] complaining to ISPs


from_name: geni

subject: [Foundation-l] complaining to ISPs

body:
On 8/4/06, Nathan Carter <cartmanau at gmail.com> wrote:
> Is there already a section within the foundation which can deal with this?
> If not is some form of "abuse reporting" procedure needed?
> I think the best way to do it is for the foundation itself to do the dirty
> work so to speak, which should avoid any privacy issues.
> Cheers,
> Nathan Carter (Cartman02au)

While that may be an option in the short term in the long run this is
likely to become more common and thus it is probably best that there
to be a way to deal with this below foundation level.

--
geni


subject: [Foundation-l] complaining to ISPs


subject: [Foundation-l] complaining to ISPs

body:
James Hare wrote:

>Oooh! War of attrition!
>
>James approves.
>
>

One other thing I have done is create two server arrays -- one external
WikiGadugi network, and an internal one. The reason
"recent changes" (??? ???????) never shows activity and the site
undergoes database purges and updates is the actual
community server is behind a proxy -- people have to be given access. I
take the translated XML dumps which are redumped
from the actual community server and importDump them into the external
server for public viewing. I did this for two reasons.

1. Vandals get smashed and their bogus junk gets blasted out of the
database every two weeks.
2. My community has privacy of their editing history to the General
Public (but not to each other).

Most of the content on Wikipedia talk pages that gets scraped by Google
really does not belong in the public view, or anywhere else public.
It's public to my community, not to the outside world.

Jeff

>
>On 8/4/06, Jeffrey V. Merkey <jmerkey at wolfmountaingroup.com> wrote:
>
>
>>geni wrote:
>>
>>
>>
>>>On 8/4/06, Nathan Carter <cartmanau at gmail.com> wrote:
>>>
>>>
>>>
>>>
>>>>Is there already a section within the foundation which can deal with
>>>>
>>>>
>>this?
>>
>>
>>>>If not is some form of "abuse reporting" procedure needed?
>>>>I think the best way to do it is for the foundation itself to do the
>>>>
>>>>
>>dirty
>>
>>
>>>>work so to speak, which should avoid any privacy issues.
>>>>Cheers,
>>>>Nathan Carter (Cartman02au)
>>>>
>>>>
>>>>
>>>>
>>>While that may be an option in the short term in the long run this is
>>>likely to become more common and thus it is probably best that there
>>>to be a way to deal with this below foundation level.
>>>
>>>
>>>
>>>
>>>
>>Most of the ISP's ignore these reports, including DCMA takedown notices
>>(they just throw them in the trash). One very effective
>>method that both works and gets their attention I have used at
>>WikiGadugi dissuades 100% of vandals, gets the ISP's attention, and
>>gets the problem fixed immediately. It is as follows:
>>
>>1. When I first setup WikiGadugi, I had the distinct pleasure of being
>>visited by WillyonWheels from the UK.
>>2. I tried the whole page move vandal, blocking, blah blah blah, it only
>>made him more persistent.
>>3. I wrote a shim program into IP tables as a MediaWiki addon that does
>>WHOIS lookup everytime
>>the IP address gets used to write a page and saves not only the IP
>>address for the write, but the ISP range as well.
>>4. The shim checks a file not visible where I record IP addresses for
>>vandal addresses and autoblocks the entire ISP IP range AT THE FIREWALL
>>blocking both reading and writing, shutting down all access for that ISP.
>>
>>I had someone from the UK email me (my email servers are on a separate
>>network) from this ISP range. It was
>>blueyonder.uk and apparently, other universities were studying the
>>Cherokee translation and wondered why the site was down. It was not long
>>before the ISP notified me a certain account was "suspended".
>>
>>I would suggest creating a banned ISP listing with ranges based on
>>persistent vandals, and when they suspend the accounts, you will unblock
>>them
>>at the firewall. It fixed my vandalism problems. I have 0% vandalism at
>>the WikiGadugi site and I stuck it to WillyOnWheels.
>>
>>Jeff
>>_______________________________________________
>>foundation-l mailing list
>>foundation-l at wikimedia.org
>>http://mail.wikipedia.org/mailman/listinfo/foundation-l
>>
>>
>>
>_______________________________________________
>foundation-l mailing list
>foundation-l at wikimedia.org
>http://mail.wikipedia.org/mailman/listinfo/foundation-l
>
>
>


subject: [Foundation-l] complaining to ISPs


subject: [Foundation-l] complaining to ISPs

body:
James Hare wrote:

>Jeff Merkey -- defender of American Indians, defeater of Asian ISPs.
>
>Wikipedia could technically set up such an extreme system, but it would
>cause too many people to be blocked at once, and thus less people can edit
>(Jimmy-Jimmy seems to like allowing everyone to edit). Also, such high
>levels of blocking could give us a bad name.
>
>But I'd implement it.
>
>

Well, the only way to get an ISPs attention is to it them in their
pocketbooks. If their users cannot use
the site and are "BANNED" on a public list -- you can very well bet they
will jump to start swatting
these gadfly vandals and pretty quick.

Jeff

>
>On 8/4/06, Jeff V. Merkey <jmerkey at wolfmountaingroup.com> wrote:
>
>
>>James Hare wrote:
>>
>>
>>
>>>Oooh! War of attrition!
>>>
>>>James approves.
>>>
>>>
>>>
>>>
>>I had to do the same thing to chinanet and several other asian ISPs as
>>well -- they remain blocked -- permanently.
>>
>>Jeff
>>
>>
>>
>>>On 8/4/06, Jeffrey V. Merkey <jmerkey at wolfmountaingroup.com> wrote:
>>>
>>>
>>>
>>>
>>>>geni wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>On 8/4/06, Nathan Carter <cartmanau at gmail.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>Is there already a section within the foundation which can deal with
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>this?
>>>>
>>>>
>>>>
>>>>
>>>>>>If not is some form of "abuse reporting" procedure needed?
>>>>>>I think the best way to do it is for the foundation itself to do the
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>dirty
>>>>
>>>>
>>>>
>>>>
>>>>>>work so to speak, which should avoid any privacy issues.
>>>>>>Cheers,
>>>>>>Nathan Carter (Cartman02au)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>While that may be an option in the short term in the long run this is
>>>>>likely to become more common and thus it is probably best that there
>>>>>to be a way to deal with this below foundation level.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>Most of the ISP's ignore these reports, including DCMA takedown notices
>>>>(they just throw them in the trash). One very effective
>>>>method that both works and gets their attention I have used at
>>>>WikiGadugi dissuades 100% of vandals, gets the ISP's attention, and
>>>>gets the problem fixed immediately. It is as follows:
>>>>
>>>>1. When I first setup WikiGadugi, I had the distinct pleasure of being
>>>>visited by WillyonWheels from the UK.
>>>>2. I tried the whole page move vandal, blocking, blah blah blah, it only
>>>>made him more persistent.
>>>>3. I wrote a shim program into IP tables as a MediaWiki addon that does
>>>>WHOIS lookup everytime
>>>>the IP address gets used to write a page and saves not only the IP
>>>>address for the write, but the ISP range as well.
>>>>4. The shim checks a file not visible where I record IP addresses for
>>>>vandal addresses and autoblocks the entire ISP IP range AT THE FIREWALL
>>>>blocking both reading and writing, shutting down all access for that
>>>>
>>>>
>>ISP.
>>
>>
>>>>I had someone from the UK email me (my email servers are on a separate
>>>>network) from this ISP range. It was
>>>>blueyonder.uk and apparently, other universities were studying the
>>>>Cherokee translation and wondered why the site was down. It was not long
>>>>before the ISP notified me a certain account was "suspended".
>>>>
>>>>I would suggest creating a banned ISP listing with ranges based on
>>>>persistent vandals, and when they suspend the accounts, you will unblock
>>>>them
>>>>at the firewall. It fixed my vandalism problems. I have 0% vandalism at
>>>>the WikiGadugi site and I stuck it to WillyOnWheels.
>>>>
>>>>Jeff
>>>>_______________________________________________
>>>>foundation-l mailing list
>>>>foundation-l at wikimedia.org
>>>>http://mail.wikipedia.org/mailman/listinfo/foundation-l
>>>>
>>>>
>>>>
>>>>
>>>>
>>>_______________________________________________
>>>foundation-l mailing list
>>>foundation-l at wikimedia.org
>>>http://mail.wikipedia.org/mailman/listinfo/foundation-l
>>>
>>>
>>>
>>>
>>>
>>_______________________________________________
>>foundation-l mailing list
>>foundation-l at wikimedia.org
>>http://mail.wikipedia.org/mailman/listinfo/foundation-l
>>
>>
>>
>_______________________________________________
>foundation-l mailing list
>foundation-l at wikimedia.org
>http://mail.wikipedia.org/mailman/listinfo/foundation-l
>
>
>


subject: [Foundation-l] complaining to ISPs


from_name: Aerik Sylvan

subject: [Wikitech-l] Wikipedia Suggest

body:
Julien Lemoine wrote:

> With this solution, you will not be able to handle a lot of queries per
> second (it will probably take more than 1 second to query 2.3 millions
> of entries). If you use a trie (yes I know this word now :)), everything
> is pre-computed and a query will use very few cpu and you will be able
> to handle a lot of queries per second.

:-)

I'm sorry I hadn't taken the time to read about how you are doing it - I
just looked up what a "trie" is, and I see why it's a hurdle - the whole
structure is based on the first x characters.  It sounds like you are using
a file to store the trie datastructure, and not using SQL at all?  I've got
to say that I only know basic SQL, PHP, and some C - I don't have a
background to speak intelligently about optimising code for zillions of
hits, but I would approach this by taking all the article titles, and
creating an index with every significant word in the title.  After that, I'm
afraid I'd have to use SQL, and likely fall back onto using "LIKE" somewhere
if I was going to return partial-word matches (an "obvious and naive"
solution).  For whole word matches (but matching multiple words, or words
that are not the first word in the page title) you could query the index
directly.  I suppose one could build an index of partial words... would that
be faster than using "LIKE"?  Hmm...

I'm throwing queries at my category_links table that match category_a OR
category_b, then group by page and count results to perform a category math
/ category intersections function.  It seems pretty fast, but I'm not using
ajax, it's just a posted form.  It would be fun to write an ajax
implementation though :-)

I have always assumed that MySQL is internally optimised enough that if one
sticks to simple queries and whole word matches, you get pretty good
performance - but that's an assumption on my part.  I'm very interested in
that because I'd like to know more about optimising search queries, but of
course it's not your problem to teach me.

Best Regards and good luck with your project.

And Timwi wrote:

> >I set mine up to require at least 4 characters and to
> > to break on whitespace, so the SQL (pseudocode) is WHERE LIKE(wordone)
> AND
> > LIKE(wordtwo) and so on.
>
> This is, of course, the most obvious and most naive solution. For a data
> set as large as Wikipedia's list of article titles, this is far too slow
> and inefficient and would kill the site for everyone if it was used on
> the live database.
>
>
Timwi, from your other posts I can see that you know a lot more about
computer programming than I do, but I would hope to be able to offer some
opinion without getting slapped with "obvious and naive".

Regards,
Aerik


subject: [Foundation-l] complaining to ISPs

body:
James Hare wrote:

>Jeff Merkey -- defender of American Indians, defeater of Asian ISPs.
>
>Wikipedia could technically set up such an extreme system, but it would
>cause too many people to be blocked at once, and thus less people can edit
>(Jimmy-Jimmy seems to like allowing everyone to edit). Also, such high
>levels of blocking could give us a bad name.
>
>But I'd implement it.
>
>

I'll post the extension code at wikimedia.org. I am working on shipment
of several Wikipedia appliances
to one of our first customers and some solera shipment this afternoon
:-) :-) :-)

I'll cleanup the code this evening and get it posted over the weekend.

:-)

Jeff

>
>On 8/4/06, Jeff V. Merkey <jmerkey at wolfmountaingroup.com> wrote:
>
>
>>James Hare wrote:
>>
>>
>>
>>>Oooh! War of attrition!
>>>
>>>James approves.
>>>
>>>
>>>
>>>
>>I had to do the same thing to chinanet and several other asian ISPs as
>>well -- they remain blocked -- permanently.
>>
>>Jeff
>>
>>
>>
>>>On 8/4/06, Jeffrey V. Merkey <jmerkey at wolfmountaingroup.com> wrote:
>>>
>>>
>>>
>>>
>>>>geni wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>On 8/4/06, Nathan Carter <cartmanau at gmail.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>Is there already a section within the foundation which can deal with
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>this?
>>>>
>>>>
>>>>
>>>>
>>>>>>If not is some form of "abuse reporting" procedure needed?
>>>>>>I think the best way to do it is for the foundation itself to do the
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>dirty
>>>>
>>>>
>>>>
>>>>
>>>>>>work so to speak, which should avoid any privacy issues.
>>>>>>Cheers,
>>>>>>Nathan Carter (Cartman02au)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>While that may be an option in the short term in the long run this is
>>>>>likely to become more common and thus it is probably best that there
>>>>>to be a way to deal with this below foundation level.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>Most of the ISP's ignore these reports, including DCMA takedown notices
>>>>(they just throw them in the trash). One very effective
>>>>method that both works and gets their attention I have used at
>>>>WikiGadugi dissuades 100% of vandals, gets the ISP's attention, and
>>>>gets the problem fixed immediately. It is as follows:
>>>>
>>>>1. When I first setup WikiGadugi, I had the distinct pleasure of being
>>>>visited by WillyonWheels from the UK.
>>>>2. I tried the whole page move vandal, blocking, blah blah blah, it only
>>>>made him more persistent.
>>>>3. I wrote a shim program into IP tables as a MediaWiki addon that does
>>>>WHOIS lookup everytime
>>>>the IP address gets used to write a page and saves not only the IP
>>>>address for the write, but the ISP range as well.
>>>>4. The shim checks a file not visible where I record IP addresses for
>>>>vandal addresses and autoblocks the entire ISP IP range AT THE FIREWALL
>>>>blocking both reading and writing, shutting down all access for that
>>>>
>>>>
>>ISP.
>>
>>
>>>>I had someone from the UK email me (my email servers are on a separate
>>>>network) from this ISP range. It was
>>>>blueyonder.uk and apparently, other universities were studying the
>>>>Cherokee translation and wondered why the site was down. It was not long
>>>>before the ISP notified me a certain account was "suspended".
>>>>
>>>>I would suggest creating a banned ISP listing with ranges based on
>>>>persistent vandals, and when they suspend the accounts, you will unblock
>>>>them
>>>>at the firewall. It fixed my vandalism problems. I have 0% vandalism at
>>>>the WikiGadugi site and I stuck it to WillyOnWheels.
>>>>
>>>>Jeff
>>>>_______________________________________________
>>>>foundation-l mailing list
>>>>foundation-l at wikimedia.org
>>>>http://mail.wikipedia.org/mailman/listinfo/foundation-l
>>>>
>>>>
>>>>
>>>>
>>>>
>>>_______________________________________________
>>>foundation-l mailing list
>>>foundation-l at wikimedia.org
>>>http://mail.wikipedia.org/mailman/listinfo/foundation-l
>>>
>>>
>>>
>>>
>>>
>>_______________________________________________
>>foundation-l mailing list
>>foundation-l at wikimedia.org
>>http://mail.wikipedia.org/mailman/listinfo/foundation-l
>>
>>
>>
>_______________________________________________
>foundation-l mailing list
>foundation-l at wikimedia.org
>http://mail.wikipedia.org/mailman/listinfo/foundation-l
>
>
>


subject: [Foundation-l] complaining to ISPs


from_name: Daniel Bregman

subject: [Foundation-l] Logo Discussion/Competitions

body:
After the brouhaha on the Incubator logo, I feel that the Foundation
should be well notified of the two other logo discussions currently
taking place at Meta.

Wiktionary: http://meta.wikimedia.org/wiki/Wiktionary/logo

Wikibooks: http://meta.wikimedia.org/wiki/Wikibooks/logo

It would be good if those who know of such matters could steer
discussion away from logos which might break rules.

Thanks,

Daniel -http://meta.wikimedia.org/wiki/User:Dbmag9


list: wikien-l

from_name: Delirium

subject: [WikiEN-l] Ok, so about that ISBN template...

body:
Steve Bennett wrote:

>Here's the deal. I created this ISBN template {{ISBN}} which does
>nothing very exciting. It simply substitutes the text "ISBN {{{1}}}".
>However, my reasoning for creating it seemed sound: to provide an
>abstraction layer so users don't have to learn the rather arbitrary
>syntax that Wikipedia uses to deal with ISBN codes. Instead, they can
>use the same syntax they use everywhere: {{template name|argument}}.
>
>
Sounds sensible to me.  "Magic" syntax where seemingly normal-looking
text gets transformed is evil, because it's unclear what's going on, and
prone to breakage.  MediaWiki took a great leap forward by moving from
magical CamelCase links to [[marked-up links]], for example.  Just about
the only necessary evil I can think of is the auto-date-formatting;
dates are too common to really be done any other way, although it too is
slightly evil and sometimes kicks in or doesn't kick in in unexpected ways.

-Mark


from_name: Andrew Gray

subject: [WikiEN-l] How to find an old TfD discussion?

body:
On 04/08/06, Steve Bennett <stevagewp at gmail.com> wrote:
> Hello,
>   I gather that at some stage a template that I had created, {{ISBN}}
> has been deleted. I'd very curious to read what the reasoning was. I'd
> also be curious to know why no one told me about it ;)
>
> Anyone know how I can find the discussion in the archives? I can't see
> a way other than systematically plodding through all the archives for
> each day and hoping to stumble on it. But since I can only guess it
> was sometime in the last month or two. Can't find it with Wikipedia's
> search, can't find it with google...any ideas?
>
> I'm presuming that old TfD archives are actually retained...?

http://en.wikipedia.org/wiki/Special:Whatlinkshere/Template:ISBN

gives us

http://en.wikipedia.org/wiki/Wikipedia:Templates_for_deletion/Log/2006_July_9

It's also in

http://en.wikipedia.org/wiki/Wikipedia:Templates_for_deletion/Log/Deleted/October_2005#Template:ISBN

and

http://en.wikipedia.org/wiki/Wikipedia:Templates_for_deletion/Log/Deleted/May_2005#Template:ISBN

(the May one seems to have been "this needs an ISBN!", the later two
just conventional ISBN links)

--
- Andrew Gray
  andrew.gray at dunelm.org.uk


body:
Steve Bennett wrote:

>Which auto date formatting do you mean, btw? I see these constant
>references to it when people link stuff like [[August 8]], but we
>don't seem to have any settings that return a nice result like "8th of
>August", which is what my personal preference would be...
>
That's precisely what it's for, although ordinal dates aren't currently
an option.  Go to preferences->date&time.  For year-less dates, it'll
auto-convert between [[October 28]] and [[28 October]], and for
year-havin' dates it'll auto-convert between any of these 5:
[[October 28]], [[1940]]
[[28 October]] [[1940]]
[[1940]] [[October 28]]
[[1940]]-[[10-28]]
[[1940-10-28]]

The two most common seem to be the top two, with Americans preferring
the first and many Europeans preferring the second.  I suppose another
format that was the same as the 2nd but with ordinals would please some
folks as well, but perhaps it's somewhat harder to implement.

-Mark


from_name: Raphael Wiegand

subject: [Wikitech-l] Blocked titles

body:
Steve Bennett schrieb:

>On 8/4/06, Magnus Manske <magnus.manske at web.de> wrote:
>
>
>>Currently, this is done by
>>putting a template on it and protecting the page.
>>
>>
>
>This has the advantage of allowing the user to see a human-written
>explanation of why that particular topic is banned. It would be good
>if any technical solution can maintain that property.
>
>
No; at the moment the template just says that "this page has been
deleted according to the (speedy) deletion guidelines for several times
in the past. In order to prevent further attempts in creating this
article, it has been blocked" or something like that (full text: [1]).
This explanation may be an explanation for what has happened with this
article and it may be human-written, but I don't think it will really
help the user (or reader) who coincidentally comes about it. It would be
good, if any technical solution for this problem gave the blocking
administrator the possibility to enter a reason for the block and if
this reason could be displayed directly on the page (i.e. instead of
{{MediaWiki_Noarticletext_NS_{{NAMESPACE}}}} on non-existing pages some
kind of {{MediaWiki_Noarticletext_blocked}} for blocked non-existing
pages where the comment can be seen - dunno, but it surely will be too
complicated/expensive regarding ressources ;-)

rdb

[1] http://de.wikipedia.org/wiki/Vorlage:Gesperrtes_Lemma


from_name: Steve Bennett

body:
Hello,
  I gather that at some stage a template that I had created, {{ISBN}}
has been deleted. I'd very curious to read what the reasoning was. I'd
also be curious to know why no one told me about it ;)

Anyone know how I can find the discussion in the archives? I can't see
a way other than systematically plodding through all the archives for
each day and hoping to stumble on it. But since I can only guess it
was sometime in the last month or two. Can't find it with Wikipedia's
search, can't find it with google...any ideas?

I'm presuming that old TfD archives are actually retained...?

Steve


from_name: Nick Jenkins

subject: [Wikitech-l] Wikipedia Suggest

body:
Hi Julien,

> > == UI stuff ==
> >
> > The only two constructive suggestions for the User-Interface that I
> > would make are:
>
> These two features are available now. You can use them :)

Excellent! Thank you, they work great.

A few more suggestions / comments / ideas after compiling it:

== Small addition for README file ==

Perhaps add this to the end of the README file (about lib
requirements) - (these worked for me) :

# If you're on Debian/Ubuntu, you can probably install expat & icu with:
apt-get install libicu34 libexpat1 libexpat1-dev libicu34-dev

# The compile with:
./configure
make

== Pull data from MySQL? ==

Currently the data used by cmd/Analyzer appears to be generated from
the XML data file (e.g. the 1.1 gigabyte XmlEnWikipedia.tar.bz2 file).

Could it be possible instead to generate the required data from a
MySQL database? That way the whole XML step could be avoided by just
connecting directly to the database, and generating the required data
directly from the database.

Also this way, you _may_ be able to get away with reading far less
data, and so it could potentially be quicker. Also it would be more
up-to-date, because there would be intermediate step to add to the
latency.

For example, this way you could get the number of links to each page
by doing something like this:
select pl_title, count(*) from pagelinks group by pl_title;

... and you could get the valid page names by doing something like this:
select page_title from page where page_namespace = 0;

== Maybe include JS / HTML files ==

Maybe bundle the JS / HTML / PHP / Perl files that you're using at
http://suggest.speedblue.org/ with the wikipedia-suggest.tar.gz
archive? Basically I'd like to try and reproduce what you've got
happening on your website so that I can play with it too ;-)

== Documentation for Gumbies ==

Potentially stupid question, but how do I use the resulting executable
files in the 'cmd' directory? I couldn't see documentation on this in
the wikipedia-suggest.tar.gz archive (but it is version 0.1, so it's
not unexpected). Or to put it differently, what series of commands did
you use to get your http://suggest.speedblue.org/ site working?

E.g.
wget http://www2.speedblue.org/download/XmlEnWikipedia.tar.bz2
# How did you create XmlEnWikipedia.tar.bz2 by the way? E.g. Did it
come from doing something to
http://download.wikimedia.org/enwiki/20060717/enwiki-20060717-pages-articles.xml.bz2
?

cmd/Analyzer XmlEnWikipedia.tar.bz2 fsa.bin page.bin
# (i.e. Not sure what creates fsa.bin & page.bin , or how to persuade
it to do so )

Then presumably either cmd/Query or cmd/TcpQuery is invoked on fsa.bin
and page.bin, and connected to somehow to query for results.
# What's the difference between these two versions? E.g. is one the
DiskQuery implementation, and the other the MemoryQuery
implementation? Or is it just how you connect to them (e.g. one via
TCP/IP, the other via some other method?)

Sorry to ask so many questions!

All the best,
Nick.


subject: [Foundation-l] complaining to ISPs

body:
Jack wrote:

>>I would suggest creating a banned ISP listing with ranges based on
>>persistent vandals, and when they suspend the accounts, you will unblock
>>them
>>at the firewall. It fixed my vandalism problems. I have 0% vandalism at
>>the WikiGadugi site and I stuck it to WillyOnWheels.
>>
>>Jeff
>>
>>
>
>Jeff, you shoulda made this a board election issue - you could
>probably have gotten a ton of votes from all the good editors the
>vandals are slowly driving off.
>
>
I think the other candidates can certainly use the same platform if they
wish and I'll throw the code
over the wall to the Foundation.

I decided not to run for the position because,

A) I don't have time to travel all over the place at my own expense and
run three companies too, plus WikiGadugi.
B) There are some folks in the community who would make a public stink
and Jimbo's life mierable were I to be elected/appointed,
and fracture and polarize the community (not that big a concern but not
a good starting point).
C) The Bylaws setup a 3-2 voting structure where the other board members
are always going to get overruled on any vote that
is not 5-0.
D) The B&O Insurance won't cover any claims for the foundation board of
trustees and I would have to get a policy from a group that handles
non-profits like the foundation.
E) I have too many irons in the fire as it is, and the WikiGadugi
project and my other interests will suffer.
F)  I am an excellent public speaker and look good in a suit, but a
really good PR person is what's needed to be successful in this role
who is young and vibrant to give it an excellent public image, not an
old Cherokee Indian Grandfather.

:-)

Jeff


subject: [Foundation-l] complaining to ISPs


from_name: Jennifer Hays

subject: [Wikitech-l] Alter Wiki layout in "edit mode"

body:
Hi all!

I am currently developing an extension for MediaWiki 1.6.7 where I want to
add a textarea to a page in edit mode right next to the summary field. I had
no problems creating a textarea but I don't know how I can show this
textarea only in "edit mode" next to the "Summary" textfield.

I also wanted to make a "select distinct" query to the database but I
haven't been able to find the right function or how to do that.

 It would be great is someone could help me out with that.

Thank you.

Jennifer


subject: [Foundation-l] complaining to ISPs

body:
On 8/4/06, Fred Bauder <fredbaud at ctelco.net> wrote:
> Assuming this is an English Wikipedia thing, this could become an
> arbitration committee matter.
>
> Fred

So arbcom should handle complaints to ISPs? Should I forward the email to you?

--
geni


subject: [Foundation-l] complaining to ISPs


subject: [Foundation-l] complaining to ISPs

body:
Jeff V. Merkey wrote:

>Jack wrote:
>
>
>
>>>I would suggest creating a banned ISP listing with ranges based on
>>>persistent vandals, and when they suspend the accounts, you will unblock
>>>them
>>>at the firewall. It fixed my vandalism problems. I have 0% vandalism at
>>>the WikiGadugi site and I stuck it to WillyOnWheels.
>>>
>>>Jeff
>>>
>>>
>>>
>>>
>>Jeff, you shoulda made this a board election issue - you could
>>probably have gotten a ton of votes from all the good editors the
>>vandals are slowly driving off.
>>
>>
>>
>>
>I think the other candidates can certainly use the same platform if they
>wish and I'll throw the code
>over the wall to the Foundation.
>
>I decided not to run for the position because,
>
>A) I don't have time to travel all over the place at my own expense and
>run three companies too, plus WikiGadugi.
>B) There are some folks in the community who would make a public stink
>and Jimbo's life mierable were I to be elected/appointed,
>and fracture and polarize the community (not that big a concern but not
>a good starting point).
>C) The Bylaws setup a 3-2 voting structure where the other board members
>are always going to get overruled on any vote that
>is not 5-0.
>D) The B&O Insurance won't cover any claims for the foundation board of
>trustees and I would have to get a policy from a group that handles
>non-profits like the foundation.
>E) I have too many irons in the fire as it is, and the WikiGadugi
>project and my other interests will suffer.
>F)  I am an excellent public speaker and look good in a suit, but a
>really good PR person is what's needed to be successful in this role
>who is young and vibrant to give it an excellent public image, not an
>old Cherokee Indian Grandfather.
>
>:-)
>
>Jeff
>
>
>_______________________________________________
>foundation-l mailing list
>foundation-l at wikimedia.org
>http://mail.wikipedia.org/mailman/listinfo/foundation-l
>
>
>
If they wanted me to do something like "project coordinator Native
American Languages Program",
I would do that and travel around and visit the tribes and promote
Wikipedia and that would be something that
would dovetail nicely, I do not think I can or should be on their Board.
I sit on too many Boards already
and when you are on more than three, your B&O insurance goes through the
roof ....

Jeff


subject: [Foundation-l] complaining to ISPs


from_name: Timwi

subject: [Wikitech-l] Extension Installer

body:
Mark Bell wrote:
> Folks,
>
> This may be answered somewhere but i have not found a reference.
>
> I would like stats or a graph that shows:
>
> Number of unique editors per article X number of articles with that many
> unique editors.
>
> So for example
>
> number of unique editors        Number of articles with that many unique
> editors
> 15                                               356
> 30                                               3455

MAYBE this will work (haven't tried it):

     SELECT editorcount as 'Number of unique editors', COUNT(rev_page)
         as 'Number of articles with that many unique editors'
     FROM (
         SELECT rev_page, COUNT(DISTINCT rev_user_text) as editorcount
         FROM revisions GROUP BY rev_page
     ) AS editorcounts
     GROUP BY editorcount
     ORDER BY 'Number of unique editors';

If this doesn't work (maybe MySQL doesn't support this particular way of
nesting queries), then I don't think you can do it in a single SQL
query, but here's how you could do it in several:

First, create a new table:

     CREATE TABLE editorcounts (
         page_id     int not null,
         editorcount int not null
     );

Now fill the table with data that tells you how many unique editors each
article has:

     INSERT INTO editorcounts (page_id, count)
         SELECT rev_page, COUNT(DISTINCT rev_user_text)
         FROM revisions GROUP BY rev_page;

And finally, query the new table for the data you actually want:

     SELECT editorcount as 'Number of unique editors', COUNT(page_id)
         as 'Number of articles with that many unique editors'
     FROM editorcounts
     GROUP BY editorcount ORDER BY editorcount;

You may have to create indexes to speed up this process. You may also
have to correct syntax errors if I made any, because I haven't tested
this :-)

Timwi


subject: [Wikitech-l] Extension Installer

body:
> MAYBE this will work (haven't tried it):
>
>      SELECT editorcount as 'Number of unique editors', COUNT(rev_page)
>          as 'Number of articles with that many unique editors'
>      FROM (
>          SELECT rev_page, COUNT(DISTINCT rev_user_text) as editorcount
>          FROM revisions GROUP BY rev_page
>      ) AS editorcounts
>      GROUP BY editorcount
>      ORDER BY 'Number of unique editors';

I've just tested this, and it works perfectly, except that "revisions"
should have been "revision". :-)

Timwi


subject: [Wikitech-l] Wikipedia Suggest

body:
On 8/4/06, Aerik Sylvan <aerik at thesylvans.com> wrote:
> I have always assumed that MySQL is internally optimised enough that if one
> sticks to simple queries and whole word matches, you get pretty good
> performance - but that's an assumption on my part.  I'm very interested in
> that because I'd like to know more about optimising search queries, but of
> course it's not your problem to teach me.

MySQL can only be as optimized as the algorithm that uses it.  LIKE
may be optimized, but it's impossible to make it very efficient given
what it does.

> Timwi, from your other posts I can see that you know a lot more about
> computer programming than I do, but I would hope to be able to offer some
> opinion without getting slapped with "obvious and naive".

I don't think he meant it as an insult.  In computer science, "naive"
basically means "the most obvious way to accomplish something, which
is probably not a good idea performance-wise".


from_name: Raimond Spekking

subject: [Wikitech-l] Blocked titles

body:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Magnus Manske schrieb:
> On de.wikipedia, we were discussing the topic of blocked titles, meaning
> topics that otherwise would go through a cycle of deletion and
> re-creation by trolls/vandals/fanatics. Currently, this is done by
> putting a template on it and protecting the page.
>
> This is rather ugly and has certain side effects, e.g., filling up the
> "short pages" list.

Maybe an option to filter all blocked pages from the shortpages-list? In
this case the software has not to look into the source so I assume a
filter is not so expensive?

Raymond.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFE03QAp+qUcbEr56YRAjLvAJ4rCx98OoXxmXxB7WowrXCxZxT6qACghz23
64BFEHriUaG1hHywSW/KYMg=
=NhnA
-----END PGP SIGNATURE-----


subject: [Wikitech-l] Extension Installer

references:
<44CF528C.8040905@brightbyte.de>	<732865eb0608031853s5cb06aa9y8ac0e64f09873bbd@mail.gmail.com>	<eavtra$aib$1@sea.gmane.org>
	<eavubg$c93$1@sea.gmane.org>
	<732865eb0608041001j8095287j91aef4f5bbe53ca0@mail.gmail.com>

body:
Mark Bell wrote:
> any chance you can send me the data?

What data? I don't have a copy of the Wikipedia, if that's what you
mean. I only tested this on another MediaWiki installation which has
only a few hundred pages.

Maybe someone reading this here will be generous enough to run this
little query on the toolserver (which I don't have access to)?

Timwi


subject: [Wikitech-l] Wikipedia Suggest

body:
Aerik Sylvan wrote:
>
> I wrote a "suggest" function for finding part numbers for a request form at
> work - enabling multi-word search is really useful and powerful.  I haven't
> looked at your code, so I don't know if you're using a keyword index, or
> LIKE SQL functions.  I set mine up to require at least 4 characters and to
> to break on whitespace, so the SQL (pseudocode) is WHERE LIKE(wordone) AND
> LIKE(wordtwo) and so on.

This is, of course, the most obvious and most naive solution. For a data
set as large as Wikipedia's list of article titles, this is far too slow
and inefficient and would kill the site for everyone if it was used on
the live database.

Timwi


from_name: Platonides

subject: [Wikitech-l] Wikipedia Suggest

body:
Hardy is probably already redirected to Laurel And Hardy, i guess...
Better keep it easy.

"Jay R. Ashworth" wrote:
> On Fri, Aug 04, 2006 at 08:03:22AM +0200, Julien Lemoine wrote:
>> Steve Bennett wrote:
>> > Just one more request: what about if the search string matches
>> > *within* the phrase, but not necessarily at the start. Like "Hardy"
>> > matching "Laurel and Hardy"? Is that possible with this data
>> > structure?
>> >
>> No it will not. But I don't think this will be really usefull :
>> Imagine a query with a single letter ("a" or "e" for example), it will
>> matches a least half articles without understanding the results.
>
> It would require you to apply some heuristics to determine what *not*
> to display, certainly...
>
> but let me note that the incremental search feature in the phone book
> of my Blackberry -- which displays a list of first, the entries whose
> titles *start* with the substring, and then, the entries which contain
> it elsewhere -- is pretty useful.  It interacts with the height of the
> pulldown, of course, but it is pretty useful.
>
> Cheers,
> -- jra


from_name: John Lyden

subject: [WikiEN-l] GFDL compliance

body:
On 8/4/06, Guy Chapman aka JzG <guy.chapman at spamcop.net> wrote:
> There is a picture on another project which would be good in an
> article on en:.  Only problem is, the other project is
> Uncyclopaedia...

Now that's irony.

--
John Lyden - rasputinaxp at gmail.com
"The only people for me are the mad ones, the ones who are mad to
live, mad to talk, mad to be saved, desirous of everything at the same
time..." -Kerouac


from_name: Guy Chapman aka JzG

subject: [WikiEN-l] GFDL compliance

body:
There is a picture on another project which would be good in an
article on en:.  Only problem is, the other project is
Uncyclopaedia...

I'm guessing the licensing is incompatible, and i can't trace the
original source.  Bummer, really.

Guy (JzG)
--
http://www.chapmancentral.co.uk
http://en.wikipedia.org/wiki/User:JzG


body:
On 8/4/06, Jay R. Ashworth <jra at baylink.com> wrote:
> I may well have misunderstood the OP, but it sure sounded to me like
> they actually meant to casefold on a {{template call}}.

I definitely could have been clearer - I'm only talking about the
search bar. Similarly, MediaWIki's "case folding" (nice term) only
takes place at the search level. Folding at the {{template call}}
level would certainly be risky, as when one day someone actually
implements {{template-call}}, all hell could break loose.

It wouldn't even have to be "folding" as such. It would still be
beneficial just to alert the user "Page 'foo boo moo' doesn't exist,
perhaps you meant 'foo-boo-moo'?".

I generally agree with the concerns about collapsing multiple inputs
into one output.

Steve


subject: [Wikitech-l] Wikipedia Suggest

body:
On 8/4/06, Timwi <timwi at gmx.net> wrote:
> This is, of course, the most obvious and most naive solution. For a data
> set as large as Wikipedia's list of article titles, this is far too slow
> and inefficient and would kill the site for everyone if it was used on
> the live database.

Thus far, I don't think anyone has actually suggested using the
suggest mechanism on the "live database".

Steve


body:
On 8/4/06, Andrew Gray <shimgray at gmail.com> wrote:
> http://en.wikipedia.org/wiki/Special:Whatlinkshere/Template:ISBN

Good thinking - I didn't think of that. Likewise for Steve Summit's solution.

Thanks very much!

Steve


from_name: maru dubshinki

body:
On 8/4/06, Steve Bennett <stevagewp at gmail.com> wrote:
> Now that I've read the reasoning
> (http://en.wikipedia.org/wiki/Wikipedia:Templates_for_deletion/Log/2006_July_9)...
>
> Here's the deal. I created this ISBN template {{ISBN}} which does
> nothing very exciting. It simply substitutes the text "ISBN {{{1}}}".
> However, my reasoning for creating it seemed sound: to provide an
> abstraction layer so users don't have to learn the rather arbitrary
> syntax that Wikipedia uses to deal with ISBN codes. Instead, they can
> use the same syntax they use everywhere: {{template name|argument}}.
>
> Is this a bad idea? Personally, I find the ISBN markup extremely
> arbitrary and maybe not that well thought out. We apparently only
> allow pure ISBN numbers, with no hyphens, spaces or slashes, although
> they are usually presented that way in other contexts. By wrappering
> this syntax in standard template syntax, we would clear the way for a
> future, more forgiving ISBN syntax to be implemented.
....
>
> Steve

I'm not sure what you're complaining about here. I've never had any
problems with ISBN syntax, and definitely no problems with spaces,
slashes or hyphens; ex, [[Seeds in the Heart: Japanese Literature from
Earliest Times to the Late Sixteenth Century]]'s ISBN link works fine
even with hyphens in it.

~maru


body:
Now that I've read the reasoning
(http://en.wikipedia.org/wiki/Wikipedia:Templates_for_deletion/Log/2006_July_9)...

Here's the deal. I created this ISBN template {{ISBN}} which does
nothing very exciting. It simply substitutes the text "ISBN {{{1}}}".
However, my reasoning for creating it seemed sound: to provide an
abstraction layer so users don't have to learn the rather arbitrary
syntax that Wikipedia uses to deal with ISBN codes. Instead, they can
use the same syntax they use everywhere: {{template name|argument}}.

Is this a bad idea? Personally, I find the ISBN markup extremely
arbitrary and maybe not that well thought out. We apparently only
allow pure ISBN numbers, with no hyphens, spaces or slashes, although
they are usually presented that way in other  contexts. By wrappering
this syntax in standard template syntax, we would clear the way for a
future, more forgiving ISBN syntax to be implemented.

Similarly, although the ISBN syntax is "simple" it may not be
"intuitive". What is the first reaction of people confronted with the
problem of inserting an ISBN number likely to be? "How do I make this
do that cool link thing?" What are they then most likely to try?
{{ISBN|34895342985}} maybe?

The arguments for deletion were interesting:
* How worse can we get? This perform exactly the same function as
typing ISBN+space+number, except with more characters. Let's burninate
it with fire.

In the same way that implementing getX() { return X; }} is a waste of space.

* Delete all text templates that take more characters to write than
what they produce.

I guess they wouldn't like {{--}} either. :)

* Subst but keep It's the sort of thing we'd create a redirect for if
it were an article, not a special piece of wikimarkup. It may be worth
checking 'what links here' every so often so that it can be susbted
and users who use it told the correct syntax.

The most enlightening - this person definitely thinks it's better that
we "educate" people on the "correct syntax", rather than admitting our
mistake at having a special syntax, and letting people not even have
to think about the question of formatting ISBN numbers.

* Comment This template was added to Maelor Way recently. I added a
notice to inform editors of the proper ISBN style.

Again - although the editors were using a template which took care of
"the proper ISBN style" for them, we choose instead to take away their
automatic transmission and tell them how to use a manual gearbox.

I'm just a bit dismayed by the kind of reasoning that goes on at some
of these deletion debates, where instead of "is this harmful?" we have
"is this needed?"

Comments very welcome, particularly if I'm completely misguided.

Steve


subject: [Wikitech-l] Blocked titles

body:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Simetrical schrieb:

> Why not extend the protection mechanism to be applicable to deleted
> pages?  That would be the most intuitive way, to my mind.  See also
> http://bugs.wikimedia.org/show_bug.cgi?id=2919.

Yes, think so too. I have voted for the bug now :-)

Raymond.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFE053Fp+qUcbEr56YRAsYLAJwJA+y0ns0I/t1Yww6cg5zTHxX7IwCgtezh
+VuoZ6bnE+DFq/0eILgWRAE=
=q165
-----END PGP SIGNATURE-----


body:
On 8/4/06, Delirium <delirium at hackish.org> wrote:
> Sounds sensible to me.  "Magic" syntax where seemingly normal-looking
> text gets transformed is evil, because it's unclear what's going on, and
> prone to breakage.  MediaWiki took a great leap forward by moving from
> magical CamelCase links to [[marked-up links]], for example.  Just about
> the only necessary evil I can think of is the auto-date-formatting;
> dates are too common to really be done any other way, although it too is
> slightly evil and sometimes kicks in or doesn't kick in in unexpected ways.

What about http://magic.formatting.com ? Is that a necessary evil? I
suspect it's probably not that evil.

Which auto date formatting do you mean, btw? I see these constant
references to it when people link stuff like [[August 8]], but we
don't seem to have any settings that return a nice result like "8th of
August", which is what my personal preference would be...

Using link syntax to induce special formatting is definitely not right.

Steve


subject: [Foundation-l] complaining to ISPs

body:
On 8/4/06, Jack <jackdt at gmail.com> wrote:
> Jeff, you shoulda made this a board election issue - you could
> probably have gotten a ton of votes from all the good editors the
> vandals are slowly driving off.
> --
> Jeandr?

However all the votes from the people who use NTL, Tiscali, AOL,
earthlink, whatever would rather counterbalance that

--
geni


subject: [Foundation-l] complaining to ISPs


from_name: Julien Lemoine

subject: [Wikitech-l] Wikipedia Suggest

body:
Hello Aerik,

Aerik Sylvan wrote:
> I wrote a "suggest" function for finding part numbers for a request form at
> work - enabling multi-word search is really useful and powerful.  I haven't
> looked at your code, so I don't know if you're using a keyword index, or
> LIKE SQL functions.  I set mine up to require at least 4 characters and to
> to break on whitespace, so the SQL (pseudocode) is WHERE LIKE(wordone) AND
> LIKE(wordtwo) and so on.
>
With this solution, you will not be able to handle a lot of queries per
second (it will probably take more than 1 second to query 2.3 millions
of entries). If you use a trie (yes I know this word now :)), everything
is pre-computed and a query will use very few cpu and you will be able
to handle a lot of queries per second.

Best Regards.
Julien Lemoine


subject: [Wikitech-l] Wikipedia Suggest

body:
Steve Bennett wrote:
> On 8/4/06, Timwi <timwi at gmx.net> wrote:
>
>> This is, of course, the most obvious and most naive solution. For a data
>> set as large as Wikipedia's list of article titles, this is far too slow
>> and inefficient and would kill the site for everyone if it was used on
>> the live database.
>>
>
> Thus far, I don't think anyone has actually suggested using the
> Suggest mechanism on the "live database".
>
I think Timwi was not talking about updating the suggest in live but to
put the service on
the same host/network. So the suggest needs to be very fast and use few
cpu power.

Best Regards.
Julien Lemoine


from_name: Nathan Carter

subject: [Foundation-l] complaining to ISPs

body:
On 8/4/06, geni <geniice at gmail.com> wrote:

>
> The foundation co opts my email and sends it itself. This allows it to
> keep the IP data private and removes possible barriers. On the other
> hand this could result in more work for the foundation.

Is there already a section within the foundation which can deal with this?
If not is some form of "abuse reporting" procedure needed?
I think the best way to do it is for the foundation itself to do the dirty
work so to speak, which should avoid any privacy issues.
Cheers,
Nathan Carter (Cartman02au)


subject: [Foundation-l] complaining to ISPs


subject: [Foundation-l] complaining to ISPs

body:
On 8/4/06, Nathan Carter <cartmanau at gmail.com> wrote:
>
>  On 8/4/06, geni <geniice at gmail.com> wrote:
>
> >
> > The foundation co opts my email and sends it itself. This allows it to
> > keep the IP data private and removes possible barriers. On the other
> > hand this could result in more work for the foundation.
>
>
>  Is there already a section within the foundation which can deal with
> this?
> If not is some form of "abuse reporting" procedure needed?
> I think the best way to do it is for the foundation itself to do the dirty
> work so to speak, which should avoid any privacy issues.
> Cheers,
>  Nathan Carter (Cartman02au)
>
>


subject: [Foundation-l] complaining to ISPs


subject: [Foundation-l] New Proposed WikiMedia Project

body:
 On 8/3/06, Jerome Baum <gratemyl at gmail.com> wrote:
> >
> > I just created this proposal:
> >
> > http://meta.wikimedia.org/wiki/SpeedWiki
> >
> > --
> > Yours,
> >
> > Jerome Baum
> > --
> > pegwit v8 public key
> > =b04220a315671b9351f71b6db4870ead7234316727d2c2f14b276e8575a5b3bd
> > --
> > e-gold Account Number: 3400024
> > Create an Account / Commit Payment: http://3400024.e-gold.com
> > _______________________________________________
> > foundation-l mailing list
> > foundation-l at wikimedia.org
> > http://mail.wikipedia.org/mailman/listinfo/foundation-l
> >

Hi mate,
Is there anything in your proposal which could not be done on one of the
existing projects?
Cheers,
Nathan


list: wiktionary-l

from_name: Michael Monaghan

subject: [Wiktionary-l] WZ Timeline.

body:
Hi,

Could someone estimate when a stable alpha/beta WZ might be available?
[give or take a few months].

Is http://meta.wikimedia.org/wiki/Wikidata/Timetable is the best page
to keep an eye on for this info?

Thanks,

~mm


from_name: Jack

subject: [Foundation-l] complaining to ISPs

body:
> I would suggest creating a banned ISP listing with ranges based on
> persistent vandals, and when they suspend the accounts, you will unblock
> them
> at the firewall. It fixed my vandalism problems. I have 0% vandalism at
> the WikiGadugi site and I stuck it to WillyOnWheels.
>
> Jeff

Jeff, you shoulda made this a board election issue - you could
probably have gotten a ton of votes from all the good editors the
vandals are slowly driving off.
--
Jeandr?


subject: [Foundation-l] complaining to ISPs


from_name: Michael Keppler

subject: [Wikitech-l] Wikipedia Suggest

body:
Julien Lemoine schrieb:
> Hum, I did not know the availability of these xml archives. In fact, I
> wrote scripts to grabb the whole content of wikipedia. You will have
> details about grabbing in the history section (june 2006) :
> http://suggest.speedblue.org/history.php

I suggest you remove those scripts from your page as this is definitely
bad behaviour against Wikipedia and other people should not even be
encouraged to do the same. That's what the database dumps are for.

Ciao, Michael.


from_name: brion at pobox.com

subject: [Wikitech-l] MediaWiki automated test run failure 2006-08-05

body:
An automated run of parserTests.php showed the following failures:

Running test Table security: embedded pipes (http://mail.wikipedia.org/pipermail/wikitech-l/2006-April/034637.html)... FAILED!
Running test Link containing double-single-quotes '' (bug 4598)... FAILED!
Running test Magic Word: {{CURRENTMONTHNAMEGEN}}... FAILED!
Running test Template with thumb image (wiht link in description)... FAILED!
Running test message transform: <noinclude> in transcluded template (bug 4926)... FAILED!
Running test message transform: <onlyinclude> in transcluded template (bug 4926)... FAILED!
Running test BUG 1887, part 2: A <math> with a thumbnail- math enabled... FAILED!
Running test HTML bullet list, unclosed tags (bug 5497)... FAILED!
Running test HTML ordered list, unclosed tags (bug 5497)... FAILED!
Running test HTML nested bullet list, open tags (bug 5497)... FAILED!
Running test HTML nested ordered list, open tags (bug 5497)... FAILED!
Running test Parsing optional HTML elements (Bug 6171)... FAILED!
Running test Inline HTML vs wiki block nesting... FAILED!
Running test Mixing markup for italics and bold... FAILED!
Running test 5 quotes, code coverage +1 line... FAILED!
Running test HTML Hex character encoding.... FAILED!
Running test dt/dd/dl test... FAILED!

Passed 412 of 429 tests (96.04%) FAILED!


from_name: Jeffrey V. Merkey

subject:
[Foundation-l] Native Cherokee Translations Posted
	20060804/20060805 Wikisource/Wikibooks/Wiktionary

body:
Native Cherokee Machine Translations posted to:

ftp.wikigadugi.org/wiki/xml

phwikibooks-20060804-pages-articles.xml.bz2       08/05/2006     05:40:00 AM
phwikisource-20060804-pages-articles.xml.bz2      08/05/2006     06:02:00 AM
phwiktionary-20060805-pages-articles.xml.bz2      08/05/2006     05:52:00 AM
sylwikibooks-20060804-pages-articles.xml.bz2      08/05/2006     05:50:00 AM
sylwikisource-20060804-pages-articles.xml.bz2     08/05/2006     06:14:00 AM
sylwiktionary-20060805-pages-articles.xml.bz2     08/05/2006     05:59:00 AM

www.wikigadugi.org

The Native Cherokee Translation project is a work in progress.

Jeff


from_name: Jerome Baum

subject: [Foundation-l] foundation-l Digest, Vol 29, Issue 5

body:
The idea behind the project is not to provide simple how-to's - it is
to provide people with ready-to-use code snippets, or quick-guides on
how to achieve certain (usually computer-related) things very fast,
sometimes by doing a simple copy-and-paste.
I personally am actually more interested in the computer side (i.e.
copy-paste, etc.), but I am more than sure that wiki users will be
able to make a lot from it.

> ---------- Forwarded message ----------
> From: "James Hare" <messedrocker at gmail.com>
> To: "Wikimedia Foundation Mailing List" <foundation-l at wikimedia.org>
> Date: Thu, 3 Aug 2006 17:56:11 -0700
> Subject: Re: [Foundation-l] New Proposed WikiMedia Project
> Hello Mr. Baum,
>
> Seems like a good idea, but Wikibooks seems to cover your idea at their
> How-To bookshelf.
>
> --James
>
>
> ---------- Forwarded message ----------
> From: "Nathan Carter" <cartmanau at gmail.com>
> To: "Wikimedia Foundation Mailing List" <foundation-l at wikimedia.org>
> Date: Fri, 4 Aug 2006 23:42:40 +1000
> Subject: Re: [Foundation-l] New Proposed WikiMedia Project
>
>
> Hi mate,
> Is there anything in your proposal which could not be done on one of the
> existing projects?
> Cheers,
> Nathan
>
>
>

--
Yours,

Jerome Baum
--
pegwit v8 public key
=b04220a315671b9351f71b6db4870ead7234316727d2c2f14b276e8575a5b3bd
--
e-gold Account Number: 3400024
Create an Account / Commit Payment: http://3400024.e-gold.com


subject: [Foundation-l] Proposed WikiMedia Project

body:
Sorry for the post under the wrong subject - I receive digest and
forgot to change it. Anyway:

I have created a site at http://speedwiki.elwiki.com/ which I am now
setting up. I will start it there, and hope to move to WikiMedia soon.

I will now update the links in the Meta, and then go to sleep (02:09, germany).

Maybe somebody already feels like contributing...

Anyways, good night / gute nacht!

--
Yours,

Jerome Baum
--
pegwit v8 public key
=b04220a315671b9351f71b6db4870ead7234316727d2c2f14b276e8575a5b3bd
--
e-gold Account Number: 3400024
Create an Account / Commit Payment: http://3400024.e-gold.com


from_name: Jeremy Tobacman

subject: [Wikitech-l] Unique editor statistics (was: Extension Installer)

body:
Mark,
We have csv files of revision-level header information.  Any standard stats
package (Stata, SAS, R, etc) should have no trouble with them.  I'll send
you the file (10MB) for Simple English; tell me (or Erik Garrison, cc'ed,
the creator of these files) when you're ready for the one for enwiki (3GB).
Best,
Jeremy

On 8/3/06, Simetrical <Simetrical+wikitech at gmail.com> wrote:
>
> On 8/3/06, Mark Bell <typewritermark at gmail.com> wrote:
> > I would like stats or a graph that shows:
> >
> > Number of unique editors per article X number of articles with that many
> > unique editors.
> >
> > So for example
> >
> > number of unique editors        Number of articles with that many unique
> > editors
> > 15                                               356
> > 30                                               3455
> >
> >
> > And so on. Is this even possible?
>
> Certainly.  If you know how to write in MySQL and some brand of
> computer language that supports MySQL, it shouldn't be hard to
> generate such stats from a database dump (see
> http://meta.wikimedia.org/wiki/Database_dumps).  If you don't know how
> to, maybe someone who does will have pity on you and do it for you.  I
> haven't gotten around to learning MySQL yet, but it looks to be
> probably fairly simple if you have access to a dump.
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at wikimedia.org
> http://mail.wikipedia.org/mailman/listinfo/wikitech-l
>


subject: [Wikitech-l] Wikipedia Suggest

body:
Hello Mickael,

Michael Keppler wrote:
> Julien Lemoine schrieb:
>
>> Hum, I did not know the availability of these xml archives. In fact, I
>> wrote scripts to grabb the whole content of wikipedia. You will have
>> details about grabbing in the history section (june 2006) :
>> http://suggest.speedblue.org/history.php
>>
>
> I suggest you remove those scripts from your page as this is definitely
> bad behaviour against Wikipedia and other people should not even be
> encouraged to do the same. That's what the database dumps are for.
>
>
You are right, I remove this scripts from download/history section with
a link to download.wikipedia.org.
Sorry, I did known this dump service before.

Best Regards.
Julien Lemoine.


subject: [Wikitech-l] Wikipedia Suggest

body:
Aerik Sylvan wrote:
> I'm sorry I hadn't taken the time to read about how you are doing it - I
> just looked up what a "trie" is, and I see why it's a hurdle - the whole
> structure is based on the first x characters.  It sounds like you are using
> a file to store the trie datastructure, and not using SQL at all?  I've got
> to say that I only know basic SQL, PHP, and some C - I don't have a
> background to speak intelligently about optimising code for zillions of
> hits, but I would approach this by taking all the article titles, and
> creating an index with every significant word in the title.  After that, I'm
> afraid I'd have to use SQL, and likely fall back onto using "LIKE" somewhere
> if I was going to return partial-word matches (an "obvious and naive"
> solution).  For whole word matches (but matching multiple words, or words
> that are not the first word in the page title) you could query the index
> directly.  I suppose one could build an index of partial words... would that
> be faster than using "LIKE"?  Hmm...
>
> I'm throwing queries at my category_links table that match category_a OR
> category_b, then group by page and count results to perform a category math
> / category intersections function.  It seems pretty fast, but I'm not using
> ajax, it's just a posted form.  It would be fun to write an ajax
> implementation though :-)
>
> I have always assumed that MySQL is internally optimised enough that if one
> sticks to simple queries and whole word matches, you get pretty good
> performance - but that's an assumption on my part.  I'm very interested in
> that because I'd like to know more about optimising search queries, but of
> course it's not your problem to teach me.
>
> Best Regards and good luck with your project.
>
>
Thank you.

I will try to give you a first draft of idea :
You will need a tokenizer to split the title in words and probably a
stop list to remove some word (a, of, the, ...)

Compilation: create a trie with all the words of your titles (without
words in stop list), each word will be associated to a number. This
number will give the access position in the vector V. this vector store
at a position, all the documents that contains this word.

Search : you receive one word,  you can do a search with regular
expression in you trie (as for exemple '*word*') and store results in a
vector of matchs (for speed, you will need to have a already allocated
vector, and fill it with correct matchs : for example keep the 10 best
matchs).

The problem: you algorithm will visit a lot of nodes in your trie.
The advantage: you will be really faster than the LIKE sql command, the
like command will be applied on each word of list.

Best Regards.
Julien Lemoine


subject: [Wikitech-l] Wikipedia Suggest

body:
Hello Michael,

Michael Keppler wrote:
> Julien Lemoine schrieb:
>
>> Hum, I did not know the availability of these xml archives. In fact, I
>> wrote scripts to grabb the whole content of wikipedia. You will have
>> details about grabbing in the history section (june 2006) :
>> http://suggest.speedblue.org/history.php
>>
>
> I suggest you remove those scripts from your page as this is definitely
> bad behaviour against Wikipedia and other people should not even be
> encouraged to do the same. That's what the database dumps are for.
>
You are right, I removed these scripts from download/history section with
a link to download.wikipedia.org.
Sorry, I did not known this dump service before.

Best Regards
Julien Lemoine


from_name: Lars Aronsson

subject: [Foundation-l] complaining to ISPs

body:
James Hare wrote:
> Yes... stating on a page "VERIZON IS BANNED!" for example would definitely
> make them scared and start taking actions against jerks.

Oh, really?  Have you tried?  Are you speaking from experience?

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik - http://aronsson.se


subject: [Foundation-l] complaining to ISPs


from_name: Jürgen Herz

subject: [Wikitech-l] Wikipedia Suggest

body:
Julien Lemoine wrote:

Hi,

> I have wrote a "google suggest" like service for wikipedia under GPL
> licence [1].
> By the way, I am not sure if this project will interest you, I am open
> to all comments from your community.
>
> I wrote a simple web site for english/french on
> http://suggest.speedblue.org, all software are available in the download
> section.

That reminds me of http://wiki.lumrix.net/en/. Don't know how that
works, but maybe you'll have a look.

J?rgen


from_name: Guillermo Roditi

subject: [Wikitech-l] Wikipedia Suggest

body:
Well considering the index will not be compiled in real time and probably
updated lets say, for the sake of argument, every other day, the resources
would be minimal after memcache ing and then mysql query cache. Since these
are read-often write-seldom records the query cache would really excel here.
and thats my 2?

On 8/4/06, Simetrical <Simetrical+wikitech at gmail.com> wrote:
>
> On 8/4/06, Aerik Sylvan <aerik at thesylvans.com> wrote:
> > I have always assumed that MySQL is internally optimised enough that if
> one
> > sticks to simple queries and whole word matches, you get pretty good
> > performance - but that's an assumption on my part.  I'm very interested
> in
> > that because I'd like to know more about optimising search queries, but
> of
> > course it's not your problem to teach me.
>
> MySQL can only be as optimized as the algorithm that uses it.  LIKE
> may be optimized, but it's impossible to make it very efficient given
> what it does.
>
> > Timwi, from your other posts I can see that you know a lot more about
> > computer programming than I do, but I would hope to be able to offer
> some
> > opinion without getting slapped with "obvious and naive".
>
> I don't think he meant it as an insult.  In computer science, "naive"
> basically means "the most obvious way to accomplish something, which
> is probably not a good idea performance-wise".
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at wikimedia.org
> http://mail.wikipedia.org/mailman/listinfo/wikitech-l
>


from_name: Stephen Bain

body:
On 8/5/06, Steve Bennett <stevagewp at gmail.com> wrote:
>
> However, my reasoning for creating it seemed sound: to provide an
> abstraction layer so users don't have to learn the rather arbitrary
> syntax that Wikipedia uses to deal with ISBN codes.

It's hardly a complicated syntax. It's hardly even a syntax: type
"ISBN", then a space, then the number.

> Similarly, although the ISBN syntax is "simple" it may not be
> "intuitive". What is the first reaction of people confronted with the
> problem of inserting an ISBN number likely to be? "How do I make this
> do that cool link thing?" What are they then most likely to try?
> {{ISBN|34895342985}} maybe?

I don't know about everyone else, but when I was learning wikisyntax I
did it by opening an edit window of existing pages and seeing what was
there.

In any event, the ISBN magic syntax produces a unique behaviour, a
link to an automatically generated Special:Booksources page, and so it
probably should be using its own syntax.[1] Using template syntax as
an abstraction layer to hide the real operation is likely to confuse
anyone who really wants to know how it works and takes a look at the
template code.

I'm aware of the argument against magic-word-type syntax, but there
are plenty of magic words in MediaWiki, most of which are very useful:
http://meta.wikimedia.org/wiki/Help:Magic_words

Of course there is room for the syntax for ISBNs, and other magic
words, to be improved by making them more consistent with each other.
Perhaps something like #ISBN 123456789, like the redirect syntax.

> I'm just a bit dismayed by the kind of reasoning that goes on at some
> of these deletion debates, where instead of "is this harmful?" we have
> "is this needed?"

There was some reasoning that suggested it was harmful. The comment
that it's pointless and sometimes harmful to have templates which use
more characters than the text they produce is a good one. Sometimes a
template may allow other problems to be avoided, such as {{!}}.
However you mentioned {{--}}, which is a lazy duplication of the
emdash in the toolbox below every edit window.

To conclude, the documentation on magic word syntaxes (in this case
http://en.wikipedia.org/wiki/Wikipedia:ISBN) should be improved and
made more prominent to noobs learning how to write wiki. But I think
that using templates to abstract magic words is a fairly silly
proposition, it's hiding problems rather than addressing them.

--
[1] Yes, variables such as {{PAGENAME}} also use the template syntax,
but they perform text substitution (an expected behaviour of
templates) and are distinguished by their ALL CAPS naming.

--
Stephen Bain
stephen.bain at gmail.com


from_name: stevertigo

subject: [WikiEN-l] Fakes and people who cant tell the difference

body:
Thought this was funny enough to list.

http://en.wikipedia.org/wiki/Image_talk:Gandhi_Boer_War_1899.jpg

I don't see the point in these exertions. I have provided the source, which is indeed,
a "very reputable" newspaper (italics to add to your scare-quotes) and one could
reasonably expect you to provide something other than bare and repeated assertion.

SV

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com


from_name: Sabine Cretella

subject: [Wiktionary-l] WZ Timeline.

body:
Hi Michael,

that timeline you saw is quite an old one.

As for now we have the following dates:
- versioning should work by the end of August/beginning of September
- by October there should be a publicly editable version

As you probably know sometimes unforeseen things happen - so the dates I
am giving here are purely indicative and could change a bit.

You can already edit on WiktionaryZ right now, but it is us to give you
edit rights for the wikidata part, since versioning is not working
properly and we only can give edit rights to people we know and trust. I
suppose that is understandable. Of course the ordinary wiki pages can be
edited by anyone (well, it's a wiki :-)

Wishing you a great week-end!

Sabine

Michael Monaghan schrieb:
> Hi,
>
> Could someone estimate when a stable alpha/beta WZ might be available?
> [give or take a few months].
>
> Is http://meta.wikimedia.org/wiki/Wikidata/Timetable is the best page
> to keep an eye on for this info?
>
> Thanks,
>
> ~mm
> _______________________________________________
> Wiktionary-l mailing list
> Wiktionary-l at Wikipedia.org
> http://mail.wikipedia.org/mailman/listinfo/wiktionary-l
>
>

Chiacchiera con i tuoi amici in tempo reale!
 http://it.yahoo.com/mail_it/foot/*http://it.messenger.yahoo.com


list: wikipedia-l

subject: [Wikipedia-l] Geocodes (was: Artikel ohne Koordinate)

body:
Hi all,

Like I pointed out yesterday in a message anwering in German to Stefan
K?hn and Timwi, I wrote a blog about how we could work on missing
geocodes on the German wikipedia. I mean: how the work done there could
be useful for all and how of course all wikipedias can improve quality
(I am going to write some more on that one as well soon).

So here is the link to the blog:
http://sabinecretella.blogspot.com/2006/08/geocodes-or-how-various-wikipedias-can.html

It is about co-operating on structural data - not just for one
wikipedia, but for all of them.

I just prefer to write blogs because it is easier to refer to them when
you need to do so after a longer period of time.

Best, Sabine

*****
Sabine Cretella
Translations IT-DE, EN-DE
Evangelist for WiktionaryZ
s.cretella at wordsandmore.it
skype: sabinecretella
phone +39-340-1809828
blog: http://sabinecretella.blogspot.com

Chiacchiera con i tuoi amici in tempo reale!
 http://it.yahoo.com/mail_it/foot/*http://it.messenger.yahoo.com


from_name: David

subject: [Wiktionary-l] WZ Timeline.

body:
Hi Michael,

Could you give me the web address for the wikipedia download just prior to March 2002 or the last update before March 2002.

Best,
David

 --- On Fri 08/04, Michael Monaghan < mickmon at gmail.com > wrote:
From: Michael Monaghan [mailto: mickmon at gmail.com]
To: wiktionary-l at wikipedia.org
Date: Fri, 4 Aug 2006 23:51:19 -0400
Subject: [Wiktionary-l] WZ Timeline.

Hi,Could someone estimate when a stable alpha/beta WZ might be available?[give or take a few months].Is http://meta.wikimedia.org/wiki/Wikidata/Timetable is the best pageto keep an eye on for this info?Thanks,~mm_______________________________________________Wiktionary-l mailing listWiktionary-l at Wikipedia.orghttp://mail.wikipedia.org/mailman/listinfo/wiktionary-l

_______________________________________________
Join Excite! - http://www.excite.com
The most personalized portal on the Web!


from_name: Alphax (Wikipedia email)

subject: [WikiEN-l] GFDL compliance

body:
Guy Chapman aka JzG wrote:
> There is a picture on another project which would be good in an
> article on en:.  Only problem is, the other project is
> Uncyclopaedia...
>
> I'm guessing the licensing is incompatible, and i can't trace the
> original source.  Bummer, really.
>

You've asked the uploader about it?

--
Alphax - http://en.wikipedia.org/wiki/User:Alphax
Contributor to Wikipedia, the Free Encyclopedia
"We make the internet not suck" - Jimbo Wales
Public key: http://en.wikipedia.org/wiki/User:Alphax/OpenPGP

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 569 bytes
Desc: OpenPGP digital signature
Url : http://lists.wikimedia.org/pipermail/wikien-l/attachments/20060805/28342a0a/attachment.pgp


from_name: Joe Anderson

subject: [WikiEN-l] GFDL compliance

body:
Fair use?

On 8/5/06, John Lyden <rasputinaxp at gmail.com> wrote:
>
> On 8/4/06, Guy Chapman aka JzG <guy.chapman at spamcop.net> wrote:
> > There is a picture on another project which would be good in an
> > article on en:.  Only problem is, the other project is
> > Uncyclopaedia...
>
> Now that's irony.
>
> --
> John Lyden - rasputinaxp at gmail.com
> "The only people for me are the mad ones, the ones who are mad to
> live, mad to talk, mad to be saved, desirous of everything at the same
> time..." -Kerouac
> _______________________________________________
> WikiEN-l mailing list
> WikiEN-l at Wikipedia.org
> To unsubscribe from this mailing list, visit:
> http://mail.wikipedia.org/mailman/listinfo/wikien-l
>

--
Joe Anderson

[[User:Computerjoe]] on en, fr, de, simple, Meta and Commons.


from_name: Steve Bennett

body:
On 8/4/06, Delirium <delirium at hackish.org> wrote:
> That's precisely what it's for, although ordinal dates aren't currently
> an option.  Go to preferences->date&time.  For year-less dates, it'll
> auto-convert between [[October 28]] and [[28 October]], and for
> year-havin' dates it'll auto-convert between any of these 5:
> [[October 28]], [[1940]]
> [[28 October]] [[1940]]
> [[1940]] [[October 28]]
> [[1940]]-[[10-28]]
> [[1940-10-28]]

Yeah, I confess I don't really understand the point of these different
formats. Is it just a "preference" thing, rather than an actual
cultural standard? Who writes "1940 October 28" as standard, for
instance? I would understand if the options were something like
"28/10/1940" vs "10/28/1940" (American vs British dates for instance)
but the chosen formats are just, um, odd. AFAIK hyphens between
numbers in dates are much more common in countries like France, rather
than any English speaking countries, which tend to use slashes. I
recognise the last format as being common in Japan IIRC, but who would
want to see that format in the middle of an English-speaking article?
Or do these settings affect all users of MediaWiki?

> The two most common seem to be the top two, with Americans preferring
> the first and many Europeans preferring the second.  I suppose another

European English speakers, like, British people? :) FWIW, I would find
"October 28th, 1940" the most natural.

> format that was the same as the 2nd but with ordinals would please some
> folks as well, but perhaps it's somewhat harder to implement.

Hmm.

Steve


body:
On 8/5/06, Stephen Bain <stephen.bain at gmail.com> wrote:
> It's hardly a complicated syntax. It's hardly even a syntax: type
> "ISBN", then a space, then the number.

I carefully avoided suggesting it was complicated :)

> In any event, the ISBN magic syntax produces a unique behaviour, a
> link to an automatically generated Special:Booksources page, and so it
> probably should be using its own syntax.[1] Using template syntax as

That's not that unique. Seems to me that the standard template
behaviour "put a word in {{ }} and it will be replaced with some
useful text and markup" applies here.

In fact, unless I'm mistaken, you could implement {{ISBN}} directly as
a standard template without the magic word at all:
[[[Special:Booksources/isbn={{{1}}}|ISBN {{{1}}}]]

See the following for some examples.
http://en.wikipedia.org/wiki/User:Stevage/sandbox

Notice the last one is slightly different since it uses an external link.

All of which is to say, the "magic word", while undoubtedly useful and
intuitive to those naively adding ISBN numbers (not necessarily
expecting a link), is not actually necessary. And, if any bot
substitution was to occur, I would actually find substitition *to* the
template syntax "cleaner" than substiution to the magic word syntax.

It's more portable anyway.

> an abstraction layer to hide the real operation is likely to confuse
> anyone who really wants to know how it works and takes a look at the
> template code.

Well, you could use the same argument against any templates. Using
"abstraction layers to hide the real operation" is a fundamental part
of making any complicated system usable by new users.

> I'm aware of the argument against magic-word-type syntax, but there
> are plenty of magic words in MediaWiki, most of which are very useful:
> http://meta.wikimedia.org/wiki/Help:Magic_words
>
> Of course there is room for the syntax for ISBNs, and other magic
> words, to be improved by making them more consistent with each other.
> Perhaps something like #ISBN 123456789, like the redirect syntax.

or, {{ISBN|123456780}} :)

> There was some reasoning that suggested it was harmful. The comment
> that it's pointless and sometimes harmful to have templates which use
> more characters than the text they produce is a good one. Sometimes a

Are we running out of disk space?

> template may allow other problems to be avoided, such as {{!}}.
> However you mentioned {{--}}, which is a lazy duplication of the
> emdash in the toolbox below every edit window.

Again, personally I find it cleaner, as it's available even if you're
using a different skin, with no javascript, and the list of available
characters seems to be very subject to change. Bots could certainly
substitute the template with the right character (much as they
currently substitute — with ?), but it's certainly preferable
that users use {{--}} instead of just --.

> To conclude, the documentation on magic word syntaxes (in this case
> http://en.wikipedia.org/wiki/Wikipedia:ISBN) should be improved and
> made more prominent to noobs learning how to write wiki. But I think
> that using templates to abstract magic words is a fairly silly
> proposition, it's hiding problems rather than addressing them.

Well, at least it's "hiding" them in a way that when they are
"addressed" they don't break anything. Template syntax has a lot going
for it - the actual real syntax underneath can change without breaking
anything, if it's used widely enough. IMHO, it would be a good thing
if the only syntax exposed to users was "standard' MediaWiki syntax
and templates. Users should use {{sup|.......}} rather than
<sup>.....</sup> for instance, to decouple the semantic content
(superscripted text) from the presentation layer (HTML).

Steve


from_name: Timwi

subject: [Wikitech-l] Wikipedia Suggest

body:
Aerik Sylvan wrote:
>
> I would approach this by taking all the article titles, and
> creating an index with every significant word in the title.  After that, I'm
> afraid I'd have to use SQL, and likely fall back onto using "LIKE" somewhere
> if I was going to return partial-word matches (an "obvious and naive"
> solution).

Creating an index of words is not that native anymore. :-) Also, if you
use "LIKE 'word%'" instead of "LIKE '%word%'", the index can be used
quite efficiently. However, you still run into the problem: how do you
list the top 10 most linked-to pages given just a single letter? You'd
still have to query all article titles that have a word that begins with
that letter. The trie gives you a much more direct way of looking that up.

> I have always assumed that MySQL is internally optimised enough that
> if one sticks to simple queries and whole word matches, you get
> pretty good performance

No matter how well something is optimised, you would still expect some
things to be more efficient than others. By relying on the efficient
things, Wikipedia can run on fewer and/or cheaper servers. In other
words, your queries may be sufficiently efficient that they run within a
fraction in a second on your server which is not used by anyone else,
but if the server is already as busy as Wikipedia, and then you're
adding another few thousand extra queries per second, then you can't
expect "internal optimisation" to do all the magic for you.

>>This is, of course, the most obvious and most naive solution. [...]
>
> Timwi, from your other posts I can see that you know a lot more about
> computer programming than I do, but I would hope to be able to offer some
> opinion without getting slapped with "obvious and naive".

Please read http://en.wikipedia.org/wiki/Wikipedia:Assume_good_faith .

In the same way that you hope to be able to offer your opinion, so do I.
The "getting slapped" is in your head, not in my intentions. If my
opinion, given in good faith, is offensive to you, then quite frankly
you need to lighten up. You can't expect it to be censored or suppressed
just so that you feel better.

Timwi


subject: [WikiEN-l] Interesting article from last "Signpost"

body:
Hi all,
  Apologies to those who read it, but there was an interesting article
about the value of Wikipedia in academia, based on several academic
mailing lists:

http://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2006-07-31/Listserv

It includes this rich quote: "...it is extremely useful as a
springboard for further research. It will be impossible to find
anything on "goosing" or "Dirty Sanchez" in the more respectable
(elitist?) encyclopedias, or much on more obscure forms of sexual
behaviour."

I can't wait to see this get used in future AfDs :)

Steve