User talk:Citation bot

Consistent spacing[edit]

Status: new bug
Reported by: Abductive (reasoning) 03:24, 2 August 2021 (UTC)[]

What happens: bot added a date parameter in a ref with a space before every pipe, but did not include a space
Relevant diffs/links: https://en.wikipedia.org/w/index.php?title=53W53&type=revision&diff=1036681818&oldid=1036681278
We can't proceed until: Feedback from maintainers

I know this is a minor bug, but it bugs me. I know that the bot is written to make an attempt to duplicate the formatting already present in the ref. How it could have failed here, I don't know. But more importantly, it should default to the consensus ref formatting: space,pipe,parametername,=,parametervalue. (Spaces before pipes, no spaces around the equals signs or anywhere else, except perhaps before the curly end brackets if there already was a space there.) Abductive (reasoning) 03:24, 2 August 2021 (UTC)[]

I agree. The default should be space,pipe,parametername,=,parametervalue. --BrownHairedGirl (talk) • (contribs) 15:27, 2 August 2021 (UTC)[]

Cannot fix since the the bot already uses the existing citation template as a guide. Templates that are mixes in spacing such as these cannot be done in a way that makes everyone happy. AManWithNoPlan (talk) 16:45, 2 August 2021 (UTC)[]

But how to explain the example? The bot deviated from the format of the ref it edited? Abductive (reasoning) 16:59, 2 August 2021 (UTC)[]

I see, you want the bot to add spaces to existing parameters - in particular the last one. Interesting, the bot by default does not in anyway modify spacing of existing parameters. That parameter has no trailing spaces. As far as the bot in concerned there are no spaces before pipes, just spaces at the end of parameters. AManWithNoPlan (talk) 17:14, 2 August 2021 (UTC)[]

The bot must have looked at the lack-of-space of the last parameter (before the end curly braces) to come to the conclusion that the ref was formatted that way. Perhaps it should look after the "cite xxxx" for the cue? Abductive (reasoning) 17:51, 2 August 2021 (UTC)[]

not, that is not what it did. It simply does not change the spacing of existing parameters. The existing final parameter has no ending space, so the bot does not add one. AManWithNoPlan (talk) 21:14, 2 August 2021 (UTC)[]

Ah, I see what you are saying. It slotted it in at the end. Well, I had hoped that the bot could have provided a cure to the annoying new habit of users removing all spaces from refs, making a wall of text for editors. Abductive (reasoning) 22:25, 2 August 2021 (UTC)[]

And creates annoyingly unpredictable line wraps. Does this format really have consensus? If so, bots (any bot) could create a cosmetic function for citations they edit. -- GreenC 17:04, 6 August 2021 (UTC)[]

There are some people who like the "crammed" format. I started a conversation about the formatting here, but I don't really understand what they were saying. Abductive (reasoning) 02:06, 7 August 2021 (UTC)[]

As Abductive suggests, what the bot should do ideally is to check if the first parameter's pipe following the template name is preceded by a space (or even better, if at least one of the parameters' pipe symbol is preceded by space) and if it is, it should add a space in front of pipe symbol of newly inserted parameters, no matter where they are inserted into the parameter list. If the template has no parameters yet, the bot should fall back to the "default" format "space, pipe, parameter name, equal sign, parameter value" we consistently use in all CS1/CS2 documentation and examples. (Well, IMO, this latter format would ideally be made the only format used at all, but that's a discussion beyond the scope of CB issues here.)

Yeah, it is only cosmetic, but like Abductive I too find it somewhat annoying when previously perfectly formatted citations become misaligned by bot edits.

--Matthiaspaul (talk) 13:34, 7 August 2021 (UTC)[]

While I agree, this is actually going to be hard to implement. I will need to think about it. AManWithNoPlan (talk) 18:12, 8 August 2021 (UTC)[]

Still thinking about how to do this. It will have to deal with figuring out what the last parameter before adding a parameter to the very end, but no the middle. AManWithNoPlan (talk) 00:51, 4 September 2021 (UTC)[]

Only four requests at a time?[edit]

Status: new bug
Reported by: Abductive (reasoning) 22:54, 2 August 2021 (UTC)[]

What happens: It seems that the bot can only work on four jobs at any one time.
We can't proceed until: Feedback from maintainers

I sampled the bot's edits going back a few days, and seems that the bot can only interleave, or even accept a single page, only four requests at any one time. At no point can I find five jobs interleaving, and (although this is harder to be certain about) at no point when there are four jobs interleaving can a fifth job be found, even a single page requested. Is this deliberate, and if yes, is it really a necessary constraint on the bot? Abductive (reasoning) 22:54, 2 August 2021 (UTC)[]

that is what i have observed and complained about also. I am convinced that default PHP config is 4. Someone with tool service access needs to get the bot a custom lighttd config file. AManWithNoPlan (talk) 23:03, 2 August 2021 (UTC)[]

Gah. Abductive (reasoning) 23:07, 2 August 2021 (UTC)[]

lol, you people with "jobs". the rest of us with single page requests can't get anything in no matter how many jobs. — Chris Capoccia 💬 11:20, 3 August 2021 (UTC)[]

https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Lighttpd Look at PHP and the "Default_configuration" area that starts collapsed. AManWithNoPlan (talk) 19:18, 3 August 2021 (UTC)[]

This is also part of the wider problem that the bot needs much more capacity, and also that a lot of its time is taken up speculative trawls through wide sets of articles which have not been identified as needing bot attention and which often produce little change. Huge categories are being fed to the bot which changes little over 10% of them, and most of those changes are trivia (type of quote mark in title) or have effect at all on output (removing redundant parameters or changing template type). It would help a lot if those speculative trawls were given a lower priority. --BrownHairedGirl (talk) • (contribs) 22:54, 9 August 2021 (UTC)[]

Who would decide what " speculative trawls " are? And what should the limit be? Might be hard to find something that can be agreed on. Perhaps the users that request these large categories see them as very important, while you don't. Of course it will be easy to know that certain specially created maintenance categories will give a high output, and do a lot of work, but if a user just wants to request a "normal" category they can't know how many % of the pages will actually get changed beforehand.

I agree capacity should be increased, more jobs at the same time would be such a good thing, however deciding that one page might be more important than another does not fix the root cause.

I do think there might be something going for giving priority to people that request a single (or low amount of) page(s). A person could be running the most important category that exists, but if I just want to have a single page check about a topic that I am knowledgeable about or have a big interest in, it is a hard swallow waiting for multiple thousand page jobs to finish. This actually has made me just give up a few times, leaving pages that could have been fixed and checked (with my knowledge about said subject) broken, I'm sure many can recognise themselves in this.

It is indeed important to fix high priority pages, and especially improve capacity, but lets not forget about the people who edit on topics that they enjoy, and just want to use the bot on something that might not be important according to some maintenance category, but is important to them. The more people that want to keep using the bot, the better! Redalert2fan (talk) 00:22, 10 August 2021 (UTC)[]

@Redalert2fan: I think you are missing my point, which is that there is no benefit to anyone in having the bot processing lots of articles where there is nothing for it to do. No matter how important anyone thinks an article is, there is no gain in having the bot spend ages deciding that there is nothing to.

The reason that single pages get locked out is because its capacity is being used up by these speculative trawls, by which I mean simply that they are not categories selected because they concentrate article which need the bot's attention -- these are "see if you find anything" batches, rather than "cleanup the problem here" batches.

One or two editors are repeatedly feeding it big categories on a huge variety of topics simply because they are big categories which fit under the 4,400 limit for categories. I have analysed the results, and in many cases the result is that only 10%–15% of the pages are edited, and only about half of those have non-trivial changes. So about 95% of the time that the bot spends on these huge categories is completely wasted.

When a resource is limited, it is best used by prioritising pages which have been selected on the basis that there is a high likelihood of something to do. --BrownHairedGirl (talk) • (contribs) 00:44, 10 August 2021 (UTC)[]

I see, there is no denying that there is no gain in having the bot spend ages deciding that there is nothing to.

Wouldn't it be an even quicker fix to ask these few editors why they run these not so intermediately helpful categories, and notify them of the problems it causes, and the benefits that can be gained by not requesting any category? It seems more like operator error than a bot mistake, and limiting the bots abilities for something that is caused by a few users, seems maybe questionable?

I agree with the points you make, but I don't feel like we should limit hundred of potential editors that request pages with what you would describe as "less than optimal request" because of 2 or so people. Even though we are limited, I don't think we need a strict priority system. If someone random editor want's to request a hundred pages that have their interest, we can't expect everyone to know beforehand if their request is an optimum use of the system, and see if you find anything might be an honest use case still. Obviously if as you say specific editors that seem to know what they are doing use the bot in a way that basically blocks out others for little gain all the time, action should be taken.

Some sort of priority system might indeed be a good idea, whether if it is in the way of important maintenance categories, "the pages with a high likelihood of something to do" or just giving priority to small request etc. Though it has to be a priority system for some types of requests, not a limitation for all request in my opinion, especially if the problem comes from a very minor selection of users. Redalert2fan (talk) 01:11, 10 August 2021 (UTC)[]

Max category size shrunk by one-quarter. AManWithNoPlan (talk) 13:08, 10 August 2021 (UTC)[]

Thanks, AManWithNoPlan. That's helpful, but wouldn't it be better to reduce it to the same 2200 limit size as the linked-from-page limit? --BrownHairedGirl (talk) • (contribs) 14:32, 10 August 2021 (UTC)[]

@Redalert2fan: Since early July, when I started using Citation bot in to clean up bare URLs, I have seen two editors repeatedly using the bot unproductively.

One was JamCor, who was using the bot to process the same set of almost 200 articles 3 or 4 times per day. Many of the articles were huge, taking several minutes each to process, so i estimated that about 20% of the bot's time was being taken on those 200 articles. I raised it on JamCor's talk, and they stopped, but only after the second request.

The other is Abductive, with whom I raised the problem several times on this page: see User talk:Citation_bot/Archive 27#throttling_big_category_runs. Sadly, that one persists, and I gave up making my case. When I started writing this post a few hours ago in response to you, I analysed the then-current recent contribs of the bot. Abductive had started the bot scanning Category:Use dmy dates from August 2012,and by then the bot had processed 1433 of the category's 4379 pages, but had saved an edited on only 141 of them, i.e. less than 10%. A with many of Abductive's previous big runs, I can't see any way in which this run could have been selected as a set which concentrates articles of interest to Abductive, or which concentrates articles of high-importance, or which concentrates articles that have been identified as being likely to have problems this bot can fix. The only criterion which I can see for its selection is that its size (4379 pages) is very close to the 4400 maximum size of Citation bot category jobs. A quick glance at the parent Category:Use dmy dates shows that very few categories are so close to the limit without exceeding it.

So AFAICS,the only reason for selecting this job was that it is a big set of articles which can be thrown at the bot with no more effort than cop-pasting the category title. I may of course have missed something, and if so I hope that Abductive will set me right. --BrownHairedGirl (talk) • (contribs) 14:33, 10 August 2021 (UTC)[]

I meant cut to one fourth, not cut by one fourth. So, category is now half the linked pages API. AManWithNoPlan (talk) 14:37, 10 August 2021 (UTC)[]

Cut to 1100 items! This is extreme. Grimes2 (talk) 14:53, 10 August 2021 (UTC)[]

@AManWithNoPlan: thanks. 1100 is much better.

However, that will reduce but not eliminate the problem of an editor apparently creating bot jobs just because they can. Such jobs will now require 4 visits to the webform in the course of a day, rather than just one, but that's not much extra effort. --BrownHairedGirl (talk) • (contribs) 15:13, 10 August 2021 (UTC)[]

People using the bot is not a problem. Abductive (reasoning) 18:02, 10 August 2021 (UTC)[]

Indeed, people using the bot is not a problem.

The problem is one person who repeatedly misuses the bot. --BrownHairedGirl (talk) • (contribs) 18:27, 10 August 2021 (UTC)[]

It is not possible to misuse the bot. Having the bot make the tedious decisions on what needs fixing is far more efficient than trying to work up a bunch of lists. Unfortunately, even the best list can be ruined if the API that the bot checks happens to be down. This is why it is it inadvisable to create lists that concentrate on one topic. Abductive (reasoning) 19:49, 10 August 2021 (UTC)[]

Bot capacity is severely limited. There is no limit to how much editors can use other tools to make lists, so that makes more efficient use of the bot.

Think of the bot like a hound, which is far more effective at finding quarry if started in the right place. The hound will waste a lot of time if started off miles away from the area where previous clues are.

Lots of other editors are targeting the bot far more effectively than your huge category runs. --BrownHairedGirl (talk) • (contribs) 22:11, 10 August 2021 (UTC)[]

Hey BrownHairedGirl, I agree with your ideas, but in the end there are no rules for what the bot can be used for so calling it misuse isn't a fair description. Anyone is allowed to use it for anything. Abductive can request what he wants, and creating bot jobs just because you can is allowed. In my eyes every page is valid to check (provided it isn't just a repeat of the same page or groups of pages frequentely).Redalert2fan (talk) 00:13, 11 August 2021 (UTC)[]

Just to be sure, whether that is the optimal way to use the bot or not is still a fair point of discussion. Redalert2fan (talk) 00:17, 11 August 2021 (UTC)[]

The question of self-restraint by users of an unregulated shared asset is a big topic in economics.

The article on the tragedy of the commons is an important read. It's well written but long. if you want a quick summary, see the section #Metaphoric meaning.

In this case, it would take only 4 editors indiscriminately feeding the bot with huge sets of poorly-selected articles to create a situation where 90% of the bots efforts changed nothing, and only 5% did anything non-trivial. That would be a tragic waste of the fine resource which the developers and maintainers of this bot have created, and would soon lead to calls for regulation. The question now is whether enough editors self-regulate to avoid the need for restrictions. --BrownHairedGirl (talk) • (contribs) 05:30, 11 August 2021 (UTC)[]

@AManWithNoPlan: the new limit of 1100 does not seem to have taken effect; see this[1] at 18:07, where the bots starts work on category of 1156 pages.

That may be due to expected delays in how things get implemented, but I thought it might help to note it. --BrownHairedGirl (talk) • (contribs) 18:51, 10 August 2021 (UTC)[]

Bot rebooted. AManWithNoPlan (talk) 20:15, 10 August 2021 (UTC)[]

Max category cut again to 550 and now prints out list of category pages so that people can use linked pages API instead, which also means that if the bot crashes the person can restart it where is left off instead of redoing the whole thing as it does with category code. AManWithNoPlan (talk) 20:25, 10 August 2021 (UTC)[]

Great work! Thanks. --BrownHairedGirl (talk) • (contribs) 22:02, 10 August 2021 (UTC)[]

It seems that the low-return speculative trawls have re-started. @Abductive has just run a batch job of Category:Venerated Catholics by Pope John Paul II; 364 pages, of which only 29 pages were actually edited by the bot, so 92% of the bot's efforts on this set were wasted. The lower category limit has helped, because this job is 1/10th of the size of similar trawls by Abductive before the limit was lowered ... but it's still not a good use of the bot. How can this sort of thing be more effectively discouraged? --BrownHairedGirl (talk) • (contribs) 11:57, 27 August 2021 (UTC)[]

A number of editors have pointed out to you that using the bot this way is perfectly acceptable. In addition, there are almost always four mass jobs running, meaning that users with one article can't get access to the bot. A run of 2200 longer articles takes about 22 hours to complete, so if I had started one of those, it would have locked such users out for nearly a day. By running a job that lasted less than a hour, I hoped that requests for smaller and single runs could be accommodated. And, in fact, User:RoanokeVirginia was able to use the bot as soon as my run completed. Abductive (reasoning) 18:14, 27 August 2021 (UTC)[]

@Abductive: on the contrary, you are the only editor who repeatedly wastes the bot's time in this way. It is quite bizarre that you regard setting the bot to waste its time as some sort of good use.

On the previous two occasions when you did it, the result was that the limits on job size were drastically cut. --BrownHairedGirl (talk) • (contribs) 18:47, 27 August 2021 (UTC)[]

That was in response to your complaints. Since I ran a job that was within the new constraints, I was not misusing the bot. You should request that the limits be increased on manually entered jobs, and decreased on category jobs. There is no particular reason that 2200 is the maximum. Abductive (reasoning) 18:52, 27 August 2021 (UTC)[]

@Abductive: you continue to evade the very simple point that you repeatedly set the bot to do big jobs which achieve almost nothing, thereby displacing and/or delaying jobs which do improve the 'pedia. --BrownHairedGirl (talk) • (contribs) 19:04, 27 August 2021 (UTC)[]

Using the bot to check a category for errors is an approved function of the bot. The fundamental problem is the limit of 4 jobs at a time. Also, the bot is throttled to run considerably slower than it could, which is a holdover from the time when it was less stable. The various throttlings, which as I recall were implemented in multiple places, should be re-examined and the bot re-tuned for its current capabilities. Abductive (reasoning) 19:11, 27 August 2021 (UTC)[]

This is not complicated. Whatever the bot's speed of operation, and whatever the limit on concurrent jobs, its capacity is not well used by having it trawl large sets of pages where it has nothing to do. I am surprised that you repeatedly choose to ignore that. --BrownHairedGirl (talk) • (contribs) 19:19, 27 August 2021 (UTC)[]

I am not ignoring anything. Bots exist to do tedious editing tasks. Your notion that editors have to do the tedious work before giving the bot a task is contrary to the purpose of bots. A number of proposals have been put forward to improve bot performance or relieve pressure on the bot, such as allowing multiple instances of the bot, or allowing users to run the bot from their userspace. These proposals have not been implemented. As the bot is currently configured, there will always be load problems. Abductive (reasoning) 19:29, 27 August 2021 (UTC)[]

Load problems that you are exacerbating. We've requested a million times to have better scheduling, or more ressources, but no dice so far. You're cognizant there's an issue, and you yet repeatedly feed the bot low-priority low-efficiency work. That's pretty WP:DE / WP:IDIDNTHEARTHAT behaviour from where I stand. Headbomb {t · c · p · b} 19:34, 27 August 2021 (UTC)[]

I have been holding off of all different kinds of runs lately. Check the bot's edits for the last week or so. Abductive (reasoning) 00:58, 28 August 2021 (UTC)[]

Abductive, increasing the bot's capacity would:

require a lot of work by the editors who kindly donate their time to maintain and develop this bot. WP:NOTCOMPULSORY, and they should by pressed to donate more time. Their efforts are a gift from them, not a contract.
exacerbate to some extent the usage limitations of the external tools which the bot relies on. Increasing the speed of the bot's operation will mean that those limits are encountered more frequently.

The bot will probably always have load problems, because there is so much work to be done.

Two examples:

Headbomb's jobs of getting the bot to cleanup refs to scholarly journals. That is high-value, because peer-reviewed are the gold standard of WP:Reliable sources, and it is also high labour-saving because those citations are very complex, so a big job for editors to fix manually. It is high-intensity work for the bot because many of the articles have dozens or even hundreds of citations. I dunno Headbomb's methodology for building those jobs or what numbers can be estimated from that, but assume that tens of thousands of such pages remain to be processed.
my jobs targeting bare URLs are focused on a longstanding problem of the core policy WP:V being undermined by linkrot, which may become unfixable. I have lists already prepared of 75,000 articles which need the bot's attention, and have a new methodology mostly mapped out to tackle about 300,000 more of the 450k more articles with bare URL refs.

My lists are (like Headbomb's lists) all of bot-fixable problems, so they don't waste the bot's time, but they do not tackle such high-value issues as Headbomb's list, so I regard mine as a lesser priority than Headbomb's.

So whatever the bot's capacity, there will be enough high priority high-efficiency work to keep it busy for a long time to come. it is not all helpful for that work to be delayed or displaced because one editor likes to run big job but doesn't like doing the prep work to create productive jobs.

In the last few weeks I have approached 4 editors about what seemed to me to poor use of the bot.

JamCor eventually stopped feeding the bot the same list of ~200 articles several times per day: User talk:JamCor#Citation_bot
Awkwafaba kindly and promptly agreed to stop feeding the bot sets of mostly-declined AFC submissions: User_talk:Awkwafaba#Use_of_Citation_bot
Eastmain seems for now to have stopped feeding the bot sets of mostly-declined AFC submissions: User talk:Eastmain#Citation_bot.

Only Abductive persists. --BrownHairedGirl (talk) • (contribs) 20:57, 27 August 2021 (UTC)[]

Reading the discussion above, I think that this issue is becoming increasingly adversarial and perhaps a concrete proposal for action would be a way to fix this.

This could include:

1) If it is an easy technical fix (the maintainers would need to chime in on this), bringing the PHP issue to someone with tool service access and increase the bots capacity

2) Adopt a definition/policy on "speculative trawling", perhaps with a notice on the bot page to nudge users into considering the bots limited capacity.

3) Any other ideas?

@Abductive @BrownHairedGirl @Headbomb @Redalert2fan RoanokeVirginia (talk) 23:11, 27 August 2021 (UTC)[]

Other ideas:

The idea of revisiting the bot's basic tuning on its dwell times, throttling and other patches now that stability has been improved deserves consideration. If the bot could run just a touch faster at every step, the overall load would be reduced. Abductive (reasoning) 00:46, 28 August 2021 (UTC)[]
Set aside one of the four channels for single use. Abductive (reasoning) 00:46, 28 August 2021 (UTC)[]
@Abductive: setting aside one channel for single use wouldn't help much, because in addition to huge batch jobs, Abductive has been simultaneously being flooding the bot with masses of individual page requests. See e.g. this set of 1,500 bot edits. By using my browser's Ctl-F to search in the page, I find that Abductive | #UCB_webform matches 370 edits (a batch job of 2200 pages), and a search for Abductive | #UCB_toolbar matches a further 114 pages.

So basically, Abductive has been WP:GAMING the limits by occupying one of the bot's 4 channels with a batch job, and then monopolising another channel with some sort of systematic flood of single jobs. (This also happened on at least one pervious day this week).

Note those 114 single use edits (the toolbar edits) are only the pages which were actually edited. It is likely that there were other toolbar requests which did not lead to the bot making an edit.

Note also that the 114 toolbar edits spanned a period from 00:58, 28 August 2021 to 08:36, 28 August 2021. That is after @Headbomb's warning[2] at 19:34 27 August about WP:DE and WP:IDHT behaviour ... and it is 18 days after @AManWithNoPlan cut the limit on category batches twice in response to Abductive's abuse of the higher limit.

It is also 28 days since I rejected Abductive's offer to be a meatpuppet for me (see User talk:Abductive/Archive_21#Talk_page_usage) by running some of my bareURL batches. That would have amounted to my jobs effectively taking two of the bot's four channels, which would be WP:GAMING the bot's scheduler and impeding other editor's access to the bot.

This looks to me like intentional WP:DE, which will not be resolved by anything along the lines @RoanokeVirginia's thoughtful proposals. The only solution I can see here is some sort of restraint on Abductive's use of Citation bot. --BrownHairedGirl (talk) • (contribs) 18:19, 28 August 2021 (UTC)[]
Request a Citation bot 2. I think this is something any one of us can do, correct? Abductive (reasoning) 00:46, 28 August 2021 (UTC)[]

Thanks for the thoughtful suggestions, @RoanokeVirginia.

A capacity increase would be very welcome, but it is very unlikely that any remotely feasible capacity increase could remove the need to use that capacity efficiently and effectively. So Abductive's conduct would still be a problem
a statement on effective use of the bot sounds like a good idea, but I don't expect that a nudge would have any impact on an editor who has repeatedly rejected nudges.

Having been using AWB heavily for 15 years, and run several bot tasks, I am used to the fact that big jobs usually require a lot of preparation if they are to be done efficiently and accurately. So doing prep work and test runs before feeding a big job to Citation bot is second nature to me, and probably also to Headbomb.

By contrast, Abductive appears to want to A) continually set the bot off on big runs with no clear purpose or selection criteria, just because they can, and B) objects to any prep work. The problem is the combination of A and B. Many other editors use Citation bot without extensive prep work, but they usually do so for short bursts or for bigger jobs targeted on a particular topic area. The problem with Abductive's jobs is that they lock up one of the bot's job slots near permanently, often with very low return, for no apparent reason. --BrownHairedGirl (talk) • (contribs) 15:35, 28 August 2021 (UTC)[]

I don't object to prep work, I'm just pointing out that it is inefficient. If, as you say above, you have tens to hundreds of thousands of bare url articles to run by the bot, shouldn't you be accommodated somehow? I have been running 2200 of your bare urls at a time when it looks like the load on the bot is low, and holding off on runs when it looks like the load is high. Abductive (reasoning) 18:40, 28 August 2021 (UTC)[]

@Abductive: I have not supplied you with a list of bare URLs, so you could not have have been running 2200 of your [i.e. BHG's] bare urls. You rejected my suggestion (at User talk:Abductive#A_citation_bot_job) that you tackle the pages which transclude {{Bare URL inline}}, and its transclusion count has dropped by only 84 in the last 25 days, so I doubt you were tackling that.

But my main point here is that you have been WP:GAMING the system by flooding the bot with hundreds of toolbar requests while also running a big batch job, so your holding off claim seem to me to be bogus. --BrownHairedGirl (talk) • (contribs) 19:07, 28 August 2021 (UTC)[]

As a matter of fact I have been running from {{Bare URL inline}}, and nobody is gaming the system. Using the bot for its intended purposes is legitimate. Abductive (reasoning) 19:14, 28 August 2021 (UTC)[]

@Abductive: Gaming the bot's limits is not legitimate.

And I see no evidence of your claim that you have been targeting {{Bare URL inline}}. Can you provide any evidence of that claim? --BrownHairedGirl (talk) • (contribs) 19:21, 28 August 2021 (UTC)[]

I drew a list from Category:CS1 errors: bare URL and have been running them 2200 at a time. Just now I requested a run of 2200 drawn directly from {{Bare URL inline}}. See if that brings down your metric. Don't forget that the bot isn't very good at fixing bare urls. Abductive (reasoning) 20:10, 28 August 2021 (UTC)[]

@Abductive: the bot's latest list of contribs shows a batch of 220 page submitted by you.

Your use of a webform rather than links from a page impedes scrutiny, because only the pages which have actually been edited are visible to other editors. However, checking the recent bot contribs for the pages which have been edited, the list appears to consist partly of pages which transcluded {{Bare URL inline}}, e.g. [3],[4],[5] ... but also some pages which appear to have never transcluded {{Bare URL inline}}, e.g. COVID-19 vaccination in Malaysia (see bot edit [6], and the unsuccessful WikiBlame search for "Bare URL inline"[7]).

So your current batch is a mix of {{Bare URL inline}} and something else ... and the fact that you have started that batch now does not alter the fact that the evidence so far indicates that your claim to have been processing batches of 2200 {{Bare URL inline}} pages is false. My bare URL cleanup jobs average about 30% of pages being completed cleared, and the first 100 of your latest batch shows 46/100 having bare URLs replaced. So if you had previously run batches of 2200 {{Bare URL inline}} pages, we should have seen multiple instances of 700 pages being cleared. That has not happened.

And you have still not addressed the issue of how you have been WP:GAMING the bot's limits by stashing up hundreds of toolbar pages whilst running a big batch. --BrownHairedGirl (talk) • (contribs) 20:53, 28 August 2021 (UTC)[]

I have been running the bot on thousands of members of the bare url category, and just now on the list of transclusions of {{Bare URL inline}}. Why there should be a difference in its ability to clear one more than the other I cannot say. Abductive (reasoning) 21:04, 28 August 2021 (UTC)[]

@Abductive: Thank you for finally confirming that your claims to to have previously processed batches of 2200 pages transcluding {{Bare URL inline}} were false.

Making repeated false assertions is not collaborative conduct. --BrownHairedGirl (talk) • (contribs) 21:12, 28 August 2021 (UTC)[]

I have been running the bot on thousands of bare urls, for you. I did not deliberately make a false statement. Abductive (reasoning) 21:16, 28 August 2021 (UTC)[]

@Abductive: So the evidence I can see indicates that your statements are false.

I have not provided you with any lists of articles with bare URLs. You have been processing pages in Category:CS1 errors: bare URL, which is a related-but-different issue which you chose. And in the last few hours, you started processing pages one batch of 2200 pages, some of which transclude {{Bare URL inline}} ... but until this evening, I see no other such batch.

I have asked you for evidence to support your claims, and you have provided none. If you wish to sustain your assertion that you have previously submitted batch jobs of pages which transclude {{Bare URL inline}}, please provide some diffs of the bot's edits in some batches, or better still some contribs lists which show multiple bot edits on those batches. --BrownHairedGirl (talk) • (contribs) 21:42, 28 August 2021 (UTC)[]

Nobody is misusing the bot. Requesting a category run is a function of the bot. Requesting individual runs of articles is a major purpose of the bot, and it should be noted that if all four channels are in use, nobody can request the bot to run on an article they just created. Just about any random category run these days will only achieve 30%; so what does it matter if it a big or small category? You are running the bot on a collection of articles in which you are achieving only 30% of your goal of fixing bare urls. But you are not misusing the bot either, or gaming the system by running as close to 2200 as you can each time. Abductive (reasoning) 21:04, 28 August 2021 (UTC)[]

@Abductive: it is hard to assess whether you misunderstand he issue being discussed, or whether you are being intentionally evasive.

Indeed, Requesting individual runs of articles is a major purpose of the bot ... but your rapid requests of hundreds of individual articles within a few hours while you are already running a max-size batch is in effect running a second batch, and thereby WP:GAMING the bot's limits. If that is not clear to you, then we have a big problem.

Your assertion that Just about any random category run these days will only achieve 30% is demonstrably false in two respects:

this thread has plenty of evidence of you feeding random categories to the bot, and getting edit rates of only ~10%, including one category with only a 7% edit rate.
you are conflating two difft percentages. My bot runs of pages with bare URLs usually clears on a first pass all the bare URLs on about 30% of the pages. However, on other pages in the set, it removes some of the bare URLs and/or makes other changes. So the total edit rate is usually well over 50%.

This is beyond ridiculous. --BrownHairedGirl (talk) • (contribs) 21:32, 28 August 2021 (UTC)[]

The fact that your bare urls get above 50% is one of the reasons I am assisting you in running them 2200 at a time. Individual requests fix errors, and do more to hold open a channel (from runs) than to clog a channel. Occassional low productivity runs on categories is a matter of bad luck. My runs are in no way impeding your efforts, and in fact are helping them, so you should not be at all concerned about my use of the bot. Abductive (reasoning) 21:42, 28 August 2021 (UTC)[]

@Abductive: This is ridiculous.

No, you are not running my bare URL jobs 2200 at a time. A I explained above, you are not running em at all: Category:CS1 errors: bare URL is a separate issue.
. But hundreds of rapid individual jobs do clog a channel, as surely as if they were batch. They are in effect a second channel.
My runs are in no way impeding your efforts. Again, nonsense: your routine use of two channels slows down the bot, and impedes other editors from even starting new jobs.
so you should not be at all concerned about my use of the bot. This is a severe case of WP:IDHT.

I fear that this may have to be escalated. --BrownHairedGirl (talk) • (contribs) 22:00, 28 August 2021 (UTC)[]

Don't you think your efforts should be directed towards improving this and other bots, and not at a misguided effort to get me to stop using the bot "wrong"? Abductive (reasoning) 22:05, 28 August 2021 (UTC)[]

I would dearly love to be able to focus on my work, rather than having to divert my efforts into trying dissuade one WP:IDHT editor from persistently disrupting my work and other's work by systematically abusing this bot and repeatedly making misleading and/or false assertions when i attempt to discuss the problem. --BrownHairedGirl (talk) • (contribs) 22:55, 28 August 2021 (UTC)[]

I never deliberately made false assertions, and I am not abusing the bot or disrupting anything. I am using the bot as intended, and doing a pretty good job of it. Abductive (reasoning) 23:04, 28 August 2021 (UTC)[]

If your false assertions were unintentional, then then I fear that you may not comprehend the issues adequately.

The bot has twice had to be reconfigured to prevent your abuse, so you are demonstrably not using it as intended,

And if you really think that repeatedly wasting the bot's time by feeding it huge categories which need almost no bot work is what the bot was intended for, then you have a very odd idea of what bots are intended for. --BrownHairedGirl (talk) • (contribs) 23:12, 28 August 2021 (UTC)[]

Reconfigured to make the bot use its 4 channels more effectively. And a review of the totality of my requests will show that I am getting above average results. Abductive (reasoning) 23:40, 28 August 2021 (UTC)[]

No, you have not Reconfigured to make the bot use its 4 channels more effectively. All you have done is to selfishlyand disruptively grab 2 of the 4 channels for yourself. This makes the bot work no faster, and use its time no more effectively; your WP:GAMING of the limits just gives you a bigger slice of the bot's time and denies other editors access to the bot.

You offer no evidence at all of the claimed effectiveness of your request; that is just an empty boast. It is not possible to assess that without access to the bot's logs, because the bot's contribs lists provide no way of telling whether any of your single requests led to no edit. For all we can see from the contribs list, it may well be that the edit rate for your single requests is as low as the 7% you got from a category. --BrownHairedGirl (talk) • (contribs) 04:20, 29 August 2021 (UTC)[]

At no point am I selfishly or disruptively grabbing channels. First off, that's not even possible, and second, the bot is improving the encylopedia when it makes corrections. Additionally, they often happen to be bare url articles, a project of yours. How is that selfish? Nobody is calling you selfish for running thousands of articles past the bot. Abductive (reasoning) 06:00, 29 August 2021 (UTC)[]

You are cherry-picking evidence. Why not mention my recent category run that got lucky and found and corrected at a 70% rate? And to suggest that a handful of individual requests makes it immpossible to assess my overall rate is silly, it is easy to assess my overall rate using just the category and multiple runs. But why are you even interested in my activities? As I said before they are not impeding your runs, and in fact I am running your bare url finds. Right now, for instance, I am not running anything, because the bot seems to be heavily loaded. Abductive (reasoning) 06:00, 29 August 2021 (UTC)[]

Please find another place to argue. Thanks. --Izno (talk) 16:32, 29 August 2021 (UTC)[]

@Izno: surely this is the appropriate place to address systematic misuse of the bot? --BrownHairedGirl (talk) • (contribs) 17:31, 29 August 2021 (UTC)[]

You said it yourself, you may need to escalate. I am suggesting you do so, because it seems clear to me that you will not be able to persuade Abductive here. Izno (talk) 18:08, 29 August 2021 (UTC)[]

Fair enough, @Izno. When it recurs, I will escalate. --BrownHairedGirl (talk) • (contribs) 20:11, 29 August 2021 (UTC)[]

One editor, two simultaneous batch jobs[edit]

Yet again, @Abductive is hogging the bot by running two batch jobs simultaneously.

The latest bot contribs shows that the bot is simultaneously processing both:

Category:Gambling terminology (63 pages), with an edit rate so far of 32/61 pages, i.e. 53%
Category:Papermaking (88 pages), with an edit rate of 17/88 pages, i.e. 19%

More low-return speculative trawling, using half the bot's capacity, delaying jobs targeted at bot-fixable issues, and locking out single-request jobs. This is more WP:DE. --BrownHairedGirl (talk) • (contribs) 15:19, 29 August 2021 (UTC)[]

I requested the first one, and nothing happened for a very long time. I entered the second one, and nothing happened for a very long time. I have entered more, and still nothing has happened. All of a sudden, both started running. Abductive (reasoning) 15:27, 29 August 2021 (UTC)[]

I am going to have to ask you to stop complaining about my so-called speculative trawling. Using the bot to clean up a category's citations is a legitimate use of the bot, and it is not a problem. Please strike your accusation of disruptive editing. Abductive (reasoning) 15:27, 29 August 2021 (UTC)[]

On the contrary, your long pattern of using the bot disruptively has been well-documented. This is just the latest round of the saga. I will stop complaining about your disruption when you stop the disruption.

Why were you entering a second category before the first one had even started processing? Are you trying to stash up batch jobs in some sort of private queue? --BrownHairedGirl (talk) • (contribs) 15:59, 29 August 2021 (UTC)[]

You are accusing me of acting in bad faith and of being disruptive, when in fact it is some sort of error with the bot. Please desist. Abductive (reasoning) 16:09, 29 August 2021 (UTC)[]

Please stop being disruptive, and please stop WP:GAMING the bot's queuing system.

The error in the bot's functionality was simply that it failed to block your gaming of the queue, a practice which you have already acknowledged above. If you were not gaming the queue, that bot functionality would be irrelevant. --BrownHairedGirl (talk) • (contribs) 16:13, 29 August 2021 (UTC)[]

As I said above, please find another place to argue. Thanks. --Izno (talk) 16:38, 29 August 2021 (UTC)[]

Izno, as i replied above, surely this is the place to raise misuse of the bot? --BrownHairedGirl (talk) • (contribs) 17:41, 29 August 2021 (UTC)[]

BrownHairedGirl, What most of us (in my opinion I feel) are getting at is that you have problems with 1 specific users actions, and this discussion is only between the 2 of you. While it is about how the bot is used, it seems to have long passed anything to do with the edits of the bot, what operators should do etc.. I think originally starting a discussion here was a good idea, as it should have been done, and that let to most people reconsidering the use of the bot, however this has turned in to a back and forward situation that nobody here can help. It's not about the bot anymore, it reads as a conflict between 2 people.

Now apparently this specific section shows a technical problem "One editor, two simultaneous batch jobs" It is great to report that, as it should not be possible. this is exactly the place to report technical problems.

But instead of just sticking to a detail about the problem, it instantly devolved in to back and forth accusations.

Summary: It was a good place to start a discussion about bot usage due to limit resources which should also be fixed, but it is now just a conflict between 2 people. Suggestion: If you find technical issues just report them, and wait for the operator to respond, if its also combined with what you see as "misuse" it seems we have passed this venue and should be discussed elsewhere. Redalert2fan (talk) 13:34, 30 August 2021 (UTC)[]

@Redalert2fan: the only impact of Abductive's systematic misuse of the bot is to impede other editor's use of the bot, so that belongs on the page where editors are discussing how to use the bot.

See the section below User talk:Citation bot/Archive 27#Citation bot, where Fade258 posted[8] about the bot not having processed a page after several hours of waiting. That happened because an earlier round of Abductive's gaming of the bot's queuing system had blocked new jobs from starting. Fade258's post didn't do a great job of explaining the problem they had encountered, but this was the right place to raise the no-response-from-bot problem. Editors such as Fade258 would be much less likely to find the bot overloaded if we didn't have one editor systematically occupying 2 of the 4 entries. --BrownHairedGirl (talk) • (contribs) 18:06, 30 August 2021 (UTC)[]

I am very sorry that the bot somehow allowed me to make more than one run at a time. But I did not "game" anything, and your continued accusations amount to incivility. Abductive (reasoning) 21:58, 30 August 2021 (UTC)[]

My accusations are both a) evidenced, and b) supported by your own assertions. The technical issue is that the bot mishandled to block your request for a second batch before the first one had even started; but that technical issue arose only because of your lack of restraint. --BrownHairedGirl (talk) • (contribs) 23:02, 30 August 2021 (UTC)[]

You asked if this is the place, I explained why I think it is not. I will leave it at that, I have no problem if you have issues with the edits of abductive, but if you have to fight back me as well on a question that you asked, and get another hit on abductive like that I'm no longer interested in explaining. You picked one specific section of my comment out just to be uncivil towards someone else. I Urge you to reconsider as well, it is looking for me increasingly like the bot should only do what you want.

And this is just the comment from me I think should not be posted on this page, but I have to do. Redalert2fan (talk) 22:40, 30 August 2021 (UTC)[]

@Redalert2fan: my reply to you is not a "fight", and it was in no way uncivil.

And please AGF. I do not in any way believe that the bot should only do what I want; my objections are to misuses which I have documented thoroughly. --BrownHairedGirl (talk) • (contribs) 22:57, 30 August 2021 (UTC)[]

And as I said previously, I do not agree with the term misuse being used, and now it is thrown out like a buzzword all the time, but I specificaly left out my own view on this this time above because I already know 0 progress is going to be made.

If I'm honest, whenever I want to use the bot its not abductive that is blocking out people anymore, you are taking up one of the slots all the time as well, claiming that your edits are the most important. I think that everyone should be allowed the to use this tool if they think they can improve something with it, and not keep being targeted on their edits. "AGF" huh, how about aplying this to everyone that uses the bot, you didnt do that above with abductive either. Redalert2fan (talk) 23:12, 30 August 2021 (UTC)[]

@Redalert2fan: I do not use the word misuse lightly. I do use it in this case, because I have clear evidence, documented above, that Abductive has been repeatedly misusing the bot in two ways: a) repeatedly flooding it with lists of article which lead to very low rates of change; b) repeatedly gaming the bot's queuing system to get the bot to simultaneously run two batch jobs for Abductive. Before you dismiss my complaints, please read the evidence, which is spread over several weeks in threads above.

Your assertion that I am claiming that your [BHG's] edits are the most important is demonstrably false. At no point have I made or implied any such claim. On the contrary, I have repeatedly stressed the importance of getting bot to do actual edits rater than processing pages which need no changes, and I have explicitly stated that I regard the academic journal refs targeted by Headbomb as a more important task.[9]. So please strike that slur.

Yes, I am usually taking up one of the bot' slots. But

I take up one of the bot's slots, whereas Abductive has been systematically flooding the bot with two simultaneous tasks (see evidence here[10])
Unlike Abductive, I do not set the bot off on large unproductive speculative trawls. Note that jobs which I ask the bot to do are highly visible because the bot has work to do on most pages; Abductive's low-return speculative trawls leave less trace because the bot' contribs lists does not log pages where th boy makes no edits. Note also that Abductive actually defends their long history of wasting the bot's time on huge unproductive trawls: see e.g. [11] Bots exist to do tedious editing tasks. Your notion that editors have to do the tedious work before giving the bot a task is contrary to the purpose of bots.. Those unproductive, speculative trawls by Abductive repeatedly tie up 1/4 of the bot's capacity for days on end, because Abductive will not restrict their use of the bot to batches which actually achieve something. Yet Redalert2fan somehow thinks it is very uncivil of me to point this out.

I remind you that WP:AGF explicitly says this guideline does not require that editors continue to assume good faith in the presence of obvious evidence to the contrary. In the case of Abductive, I have documented that obvious evidence to the contrary.

If you have concerns that too much of the bot's time is taken up by targeted batch jobs of large sets of pages with identified bot-fixable problems, such as those suggested by me, Headbomb, or Josve05a, then please open a discussion on that issue. I think it would be helpful to have such a meta-discussion about how to balance the bot between batch tasks and individual jobs. But please stop allowing your concerns about that to make false accusations about my motives and goals ... an please stop conflating bot overload with efforts by Abductive to use the bot unproductively and to actively game its limits. --BrownHairedGirl (talk) • (contribs) 00:21, 31 August 2021 (UTC)[]

I do not feel this is productive in any way nor do I think this is the right place for this as I said before. I'm not interested in clogging this page up further either. I will leave it here. Sorry if this is not satisfactory, but I feel it is for the best. Redalert2fan (talk) 10:42, 31 August 2021 (UTC)[]

I've been following this with a little interest best this is on my watchlist. I've not followed everything. BHG grabs my attention due to the working of bare-URL identification bot runs. But it is reasonable to discuss whether a bot startup can be modified with an algorithm to avoid unfair usage; or perhaps documenation if further. I'm not in the greatest of standing at the moment, but Wbm1058 has a few hours left on my watchlist and couldn't help but notice the comment (I think its been their for a long time). ": Secret to winning the Race Against the Machine: become an expert bot programmer, and hope that the bots don't learn to program themselves. HAL?". That comments been there for a long time, got me to dreaming about becoming an "expert bot programmer" (I've got lost from when input ceased to be from 80 column punch cards so a bit too late in life methinks). Wbm1058's seems to know something about bots and might be able to comment/mediate here? Thankyou. Djm-leighpark (talk) 23:28, 30 August 2021 (UTC)[]

A couple of thoughts. Is abductive a primary maintainer of the bot in which case could reasonably request an exemption for two bots if only was for low maintenance. The other might be a lightweight new control/monitor bot to monitor the runners and queuers and if user A was running two or more bots and another user B had a scheduled run holding for some time the control bot would chop one of users A bots and reschedule it to start from the place it left of. Actually this would likely be beyond anyones capability to do in a reasonable time period. Sorry I come up with stupid questions. Thankyou. Djm-leighpark (talk) 01:34, 31 August 2021 (UTC)[]

someone who knows how to and has permission to so needs to increase the total number of PHP proceses from 2*2 to something bigger/ AManWithNoPlan (talk) 01:47, 31 August 2021 (UTC)[]

not to state the obvious, but shouldn't there be a place were we can find someone like that? Redalert2fan (talk) 12:00, 31 August 2021 (UTC)[]

multiple efforts have been made. I personally do not plan to try again. Not worth wasting my time. AManWithNoPlan (talk) 12:57, 31 August 2021 (UTC)[]

@AManWithNoPlan: If you have moment, please could you tell me whether my understanding is correct?

AIUI:

the bot processes one page at a time, and its throughput is constrained by the time it takes the bot's core to parse each page, invoke zotero etc, then process the results and save any changes.
there is only one instance of that core functionality
the order in which pages are processed involves taking pages one at a time from each of the 4 PHP processes in turn (however such turns are allocated)

So AIUI adding more PHP processes would allow the bot's time to be shared between more users, but would not increase overall throughput. Is that correct? --BrownHairedGirl (talk) • (contribs) 22:10, 31 August 2021 (UTC)[]

All PHP jobs are seperate with no shared processing. We are way below our memory and cpu limits of the tool server with 4 processes. The URL expander is run by wikipedia, so that might be a bottleneck, but only for that step. AManWithNoPlan (talk) 01:15, 1 September 2021 (UTC)[]

@Legoktm: per this discussion is this process a violation of Toolforge Rule #6: "Do not provide direct access to Cloud Services resources to unauthenticated users"? Or has this been Toolforge admin vetted? wbm1058 (talk) 15:53, 1 September 2021 (UTC)[]

Everyone is authenticated. Headbomb {t · c · p · b} 15:55, 1 September 2021 (UTC)[]

Perhaps, but I'm hearing complaints that the bot "allows for anyone to make arbitrary queries". – wbm1058 (talk) 16:00, 1 September 2021 (UTC)[]

Where at? Headbomb {t · c · p · b} 16:09, 1 September 2021 (UTC)[]

BrownHairedGirl, you, and maybe others have complained that Abductive makes arbitrary queries. Essentially you seem to be complaining that Abductive launches "DoS". wbm1058 (talk) 16:14, 1 September 2021 (UTC)[]

"DDoS" on Citation Bot, maybe, not DDoS against Toolforge. Headbomb {t · c · p · b} 16:23, 1 September 2021 (UTC)[]

His requests did fix stuff, they are not vandalism or something like that. The concern from some was/is that due to the limited capacity the bot has they were not optimal, out of large request only a very small amount of pages were edited, the rest there was nothing, or only minor fixes were made. Currently it is better to prioritise what the bot is being used for, pages there is some certainty ahead that large or more important fixes will be done, and prevent the bot from spending time on checking pages there is nothing to fix. BrownHairedGirl seems to have developed a list of pages that fit these criteria.

Now if the capacity was larger there wouldn't be much of a problem with those requests from Abductive, since they did contain some pages that were fixed.

It is not a case of someone just filling up the queue with the same pages 24/7 nor knowingly feeding pages that have nothing to do to specifically block others from using the bot. Even though some like me feel like anyone should be able to request a page or list they want to have checked, It currently is just a case of how we all use this limited resource together in the most efficient way. Redalert2fan (talk) 16:29, 1 September 2021 (UTC)[]

@Redalert2fan: my complaint about Abductive is that they:

routinely fill one of the bot's 4 channels with large sets of pages where the bot has v little to do. This waste of the bot's resources could be avoided by testing samples of these sets before processing the whole lot, but even tho I showed Abductive how to do that, they strenuously resist the principle of preparation work.
repeatedly fill a second channel of the bot's 4 channels by piling up hundreds of individual page requests while the bot is already processing a batch of theirs.
This works because the bot won't allow a user to run two batches simultaneously, but it does allow a user to submit an individual job while the batch is being processed. That's good bot design, because it allows an editor to submit a batch and not be locked out from checking an individual page that they have been working on ... but Abductive has deliberately gaming this system by submitting hundreds of individual pages while their batch is running.

@Wbm1058: I am not sure whether Abductive's use of the bot counts as DDoS. I don't think that Abductive's goal is to deprive others of the use of the bot, just that Abductive shows i) a reckless disregard for how the effect of their actions is to deny others use of the bot; ii) no particular interest in any type of article or any topic area, just a desire to feed lots of pages to the bot.

I agree with Redalert2fan that this is about how we all use this limited resource together in the most efficient way. I think we need some guidelines on how to do that, and I will try to draft something. --BrownHairedGirl (talk) • (contribs) 04:46, 6 September 2021 (UTC)[]

In an ideal world, the bot would only allow a maximum of ONE batch job at a time and that it would have a queue for submission of such batch jobs. It is immensely frustrating to 'ordinary' editors who, having spent a lot of time on an extensive cleaned up and want to get the citations checked, get no response or (eventually) a failure. Batch jobs are by definition not needed in real time and, until there is such a queuing mechanism, may I suggest that the guideline states an absolute ban on submitting more than one run before the preceding one has finished (subject to a 48 hour [72 hour?] timeout). If there is any way to encapsulate the rest of my frustration in the guideline, I would be most grateful. --John Maynard Friedman (talk) 10:39, 6 September 2021 (UTC)[]

@John Maynard Friedman: I share your frustration, which is why I have repeatedly raised this issue.

However, I think your suggested remedy is far too extreme.

I watch the bot contribs very closely, because I follow behind it cleaning up as many as I can of the pages where it seems not to have fixed any bare URLs.

It is quite rare for all 4 channels to be simultaneously in use by batch jobs, and so long as one channel is free then individual requests such as yours will be processed within a few minutes ... unless Abductive has gamed the system by stashing up dozens or hundreds of individual page requests. There is no need to limit the bot to one batch job at a time, and any attempt to do so would seriously and unnecessarily impede the bot's capacity to process batches.

I think that if there is some throttling of batch jobs, a distinction should be made between batches which have been selected because they concentrate bot-fixable problems (such as my sets of articles with bare URLs, or Headbomb's sets of articles citing scholarly journals) and what I call "speculative trawls", i.e. sets of articles selected on the chance that the bot may find something.

The speculative trawls by Abductive were a serious drain on the bot when the category limit was 4,400 pages and Abductive was stashing up requests for categories which fell just under that limit, often with very low returns. In response to that abuse, the size limit for categories was cut twice, initially to 1,100 and then to 550.

Abductive continues to run a lot of speculative trawls based purely on the size of the category. For example, this morning (see 2000 bot edits) they have fed the bot first Category:Articles with specifically marked weasel-worded phrases from August 2021 (497 pages) and then Category:Articles with specifically marked weasel-worded phrases from July 2021 (460 pages). Neither category concentrates pages by topic, so there is no topic-of-interest involved, and both categories are based on an attribute which the bot cannot fix. The only reason I can see for selecting them is that they fall just under the bot's category size limit.

The bot has not had many other jobs this morning, so Abductive's speculative trawls haven't significantly delayed anything else. But at most other times, this use of the bot does create bottlenecks.

We don't need to hobble the bot's throughput of targeted batch jobs just to restrain one editor who shows reckless disregard for the effect on others. --BrownHairedGirl (talk) • (contribs) 13:19, 6 September 2021 (UTC)[]

Instead of uselessly complaining about prefectly legitimate uses of the bot, concerned senior editors should go to the Village Pump and request that this be fixed in a way that the bot operator has suggested, but which is outside his control. Abductive (reasoning) 23:17, 7 September 2021 (UTC)[]

Sigh.

Gaming the queuing system to deprive other editors of prompt processing of their requests is not a legitimate use of the bot.

Repeatedly wasting the bot's time on low-return speculative trawls is not a legitimate use of the bot.

Regardless of what improvements might be made to the bot, Wikipedia is a collaborative project, and editors should not systematically disrupt the work of others. --BrownHairedGirl (talk) • (contribs) 06:19, 8 September 2021 (UTC)[]

Repeatedly complaining here will accomplish nothing. The jobs I request from the bot have been fixing lots of high-readership articles, which is one of the metrics I use to select more runs. I have been refraining from starting runs when there are already three big runs going, but at some point new editors are going to discover the bot and then there will be times when there are four large runs going. It would be advisable for you to use your energy to cajole the technical folks at the Village Pump into doing something. Abductive (reasoning) 18:38, 8 September 2021 (UTC)[]

For goodness sake. Category:Articles with specifically marked weasel-worded phrases from August 2021 are collections of articles by a cleanup tag, not by readership. --BrownHairedGirl (talk) • (contribs) 20:17, 8 September 2021 (UTC)[]

True, but if you look at the articles in those categories, they tend to be ones that attract readers, attract users who add questionable material to them, and then attract editors who add the weasel word tags. And the bot is correcting a reasonably high percentage of them. Win-win. Abductive (reasoning) 20:23, 8 September 2021 (UTC)[]

According to the pageviews tool, the 492 pages in that category have a combined average pageviews of 413,000 per day, or 839 per page per day, and there are 217 articles in that category with less than 100 views per day, and 71 with less than 10 views per day. That's hardly high-readership, so the conclusion that articles with weasel word tags attract readers isn't true. * Pppery * _{it has begun...} 00:52, 9 September 2021 (UTC)[]

Wikipedia gets about 255 million pageviews a day. There are 6,373,026 articles on Wikipedia (492/6,373,026)*255,000,000 = 19,686, which is 21 times less than 413,000. So you are wrong by more than an order of magnitude. A simple glance at the articles in the category would reveal that there are a lot of high-interest articles in there. For example, the bot just fixed Deborah Birx. Heard of her, right? Abductive (reasoning) 08:30, 9 September 2021 (UTC)[]

Cherrypicking one example out of dozens of sets of hundreds of articles is no help. And averages obscure the fact that as ppery has shown, a significant proportion of these articles have low page views.

If you want to set the bot to work on highly-viewed articles, then use a methodology which selects only highly-viewed articles, then work through it systematically, avoiding duplicates in your list and articles which the bot has already processed in the last few months. --BrownHairedGirl (talk) • (contribs) 01:04, 11 September 2021 (UTC)[]

My activities are helping build the encyclopedia. Removing duplicates is not the best idea, since the bot often takes more than one pass to make all possible corrections. My offer still stands; if you like, place a list of articles in User:Abductive/CS1 errors: bare URL, let me know, and I will run the bot on them. Otherwise you should not concern yourself with other users' legitimate uses of the bot. Abductive (reasoning) 03:06, 12 September 2021 (UTC)[]

Arguably, your activities are hurting the encyclopedia more than helping, by acting as a denial of service attack on Citation bot that prevents others from using it more effectively. —David Eppstein (talk) 04:11, 12 September 2021 (UTC)[]

That presupposes that other large runs are somehow more helpful, which they are not. Also, please note that I make an effort to avoid running the bot when there are three other large jobs runnning, while other users do not. Abductive (reasoning) 04:41, 12 September 2021 (UTC)[]

Well, that summarizes the dispute, doesn't it? In your own mind, your searches are better than anyone else's, despite all evidence to the contrary, and so you think you deserve more access to the bot than anyone else, and will game the system to take that access because you think you deserve it. —David Eppstein (talk) 05:49, 12 September 2021 (UTC)[]

That is incorrect. I have an equal right, and I use the bot judiciously. I do not game the system, and I consider such accusations to be uncivil. Abductive (reasoning) 05:57, 12 September 2021 (UTC)[]

"I have an equal right". Not when your use of the bot is disruptive, low-efficiency, and prevents others from using it more effectively. A run on a random non-citation-related maintenance category is not as "helpful" as targeted cleanup runs, especially when it makes running the bot on individual articles crash/extremely slow because of a lack of ressources. Headbomb {t · c · p · b} 13:34, 12 September 2021 (UTC)[]

@Abductive: I have posted[12] below at #Bot_still_being_abused_by_Abductive an analysis of the bot's latest 1500 edits.

It shows that:

Your batch requests continue to be poorly chosen;
You continue to game the queueing system by flooding the bot with single-page requests while the batches are being processed, amounting to 73% of single-page requests in that period. I estimate the rate of your single-page requests to be one every 3 minutes over 9 hours.
Your choice of single-page requests has no evident basis, and includes a significant proportion of pages (6 of the sampled 30) which wasted the bot's time by follow closely a prev edit by the bot.

So:

Your claim to use the bot judiciously is demonstrably false.
Your claim that you do not game the system is demonstrably false.
Your labelling of @David Eppstein's complaint as uncivil is bogus, because David's complaint was civilly-worded and is demonstrably true.

I also note your offer above[13] to process batches selected by me. That amounts to another attempt to game the queueing system, in this case by inviting me to effectively use two of the bot's 4 channels. I explained this before, so I am surprised to see you suggesting it again. --BrownHairedGirl (talk) • (contribs) 15:13, 12 September 2021 (UTC)[]

@Wbm1058, I'm not super familiar with Citation bot, which Cloud Services resources do you think are being directly provided? If it just allows users to trigger a bot that runs on some pages I think that's fine since that's not directly letting users e.g. make SQL queries or execute bash commands. Please let me know if I missed something. Legoktm (talk) 17:08, 1 September 2021 (UTC)[]

Convert to cite biorxiv[edit]

Status: new bug
Reported by: Headbomb {t · c · p · b} 06:25, 10 September 2021 (UTC)[]

What should happen: [14]
We can't proceed until: Feedback from maintainers

Bot still being abused by Abductive[edit]

Okay, I'm done. BrownHairedGirl, if you believe Abductive is in the wrong, either ask the community via RFC or take it to WP:ANI. Enough is enough. --Izno (talk) 01:22, 14 September 2021 (UTC)[]

@Abductive continues to game the bot's queueing system by make large numbers of single page requests while the bot is processing a batch job requested by Abductive. This has the effect of denying other editors access to the bot, because two of the bot's four channels are in use by Abductive. Both the batch jobs and the single-page requests are poorly-chosen.

See the bot's most recent 1,500 edits, spanning 9 hours. the following observations are all based on that set;

the bot made 223 edits to pages requested by Abductive as batches, from "Category:Articles with specifically marked weasel-worded phrases from month year". (to verify, search the page for Category:Articles with specifically marked weasel-worded phrases from)
The total number of pages scanned by the bot in those batches requested by Abductive is 561 (149 from Category:Articles with specifically marked weasel-worded phrases from April 2020 + 245 from Category:Articles with specifically marked weasel-worded phrases from March 2020 + 167 from Category:Articles with specifically marked weasel-worded phrases from February 2020).
That is an edit rate of 40% (223/561), which initially seems impressive, but on closer scrutiny is misleading. I checked the most recent 30 edits in this set, and found that 7 of them were either purely-cosmetic edits which make no difference to readers, or trivial edits which changed only the type of quote mark. That leaves only 31% potentially significant edits, which is a low return on the bot's efforts.
Of this set of 1500 edits, the bot made 82 edits from single-page requests (to verify, search the page for | #UCB_toolbar)
60 of those 82 edits (i.e. 73%) were from from single-page requests by Abductive (to verify, search the page for Abductive | #UCB_toolbar)
Note that the bot's contribs log only shows pages that were edited. Pages which were analysed but not edited still use one of the bot's 4 channels, but are not shown in the contribs list. So the count of single-page requests by Abductive is almost certainly significantly higher than 60. Given that Abductive's concurrent batch requests get about a 30% edit rate, it seems reasonable to assume that Abductive's single-page requests get a similar edit rate, which would put the total number of Abductive's single-page requests at 200 in this 9-hour period, i.e. 22 per hour, or one every 3 minutes.
I examined the 30 most recent bot edits from single-page requests by Abductive. (To verify, search the page for Abductive | #UCB_toolbar):
- the selection criteria are not evident: none of those 30 pages had recently been edited by Abductive.
- Six of the 30 pages were bizarre choices:
  1. Junior Senior (TV series) — the 2 most recent edits to the page are by Citation bot, in each case requested by Abductive
  2. Neurometric function — the 2 of the 3 most recent edits to the page are by Citation bot; the intervening human edit was trivial
  3. Neurolaw — the 2 most recent edits to the page are by Citation bot, the latest requested by Abductive
  4. Neuroimmunology — the 1st and 6th most recent edits to the page are by Citation bot. $he intervening edits are 3 bot edits and 1 human dab edit, so nothin to make more work for the bot.
  5. Neural backpropagation — the 2 most recent edits to the page are by Citation bot, the latest requested by Abductive
  6. Neuroanatomy of intimacy — the 2 most recent edits to the page are by Citation bot, the latest requested by Abductive

I am posting this just to put it on record, without any hope of Abductive ceasing to abuse their access to the bot. --BrownHairedGirl (talk) • (contribs) 14:40, 12 September 2021 (UTC)[]

The individual articles via UCB_toolbar are fine. It's the batch runs that aren't (e.g. Category:Articles with specifically marked weasel-worded phrases from April 2020). Headbomb {t · c · p · b} 14:44, 12 September 2021 (UTC)[]

@Headbomb: Even if both sets were fine on their own (and as above I think that neither is OK), the combination of the two leaves Abductive using 2 of the bot's 4 channels for much of the time. That impedes other editors who want to check a page. --BrownHairedGirl (talk) • (contribs) 15:17, 12 September 2021 (UTC)[]

A single article will 'hog' the bot for a few seconds to a few minutes tops, which lets other requests in just fine, unless there's several dozens of them made at the same time. Headbomb {t · c · p · b} 15:28, 12 September 2021 (UTC)[]

@Headbomb: Yes, but AFAICS dozens of them have been made at about the same time. I estimate >3 per minute on average over 9 hours, but I doubt they were evenly-spaced over that long a period; much more likely that they were clustered.

I was running some smaller batches today, and when starting new batches I found significant delays (10-30 minutes) in the bot starting to process them, while Abductive's single-page requests were being processed. That's no big deal for my batches, but for someone waiting for the bot to process a page they just edited, a 10-15 minute wait is serious waste of their time. --BrownHairedGirl (talk) • (contribs) 15:37, 12 September 2021 (UTC)[]

I've been following this topic for a while and it seems that different editors have different views on when the bot should be used for batch runs. From what I can see there is plenty of documentation explaining how to use the bot but none about when the bot should be used(or how to select the articles/categories for batch runs). Is it worth having a RFC or something similar to hopefully find consensus on when the bot should be used and when it shouldn't, bearing in mind how limited the capacity of the bot is? It might be that this RFC would only apply to batch runs. If consensus is found then surely it would make it easier to take action against those who ignore it. I'm not that clued up about these things so this might not be workable, but I thought it would be worth suggesting. RicDod (talk) 16:56, 12 September 2021 (UTC)[]

@RicDod: AFAICS, only Abductive thinks that their use of the bot is appropriate. Everyone else seems willing to apply commonsense, and use the bot efficiently without inconveniencing others.

An RFC seems like a very laborious response to one editor's WP:IDHT issues.

I agree that it would be good to have some guidance on when to use the bot, but we shouldn't need to rush into that to put an end to this disruption. --BrownHairedGirl (talk) • (contribs) 17:16, 12 September 2021 (UTC)[]

If one were to look at overall usage of the bot, one would find many, many examples of what looks like people hogging the bot, and of inefficient edits. Abductive (reasoning) 18:25, 12 September 2021 (UTC)[]

Nobody else does it with anywhere near the same persistence or intensity as Abductive. --BrownHairedGirl (talk) • (contribs) 19:05, 12 September 2021 (UTC)[]

You do. Abductive (reasoning) 19:21, 12 September 2021 (UTC)[]

With targeted, high-value runs. Not crapshoot runs. Headbomb {t · c · p · b} 19:23, 12 September 2021 (UTC)[]

I have offered, and User:BrownHairedGirl has sometimes accepted, to run batches that she has targeted. The offer still stands. Abductive (reasoning) 19:45, 12 September 2021 (UTC)[]

Not true. The only batch I sent you was a randomised test set, to help you test the cleanup rate, of Category:CS1 errors: bare URL after you expressed interest in processing it.

I showed you how to test and make batches from Category:CS1 errors: bare URL, which you selected. That is now complete.

I suggested that you run batches of pages which transclude {{Bare URL inline}}. You did that.

Now have you gone back to your long-term pattern of low-return speculative trawls, and yo have repeatedly denounced the notion of selecting and processing batches. --BrownHairedGirl (talk) • (contribs) 19:53, 12 September 2021 (UTC)[]

We have discussed the numbers many times in the last few weeks. So I find it very hard to believe that a competent, good faith editor could claim either that my use of the bot is inefficient, or that I am hogging it in the same way as an editor who accompanies batches with high volume of single-page requests. --BrownHairedGirl (talk) • (contribs) 19:36, 12 September 2021 (UTC)[]

Nobody says that your use of the bot is inefficient, the bot is inefficient. We have all experienced that 10 to 20 minute delay, even when only one or two channels is in use. The bot allows people to request categories that interest them. Any category, no matter the topic, will only get about 35% of its articles fixed. I select categories based on a number of metrics, including article readership. And my selections get above average results. Abductive (reasoning) 19:43, 12 September 2021 (UTC)[]

As an example, I have just requested the bot run Category:Ranchers from Texas, an interesting topic, at 19:52. Supposedly there are 143 articles in the category. Let's see how the bot performs. Abductive (reasoning) 19:54, 12 September 2021 (UTC)[]

Category:Ranchers from Texas just showed up in the feed, 2/143, at 20:18. That's quite an unexplained delay. Abductive (reasoning) 20:25, 12 September 2021 (UTC)[]

It looks like the bot made 29 fixes to Category:Ranchers from Texas, or just above 20%. I have started a run on the interesting but randomly selected Category:General topology at 21:25, which started in 2 minutes, a lot quicker than the 26 minute delay for the last one. Let's see how this one goes. Abductive (reasoning) 21:33, 12 September 2021 (UTC)[]

@Abductive: so in response to complaints about your repeatedly feeding the bot categories which generate low return, your response is to feed the bot two more categories which also produce low return.

And to top it all, you do so at a time when the bot is already very busy.

Which part of "stop doing this" is unclear to you? --BrownHairedGirl (talk) • (contribs) 21:41, 12 September 2021 (UTC)[]

So, it sounds like you consider perfectly legitimate use of the bot on small categories to be "abuse". Any user who runs the bot on a category is therefore "abusing" it. And one channel is being taken up by your run, contributing to making it busy. Is that bad somehow? Abductive (reasoning) 21:47, 12 September 2021 (UTC)[]

For the millionth time: running the bot on a few categories containing articles in topic area where you edit is not the same thing as repeatedly selecting large numbers of big low-return categories just so that you can feed lots of pages to the bot.

Please stop pretending that the distinction is unclear to to you.

And please stop pretending that you dont understand the distinction between high-return batches and low-return batches. --BrownHairedGirl (talk) • (contribs) 21:58, 12 September 2021 (UTC)[]

I just checked the bot's handling of my current job of 1,064 pages from 21st-century deaths, part 4 of 4.

The bot's contribs list shows 440 edits out of the 832 pages processed so far. That's a 58% edit rate.

Since by your own admission, your categories don't exceed 35%, they are wasting the bot's time. --BrownHairedGirl (talk) • (contribs) 20:08, 12 September 2021 (UTC)[]

Bizarre.

You just said above that my use of the bot was inefficient, but now you deny that.

The batches I select get over 50% of pages edited, and Headbomb's batches get even more. So there is no need to waste the bot's time on low-return sets. --BrownHairedGirl (talk) • (contribs) 19:57, 12 September 2021 (UTC)[]

I did not say you were inefficient. I said you use the bot more than I do. Abductive (reasoning) 20:00, 12 September 2021 (UTC)[]

"Above average results" not at 35% on random categories unrelated to a topic or citation-related cleanup category. My runs get 85-90% edit rates. Headbomb {t · c · p · b} 19:56, 12 September 2021 (UTC)[]

You're doing a great job, but your runs aren't category runs. Abductive (reasoning) 20:13, 12 September 2021 (UTC)[]

See User talk:BrownHairedGirl#Since you're curious. I do plenty of category runs. I just pick my categories sanely, instead of unrelated categories like Category:Articles with specifically marked weasel-worded phrases from April 2020. Headbomb {t · c · p · b} 20:20, 12 September 2021 (UTC)[]

@Headbomb: note how Abductive again completely misses the point. There is no requirement for Abductive to use the bot at all, or to use it on categories.

The problem is remains that Abductive keeps on feeding the bot batches with low return. They should stop processing such batches. If they can find a way of selecting batches with higher return, then fine, run those batches ... but just stop running these low-return batches. No batches from abductive would be better than this lot. --BrownHairedGirl (talk) • (contribs) 20:46, 12 September 2021 (UTC)[]

You have an idea of how the bot should be used. Your idea is wrong-headed, and you attempt to impose it on other editors. I am demonstrating that category runs, an allowed function of the bot, have a low rate of return, and that's not anybody's fault. I also refrain from using the bot when it is busy. Abductive (reasoning) 21:55, 12 September 2021 (UTC)[]

You are demonstrating that category runs, when poorly chosen, have a low rate of return, and that's the user's fault. — JohnFromPinckney (talk / edits) 22:03, 12 September 2021 (UTC)[]

The bot completed the run on randomly selected Category:General topology, making fixes to 52 out of 174 articles, just under 30%. Any time a user runs a category, we can expect this, and it is not a problem; the bot is functioning as intended. My choices exceed this level, and have the added benefit of being to high-visibility articles. I also reiterate that I refrain from running the bot when it appears busy. Abductive (reasoning) 22:28, 12 September 2021 (UTC)[]

Sigh. The problem is that a 30% edit rate is half what I achieve with my targeted edits, and 1/3 of what Headbomb achieves. So it's a highly-inefficient use of the bot's time.

It is true that any time a user runs a category, we can expect this ... so stop using categories for high-volume general cleanup.

You know the problem, and you know the solution. --BrownHairedGirl (talk) • (contribs) 23:19, 12 September 2021 (UTC)[]

The random sample category runs above show that my method of choosing categories gets superior results. So they are the opposite of speculative trawling. Furthermore, your idea of proper bot usage, if implemented, would preclude all category runs by all users. Abductive (reasoning) 23:30, 12 September 2021 (UTC)[]

More denialism.

Your method of choosing categories frequently leads to edit rates in the low teens: I have documented many such cases above, so your denials are untruths which must be known to you to be untrue.

Even the best cases you mention get edit rates of ~35%, which is about half of what i get and about a third of what Headbomb gets. his has already been demonstrated to you with documented example, so your denial is another untruth which must be known to you to be untrue.

No, my idea would not preclude all category runs by all users. Category batches are an inefficient but handy way of cleaning up a set of articles concentrating topics which an editor works on. However, that inefficiency makes them a very bad way of making a topic-blind selection articles in need of cleanup, and the problem remains that you are repeatedly using categories for a task to which they are ill-suited. --BrownHairedGirl (talk) • (contribs) 11:43, 13 September 2021 (UTC)[]

Comment I'd just like to be able to use the bot for single pages, but more often than not I get a failed message after about 10 minutes. I know other users have given up using the bot because of these problems. --John B123 (talk) 20:34, 12 September 2021 (UTC)[]
That delay exactly why I have taken the time yet again to document how Abductive's misuse of the bot is blocking such single uses.

As Headbomb notes, the article will still be processed eventually. But you shouldn't have to wait so long for that to happen. --BrownHairedGirl (talk) • (contribs) 20:40, 12 September 2021 (UTC)[]

@John B123: btw, if you just click on 'expand citations' in your sidebar, the bot might look like it crashes, but it will still get to the article eventually. Might take 30 minutes, might take an hour, but it does get processed. Headbomb {t · c · p · b} 20:38, 12 September 2021 (UTC)[]

@Headbomb: Thanks, I'll give that a try. I've been using the 'Citations' button under the edit box which gives a pop-up window saying "Error: Citations request failed" after about 10 minutes if it hasn't run by then. --John B123 (talk) 20:58, 12 September 2021 (UTC)[]

@Headbomb: Tried that and got a browser error page with "citations.toolforge.org unexpectedly closed the connection." after about 10 minutes. --John B123 (talk) 21:02, 12 September 2021 (UTC)[]

@John B123: which page? Headbomb {t · c · p · b} 21:07, 12 September 2021 (UTC)[]

@Headbomb: It was either 8th Canadian Hussars (Princess Louise's) or 2/1st Pioneer Battalion (Australia). --John B123 (talk) 21:16, 12 September 2021 (UTC)[]

@John B123: ran on both, without issue. 8th had an edit, 2/1st had nothing to do. Headbomb {t · c · p · b} 21:48, 12 September 2021 (UTC)[]

@Headbomb: Ok, thanks for having a look. --John B123 (talk) 21:52, 12 September 2021 (UTC)[]

That resolved John B123's issue, one editor's issue and over just two articles – not by discovering any new underlying cause but just by you being lucky that the available slots were not all being taken up batch runs of marginal value at the time of trying. John B123's experience is exactly as I described earlier and why I really hesitate to use the ✅ Citations box unless I don't care that it is likely to hang or fail. I have to ask again: why is more than one batch jobs ever allowed to run at a time? Why can't requests to run a batch job just sit in a queue waiting for capacity to become available, they are not time critical. --John Maynard Friedman (talk) 22:26, 13 September 2021 (UTC)[]

Abductive again. What to do?[edit]

Today Abductive has had the bot run yet another low-return speculative trawl, this time on Category:Edible fungi

Category:Edible fungi has 511 articles, conveniently close to the limit of 550 for categories. (Abductive has a long history of selecting categories whose size is close to the limit) × As can be seen from these 700 bot edits (search the page for Category:Edible fungi), the bot edited only 111 of those 511 articles — a low edit rate of just under 22%.

The bot worked its way through this set from some time before 15:54 UTC to some time after 20:16UTC. This usually is the busiest period of the day for the bot, and today was no exception. The result of the bot trawling through Abductive's set — and deciding that in 78% of cases it had nothing to do — is that the bot was mostly unavailable to other editors for about four-and-a-half hours, because there was no slot available.

Note too that this follows Abductive's claims yesterday that their category-based batches were fine, because they assert that by claimed careful selection they exceed a claimed 30% average for category jobs. That argument about other categories is bogus, because selection other editors achieve much higher edit rates by using other election methods ... and the claim is also bogus, because the choice of this category was so poor.

I have now documented incidents like this repeatedly for about a month. AManWithNoPlan has twice reduced the size limit on category batches to curtail Abductive's abuses, an in the discussions I have seen nobody supporting Abductive's use of the bot, but about six editors denouncing this abuse of the bot. At least one editor has described it as a form of denial-of-service. I cannot know whether DOS is Abductive's intention, but the effect is DOS; and since this has been repeatedly pointed out to Abductive, their disruptive conduct is at best reckless.

So I think it's time to ask Abductive to stop running any batch jobs on Citation bot, however they are constructed. Any thoughts? --BrownHairedGirl (talk) • (contribs) 21:38, 13 September 2021 (UTC)[]

Other users are also running categories. Other users are also running jobs and simultaneously asking the bot to fix individual pages. The bot was not clogged. So there is either a problem with using the bot as intended, for all users except User:BrownHairedGirl, or there is only attention paid to one user for no good reason. Abductive (reasoning) 22:30, 13 September 2021 (UTC)[]

The sensible use of the bot is 'on demand, real-time, one article at a time, manually as required'. Maybe you think you are being one of the gnomes but actually you are being a goblin - I'm sorry but there is a limit to presumption of good faith. Yes, you are engaged in a denial of service attack. Unless and until a batch job queuing mechanism is developed, the facility needs to be removed from all editors, no matter how noble their motivation. Wikipedia editing is no longer the preserve of a few people: what was ok ten years ago is no longer ok, it does't scale. --John Maynard Friedman (talk) 22:41, 13 September 2021 (UTC)[]

It is important to note here that there was always a channel open during the period in question. Abductive (reasoning) 23:12, 13 September 2021 (UTC)[]

Is it possible to have two versions of the bot, the second version of the bot running entirely separately from the original bot and only capable of handling single page requests? --John B123 (talk) 22:58, 13 September 2021 (UTC)[]

Or simply reserve one/two of the four slots for individual runs. That said, only Adbuctive's use is problematic here, so removing batch runs from Abductive is also a solution. Headbomb {t · c · p · b} 22:58, 13 September 2021 (UTC)[]

Does each channel have its own resources, or are the four channels just 4 separate inputs? --John B123 (talk) 23:15, 13 September 2021 (UTC)[]

@John B123: AIUI, from the explanations by AManWithNoPlan, each channel is a separate process ... but for the lookups they all use a common set of external APIs, some of which are overloaded.

Because of that, it is very noticeable that when the bot is processing a page with a lot of refs, other jobs slow down markedly. --BrownHairedGirl (talk) • (contribs) 23:20, 13 September 2021 (UTC)[]

Ideally, they would give any senior editor who asks their own instance of the bot. Abductive (reasoning) 23:43, 13 September 2021 (UTC)[]

More instances of the bot would not remove the bottleneck of the overloaded APIs. In fact, it would probably make it much worse. --BrownHairedGirl (talk) • (contribs) 23:45, 13 September 2021 (UTC)[]

Given that you plan to run 100,000s of articles past the bot, to fix bare urls—which is only a subset of what the bot does—isn't the heavy use going to continue to be a problem for the foreseeable future? Abductive (reasoning) 23:48, 13 September 2021 (UTC)[]

And that's fine, since it's a targetted, high-priority task, with big payoffs. Headbomb {t · c · p · b} 23:53, 13 September 2021 (UTC)[]

Is it fine for the single use editors? I think your suggestion of somehow reserving two slots for single use editors is the best. It would be you and BrownHairedGirl doing those sorts of runs, and the rest of us getting guaranteed usage with no delays. Abductive (reasoning) 00:01, 14 September 2021 (UTC)[]

Abductive, when the bot fixes a page with bare URLs, it also makes any other fixes within its scope. I don't have some special version of the bot which does only bare URLs.

The heavy use is normally a problem only when you push the bot up against its limits by your low-return speculative trawls. --BrownHairedGirl (talk) • (contribs) 00:02, 14 September 2021 (UTC)[]

The bot, with or without people using it on categories, will always be used heavily. Any solution to this problem needs to address this. Abductive (reasoning) 00:04, 14 September 2021 (UTC)[]

The problem is not heavy use. The problem is systematic abuse by Abductive . --BrownHairedGirl (talk) • (contribs) 00:09, 14 September 2021 (UTC)[]

Again, using the bot to make the tedious determinations of finding things to fix is a legitimate use of the bot. It is not abuse, and I have been showing restraint. I used to be in User:BrownHairedGirl's position of keeping the bot running at all times, and I have ceded that role to her. Abductive (reasoning) 00:15, 14 September 2021 (UTC)[]

You put nearly 1,000 articles through the bot this evening, with a combined edit rate of under 20%. You did that after yesterday's calls for you to stop.

And you call that showing restraint?!?!?!?!?! --BrownHairedGirl (talk) • (contribs) 00:22, 14 September 2021 (UTC)[]

PS And no, for the umpteenth time, using the bot to make the tedious determinations of finding things to fix is not a legitimate use of the bot.

It wastes the bot's limited capacity, when it could be doing more of the targeted jobs which have a higher rate of return. Many other editors have told you this, many times, in many different ways. --BrownHairedGirl (talk) • (contribs) 00:26, 14 September 2021 (UTC)[]

How many articles did you run past the bot in the same time period? As you mention above, it is the number of citations that count as work for the bot. Many of the articles are that are from categories have one or two citations (or even zero), so true load comparions need take that in account. In any case, any category will have this low rate of return. Abductive (reasoning) 00:30, 14 September 2021 (UTC)[]

He has done it again. 20 minutes after I posted above Abductive wasting the bot's time with Category:Edible fungi, he set the bot to work on Category:Kung fu films. The bot has now finished processing the 410 pages in Category:Kung fu films, and as can be seen in these 300 bot contribs (search the age for Category:Kung fu films), the bot made edits to only 57 edits of the 411 pages, an edit rate of just under 14%.
I do not know why he did this, but if Abductive was actively trying to demonstrate contempt for those asking for restraint, this resource-wasting job could hardly have been better chosen. --BrownHairedGirl (talk) • (contribs) 23:57, 13 September 2021 (UTC)[]

No, that was many, many hours ago. Besides, any user could have requested a run on a category, would you be upset with them too? Abductive (reasoning) 00:01, 14 September 2021 (UTC)[]

Not true. That job made its first edit[15] at 21:58 UTC, only just over 2 hours ago.

And yes, any user could have requested a run a low-return category. But time after time, any user doesn't waste the bot's time in that way.

It's Abductive who does it repeatedly.

It's Abductive who rejects calls for restraint. --BrownHairedGirl (talk) • (contribs) 00:08, 14 September 2021 (UTC)[]

The fundamental problem is that we have all been running the bot so much that is has fixed most articles by now, so that category runs aren't going to have a high enough rate of return. Abductive (reasoning) 00:12, 14 September 2021 (UTC)[]

@Abductive: Exactly!!!!

So why oh why oh why oh why oh why do you keep on wasting the bot's time with these big category jobs when you know that they have a low rate of return?

And why have you posted dozens of times in the last few weeks to insist that will continue to do so?

What on earth are you trying to achieve by this sustained disruption? --BrownHairedGirl (talk) • (contribs) 00:20, 14 September 2021 (UTC)[]

You should look through the suggestions that people have made in the last few weeks. Technical solutions are called for. And I reiterate; I am not disrupting the bot. Using the bot on categories is something that users besides me are doing. Are you saying that edible fungi and kung-fu movies, had they been called for by someone else, are bad? Or okay with you? And what constraint should be put on the bot to level the playing field for all users? Abductive (reasoning) 00:25, 14 September 2021 (UTC)[]

Quit the evasion. Just stop wasting the bot's time with low-return speculative trawls which displace and/or delay productive uses. --BrownHairedGirl (talk) • (contribs) 00:28, 14 September 2021 (UTC)[]

All users will be unproductive in your eyes. You even try to police my single runs. Abductive (reasoning) 00:33, 14 September 2021 (UTC)[]

For goodness sake, stop the evasion.

You finally acknowledged that that category runs aren't going to have a high enough rate of return. So just stop doing them, and quit wasting everyone's time with endless deflections and denials and bogus claims. --BrownHairedGirl (talk) • (contribs) 00:49, 14 September 2021 (UTC)[]

There is no evasion. I respond to everything you say. I have made many compromises in how I use the bot, and in particular I avoid using it when three channels are in use (usually two), to leave availablity for myself and others to run the bot on articles we are working on. Abductive (reasoning) 01:03, 14 September 2021 (UTC)[]

Local mode?[edit]

Has the ability to run this bot offline (with sufficient API rate limiting, both for the mediawiki API and the data provider APIs) been considered? That's one way to solve the discussions over capacity. It seems strange to me that in this day and age a computing resource like this has limited capacity. Enterprisey (talk!) 02:15, 14 September 2021 (UTC)[]

I'd be more than happy to host something, if someone can walk me through it. I have a spare windows machine, but I could install another OS on it. That said, a better scheduler/queuing system would fix all issues. Put all individual request at the top of the queue, put all batch requests at the bottom, problem solved. Headbomb {t · c · p · b} 03:07, 14 September 2021 (UTC)[]

I would also be happy to run my bare URL batches offline (if walked through it), and could use my backup machine for it (dual boot Win10/OpenSUSE). I would welcome being able to run bigger batches, instead of having to chop them up into small chunks.

However, AFAICS the bottleneck is in the data provider APIs, which already seem to be at their limit when Citation bot is using all 4 channels: my recent batches of biographies run at up to ~350 pages/hr when there are zero or one other jobs running, but slow to #200 pages/hour with two other concurrent jobs, and down to as low as 80 pages/hr with three other jobs. The slowdown is esp severe esp if the other jobs include pages with lots of refs: the bot's overall edit rate can fall from a peak of ~8 pages/min to 0.2 pages/min

So allowing more processes to use the APIs may seriously degrade the throughput of batch jobs. To avoid wasting the APIs' limited resources, we would still need some restraints on low-return speculative trawls such as those which I documented above.[16][17][18]

It seems to me that the real issue here is the limited capacity of the APIs. Is there any way those API bottlenecks can be addressed? @AManWithNoPlan: who runs these APIs? --BrownHairedGirl (talk) • (contribs) 09:20, 14 September 2021 (UTC)[]

Bibcodes use an API key. URLs us wikipedia zotero. Those are two bottlenecks. Neither of those are accessed when run in slow mode. If you look at category code, it actually can be run on the command line - I assume that works. Of course, you need keys etc.. AManWithNoPlan (talk) 11:35, 14 September 2021 (UTC)[]

@AManWithNoPlan: by slow mode, do you mean with the option "Thorough mode" selected? --BrownHairedGirl (talk) • (contribs) 13:07, 14 September 2021 (UTC)[]

Exactly. AManWithNoPlan (talk) 13:47, 14 September 2021 (UTC)[]

Thanks. So am I right in thinking that means that if this thorough-mode task was run from a user's own computer, it would share no APIs or other resources with Citation bot?

If so, it could run at whatever speed WP:BAG allows, and take a lot of load off Citation bot. --BrownHairedGirl (talk) • (contribs) 14:08, 14 September 2021 (UTC)[]

Yes, Zotero/URLs is shared. Although, you probably don't have a BibCode API key, so you would need on of those - each key would be different. AManWithNoPlan (talk) 14:11, 14 September 2021 (UTC)[]

Just to clarify, why would slow/thorough mode make fewer queries, and how can it get away with doing that? Enterprisey (talk!) 08:16, 15 September 2021 (UTC)[]

Slow/thorough mode expands URLs and adds bibcodes. That makes the process take longer (slower) since there are more queries per page. AManWithNoPlan (talk) 11:38, 15 September 2021 (UTC)[]

Accessing the bibcode API is still required for slow mode, then? I interpreted Neither of those are accessed when run in slow mode as saying that the APIs are not accessed during slow mode, but that doesn't make total sense to me. Enterprisey (talk!) 23:32, 15 September 2021 (UTC)[]

My phone ate my edit. Should be except in slow mode. AManWithNoPlan (talk) 00:58, 16 September 2021 (UTC)[]

Caching?[edit]

I understand from the earlier arguments that the bot wastes much time looking up the metadata for citations that it already had processed in earlier runs, especially when run on batches of pages. How much (if any) storage space is available to the bot? If it doesn't already, could it cache the metadata of citations it processes (or the resulting template code, or maybe just a hash by which to recognize a previously-seen citation), so that it can waste less time when encountering an already-processed citation? —2d37 (talk) 03:04, 16 September 2021 (UTC)[]

URL data is cached on the zotero server - which the bot queries. Whether or not DOIs work is cached. Nothing else is cached. Most of the APIs are really fast other than URLs. BibCodes might be useful, but since queries are batched then there would be litte to gain. AManWithNoPlan (talk) 13:03, 16 September 2021 (UTC)[]

No new jobs[edit]

For the last 35 minutes, the bot has been editing only individual page requests by Abductive, which as usual are stacked high. See the latest bot contribs.

No edits have been made on behalf of any other editor since my batch ended at 21:24 etc. The bot has given no response either to my batch request, or to an individual page request I did as a test.

What's going on? Why is everyone else apparently locked out? --BrownHairedGirl (talk) • (contribs) 22:16, 24 September 2021 (UTC)[]

OK, the most recent contribs show that the bot let FMSky back in at 22:40, and me at 22:44. But it's weird that for an hour, only one editor got anything done. --BrownHairedGirl (talk) • (contribs) 22:52, 24 September 2021 (UTC)[]

The bot saved up a bunch of my requests, then disgorged them all at once. Sorry about that. Abductive (reasoning) 23:02, 24 September 2021 (UTC)[]

@Abductive: Had you also submitted category requests? --BrownHairedGirl (talk) • (contribs) 23:28, 24 September 2021 (UTC)[]

Not at that time. Abductive (reasoning) 00:42, 25 September 2021 (UTC)[]

Abductive, a good start would be not to submit "a bunch of my jobs". Do one, wait until it has finished before you submit another. Given how this bot shares a general resource, I assume that you are causing widespread collateral damage. Your runs are making the tool unusable to normal editors. I have stopped using this tool to fill out citations in articles as I edit them because I now expect failure to be the only option. --John Maynard Friedman (talk) 08:23, 25 September 2021 (UTC)[]

@John Maynard Friedman: Give it a try, the bot will get to the job eventually. Also, as can be seen here, the bot has inexplicable delays in spite of high or low apparent usage. Sometimes, the bot looks like it has only one or two or three channels in use, but won't make edits for a considerable period. At other times, it looks like there are four channels in use, but the request is processed in a matter of minutes. Abductive (reasoning) 00:03, 26 September 2021 (UTC)[]

Do you really believe that I am so stupid as to report experiences that don't actually happen, that it is just incompetence? I'm not talking about me giving up on a single article request because apparently nothing has happened for ten minutes. I mean a request that ends after N minutes with an explicit failure message. Repeatedly.

Stop trying to find other people to blame. Your current use of multiple concurrent batch rubs is a denial of service attack. Your behaviour is unambiguously WP: disruptive. Go find something useful to do with your life. There are thousands of articles that need personal hands-on editing. --John Maynard Friedman (talk) 08:05, 26 September 2021 (UTC)[]

If I was to blame, how do you explain the fact that I experience the same thing all the time? When I try to use the bot, and I have no outstanding requests, it's still hit or miss if I get my request done in a timely manner, or in a few minutes, or never. I don't run multiple batch jobs, and in fact that's impossible. Abductive (reasoning) 09:46, 26 September 2021 (UTC)[]

You wrote above The bot saved up a bunch of my requests, then disgorged them all at once. You should not have had "a bunch of jobs". If you know how to write a batch run then you definitely know by now not to submit another job until the previous one has finished or 24 hours has elapsed without a response. What is "a timely manner" for a background batch run? As opposed to a single editor who has worked on a single page and needs to verify the citations or collect the remaining metadata from the DOI? Answer: a week v ten minutes max.

No, you are not the only cause of this problem: I see other bot runs doing marginal value activities like inserting {{reflist talk}} into talk pages [if editors of that page cared, they'd have done it already]; fixing lint errors on signatures and so on. All well-meaning folk who fail, before they do it, to work out the cost/benefit ratio - which is what BrownHairedGirl has been banging on about. But yours is the most annoying because of the effect it has on one-shot use of this tool. You know that the server has limited capacity. Maybe it is not just you but it is your head that is above the parapet so don't complain when it gets shot. --John Maynard Friedman (talk)

I'm pretty sure that the average user wants the bot to correct more than it does, but there are endless complaints from article owners about so-called "cosmetic" edits, so these hae been stifled. Abductive (reasoning) 11:30, 26 September 2021 (UTC)[]

No. The average editor (the average user is a reader) objects to resources being wasted on edits that have no visible effect. If an article is being edited anyway, such changes can be made en passant: the objection is not to the change but to pettifogging changes that need to be verified. This has nothing to do with WP:OWN but everything to do with maintaining a reader-focussed approach. --John Maynard Friedman (talk) 12:08, 26 September 2021 (UTC)[]

@John Maynard Friedman: see my proposal below, at #Proposed new rule for using Citation bot. --BrownHairedGirl (talk) • (contribs) 15:04, 27 September 2021 (UTC)[]

Proposed new rule for using Citation bot[edit]

In the last week, there have been several prolonged periods when access to this bot has been severely impeded by the actions of two editors: @Abductive and @Whoop whoop pull up. Both editors have repeatedly flooded the bot with series of category requests made alongside long series of individual page requests.

The effect of this practice is to allow one editor to lock up two of the bot's four channels for significant lengths of time. Even when some channels appear not to be in use, the vast queues of requests from these two editors prevents the bot from processing requests by others.

Whatever these editors' intentions, the repeated effect of their actions is similar to a denial-of-service attack.

Most of this disruption could be avoided by a simple rule, which I set out below. It may be that this could be implemented by technical restrictions, but I do not want to make assumptions about either the technical possibilities or the willingness of the maintainers to volunteer their time to code any changes. --BrownHairedGirl (talk) • (contribs) 14:36, 27 September 2021 (UTC)[]

Proposed new rule: One request at a time[edit]

When an editor requests that the bot process either a batch of articles or a single article, they must not make another request until that article or batch has been processed. An exception is permitted where an editor has within the last 24 hours edited an article to add or modify a reference; in that case they may make a single page request for that article before other requests have finished processing.

Discussion and survey of Proposed new rule[edit]

Add your comments and/or support/oppose here

Support, as proposer. --BrownHairedGirl (talk) • (contribs) 14:37, 27 September 2021 (UTC)[]
Oppose - Please stop trying to WP:OWN Citation bot, @BrownHairedGirl. If you want a citation improver all to yourself, go get your own copy of Citation bot - just like other heavily-used bots have multiple duplicate instances to spread out their tasks between. Also, I'd like to note that you tie up Citation bot essentially 24/7 with enormous pageLinked jobs of thousands of articles, whereas I run category and/or individual jobs for much smaller fractions of the day, and most of my category runs (don't know about Abductive's) are smaller categories taking little time to run. Additionally, whether or not an editor has added or removed one or more reference(s) from an article has no bearing on whether the article has other citation errors that need to be fixed. Striking in light of my concerns being addressed by BrownHairedGirl's suggestions below, but Comment that the fact that this is causing problems is a sign of Citation bot being severely overloaded, illustrating the urgent need for additional duplicate instances of Citation bot and/or additional processing channels per instance of Citation bot. Whoop whoop pull up ^{Bitching Betty ⚧ Averted crashes} 15:07, 27 September 2021 (UTC)[]

@Whoop whoop pull up: asking editors to stop simultaneously using two of the bot's four channels is about sharing resources collaboratively. It is the complete opposite of WP:OWNership.
Also, your description of Abductive's usage is false: over the last few months I have documented on this page many cases where Abductive has tied up the bot for ages with low return category requests. --BrownHairedGirl (talk) • (contribs)

Insufficient capacity on Citation bot's part marks a problem with the bot (which should be taken up with its developer, or by creating additional instances of Citation bot to spread out the load, as per accepted practice for other highly-used bots), not with the editors using it to improve the encyclopedia. Whoop whoop pull up ^{Bitching Betty ⚧ Averted crashes} 15:23, 27 September 2021 (UTC)[]

The developers are volunteers who have kindly donated their time and skills to create the bot. Like any tool, it has limitations, including limited capacity, and while those limitations exist editors should collaboratively to share the resource effectively. Locking up two of the four channels is disruptive to that collaboration ... in other words, it avoidably disrupting other editors who are also using it to improve the encyclopedia. If you want to process lots of articles, submit a single batch request. --BrownHairedGirl (talk) • (contribs) 15:30, 27 September 2021 (UTC)[]

Submitting a single huge batch request (as you routinely do) would take exactly the same number of channel-hours as submitting many individual requests, the only differences being whether the job(s) lock one channel continuously or several channels intermittently - additionally, large batch jobs have the disadvantage of requiring a great deal of manual typing to assemble a pipe-separated list of article names and then paste the whole shebang into the box on the Citation bot console, while submitting individual jobs via the toolbar is quite literally point-and-click. I reiterate my earlier suggestion for having multiple duplicate instances of Citation bot to multiply its capacity. Whoop whoop pull up ^{Bitching Betty ⚧ Averted crashes} 15:39, 27 September 2021 (UTC)[]

The purpose of this proposal is to restrict editors to one bot channel at a time, so that others are not locked out of the bot.
More instances of the bot or more channels would of course be great, but for now we have one bot with only four channels, and this proposal addresses the problem that some editors are using it in ways which avoidably disrupt others. Making a list of pages is easy, and using a text editor's search-and-replace function to add pipes is also easy. --BrownHairedGirl (talk) • (contribs) 15:52, 27 September 2021 (UTC)[]

How would one use search-and-replace to pipe-separate the list, given that most methods of exporting a list of article titles separate the individual page titles either with line breaks (which text editors generally don't support search or replace functions on) or with spaces (which would require huge amounts of manual work to keep the search-and-replace from splitting multiword titles)? Whoop whoop pull up ^{Bitching Betty ⚧ Averted crashes} 16:21, 27 September 2021 (UTC)[]

Any text editor which supports regex will do it very simply, by replacing \n. For example, on Windows there is Notepad++; on Linux try Kate. There are lots of alternatives for every platform. --BrownHairedGirl (talk) • (contribs) 16:31, 27 September 2021 (UTC)[]

Will look into that, thanx! :-)
Update: Kate is already exceeding expectations. Whoop whoop pull up ^{Bitching Betty ⚧ Averted crashes} 17:44, 27 September 2021 (UTC)[]

- In response to Whoop's comment whether the article has other citation errors that need to be fixed: I agree, but that comment misses the point.
  This is not about whether the article needs bot attention, but whether it needs attention now. The purpose of that exception is to allow an editor to use the bot interactively, e.g. by adding a ref which consists only of an isbn or doi, and immediately trying to have the bot fill that ref. That interactive usage is hugely helpful when adding refs, which I why I propose it as an exception to the one-job-at-a-time rule. Other pages which are not being currently can wait until other asks have finished. --BrownHairedGirl (talk) • (contribs) 15:45, 27 September 2021 (UTC)[]

Is that really a good use for Citation bot, given that the editor could easily take the time to fill in their new reference themselves without consuming any of the limited Citation bot resources that so concern you? Whoop whoop pull up ^{Bitching Betty ⚧ Averted crashes} 15:56, 27 September 2021 (UTC)[]

Sometimes, it is a very good use of the bot. For example, if a book or journal paper has multiple authors and editors, the bot can take a ref which consist of only an isbn or doi and fill those out quickly, instead of taking lots of that editor's time. See User:Headbomb/Tips and Tricks. --BrownHairedGirl (talk) • (contribs) 16:08, 27 September 2021 (UTC)[]

And that is exactly one of the reasons that I [used to] use the tool. I strongly support the proposal.

For as long as (a) the resource is limited and (b) there is no mechanism for batch runs to have background priority, this proposal is essential. Airy handwaving that the service needs to improve is just fatuous. To accuse BHG of WP:OWN displays remarkable chutzpah - or just plain hypocrisy. As I have already said above, hogging all channels with batch requests, especially for 'nice to have' changes, is unambiguously WP:DISRUPTIVE (insidious type). BHG's proposal offers a reasonable compromise between types of use. Let's work towards consensus rather than have to refer the dispute upwards. --John Maynard Friedman (talk) 17:53, 27 September 2021 (UTC)[]

Support I'm failing to see a good reason not to support this. * Pppery * _{it has begun...} 17:55, 27 September 2021 (UTC)[]
Comment. Many thanks to @Whoop whoop pull up for withdrawing their oppose. I have a lot experiencing in preparing batch jobs, so if Whoops or anyone else wants assistance, I will try to help. --BrownHairedGirl (talk) • (contribs) 18:59, 27 September 2021 (UTC)[]
I have a number of concerns that I hope will be addressed. First, if a user runs a category, the rate of edits will be low, and the rate of substantive edits will be even lower. A category that by some rare confluence of circumstances has never been run may get a return rate in the 75% range. A random collection of articles gets a return in the 50% range, but only about 10% of these will be substantive. A run of a category of stubs gets a low rate of return (perhaps 20-25%), but goes very quickly as many stubs have zero, one or two refs. A second run of a category that has recently been run gets an edit rate around 12%, and includes edits that the bot should have made the first time around. Long articles take a long time for the bot to run, but more often find at least one edit to make; so running a bunch of them will look like a good use of the bot on a percent basis, but not actually be something to be emulated. As an example of this, just choosing to run articles that contain any set of search terms will find longer articles, the more the search terms narrow down the list, the longer the articles and the more likely are edits. Second, experienced users of the bot have been making efforts to find articles that need bot attention, or remove articles from their runs that have recently been run. I myself have a spreadsheet with weeks of bot activity on it which I use. But how can we expect all users to do this? Third (and related), users will discover the bot, and run sub-optimal categories and batch jobs. I fear that unless they are allowed some leeway at first and then gently guided towards this rule, rancor will continue. The sub-optimal runs can be used to ascertain bot performance on the average category. Similarly, if a user wants to test a method of discovering articles that the bot will make edits to, they may occasionally fail to reach a threshold of edits that will please intensive users of the bot. Fourth, as User:Whoop whoop pull up notes, it would be a shame if the rule make it seem like the problem of demand is solved, and progress towards technical solutions stalls. Fifth (and related), even if everybody follows the rule as proposed, it will only take four runs to lock out single-article users. This will happen a lot, and without, for example, an instance of the bot for them, there will continue to be complaints and rancor ("You big users have banded together at the little guy's expense!") So, I think the rule would be a good temporary measure, if other big users agree among themselves to refrain from running the bot if there are three medium or large runs already going, and agree to refrain from complaining about mid-return runs that otherwise abide by the bot limits and the rule. Abductive (reasoning) 20:33, 27 September 2021 (UTC)[]
- Most of Abductive's over-long and poorly-laid-out comment has nothing to do with this proposal. This proposal is not about Abductive's disruption-by-low-return-speculative-trawl; this proposal is about Abductive's disruption-by-lots-of-individual-requests-alongside-a-batch.
  Also, this proposal will neither delay nor hasten any improvements to the bot's capacity. This is a proposal to remove one form of disruption to the bot as currently configured.
  Finally, of course new users of the bot should be treated gently. It seems rather disingenuous of Abductive to try to invoke new users to deflect attention away from the fact that Abductive has been using the bot disruptively for months. --BrownHairedGirl (talk) • (contribs) 21:36, 27 September 2021 (UTC)[]
Support on batch requests, individual articles are pretty irrelevant. That said, simply removing batch request privilege from Abductive would also be sufficient in solving the issues. Headbomb {t · c · p · b} 20:51, 27 September 2021 (UTC)[]

Of course that will not work, as it does not solve the problem of demand and single users will continue to get locked out. I propose that the rule be amended to state that no user will be prevented from using the bot based on arbitrary notions of correct usage, and that users should never start a job when there are three jobs running. Abductive (reasoning) 20:56, 27 September 2021 (UTC)[]

Question - how would one tell how many jobs are currently running? The Citation bot console doesn't provide this information. Whoop whoop pull up ^{Bitching Betty ⚧ Averted crashes} 22:31, 27 September 2021 (UTC)[]

I despair of the comment about arbitrary notions of correct usage. There is nothing at all arbitrary about deploring tying up bot for ages with low-return-batch-jobs. There is nothing at all arbitrary about deploring the practice of one editor systematically tying up two of the bot's channels. --BrownHairedGirl (talk) • (contribs) 21:40, 27 September 2021 (UTC)[]

Off-topic

As long as rules are applied equally to all users, there should not be a problem. All category runs, by any user, are going to be low-return, and the present setting of 550 (or any setting other than 0) will allow users to use the bot on categories. Abductive (reasoning) 21:52, 27 September 2021 (UTC)[]

IIRC, the limit is 500, not 550. Whoop whoop pull up ^{Bitching Betty ⚧ Averted crashes} 22:33, 27 September 2021 (UTC)[]

I guess I didn't notice, because I have been refraining from running large categories until a compromise was reached. Abductive (reasoning) 23:17, 27 September 2021 (UTC)[]

Enough already of that disingenuous nonsense: only Abductive repeatedly engages in large, low-return speculative trawls. And this discussion is NOT about those speculative trawls, so please stop that off-topic stuff. --BrownHairedGirl (talk) • (contribs) 22:04, 27 September 2021 (UTC)[]

There is clearly no consensus on what is on or off topic, which basically means the issues cannot be separated. If the rule as proposed and amended is implemented, then the bot will never get overloaded. Isn't that the goal? Or is there a plan to introduce another rule proposal later to continue to attempt to monitor and control people's "incorrect" usage of the bot? Abductive (reasoning) 23:09, 27 September 2021 (UTC)[]

If the rule as proposed and amended is implemented, then the bot will never get overloaded. Isn't that the goal? Or is there a plan to introduce another rule proposal later to continue to attempt to monitor and control people's "incorrect" usage of the bot? Abductive (reasoning) 00:09, 28 September 2021 (UTC)[]

For goodness sake, please stop this disruption.

No, of course this proposal will not mean that the bot will never get overloaded. This tackles one cause of overload, but there are other ways in which it can be overloaded, most notably by Abductive's repeated DOS attacks.

If this proposal is implemented, we will have to see whether the other forms of disruption resume. It may be that Headbomb is right, and that most effective remedy would be to ban Abductive. Or maybe we need another rule. Or maybe Abductive will overcome his severe IDHT problem, and listen to the many editors who have asked him to stop.

But here in this section, we are discussing a specific proposal. Please stop going off-topic. — Preceding unsigned comment added by BrownHairedGirl (talk • contribs) 00:25, 28 September 2021 (UTC)[]

How about this: everybody here agrees to abide by the proposed rule, including the amendment to not start a run if there are three runs already going, and we see how it goes? Abductive (reasoning) 00:35, 28 September 2021 (UTC)[]

Your proposed amendment was just a ruse to try to legitmate your disruptive speculative trawls. Your continued disruption of this discussion has been noted. Again, please stop. --BrownHairedGirl (talk) • (contribs) 01:06, 28 September 2021 (UTC)[]

More flooding. Another example of why I proposed this rule.
This evening, we seem to have another instance of the bot being locked by a flood of individual requests. For the last 40 minutes, the bot has made no edits except (mostly large) individual pages requested by Abductive: see recent bot contribs. I have had no response to either a batch request made over 30 minutes ago, or an individual page request made ~15 minutes ago as a test.
I can live with a delay to my batch job, but it is highly disruptive to anyone trying to use the bot to fill in refs on an article which they are improving. --BrownHairedGirl (talk) • (contribs) 21:07, 30 September 2021 (UTC)[]

It is odd how the bot seems to be unresponsive to requests, even if it appears that channels are open, then do a bunch at once. But over the last few days, there have been long periods where big users (not me), have monopolized all the channels without checking to see if there are already three big jobs running. So intense demand remains a problem. Abductive (reasoning) 21:51, 30 September 2021 (UTC)[]

Long periods of unresponsiveness without results to show for it could potentially be due to the bot chewing through a long run of articles that it was fed but that didn't actually need any fixes; in that situation, the bot would be making no edits for a potentially-considerable length of time, thus seeming to be stalled. Whoop whoop pull up ^{Bitching Betty ⚧ Averted crashes} 01:27, 3 October 2021 (UTC)[]

Bot makes cosmetic edits solely to bypass citation template redirects[edit]

Status: new bug
Reported by: BrownHairedGirl (talk) • (contribs) 22:53, 29 September 2021 (UTC)[]

What happens: bot makes edits solely to bypass a redirect to a CS1/CS2 template. This makes absolutely zero difference to how the page is rendered.
What should happen: In the example below, nothing should happen. Per WP:NOTBROKEN and WP:COSMETICBOT, such changes should be made only if they are accompanied by a non-cosmetic change.
Relevant diffs/links: [19], [20], [21], [22]
We can't proceed until: Feedback from maintainers

Arbitrary cite type changes[edit]

Some observations of Citation bot's template-changing behaviour from today:

In this edit, the bot classifies PC Gamer -- within the same edit -- as both a magazine and a newspaper (in the latter case renaming it "Pc Gamer"). PC Gamer has a website and a magazine. However, none of the online articles makes it into the mag and only a few printed ones ever get online, so why conflate the two? In no instance is PC Gamer a newspaper. The newspaper change also occurred here.
Citation bot changes GameStar, as seen here, and Rock Paper Shotgun, here, into journals. GameStar is a magazine–website hybrid like PC Gamer, while Rock Paper Shotgun is a website only; neither is a journal. I previously complained that Rock Paper Shotgun was being turned into a newspaper for no reason.
The bot inconsistently classifies MCV/Develop as a magazine and a journal.

These are from only the past handful of edits that popped up on my watchlist. I find these kinds of changes to be arbitrary and, given the inconsistency and the fact that these template changes do not actually affect the rendered article, the bot should not perform these changes unless there is a clear point in doing so. IceWelder [✉] 08:31, 1 October 2021 (UTC)[]

Added new array of works that often are both magazines and websites that will not be changed. AManWithNoPlan (talk) 11:59, 1 October 2021 (UTC)[]

Job stalled, refuses to die[edit]

My overnight batch job of 2,199 articles ("Vehicles, part 2 of 3") stalled at 08:10 UTC. This can be seen in the latest set of bot edits, where the last edit to this set is item #140, an edit[23] to Driver (series).

I tried to remedy this at about 08:40 by using https://citations.toolforge.org/kill_big_job.php, which promptly responded Existing large job flagged for stopping.

But over an hour later, attempts to start a new job (with a new list) using https://citations.toolforge.org/linked_pages.php get still get a response of Run blocked by your existing big run.

Meanwhile, the bot was happy to process an individual page request from me: see [24] at 09:49. --BrownHairedGirl (talk) • (contribs) 10:05, 2 October 2021 (UTC)[]

Rebooted. AManWithNoPlan (talk) 11:28, 2 October 2021 (UTC)[]

Thanks, @AManWithNoPlan. I tried again to start a new batch, and the bot began work promptly.[25] --BrownHairedGirl (talk) • (contribs) 11:46, 2 October 2021 (UTC)[]

Job dropped[edit]

My overnight batch of 2,198 articles ("Vehicles, part 3 of 3") was dropped at 08:00 UTC this morning, after this edit[26] to National Tyre Distributors Association, which was #661 in the list. See the bot contribs for that period, where that edit is #126 in the contribs list.

When I spotted it, I was able to start a new batch at 10:48[27], so the overnight batch wasn't stuck like yesterday.

This is a bit tedious. --BrownHairedGirl (talk) • (contribs) 11:14, 3 October 2021 (UTC)[]

Another job stalled, refuses to die[edit]

After my big overnight job of 2,198 pages was dropped (see above #Job dropped), I ran another small job of 100 pages. That was processed successfully.

I then resumed the overnight job "Vehicles, part 3 of 3 resumed" (1,537 pages) ... but it has stalled.

See the latest bot contribs: that batched started processing at 11:41[28], but stalled after its fifth edit[29], at 11:42.

At 12:14 I used https://citations.toolforge.org/kill_big_job.php to try to kill this stalled job. Now 15 minutes later, I still can't start a new job: the response is Run blocked by your existing big run. --BrownHairedGirl (talk) • (contribs) 12:31, 3 October 2021 (UTC)[]

I have being re-trying intermittently throughout the day, but the bot still says Run blocked by your existing big run.

--BrownHairedGirl (talk) • (contribs) 19:25, 3 October 2021 (UTC)[]

Caps: GSA Today[edit]

Status: new bug
Reported by: Headbomb {t · c · p · b} 14:38, 3 October 2021 (UTC)[]

What happens: [30]
What should happen: [31]
We can't proceed until: Feedback from maintainers

User talk:Citation bot

Contents

Consistent spacing[edit]

Only four requests at a time?[edit]

One editor, two simultaneous batch jobs[edit]

Convert to cite biorxiv[edit]

Bot still being abused by Abductive[edit]

Abductive again. What to do?[edit]

Local mode?[edit]

Caching?[edit]

No new jobs[edit]

Proposed new rule for using Citation bot[edit]

Proposed new rule: One request at a time[edit]

Discussion and survey of Proposed new rule[edit]

Bot makes cosmetic edits solely to bypass citation template redirects[edit]

Arbitrary cite type changes[edit]

Job stalled, refuses to die[edit]

Job dropped[edit]

Another job stalled, refuses to die[edit]

Caps: GSA Today[edit]

Navigation menu

User talk:Citation bot

Consistent spacing[edit]

Only four requests at a time?[edit]

One editor, two simultaneous batch jobs[edit]

Convert to cite biorxiv[edit]

Bot still being abused by Abductive[edit]

Abductive again. What to do?[edit]

Local mode?[edit]

Caching?[edit]

No new jobs[edit]

Proposed new rule for using Citation bot[edit]

Proposed new rule: One request at a time[edit]

Discussion and survey of Proposed new rule[edit]

Bot makes cosmetic edits solely to bypass citation template redirects[edit]

Arbitrary cite type changes[edit]

Job stalled, refuses to die[edit]

Job dropped[edit]

Another job stalled, refuses to die[edit]

Caps: GSA Today[edit]

Navigation menu

Search