wiki/jogobot-red
1
0
Fork 0
Code Issues 4 Pull Requests Releases Wiki Activity
276 Commits 2 Branches 14 Tags
68b81b1111b6c0f33995a93886e8b7ed4a65b6e1
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
golderweb 68b81b1111 missingnotice: Implement treat_redfam 
For each redfam, we need to check weather related redundance discussion
exists and if there are missing notices. For those redfams return links
to discussion and articles missing notice.

Issue #64 (#64)
 2018-09-18 16:29:28 +02:00
golderweb 389c48605e redfam: Make get_disc_link() able to return wikilink 
Issue #64 (#64)
 2018-09-18 16:29:22 +02:00
golderweb 95af95aca6 missingnotice: Implement article selection 
Issue #64 (#64)
 2018-09-18 16:29:17 +02:00
golderweb 99adad873e missingnotice_test: Test article query 
Issue #64 (#64)
 2018-09-18 16:29:01 +02:00
golderweb dbcc2717d7 missingnotice: Implement article query 
Issue #64 (#64)
 2018-09-18 16:28:56 +02:00
golderweb e5a45fa692 tests: Add test script for missingnotice 
Issue #64 (#64)
 2018-09-18 16:28:44 +02:00
golderweb 63d3f837e9 red.py: Introduce subtask missingnotice 
Issue #64 (#64)
 2018-09-18 16:28:38 +02:00
golderweb cfb3e8e37c bots: Add basic structure for MissingNoticeBot 
Issue #64 (#64)
 2018-09-18 16:28:29 +02:00
golderweb dfffe97200 redfam: Add method to check disc section 
Sometimes disc sections are disapering since the heading is changed
and the famhash changes, so we get a new redfam. Mark those as absent

Issue #64 (#64)
 2018-09-18 16:28:22 +02:00
golderweb 246e94c228 redfam: Add generator for open redfams to Worker 
Issue #64 (#64)
 2018-09-18 16:27:51 +02:00
golderweb 181486c718 Merge branch 'release-1.1.1' back into develop 2018-09-17 17:23:12 +02:00
golderweb 3fbfd4ccd7 Prepare release-1.1.1 2018-08-12 11:46:40 +02:00
golderweb 50b0e142ec Merge branch 'i#71-moved-page-exists' into develop 2018-08-12 11:43:18 +02:00
golderweb 14db996a43 redfam: Check if moved page exists 
To prevent creation of orphaned diskpages in case of special movement
constructs

Issue #71 (#71)
 2018-08-12 11:41:50 +02:00
golderweb 110589cb5b Merge branch 'release-1.1' back into develop 2018-08-12 11:15:30 +02:00
golderweb a466ab4e74 Prepare release-1.1 2018-05-17 12:41:06 +02:00
golderweb 860a285ab0 Merge branch 'i#68-exclude-users' into develop 2018-05-17 12:36:37 +02:00
golderweb 2c105336b0 RedFamWorker: Exclude users and user talkpages 
Users can't be part of valid redundances

Issue #68 (#68)
 2018-05-17 12:35:38 +02:00
golderweb ea85ca731f Merge branch 'i#69-already-talkpage' into develop 2018-05-17 12:28:17 +02:00
golderweb 6e119ea98f RedFamWorker: Improve talkpagetoggling 
Do not toggle to main page if we have already a talkpage and vice versa

Issue #69 (#69)
 2018-05-17 12:26:37 +02:00
golderweb 67aaf3cbbe Merge branch 'i#70-follow-moved-pages' into develop 2018-05-17 12:24:00 +02:00
golderweb fa13e2a5cf Follow moved pages 
Keep notice together with content
https://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:Jogo.obb&oldid=176464377#Redundanzhinweis_zu_zwischenzeitlich_verschobenen_Artikeln

Issue #70 (#70)
 2018-05-17 12:18:13 +02:00
golderweb 562e689418 Merge branch 'release-1.0' back into develop 2017-11-05 12:30:18 +01:00
golderweb 93447d8dc6 Prepare release v1.0 
Update Copyright Notices
Version information
 2017-11-05 12:25:13 +01:00
golderweb 1b6faf9e53 Use own db for red-task 
Since we have several tables and sometimes need to create a copy on
replication servers.
 2017-11-05 12:17:05 +01:00
golderweb b4c193eedc Disable echoing of SQLAlchemy Egine 
We don't need this extensive output for production
 2017-11-05 12:07:38 +01:00
golderweb 788a3df0cd Update jogobot-submodule to v0.1 2017-11-05 12:00:28 +01:00
golderweb 04f591b466 Merge branch 'fs#161-add-article-titles' into develop 2017-11-05 11:24:15 +01:00
golderweb 9640467f69 markpages: Use redarticle attribute of Page 
Instead of trying to reconstruct our db article title, use the one added
to Page-object by redfam.article_generator

Related Task: [FS#161](https://fs.golderweb.de/index.php?do=details&task_id=161)
 2017-11-05 11:22:43 +01:00
golderweb bfec2abf98 markpages: Get rid of PageWithTalkPageGenerator 
Since redfam.article_generator can yield talkpage with additional
information about redfam and current article from db, we do not need it
anymore.

Related Task: [FS#161](https://fs.golderweb.de/index.php?do=details&task_id=161)
 2017-11-05 11:20:55 +01:00
golderweb 20103d589d redfam: article_generator add redfam info to page 
Add reference to redfam object and article title from db to Page object
since Page.title() may differe (short Namespaces, anchors, special chars)

Related Task: [FS#161](https://fs.golderweb.de/index.php?do=details&task_id=161)
 2017-11-05 11:18:53 +01:00
golderweb e18aa96a84 redfam: article_generator can return talkpage 
To make pywikibot.pagegenerators.PageWithTalkPageGenerators unneccessary
so we can manipulate talkpage object directly

Related Task: [FS#161](https://fs.golderweb.de/index.php?do=details&task_id=161)
 2017-11-05 11:15:04 +01:00
golderweb 1dd4c7f87e Merge branch 'test-v7' back into develop 2017-11-02 18:57:59 +01:00
golderweb 33b2e47312 Describe version test-v7 2017-10-28 22:43:53 +02:00
golderweb 3bd17ce692 Merge branch 'fs#160-urlencoded-chars' into develop 2017-10-28 22:36:55 +02:00
golderweb 5f4640d5ff Replace urlencoded chars with unicode equivalent 
Otherwise we get value errors while marking since pwb replaces those

Related Task: [FS#160](https://fs.golderweb.de/index.php?do=details&task_id=160)
 2017-10-28 22:35:25 +02:00
golderweb 7e0456ae4f Merge branch 'test-v6' back into develop 2017-10-28 22:34:30 +02:00
golderweb 108b7aa331 Describe version test-v6 2017-10-28 18:46:30 +02:00
golderweb a3adf31b89 Merge branch 'fs#86-activate-status-api' into develop 2017-10-28 18:44:42 +02:00
golderweb 614f288bb9 Activate jogobot status api for onwiki disabling 
Related Task: [FS#86](https://fs.golderweb.de/index.php?do=details&task_id=86)
 2017-10-28 18:44:05 +02:00
golderweb c450a045bf Merge branch 'fs#159-space-before-anchor' into develop 2017-10-28 18:43:13 +02:00
golderweb 84802cf521 Remove leading or trailing spaces in articles 
Some articles contain spaces between title and anchor part which will
be stripped now

Related Task: [FS#159](https://fs.golderweb.de/index.php?do=details&task_id=159)
 2017-10-28 18:41:06 +02:00
golderweb 5f6c443ba8 Merge branch 'test-v5' back into develop 2017-10-28 18:17:01 +02:00
golderweb 0c135ef1bb Describe version test-v5 2017-09-23 23:50:42 +02:00
golderweb 8b8221cfcd Merge branch 'fs#152-respect-always-flag' into develop 2017-09-23 23:49:59 +02:00
golderweb bdccc8417c Set always in Pywikibot.Basebot 
If cmdline param -always is set, set the related option in
Pywikibot.Basebot Object for automatic edits with out further requests

Related Task: [FS#152](https://fs.golderweb.de/index.php?do=details&task_id=152)
 2017-09-23 23:49:41 +02:00
golderweb a70835c58a Merge back branch 'test-v4' into develop 2017-09-23 23:48:25 +02:00
golderweb ec2b84df2a Add requirements 
To make setup of environment for this module easier
 2017-09-23 21:09:58 +02:00
golderweb 88848cb084 Prepare Version test-v4 for release 
Add a README.md file for this project
 2017-09-23 20:32:13 +02:00
golderweb 5057aed0d3 Merge branch 'fs#157-lowercase-title' into develop 2017-09-09 21:47:03 +02:00
golderweb 02e53475f1 Prevent lowercase article titles in Parser 
Since real lowercase article titles are not allowed, make sure to
convert all first letters of article titles to uppercase. This is
neccessary since pywikibot will return article titles like this.

Related Task: [FS#157](https://fs.golderweb.de/index.php?do=details&task_id=157)
 2017-09-09 21:35:36 +02:00
golderweb d6f9b460c9 Merge branch 'fs#156-dbapi-charset' into develop 2017-09-02 22:13:20 +02:00
golderweb ff03ca8f13 Explicitly set charset for PyMySQL-Connection 
Since PyMySQL-Connection otherwise uses charset 'latin-1', explicitly
set connection charset to 'utf8'

http://docs.sqlalchemy.org/en/rel_1_0/dialects/mysql.html#charset-selection
http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html?highlight=url#sqlalchemy.engine.url.URL

Related Task: [FS#156](https://fs.golderweb.de/index.php?do=details&task_id=156)
 2017-09-02 22:10:25 +02:00
golderweb 88692ca678 Merge branch 'fs#155-article-surouding-space' into develop 2017-09-02 22:08:31 +02:00
golderweb d9b4fcc0bd Strip spaces before adding articles to redfam 
Some article links have surounding spaces in their linktext. Remove them
before adding article to RedFam to have a cannonical title

Related Task: [FS#155](https://fs.golderweb.de/index.php?do=details&task_id=155)
 2017-09-02 22:06:30 +02:00
golderweb 22ff78ea98 Merge branch 'fs#154-categorie-colons-missing' into develop 2017-09-02 16:02:45 +02:00
golderweb b3cfcdc259 Improve title detection to get correct behaviour 
Make sure that categorie links are starting with colon and non article
pages are returned with namespace.

Related Task: [FS#154](https://fs.golderweb.de/index.php?do=details&task_id=154)
 2017-09-02 15:59:34 +02:00
golderweb b3e0ace2f4 Merge branch 'fs#153-nested-templates' into develop 2017-09-02 14:25:21 +02:00
golderweb f8002c85da Do not search for templates recursivly 
Since nested templates did not get an index in global wikicode object
searching for index of an nested template results in ValueError

Related Task: [FS#153](https://fs.golderweb.de/index.php?do=details&task_id=153)
 2017-09-02 14:23:25 +02:00
golderweb 49bc05d29b Merge branch 'fs#151-normalize-article-titles-anchor' into develop 2017-09-02 13:36:17 +02:00
golderweb 8a26b6d92a Normalize article titles with anchors 
In our db article titles with anchors are stored with underscores in
anchor string. Therefore we need to replace spaces in anchor string
given by pywikibot.Page.title().

Related Task: [FS#151](https://fs.golderweb.de/index.php?do=details&task_id=151)
 2017-08-25 18:11:41 +02:00
golderweb 49a8230d76 Merge branch 'fs#141-place-notice-after-comment' into develop 2017-08-25 17:11:28 +02:00
golderweb 31c10073a2 Prevent index errors searching for comments 
Make sure not to exceed existing indexes of wikicode object while trying
to search for comments

Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141)
 2017-08-25 17:09:38 +02:00
golderweb 642a29b022 Improve regex for blank lines 
Do not match consecutive linebreaks as one

Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141)
 2017-08-24 18:47:18 +02:00
golderweb 2f90751dc2 Merge branch 'fs#146-famhash-generator' into develop 2017-08-24 12:27:54 +02:00
golderweb 024be69fe1 Use famhash as generator 
If famhash is defined, fetch explicitly that redfam from db and work
only on this

Related Task: [FS#146](https://fs.golderweb.de/index.php?do=details&task_id=146)
 2017-08-24 12:27:13 +02:00
golderweb b6d7268a7f select by famhash: Add methods to get param in bot 
We need a method as callback to get bot specific params passed through
to our bot class.
Introduce -famhash parameter to work on specific famhash

Related Task:[FS#146](https://fs.golderweb.de/index.php?do=details&task_id=146)
 2017-08-24 12:27:13 +02:00
golderweb 526184c1e1 Merge branch 'fs#148-articles-mixed-up' into develop 2017-08-24 12:26:53 +02:00
golderweb 3aa6c5fb1c Disable PreloadingGenerator temporarily 
PreloadingGenerator mixes up yielded Pages. This is very unconvenient
for semi-automatic workflow with manual checks as the articles of the
RedFams were not following each other.

Related Task: [FS#148](https://fs.golderweb.de/index.php?do=details&task_id=148)
 2017-08-24 12:23:17 +02:00
golderweb ec8f459db5 Merge branch 'fs#138-marked-articles-shown-again' into develop 2017-08-24 12:19:24 +02:00
golderweb 3b2cb95f36 Do not fetch marked redfams from db 
Exclude marked Redfams from DB-Query to prevent marking them again

Related Task: [FS#138](https://fs.golderweb.de/index.php?do=details&task_id=138)
 2017-08-24 12:09:43 +02:00
golderweb 41e5cc1a9d Merge branch 'fs#141-place-notice-after-comment' into develop 2017-08-24 12:06:03 +02:00
golderweb 9b9d50c4d2 Improve detection of empty lines 
Search with RegEx as empty lines could also contain spaces

Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141)
 2017-08-24 12:04:45 +02:00
golderweb a755288700 Merge branch 'fs#147-templates-in-heading' into develop 2017-08-23 14:55:43 +02:00
golderweb 14ec71dd09 Rewrite get_disc_link to handle special cases 
Use methods of pywikibot site-object and mwparser to get rid of any
special elements like templates or links in headings for construction
of our disc link.
Replace   by hand as it otherwise will occur as normal space and
wont work

Related Task: [FS#147](https://fs.golderweb.de/index.php?do=details&task_id=147)
 2017-08-23 14:53:22 +02:00
golderweb e283eb78ac Merge branch 'fs#140-also-mark-redirects' into develop 2017-08-22 21:59:22 +02:00
golderweb cc02006fd2 Do not exclude redirects from beeing marked 
In accordance with Zulu55 redirect discussion pages should also get
a notice, therefore do not exclude redirects.

Related Task: [FS#140](https://fs.golderweb.de/index.php?do=details&task_id=140)
 2017-08-22 21:59:07 +02:00
golderweb 37b0cbef08 Merge branch 'fs#138-marked-articles-shown-again' into develop 2017-08-22 21:58:22 +02:00
golderweb 4137d72468 Look for existing notice by simple in-check 
To detect maybe uncommented notices already present, check for them
using just a simple python x in y check over whole wikicode

Related Task: [FS#138](https://fs.golderweb.de/index.php?do=details&task_id=138)
 2017-08-22 21:56:43 +02:00
golderweb cd87d1c2bb Fix already marked articles was reshown bug 
Since we search for matching states for articles to include or exclude
in a loop, we could not control the outer loop via default break/
continue. Python docs recommend using Exceptions and try/except
structures to realise that most conveniently.

https://docs.python.org/3/faq/design.html#why-is-there-no-goto

Related Task: [FS#138](https://fs.golderweb.de/index.php?do=details&task_id=138)
 2017-08-22 21:45:58 +02:00
golderweb 456b2ba3d4 Merge branch 'fs#141-place-notice-after-comment' into develop 2017-08-21 22:11:51 +02:00
golderweb 47b85a0b5e Add missing line break if there is no template 
To make sure our notice template resides in its own line in every case

Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141)
 2017-08-21 22:09:59 +02:00
golderweb a6fdc974bd Merge branch 'fs#144-PyMySQL-instead-oursql' into develop 2017-08-21 13:58:34 +02:00
golderweb 30de2a2e12 Replace oursql with PyMySQL 
Since this is prefered on toolsforge and works out of the box after
installing via pip, replace oursql which caused some problems.
Especially oursql was not able to connect to db via ssh tunnel.

Related Task: [FS#144](https://fs.golderweb.de/index.php?do=details&task_id=144)
 2017-08-21 13:55:33 +02:00
golderweb 4a6855cf7b Merge branch 'fs#141-place-notice-after-comment' into develop 2017-08-21 13:51:32 +02:00
golderweb 8422d08cb6 Keep comments and leading templates together 
Prevent spliting up existing comments and templates as often those are
documenting archiv templates behaviour

Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141)
 2017-08-21 13:49:34 +02:00
golderweb ed78501821 Merge branch 'fs#115-lsec-no-template' into test-v3 2017-08-21 13:16:28 +02:00
golderweb 34e7e0d3be Prevent index Error if no template in leadsec 
Check if there is a template in leadsec before accessing list item to
prevent IndexErrors

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=115 FS#115]
 2017-03-11 12:22:10 +01:00
golderweb f9f081d072 Merge branch 'fs#114-remove-underscores-in-articles' into test-v3 2017-03-11 11:43:49 +01:00
golderweb 0f930082b4 Also canonicalise anchor parts of articles 
Replace spaces in anchors with underscores as spaces are not correct
there

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=114 FS#114]
 2017-03-11 11:40:41 +01:00
golderweb 80c94ccf4f Replace underscores in article titles 
Remove underscores in article titles and replace with spaces to have
canonical state for all articles
Therefore we need to split title and posible anchors in heading parser

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=114 FS#114]
 2017-03-11 11:30:19 +01:00
golderweb 37704c6661 Replace pywikibot.showDiff with patched version 
Pywikibot.bot.userPut does not support setting the value of diff context
so it is always zero. Therefore we need to patch either userPut or
showDiff to get some context.

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=113 FS#113]
 2017-03-11 10:39:31 +01:00
golderweb 4e4be1c6d0 Merge branch 'fs#110-markpages-status-problems' into test-v3 2017-03-11 00:06:00 +01:00
golderweb 3e69a1c77e Remove problem indicating stati when set marked 
Remove states which are indicating problems in previous runs if
successfully marked article and also whole RedFam

[https://fs.golderweb.de/index.php?do=details&task_id=112 FS#112]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=110 FS#110]
 2017-03-11 00:03:42 +01:00
golderweb 56f326b568 Fix error all current redfams marked when quit 
Restructure update_status to make sure, marked is only set when all
articles are marked or gone (means deleted or redirect)

[https://fs.golderweb.de/index.php?do=details&task_id=111 FS#111]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=110 FS#110]
 2017-03-10 23:45:48 +01:00
golderweb 868894a38b Format fixes 
Set locale to de_DE.utf-8 for whole Task

Make sure Template is added in own source line
 2017-03-10 23:28:24 +01:00
golderweb 65de6decb2 markpages: Filter redirects 
Do not mark redirects discussion pages
 2017-03-10 21:51:59 +01:00
golderweb 889be30a47 Merge branch 'fs#109-redpage-redfam-join-type' into test-v3 2017-03-09 15:33:05 +01:00
golderweb 147e96d388 Add Wrapperclass for Parser to RedPage 
Add a wrapper class to overwrite type of Items returned by
RedPage.redfams relationship

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=109 FS#109]
 2017-03-09 15:30:51 +01:00
golderweb 76666aa294 Merge branch 'fs#25-mark-done' into test-v3 2017-03-09 10:50:14 +01:00