wiki/jogobot-red
1
0
Fork 0
Code Issues 4 Pull Requests Releases Wiki Activity
294 Commits 2 Branches 14 Tags
a35546e53d7b494f306ac50118271a20610eed6e
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
golderweb a35546e53d Fix removed pywikibot config property db_hostname 
https://phabricator.wikimedia.org/rPWBC2d73643f70a3f3289ff83e7ec142727d79d2649c
 2021-03-07 11:48:07 +01:00
golderweb e6ffd7d14a Merge branch 'hotfix-1.2.1' 2019-06-08 13:40:09 +02:00
golderweb 1b82de1eab Fix #72: Markpages terminates with error 
Filtering templates with wikicode as match param does not work anymore,
explicitly cast to string
 2019-06-08 13:38:25 +02:00
golderweb 466b9da886 Merge branch 'release-1.2' 2018-10-05 12:37:23 +02:00
golderweb b80f5bd2c9 Prepare release v1.2 2018-10-05 12:36:49 +02:00
golderweb ff2421b63e Merge branch 'remove-jogobot-submodule' into develop 2018-10-05 11:42:08 +02:00
golderweb 236ba6a870 Update requirements 2018-10-05 11:40:25 +02:00
golderweb 0df2017387 Remove submodule jogobot 
As now installable via pip
 2018-10-05 11:37:56 +02:00
golderweb b9faed8847 Merge branch 'i#64-missingnotice' into develop 2018-10-05 11:35:50 +02:00
golderweb 5cdccaeec6 missingnotice: Add RedFam counter output 
Signalize bot is working, since processing RedFams takes about
15 minutes with out any output right now

Issue #64 (#64)
 2018-09-25 17:51:01 +02:00
golderweb 54d8b8ea3b missingnotice: Disable verbose logging of sqlalchemy 
Issue #64 (#64)
 2018-09-18 16:29:54 +02:00
golderweb dea5a393ad missingnotice: Call RedFamWorker.flush_db_cache 
To write status changes to db

Issue #64 (#64)
 2018-09-18 16:29:50 +02:00
golderweb f021a13202 missingnotice: Implement run() 
The bots working sequence, using previously implemented methods to
update the list of missing notices

Issue #64 (#64)
 2018-09-18 16:29:45 +02:00
golderweb 4c8ba95534 missingnotice: Implement update_page() 
This method updates the content of the configured or given wikipage with
the generated lines

Issue #64 (#64)
 2018-09-18 16:29:40 +02:00
golderweb 9804db212f missingnotice: Implement format_row() 
With this method, the links to redundance discussions and articles
missining notice are concatenated and formated

Issue #64 (#64)
 2018-09-18 16:29:34 +02:00
golderweb 68b81b1111 missingnotice: Implement treat_redfam 
For each redfam, we need to check weather related redundance discussion
exists and if there are missing notices. For those redfams return links
to discussion and articles missing notice.

Issue #64 (#64)
 2018-09-18 16:29:28 +02:00
golderweb 389c48605e redfam: Make get_disc_link() able to return wikilink 
Issue #64 (#64)
 2018-09-18 16:29:22 +02:00
golderweb 95af95aca6 missingnotice: Implement article selection 
Issue #64 (#64)
 2018-09-18 16:29:17 +02:00
golderweb 99adad873e missingnotice_test: Test article query 
Issue #64 (#64)
 2018-09-18 16:29:01 +02:00
golderweb dbcc2717d7 missingnotice: Implement article query 
Issue #64 (#64)
 2018-09-18 16:28:56 +02:00
golderweb e5a45fa692 tests: Add test script for missingnotice 
Issue #64 (#64)
 2018-09-18 16:28:44 +02:00
golderweb 63d3f837e9 red.py: Introduce subtask missingnotice 
Issue #64 (#64)
 2018-09-18 16:28:38 +02:00
golderweb cfb3e8e37c bots: Add basic structure for MissingNoticeBot 
Issue #64 (#64)
 2018-09-18 16:28:29 +02:00
golderweb dfffe97200 redfam: Add method to check disc section 
Sometimes disc sections are disapering since the heading is changed
and the famhash changes, so we get a new redfam. Mark those as absent

Issue #64 (#64)
 2018-09-18 16:28:22 +02:00
golderweb 246e94c228 redfam: Add generator for open redfams to Worker 
Issue #64 (#64)
 2018-09-18 16:27:51 +02:00
golderweb 181486c718 Merge branch 'release-1.1.1' back into develop 2018-09-17 17:23:12 +02:00
golderweb 4f31b1a792 Merge branch 'release-1.1.1' 2018-08-12 11:48:17 +02:00
golderweb 3fbfd4ccd7 Prepare release-1.1.1 2018-08-12 11:46:40 +02:00
golderweb 50b0e142ec Merge branch 'i#71-moved-page-exists' into develop 2018-08-12 11:43:18 +02:00
golderweb 14db996a43 redfam: Check if moved page exists 
To prevent creation of orphaned diskpages in case of special movement
constructs

Issue #71 (#71)
 2018-08-12 11:41:50 +02:00
golderweb 110589cb5b Merge branch 'release-1.1' back into develop 2018-08-12 11:15:30 +02:00
golderweb 5c277495a3 Merge branch 'release-1.1' 2018-05-17 12:41:35 +02:00
golderweb a466ab4e74 Prepare release-1.1 2018-05-17 12:41:06 +02:00
golderweb 860a285ab0 Merge branch 'i#68-exclude-users' into develop 2018-05-17 12:36:37 +02:00
golderweb 2c105336b0 RedFamWorker: Exclude users and user talkpages 
Users can't be part of valid redundances

Issue #68 (#68)
 2018-05-17 12:35:38 +02:00
golderweb ea85ca731f Merge branch 'i#69-already-talkpage' into develop 2018-05-17 12:28:17 +02:00
golderweb 6e119ea98f RedFamWorker: Improve talkpagetoggling 
Do not toggle to main page if we have already a talkpage and vice versa

Issue #69 (#69)
 2018-05-17 12:26:37 +02:00
golderweb 67aaf3cbbe Merge branch 'i#70-follow-moved-pages' into develop 2018-05-17 12:24:00 +02:00
golderweb fa13e2a5cf Follow moved pages 
Keep notice together with content
https://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:Jogo.obb&oldid=176464377#Redundanzhinweis_zu_zwischenzeitlich_verschobenen_Artikeln

Issue #70 (#70)
 2018-05-17 12:18:13 +02:00
golderweb 562e689418 Merge branch 'release-1.0' back into develop 2017-11-05 12:30:18 +01:00
golderweb ae1ee7d6a5 Merge branch 'release-1.0' 2017-11-05 12:28:21 +01:00
golderweb 93447d8dc6 Prepare release v1.0 
Update Copyright Notices
Version information
 2017-11-05 12:25:13 +01:00
golderweb 1b6faf9e53 Use own db for red-task 
Since we have several tables and sometimes need to create a copy on
replication servers.
 2017-11-05 12:17:05 +01:00
golderweb b4c193eedc Disable echoing of SQLAlchemy Egine 
We don't need this extensive output for production
 2017-11-05 12:07:38 +01:00
golderweb 788a3df0cd Update jogobot-submodule to v0.1 2017-11-05 12:00:28 +01:00
golderweb 04f591b466 Merge branch 'fs#161-add-article-titles' into develop 2017-11-05 11:24:15 +01:00
golderweb 9640467f69 markpages: Use redarticle attribute of Page 
Instead of trying to reconstruct our db article title, use the one added
to Page-object by redfam.article_generator

Related Task: [FS#161](https://fs.golderweb.de/index.php?do=details&task_id=161)
 2017-11-05 11:22:43 +01:00
golderweb bfec2abf98 markpages: Get rid of PageWithTalkPageGenerator 
Since redfam.article_generator can yield talkpage with additional
information about redfam and current article from db, we do not need it
anymore.

Related Task: [FS#161](https://fs.golderweb.de/index.php?do=details&task_id=161)
 2017-11-05 11:20:55 +01:00
golderweb 20103d589d redfam: article_generator add redfam info to page 
Add reference to redfam object and article title from db to Page object
since Page.title() may differe (short Namespaces, anchors, special chars)

Related Task: [FS#161](https://fs.golderweb.de/index.php?do=details&task_id=161)
 2017-11-05 11:18:53 +01:00
golderweb e18aa96a84 redfam: article_generator can return talkpage 
To make pywikibot.pagegenerators.PageWithTalkPageGenerators unneccessary
so we can manipulate talkpage object directly

Related Task: [FS#161](https://fs.golderweb.de/index.php?do=details&task_id=161)
 2017-11-05 11:15:04 +01:00
golderweb 1dd4c7f87e Merge branch 'test-v7' back into develop 2017-11-02 18:57:59 +01:00
golderweb 33b2e47312 Describe version test-v7 2017-10-28 22:43:53 +02:00
golderweb 3bd17ce692 Merge branch 'fs#160-urlencoded-chars' into develop 2017-10-28 22:36:55 +02:00
golderweb 5f4640d5ff Replace urlencoded chars with unicode equivalent 
Otherwise we get value errors while marking since pwb replaces those

Related Task: [FS#160](https://fs.golderweb.de/index.php?do=details&task_id=160)
 2017-10-28 22:35:25 +02:00
golderweb 7e0456ae4f Merge branch 'test-v6' back into develop 2017-10-28 22:34:30 +02:00
golderweb 108b7aa331 Describe version test-v6 2017-10-28 18:46:30 +02:00
golderweb a3adf31b89 Merge branch 'fs#86-activate-status-api' into develop 2017-10-28 18:44:42 +02:00
golderweb 614f288bb9 Activate jogobot status api for onwiki disabling 
Related Task: [FS#86](https://fs.golderweb.de/index.php?do=details&task_id=86)
 2017-10-28 18:44:05 +02:00
golderweb c450a045bf Merge branch 'fs#159-space-before-anchor' into develop 2017-10-28 18:43:13 +02:00
golderweb 84802cf521 Remove leading or trailing spaces in articles 
Some articles contain spaces between title and anchor part which will
be stripped now

Related Task: [FS#159](https://fs.golderweb.de/index.php?do=details&task_id=159)
 2017-10-28 18:41:06 +02:00
golderweb 5f6c443ba8 Merge branch 'test-v5' back into develop 2017-10-28 18:17:01 +02:00
golderweb 0c135ef1bb Describe version test-v5 2017-09-23 23:50:42 +02:00
golderweb 8b8221cfcd Merge branch 'fs#152-respect-always-flag' into develop 2017-09-23 23:49:59 +02:00
golderweb bdccc8417c Set always in Pywikibot.Basebot 
If cmdline param -always is set, set the related option in
Pywikibot.Basebot Object for automatic edits with out further requests

Related Task: [FS#152](https://fs.golderweb.de/index.php?do=details&task_id=152)
 2017-09-23 23:49:41 +02:00
golderweb a70835c58a Merge back branch 'test-v4' into develop 2017-09-23 23:48:25 +02:00
golderweb ec2b84df2a Add requirements 
To make setup of environment for this module easier
 2017-09-23 21:09:58 +02:00
golderweb 88848cb084 Prepare Version test-v4 for release 
Add a README.md file for this project
 2017-09-23 20:32:13 +02:00
golderweb 5057aed0d3 Merge branch 'fs#157-lowercase-title' into develop 2017-09-09 21:47:03 +02:00
golderweb 02e53475f1 Prevent lowercase article titles in Parser 
Since real lowercase article titles are not allowed, make sure to
convert all first letters of article titles to uppercase. This is
neccessary since pywikibot will return article titles like this.

Related Task: [FS#157](https://fs.golderweb.de/index.php?do=details&task_id=157)
 2017-09-09 21:35:36 +02:00
golderweb d6f9b460c9 Merge branch 'fs#156-dbapi-charset' into develop 2017-09-02 22:13:20 +02:00
golderweb ff03ca8f13 Explicitly set charset for PyMySQL-Connection 
Since PyMySQL-Connection otherwise uses charset 'latin-1', explicitly
set connection charset to 'utf8'

http://docs.sqlalchemy.org/en/rel_1_0/dialects/mysql.html#charset-selection
http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html?highlight=url#sqlalchemy.engine.url.URL

Related Task: [FS#156](https://fs.golderweb.de/index.php?do=details&task_id=156)
 2017-09-02 22:10:25 +02:00
golderweb 88692ca678 Merge branch 'fs#155-article-surouding-space' into develop 2017-09-02 22:08:31 +02:00
golderweb d9b4fcc0bd Strip spaces before adding articles to redfam 
Some article links have surounding spaces in their linktext. Remove them
before adding article to RedFam to have a cannonical title

Related Task: [FS#155](https://fs.golderweb.de/index.php?do=details&task_id=155)
 2017-09-02 22:06:30 +02:00
golderweb 22ff78ea98 Merge branch 'fs#154-categorie-colons-missing' into develop 2017-09-02 16:02:45 +02:00
golderweb b3cfcdc259 Improve title detection to get correct behaviour 
Make sure that categorie links are starting with colon and non article
pages are returned with namespace.

Related Task: [FS#154](https://fs.golderweb.de/index.php?do=details&task_id=154)
 2017-09-02 15:59:34 +02:00
golderweb b3e0ace2f4 Merge branch 'fs#153-nested-templates' into develop 2017-09-02 14:25:21 +02:00
golderweb f8002c85da Do not search for templates recursivly 
Since nested templates did not get an index in global wikicode object
searching for index of an nested template results in ValueError

Related Task: [FS#153](https://fs.golderweb.de/index.php?do=details&task_id=153)
 2017-09-02 14:23:25 +02:00
golderweb 49bc05d29b Merge branch 'fs#151-normalize-article-titles-anchor' into develop 2017-09-02 13:36:17 +02:00
golderweb 8a26b6d92a Normalize article titles with anchors 
In our db article titles with anchors are stored with underscores in
anchor string. Therefore we need to replace spaces in anchor string
given by pywikibot.Page.title().

Related Task: [FS#151](https://fs.golderweb.de/index.php?do=details&task_id=151)
 2017-08-25 18:11:41 +02:00
golderweb 49a8230d76 Merge branch 'fs#141-place-notice-after-comment' into develop 2017-08-25 17:11:28 +02:00
golderweb 31c10073a2 Prevent index errors searching for comments 
Make sure not to exceed existing indexes of wikicode object while trying
to search for comments

Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141)
 2017-08-25 17:09:38 +02:00
golderweb 642a29b022 Improve regex for blank lines 
Do not match consecutive linebreaks as one

Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141)
 2017-08-24 18:47:18 +02:00
golderweb 2f90751dc2 Merge branch 'fs#146-famhash-generator' into develop 2017-08-24 12:27:54 +02:00
golderweb 024be69fe1 Use famhash as generator 
If famhash is defined, fetch explicitly that redfam from db and work
only on this

Related Task: [FS#146](https://fs.golderweb.de/index.php?do=details&task_id=146)
 2017-08-24 12:27:13 +02:00
golderweb b6d7268a7f select by famhash: Add methods to get param in bot 
We need a method as callback to get bot specific params passed through
to our bot class.
Introduce -famhash parameter to work on specific famhash

Related Task:[FS#146](https://fs.golderweb.de/index.php?do=details&task_id=146)
 2017-08-24 12:27:13 +02:00
golderweb 526184c1e1 Merge branch 'fs#148-articles-mixed-up' into develop 2017-08-24 12:26:53 +02:00
golderweb 3aa6c5fb1c Disable PreloadingGenerator temporarily 
PreloadingGenerator mixes up yielded Pages. This is very unconvenient
for semi-automatic workflow with manual checks as the articles of the
RedFams were not following each other.

Related Task: [FS#148](https://fs.golderweb.de/index.php?do=details&task_id=148)
 2017-08-24 12:23:17 +02:00
golderweb ec8f459db5 Merge branch 'fs#138-marked-articles-shown-again' into develop 2017-08-24 12:19:24 +02:00
golderweb 3b2cb95f36 Do not fetch marked redfams from db 
Exclude marked Redfams from DB-Query to prevent marking them again

Related Task: [FS#138](https://fs.golderweb.de/index.php?do=details&task_id=138)
 2017-08-24 12:09:43 +02:00
golderweb 41e5cc1a9d Merge branch 'fs#141-place-notice-after-comment' into develop 2017-08-24 12:06:03 +02:00
golderweb 9b9d50c4d2 Improve detection of empty lines 
Search with RegEx as empty lines could also contain spaces

Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141)
 2017-08-24 12:04:45 +02:00
golderweb a755288700 Merge branch 'fs#147-templates-in-heading' into develop 2017-08-23 14:55:43 +02:00
golderweb 14ec71dd09 Rewrite get_disc_link to handle special cases 
Use methods of pywikibot site-object and mwparser to get rid of any
special elements like templates or links in headings for construction
of our disc link.
Replace   by hand as it otherwise will occur as normal space and
wont work

Related Task: [FS#147](https://fs.golderweb.de/index.php?do=details&task_id=147)
 2017-08-23 14:53:22 +02:00
golderweb e283eb78ac Merge branch 'fs#140-also-mark-redirects' into develop 2017-08-22 21:59:22 +02:00
golderweb cc02006fd2 Do not exclude redirects from beeing marked 
In accordance with Zulu55 redirect discussion pages should also get
a notice, therefore do not exclude redirects.

Related Task: [FS#140](https://fs.golderweb.de/index.php?do=details&task_id=140)
 2017-08-22 21:59:07 +02:00
golderweb 37b0cbef08 Merge branch 'fs#138-marked-articles-shown-again' into develop 2017-08-22 21:58:22 +02:00
golderweb 4137d72468 Look for existing notice by simple in-check 
To detect maybe uncommented notices already present, check for them
using just a simple python x in y check over whole wikicode

Related Task: [FS#138](https://fs.golderweb.de/index.php?do=details&task_id=138)
 2017-08-22 21:56:43 +02:00
golderweb cd87d1c2bb Fix already marked articles was reshown bug 
Since we search for matching states for articles to include or exclude
in a loop, we could not control the outer loop via default break/
continue. Python docs recommend using Exceptions and try/except
structures to realise that most conveniently.

https://docs.python.org/3/faq/design.html#why-is-there-no-goto

Related Task: [FS#138](https://fs.golderweb.de/index.php?do=details&task_id=138)
 2017-08-22 21:45:58 +02:00
golderweb 456b2ba3d4 Merge branch 'fs#141-place-notice-after-comment' into develop 2017-08-21 22:11:51 +02:00
golderweb 47b85a0b5e Add missing line break if there is no template 
To make sure our notice template resides in its own line in every case

Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141)
 2017-08-21 22:09:59 +02:00