wiki/jogobot-red
1
0
Fork 0
Code Issues 4 Pull Requests Releases Wiki Activity
194 Commits 2 Branches 14 Tags
642a29b022ade48b8e9e11d93d3ff380b0de737b
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
golderweb 642a29b022 Improve regex for blank lines 
Do not match consecutive linebreaks as one

Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141)
 2017-08-24 18:47:18 +02:00
golderweb 9b9d50c4d2 Improve detection of empty lines 
Search with RegEx as empty lines could also contain spaces

Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141)
 2017-08-24 12:04:45 +02:00
golderweb 47b85a0b5e Add missing line break if there is no template 
To make sure our notice template resides in its own line in every case

Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141)
 2017-08-21 22:09:59 +02:00
golderweb 8422d08cb6 Keep comments and leading templates together 
Prevent spliting up existing comments and templates as often those are
documenting archiv templates behaviour

Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141)
 2017-08-21 13:49:34 +02:00
golderweb ed78501821 Merge branch 'fs#115-lsec-no-template' into test-v3 2017-08-21 13:16:28 +02:00
golderweb 34e7e0d3be Prevent index Error if no template in leadsec 
Check if there is a template in leadsec before accessing list item to
prevent IndexErrors

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=115 FS#115]
 2017-03-11 12:22:10 +01:00
golderweb f9f081d072 Merge branch 'fs#114-remove-underscores-in-articles' into test-v3 2017-03-11 11:43:49 +01:00
golderweb 0f930082b4 Also canonicalise anchor parts of articles 
Replace spaces in anchors with underscores as spaces are not correct
there

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=114 FS#114]
 2017-03-11 11:40:41 +01:00
golderweb 80c94ccf4f Replace underscores in article titles 
Remove underscores in article titles and replace with spaces to have
canonical state for all articles
Therefore we need to split title and posible anchors in heading parser

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=114 FS#114]
 2017-03-11 11:30:19 +01:00
golderweb 37704c6661 Replace pywikibot.showDiff with patched version 
Pywikibot.bot.userPut does not support setting the value of diff context
so it is always zero. Therefore we need to patch either userPut or
showDiff to get some context.

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=113 FS#113]
 2017-03-11 10:39:31 +01:00
golderweb 4e4be1c6d0 Merge branch 'fs#110-markpages-status-problems' into test-v3 2017-03-11 00:06:00 +01:00
golderweb 3e69a1c77e Remove problem indicating stati when set marked 
Remove states which are indicating problems in previous runs if
successfully marked article and also whole RedFam

[https://fs.golderweb.de/index.php?do=details&task_id=112 FS#112]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=110 FS#110]
 2017-03-11 00:03:42 +01:00
golderweb 56f326b568 Fix error all current redfams marked when quit 
Restructure update_status to make sure, marked is only set when all
articles are marked or gone (means deleted or redirect)

[https://fs.golderweb.de/index.php?do=details&task_id=111 FS#111]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=110 FS#110]
 2017-03-10 23:45:48 +01:00
golderweb 868894a38b Format fixes 
Set locale to de_DE.utf-8 for whole Task

Make sure Template is added in own source line
 2017-03-10 23:28:24 +01:00
golderweb 65de6decb2 markpages: Filter redirects 
Do not mark redirects discussion pages
 2017-03-10 21:51:59 +01:00
golderweb 889be30a47 Merge branch 'fs#109-redpage-redfam-join-type' into test-v3 2017-03-09 15:33:05 +01:00
golderweb 147e96d388 Add Wrapperclass for Parser to RedPage 
Add a wrapper class to overwrite type of Items returned by
RedPage.redfams relationship

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=109 FS#109]
 2017-03-09 15:30:51 +01:00
golderweb 76666aa294 Merge branch 'fs#25-mark-done' into test-v3 2017-03-09 10:50:14 +01:00
golderweb db39bb5ff4 Merge branch 'fs#88-mark-pages-bot' into fs#25-mark-done 2017-03-09 10:49:16 +01:00
golderweb ec7880207b Merge branch 'fs#95-sqlalchemy' into fs#88-mark-pages-bot 2017-03-09 10:48:10 +01:00
golderweb 4aaacf1443 Add redfams to redpage-obj after parsing 
To have redfams available for updates immediately after parsing. Double
redfams then will be seen as Update.

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=108 FS#108]
 2017-03-09 10:25:52 +01:00
golderweb 281f1c49a8 mysqlred: Set family via pywikibot 
Get family/language part of table names from PyWikiBot Site
 2017-03-09 00:12:41 +01:00
golderweb 3fe47e666f Fix polymorphism problem with relationships 
Since we are using subclasses of the ORM mapped classes, disable
typechecks for ORM relations
 2017-03-09 00:12:41 +01:00
golderweb e16925197c Fix pep8.. compliance 
To be concordant with the coding styles fix pep8 compliance
 2017-03-09 00:12:31 +01:00
golderweb 9ba7d2e517 Change redfam generator filters 
Change and clear up the filters in redfam generator to keep track of
article status and use positive conditionals
 2017-03-09 00:10:51 +01:00
golderweb 844fee52ae Make markpages using new DB/Class structure 
Update markpages and RedFamWorker-Code to use the new sqlalchemy based
DB ORM Interface
 2017-03-09 00:10:51 +01:00
golderweb 43e31c108a Working RedFamWorker query 
Modify RedfamWorker class to work with new DB API
 2017-03-09 00:10:50 +01:00
golderweb 89b50e3312 Remove old status API 
Now we use the methods of status object directly
 2017-03-09 00:10:50 +01:00
golderweb bf8e47f916 Improve new status API 
Make sure state changes are only detected as such by sqlalchemy if they
are real changes
 2017-03-09 00:10:50 +01:00
golderweb 467f829af2 Some cleanups 
Remove old commented out code from manual mysql solution
 2017-03-09 00:10:50 +01:00
golderweb 6e973369cd sqlalchemy working for parser 
Needs some testing, presumably contains some bugs
 2017-03-09 00:08:48 +01:00
golderweb 0ebf307bb8 Add markpages as subtask 
Markpages is a subtask of our Red-Bot

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]

# The following line will be added automatically
# Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]
 2016-11-05 19:41:22 +01:00
golderweb 4e4d5005fd Merge branch 'fs#89-article-status' into fs#88-mark-pages-bot 2016-11-05 19:39:30 +01:00
golderweb 65fb2ecb28 Generate Fam status based on article status 
Some article states should be reflected in the RedFam status

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]
 2016-11-05 19:27:56 +01:00
golderweb d55c81c97b Set article status when worked on talkpage 
To detect whole redfam status after run over all articles

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]
 2016-08-30 18:05:51 +02:00
golderweb 870ed4bf25 Update redfam.article_generator use article status 
To be able to filter articles by status of that article

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]
 2016-08-30 17:47:02 +02:00
golderweb e13320820c Add API to manage status per article 
To be able to track changes to articles to update redfam status

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]
 2016-08-30 17:45:18 +02:00
golderweb 4ae562590e Merge branch 'fs#94-data-structure' into fs#88-mark-pages-bot 2016-08-30 17:37:31 +02:00
golderweb 6149dcdb8b Apply changes to data structure 
See related ticket

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=94 FS#94]
 2016-08-30 14:28:28 +02:00
golderweb f021f2ea60 Merge branch 'fs#93-update-talkpage-template' into fs#88-mark-pages-bot 2016-08-30 12:08:57 +02:00
golderweb 8c56125a7b Update talkpage notice template 
Exact date is not necessary and end could be ommited if of the same
month

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=93 FS#93]
 2016-08-30 12:07:11 +02:00
golderweb c19f642d11 Merge branch 'fs#92-mark-done-edit-summary' into fs#88-mark-pages-bot 2016-08-30 11:51:07 +02:00
golderweb 20b811bc2a Make sure edit summary starts with bot 
Due to bot policy all edit summaries of bot edits have to start with
"Bot:"

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=92 FS#92]
 2016-08-30 11:48:07 +02:00
golderweb 59d4d23c83 Set edit summary for each edit 
Each edit of bot needs a edit summary

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=92 FS#92]
 2016-08-30 11:33:54 +02:00
golderweb 2b93e4cf16 Check if notice is present before add 
To prevent duplications we need to check wether notice is already
present on talkpage

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]
 2016-08-28 21:39:54 +02:00
golderweb 9beca7f6c9 Implement method to add notice to disk page 
Adds the generated notice to the talkpage and starts the saving of
the page

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]
 2016-08-28 21:33:36 +02:00
golderweb c4d8a95672 Implement build_generator-method 
Build_generator will add the redfam_talkpages_generator to the
genFactory, build a generator of the genFactory and sets self.gen
which is used as generator for run()

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]
 2016-08-28 18:13:27 +02:00
golderweb da4f9b5d6b Add wrapper-generator to redfam.article_generator 
We need a wrapper around redfam.article_generator to pass it to
pagegenerators.PageWithTalkPageGenerator and to add a reference to
related redfam to each pywikibot.page-object before yielding it

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]
 2016-08-28 18:09:04 +02:00
golderweb ecc78bef96 Import needed modules and add redfams-generator 
We will need a couple of modules to build the needed generator
Also we will need a generator with redfams to work with

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]
 2016-08-28 18:01:02 +02:00
golderweb efa919ff27 Add new bot with basic structure 
We need a bot to work on pages which are subjects of redfams and on the
belonging talk page

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]
 2016-08-28 16:39:32 +02:00
golderweb 72c6165de8 Merge branch 'fs#87-redfam-article-generator' into fs#25-mark-done 2016-08-28 16:20:53 +02:00
golderweb c0b18f88e5 Add filter options to redfam.article_generator 
To give the posibility to filter not existing pages or redirect pages or
vice versa.

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=87 FS#87]
 2016-08-28 15:06:17 +02:00
golderweb e5989305a4 Add a generator to redfam yielding article pages 
To work on articles of a redfam a generator which yields belonging
articles is necessary

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=87 FS#87]
 2016-08-28 13:20:13 +02:00
golderweb 8ce6f03641 Merge branch 'fs#29-generate-articledisc-notice' into fs#25-mark-done 2016-08-27 19:50:18 +02:00
golderweb 6717fa4fba Add method to generate notice for article discpage 
We need a method to generate the template to add to article discpages

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=29 FS#29]
 2016-08-27 19:49:50 +02:00
golderweb 8acba7d0f9 Merge branch 'fs#81-get-reddisc-link' into fs#25-mark-done 2016-08-27 19:48:24 +02:00
golderweb 3723aba578 Add a method to get link to related reddisc 
To generate notices or other stuff it is necessary to add links to the
related reddisc.
This method returns a wikilink to text the redfam's reddisc.

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=81 FS#81]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=81 FS#81]
 2016-08-27 19:48:03 +02:00
golderweb 9d3bc74c80 Merge branch 'fs#26-done-redfam-gen' into fs#25-mark-done 2016-08-27 19:46:41 +02:00
golderweb b36dc250d2 Request information about reddisc page for redfams 
To generate links to related reddisc it is necessary to get at least the
Title of the related reddisc page. As saving the same data in db is
worse, we retreive it via join from red_pages table

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]
 2016-08-27 19:44:33 +02:00
golderweb 4055dc52d8 Make it possible to get a RedPage-Object by pageid 
When working on redfams it is necessary to have information about redpage

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]
 2016-08-27 19:44:33 +02:00
golderweb 594130c8a6 Restore changes from 45df35431 
Documented to prevent deleting again

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]
 2016-08-27 19:44:33 +02:00
golderweb b271a0b0b1 Add generator wrapper to fetch RedFams by status and ending 
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]
 2016-08-27 19:44:33 +02:00
golderweb ad088126e7 Define method to update Status after Working with RedFam 
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]
 2016-08-27 19:44:33 +02:00
golderweb 151c22a735 Add fetched mysql_data to _mysql-Object of parent class for using change-method to update db 
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]
 2016-08-27 19:44:33 +02:00
golderweb a97d8c722e Move handling of mysql-Connection from RedFamParser and RedFamWorker to RedFam-Class and make it protected instead of private 
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]
 2016-08-27 19:44:33 +02:00
golderweb 58dfd8c86a For RedFamilies not fetched individually we need to provide the fam hash as index 
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]
 2016-08-27 19:44:33 +02:00
golderweb 9481116777 Add new generator-method to fetch RedFams by Status and Ending 
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]
 2016-08-27 19:44:33 +02:00
golderweb eaa7596a8f Merge branch 'fs#70-refactoring' into test-v3 2016-08-27 19:40:00 +02:00
golderweb 449d83d7b5 Merge branch 'fs#82-subtask-wrapper' into fs#70-refactoring 2016-08-27 19:19:40 +02:00
golderweb 4ac9b305f5 Merge branch 'fs#85-move-start-api-to-jogobot' into fs#82-subtask-wrapper 2016-08-27 19:17:10 +02:00
golderweb 604b7bd8b7 Now use Bot-Start API from jogobot framework 
API was moved to jogobot to share with other tasks

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85]
 2016-08-27 19:04:13 +02:00
golderweb d0fa15d0ed Update jogobot module to get standart Start-API 
[FS#84]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85]
 2016-08-27 18:50:36 +02:00
golderweb 71e41bfed3 Merge branch 'fs#83-wrapper-compatibility' into fs#82-subtask-wrapper 2016-08-27 18:24:14 +02:00
golderweb 2be0a8903d Adjust constructor for wrapper-script 
The new wrapper-script calls a standardized API
We need to be conform with that

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=83 FS#83]
 2016-08-27 17:02:51 +02:00
golderweb 0ceb2e6e83 Add methods to build gen to DiscussionParser 
With the new wrapper script the Bot gets a GenFactory and has to build
a generator out of it by its own

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=83 FS#83]
 2016-08-27 16:58:20 +02:00
golderweb 3540cc2a7d Move functional sections to functions in main() 
To make main() function less complicated functional sections are moved
to dedicated functions

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]
 2016-08-27 15:40:09 +02:00
golderweb 460d2db183 Add Bot run with exception handling 
Errors, especially caused by missing run-method, need to be catched to
provide information in Logfile.
And also to get information wether bot run was successfull

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]
 2016-08-27 15:40:09 +02:00
golderweb 156f117b18 Add Bot initiation with exception handling 
Bot initiation needs to catch errors by Bot to enforce at least a basic
logging.
And also to be sure Init was successfull before starting bot.

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]
 2016-08-27 15:40:09 +02:00
golderweb 1679e2ad6a Prepare environment for starting subtasks 
Before init and run bot we need to provide a environment for it,
like parsed args

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]
 2016-08-27 15:40:09 +02:00
golderweb b88efb6bdd Reflect stucture changes in Code 
Since bot class is moved to separate dir/file we need to do some changes
to rebuild functionality

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]
 2016-08-27 15:40:09 +02:00
golderweb 177a8f920f Prepare new structure to use subtasks 
To have only one entry point for the bot we want to have a single file
(red.py) which is calling the specfic task class from bots dir with a
standardized call

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]
 2016-08-27 15:40:09 +02:00
golderweb 0549cbd2c2 Merge branch 'fs#80-remove-deprecated-methods' into fs#70-refactoring 2016-08-25 22:43:57 +02:00
golderweb 78eda10562 Remove deprecated methods 
Deprecated functions which are not used anymore can be removed to make
code more clearer and improve maintainability

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=80 FS#80]
 2016-08-25 22:41:13 +02:00
golderweb 510771509b Merge branch 'fs#79-mysql-table-prefix' into fs#70-refactoring 2016-08-25 13:15:23 +02:00
golderweb 71b99b5f58 Delay definition of db_table_prefix 
db_table_prefix should be defined at init of MysqlRed and not at import
to have cmdline args already parsed
Otherwise it uses default family

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=79 FS#79]
 2016-08-25 13:06:32 +02:00
golderweb 77d1de4473 Add a tablename prefix depending on Site 
To be able to run the bot on different wikis the db tables should be
named pywikibot.Site dependend and changed automatically

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=79 FS#79]
 2016-08-24 23:53:10 +02:00
golderweb cac04f344f Merge branch 'fs#74-helpermodules-lib' into fs#70-refactoring 2016-08-24 23:09:33 +02:00
golderweb e28acf88d1 Introduce new directory structure 
To clarify which is a bot and which are helper scripts

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=74 FS#74]
 2016-08-24 22:41:41 +02:00
golderweb af48888535 Merge branch 'fs#78-redfam-section-false-positives' into fs#70-refactoring 2016-08-24 20:05:04 +02:00
golderweb ac54aea698 Use callback to detect redfam.section 
Detecting redfam-Sections via RegExp caused some false positives due to
wrong formated things in wikisyntax. See Task

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=78 FS#78]
 2016-08-24 20:02:48 +02:00
golderweb 2deb02fe47 Merge branch 'fs#77-errors-on-old-archives' into fs#70-refactoring 2016-08-24 19:59:53 +02:00
golderweb 1e4c8646bf Reparse redfam-heading with mwparser 
See related ticked for detailed failure explanation

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=77 FS#77]
 2016-08-24 19:57:25 +02:00
golderweb fe2810f07c Merge branch 'fs#76-redfam-without-dates' into fs#70-refactoring 2016-08-24 17:01:00 +02:00
golderweb ab430e0085 Use month of reddisc as beginning if missing 
Construct a fictive but sensfull beginning if we cant detect one
Needed since beginning is mandatory

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=76 FS#76]
 2016-08-24 16:56:54 +02:00
golderweb 95be313859 Pass reddisc pywikibot.page object to redfam 
To access page information like page title (eg. to get dates from it)
of the reddisc page

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=76 FS#76]
 2016-08-24 16:53:45 +02:00
golderweb 0bb0b2d957 Make sure var beginning is always defined 
To prevent unbound Errors caused by using undeclared variable beginning
if the redfam-section does not contain any timestamp

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=76 FS#76]
 2016-08-24 16:51:23 +02:00
golderweb db32c9e8f6 Merge branch 'fs#75-mysql-flush-error-false-reddisc' into fs#70-refactoring 2016-08-24 16:49:37 +02:00
golderweb bd2d221c48 Prevent flush from creating cursor without con 
MysqlRed.flush() tried to create a cursor in any case. If there was no
connection (because the subclasses haven't been instantiated an oursql
Error occured.
Instead, check before if there is a connection and otherwise raise an Error

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]
 2016-08-24 16:46:51 +02:00
golderweb ee8ebbc8bc Make sure only flush db if there are redfams 
To prevent from doing unnecessary stuff and trying to use not existing
db connection

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]
 2016-08-24 15:47:43 +02:00
golderweb dcc4851513 Check reddisc page titles against regex 
To prevent parsing Pages which have been categorized in configured cats
wrong or are given via cmd params
Parsing them results in unexpected behaviour

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]
 2016-08-24 15:27:42 +02:00