100 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
golderweb	cc02006fd2	Do not exclude redirects from beeing marked ... In accordance with Zulu55 redirect discussion pages should also get a notice, therefore do not exclude redirects. Related Task: [FS#140](https://fs.golderweb.de/index.php?do=details&task_id=140)	2017-08-22 21:59:07 +02:00
golderweb	37b0cbef08	Merge branch 'fs#138-marked-articles-shown-again' into develop	2017-08-22 21:58:22 +02:00
golderweb	4137d72468	Look for existing notice by simple in-check ... To detect maybe uncommented notices already present, check for them using just a simple python x in y check over whole wikicode Related Task: [FS#138](https://fs.golderweb.de/index.php?do=details&task_id=138)	2017-08-22 21:56:43 +02:00
golderweb	cd87d1c2bb	Fix already marked articles was reshown bug ... Since we search for matching states for articles to include or exclude in a loop, we could not control the outer loop via default break/ continue. Python docs recommend using Exceptions and try/except structures to realise that most conveniently. https://docs.python.org/3/faq/design.html#why-is-there-no-goto Related Task: [FS#138](https://fs.golderweb.de/index.php?do=details&task_id=138)	2017-08-22 21:45:58 +02:00
golderweb	456b2ba3d4	Merge branch 'fs#141-place-notice-after-comment' into develop	2017-08-21 22:11:51 +02:00
golderweb	47b85a0b5e	Add missing line break if there is no template ... To make sure our notice template resides in its own line in every case Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141)	2017-08-21 22:09:59 +02:00
golderweb	a6fdc974bd	Merge branch 'fs#144-PyMySQL-instead-oursql' into develop	2017-08-21 13:58:34 +02:00
golderweb	30de2a2e12	Replace oursql with PyMySQL ... Since this is prefered on toolsforge and works out of the box after installing via pip, replace oursql which caused some problems. Especially oursql was not able to connect to db via ssh tunnel. Related Task: [FS#144](https://fs.golderweb.de/index.php?do=details&task_id=144)	2017-08-21 13:55:33 +02:00
golderweb	4a6855cf7b	Merge branch 'fs#141-place-notice-after-comment' into develop	2017-08-21 13:51:32 +02:00
golderweb	8422d08cb6	Keep comments and leading templates together ... Prevent spliting up existing comments and templates as often those are documenting archiv templates behaviour Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141)	2017-08-21 13:49:34 +02:00
golderweb	ed78501821	Merge branch 'fs#115-lsec-no-template' into test-v3	2017-08-21 13:16:28 +02:00
golderweb	34e7e0d3be	Prevent index Error if no template in leadsec ... Check if there is a template in leadsec before accessing list item to prevent IndexErrors Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=115 FS#115]	2017-03-11 12:22:10 +01:00
golderweb	f9f081d072	Merge branch 'fs#114-remove-underscores-in-articles' into test-v3	2017-03-11 11:43:49 +01:00
golderweb	0f930082b4	Also canonicalise anchor parts of articles ... Replace spaces in anchors with underscores as spaces are not correct there Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=114 FS#114]	2017-03-11 11:40:41 +01:00
golderweb	80c94ccf4f	Replace underscores in article titles ... Remove underscores in article titles and replace with spaces to have canonical state for all articles Therefore we need to split title and posible anchors in heading parser Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=114 FS#114]	2017-03-11 11:30:19 +01:00
golderweb	37704c6661	Replace pywikibot.showDiff with patched version ... Pywikibot.bot.userPut does not support setting the value of diff context so it is always zero. Therefore we need to patch either userPut or showDiff to get some context. Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=113 FS#113]	2017-03-11 10:39:31 +01:00
golderweb	4e4be1c6d0	Merge branch 'fs#110-markpages-status-problems' into test-v3	2017-03-11 00:06:00 +01:00
golderweb	3e69a1c77e	Remove problem indicating stati when set marked ... Remove states which are indicating problems in previous runs if successfully marked article and also whole RedFam [https://fs.golderweb.de/index.php?do=details&task_id=112 FS#112] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=110 FS#110]	2017-03-11 00:03:42 +01:00
golderweb	56f326b568	Fix error all current redfams marked when quit ... Restructure update_status to make sure, marked is only set when all articles are marked or gone (means deleted or redirect) [https://fs.golderweb.de/index.php?do=details&task_id=111 FS#111] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=110 FS#110]	2017-03-10 23:45:48 +01:00
golderweb	868894a38b	Format fixes ... Set locale to de_DE.utf-8 for whole Task Make sure Template is added in own source line	2017-03-10 23:28:24 +01:00
golderweb	65de6decb2	markpages: Filter redirects ... Do not mark redirects discussion pages	2017-03-10 21:51:59 +01:00
golderweb	889be30a47	Merge branch 'fs#109-redpage-redfam-join-type' into test-v3	2017-03-09 15:33:05 +01:00
golderweb	147e96d388	Add Wrapperclass for Parser to RedPage ... Add a wrapper class to overwrite type of Items returned by RedPage.redfams relationship Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=109 FS#109]	2017-03-09 15:30:51 +01:00
golderweb	76666aa294	Merge branch 'fs#25-mark-done' into test-v3	2017-03-09 10:50:14 +01:00
golderweb	db39bb5ff4	Merge branch 'fs#88-mark-pages-bot' into fs#25-mark-done	2017-03-09 10:49:16 +01:00
golderweb	ec7880207b	Merge branch 'fs#95-sqlalchemy' into fs#88-mark-pages-bot	2017-03-09 10:48:10 +01:00
golderweb	4aaacf1443	Add redfams to redpage-obj after parsing ... To have redfams available for updates immediately after parsing. Double redfams then will be seen as Update. Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=108 FS#108]	2017-03-09 10:25:52 +01:00
golderweb	281f1c49a8	mysqlred: Set family via pywikibot ... Get family/language part of table names from PyWikiBot Site	2017-03-09 00:12:41 +01:00
golderweb	3fe47e666f	Fix polymorphism problem with relationships ... Since we are using subclasses of the ORM mapped classes, disable typechecks for ORM relations	2017-03-09 00:12:41 +01:00
golderweb	e16925197c	Fix pep8.. compliance ... To be concordant with the coding styles fix pep8 compliance	2017-03-09 00:12:31 +01:00
golderweb	9ba7d2e517	Change redfam generator filters ... Change and clear up the filters in redfam generator to keep track of article status and use positive conditionals	2017-03-09 00:10:51 +01:00
golderweb	844fee52ae	Make markpages using new DB/Class structure ... Update markpages and RedFamWorker-Code to use the new sqlalchemy based DB ORM Interface	2017-03-09 00:10:51 +01:00
golderweb	43e31c108a	Working RedFamWorker query ... Modify RedfamWorker class to work with new DB API	2017-03-09 00:10:50 +01:00
golderweb	89b50e3312	Remove old status API ... Now we use the methods of status object directly	2017-03-09 00:10:50 +01:00
golderweb	bf8e47f916	Improve new status API ... Make sure state changes are only detected as such by sqlalchemy if they are real changes	2017-03-09 00:10:50 +01:00
golderweb	467f829af2	Some cleanups ... Remove old commented out code from manual mysql solution	2017-03-09 00:10:50 +01:00
golderweb	6e973369cd	sqlalchemy working for parser ... Needs some testing, presumably contains some bugs	2017-03-09 00:08:48 +01:00
golderweb	0ebf307bb8	Add markpages as subtask ... Markpages is a subtask of our Red-Bot Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89] # The following line will be added automatically # Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]	2016-11-05 19:41:22 +01:00
golderweb	4e4d5005fd	Merge branch 'fs#89-article-status' into fs#88-mark-pages-bot	2016-11-05 19:39:30 +01:00
golderweb	65fb2ecb28	Generate Fam status based on article status ... Some article states should be reflected in the RedFam status Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]	2016-11-05 19:27:56 +01:00
golderweb	d55c81c97b	Set article status when worked on talkpage ... To detect whole redfam status after run over all articles Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]	2016-08-30 18:05:51 +02:00
golderweb	870ed4bf25	Update redfam.article_generator use article status ... To be able to filter articles by status of that article Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]	2016-08-30 17:47:02 +02:00
golderweb	e13320820c	Add API to manage status per article ... To be able to track changes to articles to update redfam status Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]	2016-08-30 17:45:18 +02:00
golderweb	4ae562590e	Merge branch 'fs#94-data-structure' into fs#88-mark-pages-bot	2016-08-30 17:37:31 +02:00
golderweb	6149dcdb8b	Apply changes to data structure ... See related ticket Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=94 FS#94]	2016-08-30 14:28:28 +02:00
golderweb	f021f2ea60	Merge branch 'fs#93-update-talkpage-template' into fs#88-mark-pages-bot	2016-08-30 12:08:57 +02:00
golderweb	8c56125a7b	Update talkpage notice template ... Exact date is not necessary and end could be ommited if of the same month Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=93 FS#93]	2016-08-30 12:07:11 +02:00
golderweb	c19f642d11	Merge branch 'fs#92-mark-done-edit-summary' into fs#88-mark-pages-bot	2016-08-30 11:51:07 +02:00
golderweb	20b811bc2a	Make sure edit summary starts with bot ... Due to bot policy all edit summaries of bot edits have to start with "Bot:" Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=92 FS#92]	2016-08-30 11:48:07 +02:00
golderweb	59d4d23c83	Set edit summary for each edit ... Each edit of bot needs a edit summary Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=92 FS#92]	2016-08-30 11:33:54 +02:00
golderweb	2b93e4cf16	Check if notice is present before add ... To prevent duplications we need to check wether notice is already present on talkpage Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]	2016-08-28 21:39:54 +02:00
golderweb	9beca7f6c9	Implement method to add notice to disk page ... Adds the generated notice to the talkpage and starts the saving of the page Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]	2016-08-28 21:33:36 +02:00
golderweb	c4d8a95672	Implement build_generator-method ... Build_generator will add the redfam_talkpages_generator to the genFactory, build a generator of the genFactory and sets self.gen which is used as generator for run() Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]	2016-08-28 18:13:27 +02:00
golderweb	da4f9b5d6b	Add wrapper-generator to redfam.article_generator ... We need a wrapper around redfam.article_generator to pass it to pagegenerators.PageWithTalkPageGenerator and to add a reference to related redfam to each pywikibot.page-object before yielding it Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]	2016-08-28 18:09:04 +02:00
golderweb	ecc78bef96	Import needed modules and add redfams-generator ... We will need a couple of modules to build the needed generator Also we will need a generator with redfams to work with Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]	2016-08-28 18:01:02 +02:00
golderweb	efa919ff27	Add new bot with basic structure ... We need a bot to work on pages which are subjects of redfams and on the belonging talk page Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]	2016-08-28 16:39:32 +02:00
golderweb	72c6165de8	Merge branch 'fs#87-redfam-article-generator' into fs#25-mark-done	2016-08-28 16:20:53 +02:00
golderweb	c0b18f88e5	Add filter options to redfam.article_generator ... To give the posibility to filter not existing pages or redirect pages or vice versa. Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=87 FS#87]	2016-08-28 15:06:17 +02:00
golderweb	e5989305a4	Add a generator to redfam yielding article pages ... To work on articles of a redfam a generator which yields belonging articles is necessary Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=87 FS#87]	2016-08-28 13:20:13 +02:00
golderweb	8ce6f03641	Merge branch 'fs#29-generate-articledisc-notice' into fs#25-mark-done	2016-08-27 19:50:18 +02:00
golderweb	6717fa4fba	Add method to generate notice for article discpage ... We need a method to generate the template to add to article discpages Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=29 FS#29]	2016-08-27 19:49:50 +02:00
golderweb	8acba7d0f9	Merge branch 'fs#81-get-reddisc-link' into fs#25-mark-done	2016-08-27 19:48:24 +02:00
golderweb	3723aba578	Add a method to get link to related reddisc ... To generate notices or other stuff it is necessary to add links to the related reddisc. This method returns a wikilink to text the redfam's reddisc. Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=81 FS#81] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=81 FS#81]	2016-08-27 19:48:03 +02:00
golderweb	9d3bc74c80	Merge branch 'fs#26-done-redfam-gen' into fs#25-mark-done	2016-08-27 19:46:41 +02:00
golderweb	b36dc250d2	Request information about reddisc page for redfams ... To generate links to related reddisc it is necessary to get at least the Title of the related reddisc page. As saving the same data in db is worse, we retreive it via join from red_pages table Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]	2016-08-27 19:44:33 +02:00
golderweb	4055dc52d8	Make it possible to get a RedPage-Object by pageid ... When working on redfams it is necessary to have information about redpage Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]	2016-08-27 19:44:33 +02:00
golderweb	594130c8a6	Restore changes from 45df35431 ... Documented to prevent deleting again Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]	2016-08-27 19:44:33 +02:00
golderweb	b271a0b0b1	Add generator wrapper to fetch RedFams by status and ending ... Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]	2016-08-27 19:44:33 +02:00
golderweb	ad088126e7	Define method to update Status after Working with RedFam ... Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]	2016-08-27 19:44:33 +02:00
golderweb	151c22a735	Add fetched mysql_data to _mysql-Object of parent class for using change-method to update db ... Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]	2016-08-27 19:44:33 +02:00
golderweb	a97d8c722e	Move handling of mysql-Connection from RedFamParser and RedFamWorker to RedFam-Class and make it protected instead of private ... Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]	2016-08-27 19:44:33 +02:00
golderweb	58dfd8c86a	For RedFamilies not fetched individually we need to provide the fam hash as index ... Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]	2016-08-27 19:44:33 +02:00
golderweb	9481116777	Add new generator-method to fetch RedFams by Status and Ending ... Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]	2016-08-27 19:44:33 +02:00
golderweb	eaa7596a8f	Merge branch 'fs#70-refactoring' into test-v3	2016-08-27 19:40:00 +02:00
golderweb	449d83d7b5	Merge branch 'fs#82-subtask-wrapper' into fs#70-refactoring	2016-08-27 19:19:40 +02:00
golderweb	4ac9b305f5	Merge branch 'fs#85-move-start-api-to-jogobot' into fs#82-subtask-wrapper	2016-08-27 19:17:10 +02:00
golderweb	604b7bd8b7	Now use Bot-Start API from jogobot framework ... API was moved to jogobot to share with other tasks Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85]	2016-08-27 19:04:13 +02:00
golderweb	d0fa15d0ed	Update jogobot module to get standart Start-API ... [FS#84] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85]	2016-08-27 18:50:36 +02:00
golderweb	71e41bfed3	Merge branch 'fs#83-wrapper-compatibility' into fs#82-subtask-wrapper	2016-08-27 18:24:14 +02:00
golderweb	2be0a8903d	Adjust constructor for wrapper-script ... The new wrapper-script calls a standardized API We need to be conform with that Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=83 FS#83]	2016-08-27 17:02:51 +02:00
golderweb	0ceb2e6e83	Add methods to build gen to DiscussionParser ... With the new wrapper script the Bot gets a GenFactory and has to build a generator out of it by its own Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=83 FS#83]	2016-08-27 16:58:20 +02:00
golderweb	3540cc2a7d	Move functional sections to functions in main() ... To make main() function less complicated functional sections are moved to dedicated functions Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]	2016-08-27 15:40:09 +02:00
golderweb	460d2db183	Add Bot run with exception handling ... Errors, especially caused by missing run-method, need to be catched to provide information in Logfile. And also to get information wether bot run was successfull Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]	2016-08-27 15:40:09 +02:00
golderweb	156f117b18	Add Bot initiation with exception handling ... Bot initiation needs to catch errors by Bot to enforce at least a basic logging. And also to be sure Init was successfull before starting bot. Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]	2016-08-27 15:40:09 +02:00
golderweb	1679e2ad6a	Prepare environment for starting subtasks ... Before init and run bot we need to provide a environment for it, like parsed args Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]	2016-08-27 15:40:09 +02:00
golderweb	b88efb6bdd	Reflect stucture changes in Code ... Since bot class is moved to separate dir/file we need to do some changes to rebuild functionality Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]	2016-08-27 15:40:09 +02:00
golderweb	177a8f920f	Prepare new structure to use subtasks ... To have only one entry point for the bot we want to have a single file (red.py) which is calling the specfic task class from bots dir with a standardized call Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]	2016-08-27 15:40:09 +02:00
golderweb	0549cbd2c2	Merge branch 'fs#80-remove-deprecated-methods' into fs#70-refactoring	2016-08-25 22:43:57 +02:00
golderweb	78eda10562	Remove deprecated methods ... Deprecated functions which are not used anymore can be removed to make code more clearer and improve maintainability Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=80 FS#80]	2016-08-25 22:41:13 +02:00
golderweb	510771509b	Merge branch 'fs#79-mysql-table-prefix' into fs#70-refactoring	2016-08-25 13:15:23 +02:00
golderweb	71b99b5f58	Delay definition of db_table_prefix ... db_table_prefix should be defined at init of MysqlRed and not at import to have cmdline args already parsed Otherwise it uses default family Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=79 FS#79]	2016-08-25 13:06:32 +02:00
golderweb	77d1de4473	Add a tablename prefix depending on Site ... To be able to run the bot on different wikis the db tables should be named pywikibot.Site dependend and changed automatically Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=79 FS#79]	2016-08-24 23:53:10 +02:00
golderweb	cac04f344f	Merge branch 'fs#74-helpermodules-lib' into fs#70-refactoring	2016-08-24 23:09:33 +02:00
golderweb	e28acf88d1	Introduce new directory structure ... To clarify which is a bot and which are helper scripts Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=74 FS#74]	2016-08-24 22:41:41 +02:00
golderweb	af48888535	Merge branch 'fs#78-redfam-section-false-positives' into fs#70-refactoring	2016-08-24 20:05:04 +02:00
golderweb	ac54aea698	Use callback to detect redfam.section ... Detecting redfam-Sections via RegExp caused some false positives due to wrong formated things in wikisyntax. See Task Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=78 FS#78]	2016-08-24 20:02:48 +02:00
golderweb	2deb02fe47	Merge branch 'fs#77-errors-on-old-archives' into fs#70-refactoring	2016-08-24 19:59:53 +02:00
golderweb	1e4c8646bf	Reparse redfam-heading with mwparser ... See related ticked for detailed failure explanation Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=77 FS#77]	2016-08-24 19:57:25 +02:00
golderweb	fe2810f07c	Merge branch 'fs#76-redfam-without-dates' into fs#70-refactoring	2016-08-24 17:01:00 +02:00
golderweb	ab430e0085	Use month of reddisc as beginning if missing ... Construct a fictive but sensfull beginning if we cant detect one Needed since beginning is mandatory Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=76 FS#76]	2016-08-24 16:56:54 +02:00