100 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
golderweb	4e4d5005fd	Merge branch 'fs#89-article-status' into fs#88-mark-pages-bot	2016-11-05 19:39:30 +01:00
golderweb	65fb2ecb28	Generate Fam status based on article status ... Some article states should be reflected in the RedFam status Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]	2016-11-05 19:27:56 +01:00
golderweb	d55c81c97b	Set article status when worked on talkpage ... To detect whole redfam status after run over all articles Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]	2016-08-30 18:05:51 +02:00
golderweb	870ed4bf25	Update redfam.article_generator use article status ... To be able to filter articles by status of that article Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]	2016-08-30 17:47:02 +02:00
golderweb	e13320820c	Add API to manage status per article ... To be able to track changes to articles to update redfam status Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]	2016-08-30 17:45:18 +02:00
golderweb	4ae562590e	Merge branch 'fs#94-data-structure' into fs#88-mark-pages-bot	2016-08-30 17:37:31 +02:00
golderweb	6149dcdb8b	Apply changes to data structure ... See related ticket Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=94 FS#94]	2016-08-30 14:28:28 +02:00
golderweb	f021f2ea60	Merge branch 'fs#93-update-talkpage-template' into fs#88-mark-pages-bot	2016-08-30 12:08:57 +02:00
golderweb	8c56125a7b	Update talkpage notice template ... Exact date is not necessary and end could be ommited if of the same month Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=93 FS#93]	2016-08-30 12:07:11 +02:00
golderweb	c19f642d11	Merge branch 'fs#92-mark-done-edit-summary' into fs#88-mark-pages-bot	2016-08-30 11:51:07 +02:00
golderweb	20b811bc2a	Make sure edit summary starts with bot ... Due to bot policy all edit summaries of bot edits have to start with "Bot:" Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=92 FS#92]	2016-08-30 11:48:07 +02:00
golderweb	59d4d23c83	Set edit summary for each edit ... Each edit of bot needs a edit summary Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=92 FS#92]	2016-08-30 11:33:54 +02:00
golderweb	2b93e4cf16	Check if notice is present before add ... To prevent duplications we need to check wether notice is already present on talkpage Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]	2016-08-28 21:39:54 +02:00
golderweb	9beca7f6c9	Implement method to add notice to disk page ... Adds the generated notice to the talkpage and starts the saving of the page Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]	2016-08-28 21:33:36 +02:00
golderweb	c4d8a95672	Implement build_generator-method ... Build_generator will add the redfam_talkpages_generator to the genFactory, build a generator of the genFactory and sets self.gen which is used as generator for run() Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]	2016-08-28 18:13:27 +02:00
golderweb	da4f9b5d6b	Add wrapper-generator to redfam.article_generator ... We need a wrapper around redfam.article_generator to pass it to pagegenerators.PageWithTalkPageGenerator and to add a reference to related redfam to each pywikibot.page-object before yielding it Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]	2016-08-28 18:09:04 +02:00
golderweb	ecc78bef96	Import needed modules and add redfams-generator ... We will need a couple of modules to build the needed generator Also we will need a generator with redfams to work with Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]	2016-08-28 18:01:02 +02:00
golderweb	efa919ff27	Add new bot with basic structure ... We need a bot to work on pages which are subjects of redfams and on the belonging talk page Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]	2016-08-28 16:39:32 +02:00
golderweb	72c6165de8	Merge branch 'fs#87-redfam-article-generator' into fs#25-mark-done	2016-08-28 16:20:53 +02:00
golderweb	c0b18f88e5	Add filter options to redfam.article_generator ... To give the posibility to filter not existing pages or redirect pages or vice versa. Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=87 FS#87]	2016-08-28 15:06:17 +02:00
golderweb	e5989305a4	Add a generator to redfam yielding article pages ... To work on articles of a redfam a generator which yields belonging articles is necessary Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=87 FS#87]	2016-08-28 13:20:13 +02:00
golderweb	8ce6f03641	Merge branch 'fs#29-generate-articledisc-notice' into fs#25-mark-done	2016-08-27 19:50:18 +02:00
golderweb	6717fa4fba	Add method to generate notice for article discpage ... We need a method to generate the template to add to article discpages Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=29 FS#29]	2016-08-27 19:49:50 +02:00
golderweb	8acba7d0f9	Merge branch 'fs#81-get-reddisc-link' into fs#25-mark-done	2016-08-27 19:48:24 +02:00
golderweb	3723aba578	Add a method to get link to related reddisc ... To generate notices or other stuff it is necessary to add links to the related reddisc. This method returns a wikilink to text the redfam's reddisc. Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=81 FS#81] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=81 FS#81]	2016-08-27 19:48:03 +02:00
golderweb	9d3bc74c80	Merge branch 'fs#26-done-redfam-gen' into fs#25-mark-done	2016-08-27 19:46:41 +02:00
golderweb	b36dc250d2	Request information about reddisc page for redfams ... To generate links to related reddisc it is necessary to get at least the Title of the related reddisc page. As saving the same data in db is worse, we retreive it via join from red_pages table Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]	2016-08-27 19:44:33 +02:00
golderweb	4055dc52d8	Make it possible to get a RedPage-Object by pageid ... When working on redfams it is necessary to have information about redpage Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]	2016-08-27 19:44:33 +02:00
golderweb	594130c8a6	Restore changes from 45df35431 ... Documented to prevent deleting again Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]	2016-08-27 19:44:33 +02:00
golderweb	b271a0b0b1	Add generator wrapper to fetch RedFams by status and ending ... Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]	2016-08-27 19:44:33 +02:00
golderweb	ad088126e7	Define method to update Status after Working with RedFam ... Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]	2016-08-27 19:44:33 +02:00
golderweb	151c22a735	Add fetched mysql_data to _mysql-Object of parent class for using change-method to update db ... Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]	2016-08-27 19:44:33 +02:00
golderweb	a97d8c722e	Move handling of mysql-Connection from RedFamParser and RedFamWorker to RedFam-Class and make it protected instead of private ... Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]	2016-08-27 19:44:33 +02:00
golderweb	58dfd8c86a	For RedFamilies not fetched individually we need to provide the fam hash as index ... Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]	2016-08-27 19:44:33 +02:00
golderweb	9481116777	Add new generator-method to fetch RedFams by Status and Ending ... Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]	2016-08-27 19:44:33 +02:00
golderweb	eaa7596a8f	Merge branch 'fs#70-refactoring' into test-v3	2016-08-27 19:40:00 +02:00
golderweb	449d83d7b5	Merge branch 'fs#82-subtask-wrapper' into fs#70-refactoring	2016-08-27 19:19:40 +02:00
golderweb	4ac9b305f5	Merge branch 'fs#85-move-start-api-to-jogobot' into fs#82-subtask-wrapper	2016-08-27 19:17:10 +02:00
golderweb	604b7bd8b7	Now use Bot-Start API from jogobot framework ... API was moved to jogobot to share with other tasks Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85]	2016-08-27 19:04:13 +02:00
golderweb	d0fa15d0ed	Update jogobot module to get standart Start-API ... [FS#84] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85]	2016-08-27 18:50:36 +02:00
golderweb	71e41bfed3	Merge branch 'fs#83-wrapper-compatibility' into fs#82-subtask-wrapper	2016-08-27 18:24:14 +02:00
golderweb	2be0a8903d	Adjust constructor for wrapper-script ... The new wrapper-script calls a standardized API We need to be conform with that Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=83 FS#83]	2016-08-27 17:02:51 +02:00
golderweb	0ceb2e6e83	Add methods to build gen to DiscussionParser ... With the new wrapper script the Bot gets a GenFactory and has to build a generator out of it by its own Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=83 FS#83]	2016-08-27 16:58:20 +02:00
golderweb	3540cc2a7d	Move functional sections to functions in main() ... To make main() function less complicated functional sections are moved to dedicated functions Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]	2016-08-27 15:40:09 +02:00
golderweb	460d2db183	Add Bot run with exception handling ... Errors, especially caused by missing run-method, need to be catched to provide information in Logfile. And also to get information wether bot run was successfull Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]	2016-08-27 15:40:09 +02:00
golderweb	156f117b18	Add Bot initiation with exception handling ... Bot initiation needs to catch errors by Bot to enforce at least a basic logging. And also to be sure Init was successfull before starting bot. Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]	2016-08-27 15:40:09 +02:00
golderweb	1679e2ad6a	Prepare environment for starting subtasks ... Before init and run bot we need to provide a environment for it, like parsed args Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]	2016-08-27 15:40:09 +02:00
golderweb	b88efb6bdd	Reflect stucture changes in Code ... Since bot class is moved to separate dir/file we need to do some changes to rebuild functionality Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]	2016-08-27 15:40:09 +02:00
golderweb	177a8f920f	Prepare new structure to use subtasks ... To have only one entry point for the bot we want to have a single file (red.py) which is calling the specfic task class from bots dir with a standardized call Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]	2016-08-27 15:40:09 +02:00
golderweb	0549cbd2c2	Merge branch 'fs#80-remove-deprecated-methods' into fs#70-refactoring	2016-08-25 22:43:57 +02:00
golderweb	78eda10562	Remove deprecated methods ... Deprecated functions which are not used anymore can be removed to make code more clearer and improve maintainability Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=80 FS#80]	2016-08-25 22:41:13 +02:00
golderweb	510771509b	Merge branch 'fs#79-mysql-table-prefix' into fs#70-refactoring	2016-08-25 13:15:23 +02:00
golderweb	71b99b5f58	Delay definition of db_table_prefix ... db_table_prefix should be defined at init of MysqlRed and not at import to have cmdline args already parsed Otherwise it uses default family Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=79 FS#79]	2016-08-25 13:06:32 +02:00
golderweb	77d1de4473	Add a tablename prefix depending on Site ... To be able to run the bot on different wikis the db tables should be named pywikibot.Site dependend and changed automatically Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=79 FS#79]	2016-08-24 23:53:10 +02:00
golderweb	cac04f344f	Merge branch 'fs#74-helpermodules-lib' into fs#70-refactoring	2016-08-24 23:09:33 +02:00
golderweb	e28acf88d1	Introduce new directory structure ... To clarify which is a bot and which are helper scripts Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=74 FS#74]	2016-08-24 22:41:41 +02:00
golderweb	af48888535	Merge branch 'fs#78-redfam-section-false-positives' into fs#70-refactoring	2016-08-24 20:05:04 +02:00
golderweb	ac54aea698	Use callback to detect redfam.section ... Detecting redfam-Sections via RegExp caused some false positives due to wrong formated things in wikisyntax. See Task Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=78 FS#78]	2016-08-24 20:02:48 +02:00
golderweb	2deb02fe47	Merge branch 'fs#77-errors-on-old-archives' into fs#70-refactoring	2016-08-24 19:59:53 +02:00
golderweb	1e4c8646bf	Reparse redfam-heading with mwparser ... See related ticked for detailed failure explanation Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=77 FS#77]	2016-08-24 19:57:25 +02:00
golderweb	fe2810f07c	Merge branch 'fs#76-redfam-without-dates' into fs#70-refactoring	2016-08-24 17:01:00 +02:00
golderweb	ab430e0085	Use month of reddisc as beginning if missing ... Construct a fictive but sensfull beginning if we cant detect one Needed since beginning is mandatory Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=76 FS#76]	2016-08-24 16:56:54 +02:00
golderweb	95be313859	Pass reddisc pywikibot.page object to redfam ... To access page information like page title (eg. to get dates from it) of the reddisc page Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=76 FS#76]	2016-08-24 16:53:45 +02:00
golderweb	0bb0b2d957	Make sure var beginning is always defined ... To prevent unbound Errors caused by using undeclared variable beginning if the redfam-section does not contain any timestamp Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=76 FS#76]	2016-08-24 16:51:23 +02:00
golderweb	db32c9e8f6	Merge branch 'fs#75-mysql-flush-error-false-reddisc' into fs#70-refactoring	2016-08-24 16:49:37 +02:00
golderweb	bd2d221c48	Prevent flush from creating cursor without con ... MysqlRed.flush() tried to create a cursor in any case. If there was no connection (because the subclasses haven't been instantiated an oursql Error occured. Instead, check before if there is a connection and otherwise raise an Error Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]	2016-08-24 16:46:51 +02:00
golderweb	ee8ebbc8bc	Make sure only flush db if there are redfams ... To prevent from doing unnecessary stuff and trying to use not existing db connection Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]	2016-08-24 15:47:43 +02:00
golderweb	dcc4851513	Check reddisc page titles against regex ... To prevent parsing Pages which have been categorized in configured cats wrong or are given via cmd params Parsing them results in unexpected behaviour Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]	2016-08-24 15:27:42 +02:00
golderweb	0ea1b0039d	Merge branch 'fs#72-rewrite-reddiscparser' into fs#70-refactoring	2016-08-24 11:23:38 +02:00
golderweb	2f878ee901	Correct filename in header ... Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72]	2016-08-24 11:20:28 +02:00
golderweb	17bfb32ded	Building generators of config cats in sep Function ... Since the main()-Function was too complex the logic to build generators out of categories provided in jogobot.conf was moved in a separate function [https://fs.golderweb.de/index.php?do=details&task_id=73 FS#73] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72] Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72]	2016-08-24 11:17:00 +02:00
golderweb	6cb92c1da7	Rewrite parse control using pywikibot.bot classes ... To use the default pywikibot.classes making life easier at some point Beeing standardconform with pywikibot in handling args Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72]	2016-08-23 21:53:44 +02:00
golderweb	a8605bcee6	Mv pages-parser.py to reddiscparser.py ... New, more meaningfull naming conventions, from redpage to reddisc (page) Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72]	2016-08-23 21:50:22 +02:00
golderweb	5d31bdd7eb	Jogobot submodule updated	2016-08-23 21:28:13 +02:00
golderweb	7f8ab1897e	Merge branch 'fs#69-deprecated-decorators-param-str' into test-v3	2016-08-23 21:26:47 +02:00
golderweb	79dbde2413	Provide Replacement to @deprecated() as str ... Since use of pywikibot-master (or Python3.5 @see ticket below) the @deprecator requires a str as param and no callable object like done before Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=69 FS#69]	2016-08-23 21:23:24 +02:00
golderweb	36a480a042	Merge branch 'fs#68-mysql-db-port' into test-v3	2016-08-23 21:16:09 +02:00
golderweb	bd9dbdfa17	Make use of declared db_host_port ... The port to connect to MySQL-Server was previously always assumed as the default one. So the library was incompatible to db's on nonstandard ports Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=68 FS#68]	2016-08-23 21:12:07 +02:00
golderweb	944bea488a	Merge branch 'restucture-parsers' into test-v3	2016-03-05 15:02:31 +01:00
golderweb	7cac294181	Merge branch 'parser-script' into restucture-parsers	2016-03-05 15:01:09 +01:00
golderweb	a24f208449	Add parse-pages.py Script	2016-03-04 20:39:41 +01:00
golderweb	0af7eb11d6	Move parsing of redfams from RedPageParser to RedFamParser.parser so RedPageParse won't do anything with redfams ... except for returning a generator of text-sections	2016-03-03 20:41:14 +01:00
golderweb	7422307985	Rewrite RedPage.parse using mwparserfromhell to make it simpler	2016-03-03 17:37:46 +01:00
golderweb	b81694c6d3	Rewrite heading_parser using mwparserfromhell to make it simpler	2016-03-03 17:30:39 +01:00
golderweb	a2dfffc74b	Let old date-extracting methods use dates_extract and mark them as deprecated	2016-03-03 17:23:44 +01:00
golderweb	163972c924	New method dates_extract which finds begining and ending at once	2016-03-03 17:20:57 +01:00
golderweb	baf4ae2a07	Merge branch 'new-structure' into test-v3	2016-03-02 17:22:04 +01:00
golderweb	10f64199ab	Remove relativ imports as we don't are in a package anymore	2016-03-02 17:19:11 +01:00
golderweb	24f1a7f516	Remove __init__.py as we won't use it as a package	2016-03-02 17:14:30 +01:00
golderweb	9113a40704	Merge branch 'warning-non-flushed-mysql-cache' into test-v3	2016-03-02 17:13:08 +01:00
golderweb	f53a5b3745	Output a warning if there are update/insert querys cached when exit programm	2016-03-02 17:10:08 +01:00
golderweb	673e49c55a	Merge branch 'jogobot' into test-v3 ... Use new jogobot package	2016-02-29 11:37:40 +01:00
golderweb	24adafeee7	Changes for new jogobot-module	2016-02-29 11:35:48 +01:00
golderweb	b26f04db8c	Use updated version of jogobot with ast.literal_eval parsed config entrys	2016-02-29 11:13:14 +01:00
golderweb	f29dfd5003	Use new jogobot module	2016-02-28 18:00:46 +01:00
golderweb	ef9c13324a	Improve documentation of MysqlRed.flush()	2015-09-20 18:17:59 +02:00
golderweb	e186f2f22b	Use dictionary with page_id / fam_hash as key for cached_insert_data to prevent double entrys	2015-09-20 17:45:07 +02:00
golderweb	7d6cd8bb30	Strip leading and trailing whitespace in Links to prevent wrong fam_hashes (when receiving redfam from db) since MySQL drops it	2015-09-19 22:44:43 +02:00
golderweb	4e21b6696a	Remove unnecessary whitespace from error messages	2015-09-19 20:51:52 +02:00
golderweb	6992f82f02	Start Implementing of RedFamWorker	2015-09-19 20:51:21 +02:00