wiki/jogobot-red
1
0
Fork 0
Code Issues 4 Pull Requests Releases Wiki Activity
109 Commits 2 Branches 14 Tags
77d1de44731b1aa51649a3e0a4a0550488dea63c
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
golderweb 77d1de4473 Add a tablename prefix depending on Site 
To be able to run the bot on different wikis the db tables should be
named pywikibot.Site dependend and changed automatically

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=79 FS#79]
 2016-08-24 23:53:10 +02:00
golderweb cac04f344f Merge branch 'fs#74-helpermodules-lib' into fs#70-refactoring 2016-08-24 23:09:33 +02:00
golderweb e28acf88d1 Introduce new directory structure 
To clarify which is a bot and which are helper scripts

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=74 FS#74]
 2016-08-24 22:41:41 +02:00
golderweb af48888535 Merge branch 'fs#78-redfam-section-false-positives' into fs#70-refactoring 2016-08-24 20:05:04 +02:00
golderweb ac54aea698 Use callback to detect redfam.section 
Detecting redfam-Sections via RegExp caused some false positives due to
wrong formated things in wikisyntax. See Task

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=78 FS#78]
 2016-08-24 20:02:48 +02:00
golderweb 2deb02fe47 Merge branch 'fs#77-errors-on-old-archives' into fs#70-refactoring 2016-08-24 19:59:53 +02:00
golderweb 1e4c8646bf Reparse redfam-heading with mwparser 
See related ticked for detailed failure explanation

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=77 FS#77]
 2016-08-24 19:57:25 +02:00
golderweb fe2810f07c Merge branch 'fs#76-redfam-without-dates' into fs#70-refactoring 2016-08-24 17:01:00 +02:00
golderweb ab430e0085 Use month of reddisc as beginning if missing 
Construct a fictive but sensfull beginning if we cant detect one
Needed since beginning is mandatory

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=76 FS#76]
 2016-08-24 16:56:54 +02:00
golderweb 95be313859 Pass reddisc pywikibot.page object to redfam 
To access page information like page title (eg. to get dates from it)
of the reddisc page

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=76 FS#76]
 2016-08-24 16:53:45 +02:00
golderweb 0bb0b2d957 Make sure var beginning is always defined 
To prevent unbound Errors caused by using undeclared variable beginning
if the redfam-section does not contain any timestamp

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=76 FS#76]
 2016-08-24 16:51:23 +02:00
golderweb db32c9e8f6 Merge branch 'fs#75-mysql-flush-error-false-reddisc' into fs#70-refactoring 2016-08-24 16:49:37 +02:00
golderweb bd2d221c48 Prevent flush from creating cursor without con 
MysqlRed.flush() tried to create a cursor in any case. If there was no
connection (because the subclasses haven't been instantiated an oursql
Error occured.
Instead, check before if there is a connection and otherwise raise an Error

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]
 2016-08-24 16:46:51 +02:00
golderweb ee8ebbc8bc Make sure only flush db if there are redfams 
To prevent from doing unnecessary stuff and trying to use not existing
db connection

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]
 2016-08-24 15:47:43 +02:00
golderweb dcc4851513 Check reddisc page titles against regex 
To prevent parsing Pages which have been categorized in configured cats
wrong or are given via cmd params
Parsing them results in unexpected behaviour

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]
 2016-08-24 15:27:42 +02:00
golderweb 0ea1b0039d Merge branch 'fs#72-rewrite-reddiscparser' into fs#70-refactoring 2016-08-24 11:23:38 +02:00
golderweb 2f878ee901 Correct filename in header 
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72]
 2016-08-24 11:20:28 +02:00
golderweb 17bfb32ded Building generators of config cats in sep Function 
Since the main()-Function was too complex the logic to build generators
out of categories provided in jogobot.conf was moved in a separate
function

[https://fs.golderweb.de/index.php?do=details&task_id=73 FS#73]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72]
 2016-08-24 11:17:00 +02:00
golderweb 6cb92c1da7 Rewrite parse control using pywikibot.bot classes 
To use the default pywikibot.classes making life easier at some point
Beeing standardconform with pywikibot in handling args

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72]
 2016-08-23 21:53:44 +02:00
golderweb a8605bcee6 Mv pages-parser.py to reddiscparser.py 
New, more meaningfull naming conventions, from redpage to reddisc (page)

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72]
 2016-08-23 21:50:22 +02:00
golderweb 5d31bdd7eb Jogobot submodule updated 2016-08-23 21:28:13 +02:00
golderweb 7f8ab1897e Merge branch 'fs#69-deprecated-decorators-param-str' into test-v3 2016-08-23 21:26:47 +02:00
golderweb 79dbde2413 Provide Replacement to @deprecated() as str 
Since use of pywikibot-master (or Python3.5 @see ticket below)
the @deprecator requires a str as param and no callable object like
done before

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=69 FS#69]
 2016-08-23 21:23:24 +02:00
golderweb 36a480a042 Merge branch 'fs#68-mysql-db-port' into test-v3 2016-08-23 21:16:09 +02:00
golderweb bd9dbdfa17 Make use of declared db_host_port 
The port to connect to MySQL-Server was previously always assumed as
the default one. So the library was incompatible to db's on nonstandard
ports

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=68 FS#68]
 2016-08-23 21:12:07 +02:00
golderweb 944bea488a Merge branch 'restucture-parsers' into test-v3 2016-03-05 15:02:31 +01:00
golderweb 7cac294181 Merge branch 'parser-script' into restucture-parsers 2016-03-05 15:01:09 +01:00
golderweb a24f208449 Add parse-pages.py Script 2016-03-04 20:39:41 +01:00
golderweb 0af7eb11d6 Move parsing of redfams from RedPageParser to RedFamParser.parser so RedPageParse won't do anything with redfams 
except for returning a generator of text-sections
 2016-03-03 20:41:14 +01:00
golderweb 7422307985 Rewrite RedPage.parse using mwparserfromhell to make it simpler 2016-03-03 17:37:46 +01:00
golderweb b81694c6d3 Rewrite heading_parser using mwparserfromhell to make it simpler 2016-03-03 17:30:39 +01:00
golderweb a2dfffc74b Let old date-extracting methods use dates_extract and mark them as deprecated 2016-03-03 17:23:44 +01:00
golderweb 163972c924 New method dates_extract which finds begining and ending at once 2016-03-03 17:20:57 +01:00
golderweb baf4ae2a07 Merge branch 'new-structure' into test-v3 2016-03-02 17:22:04 +01:00
golderweb 10f64199ab Remove relativ imports as we don't are in a package anymore 2016-03-02 17:19:11 +01:00
golderweb 24f1a7f516 Remove __init__.py as we won't use it as a package 2016-03-02 17:14:30 +01:00
golderweb 9113a40704 Merge branch 'warning-non-flushed-mysql-cache' into test-v3 2016-03-02 17:13:08 +01:00
golderweb f53a5b3745 Output a warning if there are update/insert querys cached when exit programm 2016-03-02 17:10:08 +01:00
golderweb 673e49c55a Merge branch 'jogobot' into test-v3 
Use new jogobot package
 2016-02-29 11:37:40 +01:00
golderweb 24adafeee7 Changes for new jogobot-module 2016-02-29 11:35:48 +01:00
golderweb b26f04db8c Use updated version of jogobot with ast.literal_eval parsed config entrys 2016-02-29 11:13:14 +01:00
golderweb f29dfd5003 Use new jogobot module 2016-02-28 18:00:46 +01:00
golderweb ef9c13324a Improve documentation of MysqlRed.flush() 2015-09-20 18:17:59 +02:00
golderweb e186f2f22b Use dictionary with page_id / fam_hash as key for cached_insert_data to prevent double entrys 2015-09-20 17:45:07 +02:00
golderweb 7d6cd8bb30 Strip leading and trailing whitespace in Links to prevent wrong fam_hashes (when receiving redfam from db) since MySQL drops it 2015-09-19 22:44:43 +02:00
golderweb 4e21b6696a Remove unnecessary whitespace from error messages 2015-09-19 20:51:52 +02:00
golderweb 6992f82f02 Start Implementing of RedFamWorker 2015-09-19 20:51:21 +02:00
golderweb dbcfe8f106 Add a generator to MysqlRedFam to retrieve redfams from db by status 2015-09-19 19:50:38 +02:00
golderweb 8059bb9992 Change behavior of MysqlRedFam to be able to get instance without knowen fam_hash 2015-09-19 19:49:20 +02:00
golderweb b5ca69077c Remove double appearence of heading parameter in repression of RedFam 2015-09-19 19:47:09 +02:00
golderweb 523d029fdc Fix bug causing db table cells containing empty strings 2015-09-19 19:45:34 +02:00
golderweb 4518efc504 Fix bug (Cached querys not executed) caused by class attribute protection level --> changed from private to protected 
Reformat MySQL querys to remove whitespace generated by indetation
 2015-09-18 18:08:13 +02:00
golderweb b1b37f9b9e Implement functions for flushing db query caches 2015-09-17 20:00:13 +02:00
golderweb 8dc7fe678d Fix bug caused by adding fam_hash to repr of RedFam class since it was not defined yet while output of warning caused by to many articles 2015-09-17 19:57:53 +02:00
golderweb 53f53ddb8b Implement cached querys in MysqlRedFam 2015-09-17 19:56:39 +02:00
golderweb 26f5912f88 Collect writing db querys for running once in MysqlRedPage 
Add classmethod to MysqlRed for executing collected querys
 2015-09-16 21:02:02 +02:00
golderweb 1dea5d7e84 NOT WORKING Cache SQL querys to reduce amount of querys 2015-09-16 18:31:54 +02:00
golderweb b514eb5c42 Move configuration to jogobot module 
Use custom Error classes
 2015-09-15 21:21:05 +02:00
golderweb db5bb7401e Update RedFam class to rebuild the whole structure of RedFamPaser generated object 
Move fam_hash() method from RedFamParser to RedFam
Define custom Error classes
 2015-09-15 21:19:07 +02:00
golderweb be0041804a Clean up 2015-09-13 13:19:22 +02:00
golderweb af6d483280 Use a additional module to store custom config ... 2015-09-13 13:14:37 +02:00
golderweb 7c9061d37f Use redundances as python package 2015-09-13 13:13:28 +02:00
golderweb 74b2dc727c Clean up PEP8 styleguide compatibility with flake8 2015-09-13 11:53:47 +02:00
golderweb cafe08dd7f Added license GPLv3 + Reworked code 
Better compatibility with https://www.mediawiki.org/wiki/Manual:Pywikibot/Development_guidelines
 2015-09-13 01:22:27 +02:00
golderweb ad100e96eb Remove timezone from timestamp patterns since it is not stored and causes errors on labs 2015-09-12 22:35:12 +02:00
golderweb c78480a75d Use mysql config from pywikibot user-config.py 2015-09-12 22:33:44 +02:00
golderweb 6e0d3200c2 Change visibility of attributes changed and parsed in RED_PAGE to protected 2015-09-12 14:24:45 +02:00
golderweb f50f4b664f Add additional possibilities to detect wether a red_page is an archive or not 2015-09-12 13:48:01 +02:00
golderweb e70655f679 Catch sections with more then 8 articles 2015-09-12 12:42:23 +02:00
golderweb 14e865c6e6 We need to save the red_fam section heading in database for generating section links since format is not consistent 2015-09-12 00:50:04 +02:00
golderweb a97e2cea5a Since some timestamps are broken we need to reconstruct them by regex match groups 
Prevents ValueErrors of datetime.strptime in most cases
 2015-09-11 23:47:35 +02:00
golderweb 6e64d8448e Prevent RED_PAGE from resetting status if parser is not fired 2015-09-11 22:29:16 +02:00
golderweb deaa1d855b Fix bug which prevents detecting of red_fam sectionheading when there is whitespace after closing heading tags (===) 2015-09-11 22:27:25 +02:00
golderweb 90da231712 Move the code for instanciating of RED_FAM_PARSER to the end of loop 2015-09-11 17:05:18 +02:00
golderweb 529ed09702 Use pywikibot.output instead of print 2015-09-11 16:42:12 +02:00
golderweb 1892c6d015 Fix wrong intendet line (RED_FAM_PARSER only was instantiated when ending-datetime workaround fired) 2015-09-11 16:41:04 +02:00
golderweb 99f050acd3 Add workaround to detect ending datetime if there is no done notice 2015-09-11 16:22:19 +02:00
golderweb 8b7dc5dbf2 Improve sectionheading regex to reduce false positives, require at least two links 2015-09-11 16:18:46 +02:00
golderweb 5e39ea0b06 Correct use of pywikibot page.title() 2015-09-11 14:12:26 +02:00
golderweb f8dacb53e1 Add param to RED_PAGE for predefining page status archived when using cat of archives 2015-09-11 14:10:33 +02:00
golderweb ebf7a8fe0b Add alternative syntax for done_notice 2015-09-11 13:56:16 +02:00
golderweb 47238f0f96 On archived red_fams do not delete possibly existing ending 2015-09-11 13:37:58 +02:00
golderweb 907ccad63a Detect if red_fam is archived but has no detectable ending, set status 2 anyway 2015-09-11 13:12:29 +02:00
golderweb 31ea512c41 Prevent overwriting of ending with None by only calling RED_FAM_PARSER if there was no ending before 2015-09-11 13:01:04 +02:00
golderweb 4f7cfc65cc Catch ValueErrors caused by missing points after month abreviation in RED_PAGE_PARSER.__timestamp 2015-09-10 23:09:35 +02:00
golderweb a26b92082b Implement parsing process in RED_PAGE.parse() 2015-09-10 23:08:51 +02:00
golderweb 984c269aa4 Implement classmethods for detecting sectionheading, beginning and ending in class RED_FAM_PARSER 2015-09-10 23:06:54 +02:00
golderweb 0f4ce7c4c9 Add workaround to get mysql.data dictionary while adding new pages/fams 2015-09-10 23:02:38 +02:00
golderweb 7535172d30 Update mysql db if anything of re_fam changes 2015-09-10 11:57:35 +02:00
golderweb 9e865e1c2f Some clean ups 2015-09-10 11:57:10 +02:00
golderweb 79e232d52b Add gitignore file 2015-09-10 11:46:05 +02:00
golderweb ea92415b37 Add dummy -method for parsing to RED_PAGE 2015-09-10 11:24:25 +02:00
golderweb 30e1672557 Implement methods for parsing red_fams 
Add Interfaces for RED_FAM
 2015-09-10 11:23:15 +02:00
golderweb 6d1ed33699 Add methods for MySQL actions while parsing RED_FAMs 2015-09-10 11:20:20 +02:00
golderweb 45df354315 Prevent descendant classes of MYSQL_RED from deleting connection to db 2015-09-10 11:18:22 +02:00
golderweb 73b5d87e8d Implemment handling of changed page meta data (rev_id, page_title, status) 2015-09-07 22:51:07 +02:00
golderweb cef90e4fac Handle building of MYSQL_RED_PAGE objects 2015-09-07 19:04:16 +02:00
golderweb c800a13f0c Change behavior of MYSQL_RED(_PAGE), handle save requested data in object for simple update handling 2015-09-07 19:00:04 +02:00
golderweb 0beb97d67d Use qmark parametrization style for mysql query 2015-09-07 16:21:57 +02:00
golderweb 773557a591 Use DictCursor ouf oursql to get better readability in MYSQL_RED_PAGE 2015-09-06 16:02:30 +02:00