wiki/jogobot-red
1
0
Fork 0
Code Issues 4 Pull Requests Releases Wiki Activity
128 Commits 2 Branches 14 Tags
9481116777ba73f84f9b47d5425516467f637764
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
golderweb 9481116777 Add new generator-method to fetch RedFams by Status and Ending 
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]
 2016-08-27 19:44:33 +02:00
golderweb eaa7596a8f Merge branch 'fs#70-refactoring' into test-v3 2016-08-27 19:40:00 +02:00
golderweb 449d83d7b5 Merge branch 'fs#82-subtask-wrapper' into fs#70-refactoring 2016-08-27 19:19:40 +02:00
golderweb 4ac9b305f5 Merge branch 'fs#85-move-start-api-to-jogobot' into fs#82-subtask-wrapper 2016-08-27 19:17:10 +02:00
golderweb 604b7bd8b7 Now use Bot-Start API from jogobot framework 
API was moved to jogobot to share with other tasks

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85]
 2016-08-27 19:04:13 +02:00
golderweb d0fa15d0ed Update jogobot module to get standart Start-API 
[FS#84]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=85 FS#85]
 2016-08-27 18:50:36 +02:00
golderweb 71e41bfed3 Merge branch 'fs#83-wrapper-compatibility' into fs#82-subtask-wrapper 2016-08-27 18:24:14 +02:00
golderweb 2be0a8903d Adjust constructor for wrapper-script 
The new wrapper-script calls a standardized API
We need to be conform with that

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=83 FS#83]
 2016-08-27 17:02:51 +02:00
golderweb 0ceb2e6e83 Add methods to build gen to DiscussionParser 
With the new wrapper script the Bot gets a GenFactory and has to build
a generator out of it by its own

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=83 FS#83]
 2016-08-27 16:58:20 +02:00
golderweb 3540cc2a7d Move functional sections to functions in main() 
To make main() function less complicated functional sections are moved
to dedicated functions

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]
 2016-08-27 15:40:09 +02:00
golderweb 460d2db183 Add Bot run with exception handling 
Errors, especially caused by missing run-method, need to be catched to
provide information in Logfile.
And also to get information wether bot run was successfull

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]
 2016-08-27 15:40:09 +02:00
golderweb 156f117b18 Add Bot initiation with exception handling 
Bot initiation needs to catch errors by Bot to enforce at least a basic
logging.
And also to be sure Init was successfull before starting bot.

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]
 2016-08-27 15:40:09 +02:00
golderweb 1679e2ad6a Prepare environment for starting subtasks 
Before init and run bot we need to provide a environment for it,
like parsed args

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]
 2016-08-27 15:40:09 +02:00
golderweb b88efb6bdd Reflect stucture changes in Code 
Since bot class is moved to separate dir/file we need to do some changes
to rebuild functionality

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]
 2016-08-27 15:40:09 +02:00
golderweb 177a8f920f Prepare new structure to use subtasks 
To have only one entry point for the bot we want to have a single file
(red.py) which is calling the specfic task class from bots dir with a
standardized call

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=82 FS#82]
 2016-08-27 15:40:09 +02:00
golderweb 0549cbd2c2 Merge branch 'fs#80-remove-deprecated-methods' into fs#70-refactoring 2016-08-25 22:43:57 +02:00
golderweb 78eda10562 Remove deprecated methods 
Deprecated functions which are not used anymore can be removed to make
code more clearer and improve maintainability

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=80 FS#80]
 2016-08-25 22:41:13 +02:00
golderweb 510771509b Merge branch 'fs#79-mysql-table-prefix' into fs#70-refactoring 2016-08-25 13:15:23 +02:00
golderweb 71b99b5f58 Delay definition of db_table_prefix 
db_table_prefix should be defined at init of MysqlRed and not at import
to have cmdline args already parsed
Otherwise it uses default family

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=79 FS#79]
 2016-08-25 13:06:32 +02:00
golderweb 77d1de4473 Add a tablename prefix depending on Site 
To be able to run the bot on different wikis the db tables should be
named pywikibot.Site dependend and changed automatically

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=79 FS#79]
 2016-08-24 23:53:10 +02:00
golderweb cac04f344f Merge branch 'fs#74-helpermodules-lib' into fs#70-refactoring 2016-08-24 23:09:33 +02:00
golderweb e28acf88d1 Introduce new directory structure 
To clarify which is a bot and which are helper scripts

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=74 FS#74]
 2016-08-24 22:41:41 +02:00
golderweb af48888535 Merge branch 'fs#78-redfam-section-false-positives' into fs#70-refactoring 2016-08-24 20:05:04 +02:00
golderweb ac54aea698 Use callback to detect redfam.section 
Detecting redfam-Sections via RegExp caused some false positives due to
wrong formated things in wikisyntax. See Task

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=78 FS#78]
 2016-08-24 20:02:48 +02:00
golderweb 2deb02fe47 Merge branch 'fs#77-errors-on-old-archives' into fs#70-refactoring 2016-08-24 19:59:53 +02:00
golderweb 1e4c8646bf Reparse redfam-heading with mwparser 
See related ticked for detailed failure explanation

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=77 FS#77]
 2016-08-24 19:57:25 +02:00
golderweb fe2810f07c Merge branch 'fs#76-redfam-without-dates' into fs#70-refactoring 2016-08-24 17:01:00 +02:00
golderweb ab430e0085 Use month of reddisc as beginning if missing 
Construct a fictive but sensfull beginning if we cant detect one
Needed since beginning is mandatory

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=76 FS#76]
 2016-08-24 16:56:54 +02:00
golderweb 95be313859 Pass reddisc pywikibot.page object to redfam 
To access page information like page title (eg. to get dates from it)
of the reddisc page

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=76 FS#76]
 2016-08-24 16:53:45 +02:00
golderweb 0bb0b2d957 Make sure var beginning is always defined 
To prevent unbound Errors caused by using undeclared variable beginning
if the redfam-section does not contain any timestamp

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=76 FS#76]
 2016-08-24 16:51:23 +02:00
golderweb db32c9e8f6 Merge branch 'fs#75-mysql-flush-error-false-reddisc' into fs#70-refactoring 2016-08-24 16:49:37 +02:00
golderweb bd2d221c48 Prevent flush from creating cursor without con 
MysqlRed.flush() tried to create a cursor in any case. If there was no
connection (because the subclasses haven't been instantiated an oursql
Error occured.
Instead, check before if there is a connection and otherwise raise an Error

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]
 2016-08-24 16:46:51 +02:00
golderweb ee8ebbc8bc Make sure only flush db if there are redfams 
To prevent from doing unnecessary stuff and trying to use not existing
db connection

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]
 2016-08-24 15:47:43 +02:00
golderweb dcc4851513 Check reddisc page titles against regex 
To prevent parsing Pages which have been categorized in configured cats
wrong or are given via cmd params
Parsing them results in unexpected behaviour

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]
 2016-08-24 15:27:42 +02:00
golderweb 0ea1b0039d Merge branch 'fs#72-rewrite-reddiscparser' into fs#70-refactoring 2016-08-24 11:23:38 +02:00
golderweb 2f878ee901 Correct filename in header 
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72]
 2016-08-24 11:20:28 +02:00
golderweb 17bfb32ded Building generators of config cats in sep Function 
Since the main()-Function was too complex the logic to build generators
out of categories provided in jogobot.conf was moved in a separate
function

[https://fs.golderweb.de/index.php?do=details&task_id=73 FS#73]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72]

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72]
 2016-08-24 11:17:00 +02:00
golderweb 6cb92c1da7 Rewrite parse control using pywikibot.bot classes 
To use the default pywikibot.classes making life easier at some point
Beeing standardconform with pywikibot in handling args

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72]
 2016-08-23 21:53:44 +02:00
golderweb a8605bcee6 Mv pages-parser.py to reddiscparser.py 
New, more meaningfull naming conventions, from redpage to reddisc (page)

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72]
 2016-08-23 21:50:22 +02:00
golderweb 5d31bdd7eb Jogobot submodule updated 2016-08-23 21:28:13 +02:00
golderweb 7f8ab1897e Merge branch 'fs#69-deprecated-decorators-param-str' into test-v3 2016-08-23 21:26:47 +02:00
golderweb 79dbde2413 Provide Replacement to @deprecated() as str 
Since use of pywikibot-master (or Python3.5 @see ticket below)
the @deprecator requires a str as param and no callable object like
done before

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=69 FS#69]
 2016-08-23 21:23:24 +02:00
golderweb 36a480a042 Merge branch 'fs#68-mysql-db-port' into test-v3 2016-08-23 21:16:09 +02:00
golderweb bd9dbdfa17 Make use of declared db_host_port 
The port to connect to MySQL-Server was previously always assumed as
the default one. So the library was incompatible to db's on nonstandard
ports

Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=68 FS#68]
 2016-08-23 21:12:07 +02:00
golderweb 944bea488a Merge branch 'restucture-parsers' into test-v3 2016-03-05 15:02:31 +01:00
golderweb 7cac294181 Merge branch 'parser-script' into restucture-parsers 2016-03-05 15:01:09 +01:00
golderweb a24f208449 Add parse-pages.py Script 2016-03-04 20:39:41 +01:00
golderweb 0af7eb11d6 Move parsing of redfams from RedPageParser to RedFamParser.parser so RedPageParse won't do anything with redfams 
except for returning a generator of text-sections
 2016-03-03 20:41:14 +01:00
golderweb 7422307985 Rewrite RedPage.parse using mwparserfromhell to make it simpler 2016-03-03 17:37:46 +01:00
golderweb b81694c6d3 Rewrite heading_parser using mwparserfromhell to make it simpler 2016-03-03 17:30:39 +01:00
golderweb a2dfffc74b Let old date-extracting methods use dates_extract and mark them as deprecated 2016-03-03 17:23:44 +01:00
golderweb 163972c924 New method dates_extract which finds begining and ending at once 2016-03-03 17:20:57 +01:00
golderweb baf4ae2a07 Merge branch 'new-structure' into test-v3 2016-03-02 17:22:04 +01:00
golderweb 10f64199ab Remove relativ imports as we don't are in a package anymore 2016-03-02 17:19:11 +01:00
golderweb 24f1a7f516 Remove __init__.py as we won't use it as a package 2016-03-02 17:14:30 +01:00
golderweb 9113a40704 Merge branch 'warning-non-flushed-mysql-cache' into test-v3 2016-03-02 17:13:08 +01:00
golderweb f53a5b3745 Output a warning if there are update/insert querys cached when exit programm 2016-03-02 17:10:08 +01:00
golderweb 673e49c55a Merge branch 'jogobot' into test-v3 
Use new jogobot package
 2016-02-29 11:37:40 +01:00
golderweb 24adafeee7 Changes for new jogobot-module 2016-02-29 11:35:48 +01:00
golderweb b26f04db8c Use updated version of jogobot with ast.literal_eval parsed config entrys 2016-02-29 11:13:14 +01:00
golderweb f29dfd5003 Use new jogobot module 2016-02-28 18:00:46 +01:00
golderweb ef9c13324a Improve documentation of MysqlRed.flush() 2015-09-20 18:17:59 +02:00
golderweb e186f2f22b Use dictionary with page_id / fam_hash as key for cached_insert_data to prevent double entrys 2015-09-20 17:45:07 +02:00
golderweb 7d6cd8bb30 Strip leading and trailing whitespace in Links to prevent wrong fam_hashes (when receiving redfam from db) since MySQL drops it 2015-09-19 22:44:43 +02:00
golderweb 4e21b6696a Remove unnecessary whitespace from error messages 2015-09-19 20:51:52 +02:00
golderweb 6992f82f02 Start Implementing of RedFamWorker 2015-09-19 20:51:21 +02:00
golderweb dbcfe8f106 Add a generator to MysqlRedFam to retrieve redfams from db by status 2015-09-19 19:50:38 +02:00
golderweb 8059bb9992 Change behavior of MysqlRedFam to be able to get instance without knowen fam_hash 2015-09-19 19:49:20 +02:00
golderweb b5ca69077c Remove double appearence of heading parameter in repression of RedFam 2015-09-19 19:47:09 +02:00
golderweb 523d029fdc Fix bug causing db table cells containing empty strings 2015-09-19 19:45:34 +02:00
golderweb 4518efc504 Fix bug (Cached querys not executed) caused by class attribute protection level --> changed from private to protected 
Reformat MySQL querys to remove whitespace generated by indetation
 2015-09-18 18:08:13 +02:00
golderweb b1b37f9b9e Implement functions for flushing db query caches 2015-09-17 20:00:13 +02:00
golderweb 8dc7fe678d Fix bug caused by adding fam_hash to repr of RedFam class since it was not defined yet while output of warning caused by to many articles 2015-09-17 19:57:53 +02:00
golderweb 53f53ddb8b Implement cached querys in MysqlRedFam 2015-09-17 19:56:39 +02:00
golderweb 26f5912f88 Collect writing db querys for running once in MysqlRedPage 
Add classmethod to MysqlRed for executing collected querys
 2015-09-16 21:02:02 +02:00
golderweb 1dea5d7e84 NOT WORKING Cache SQL querys to reduce amount of querys 2015-09-16 18:31:54 +02:00
golderweb b514eb5c42 Move configuration to jogobot module 
Use custom Error classes
 2015-09-15 21:21:05 +02:00
golderweb db5bb7401e Update RedFam class to rebuild the whole structure of RedFamPaser generated object 
Move fam_hash() method from RedFamParser to RedFam
Define custom Error classes
 2015-09-15 21:19:07 +02:00
golderweb be0041804a Clean up 2015-09-13 13:19:22 +02:00
golderweb af6d483280 Use a additional module to store custom config ... 2015-09-13 13:14:37 +02:00
golderweb 7c9061d37f Use redundances as python package 2015-09-13 13:13:28 +02:00
golderweb 74b2dc727c Clean up PEP8 styleguide compatibility with flake8 2015-09-13 11:53:47 +02:00
golderweb cafe08dd7f Added license GPLv3 + Reworked code 
Better compatibility with https://www.mediawiki.org/wiki/Manual:Pywikibot/Development_guidelines
 2015-09-13 01:22:27 +02:00
golderweb ad100e96eb Remove timezone from timestamp patterns since it is not stored and causes errors on labs 2015-09-12 22:35:12 +02:00
golderweb c78480a75d Use mysql config from pywikibot user-config.py 2015-09-12 22:33:44 +02:00
golderweb 6e0d3200c2 Change visibility of attributes changed and parsed in RED_PAGE to protected 2015-09-12 14:24:45 +02:00
golderweb f50f4b664f Add additional possibilities to detect wether a red_page is an archive or not 2015-09-12 13:48:01 +02:00
golderweb e70655f679 Catch sections with more then 8 articles 2015-09-12 12:42:23 +02:00
golderweb 14e865c6e6 We need to save the red_fam section heading in database for generating section links since format is not consistent 2015-09-12 00:50:04 +02:00
golderweb a97e2cea5a Since some timestamps are broken we need to reconstruct them by regex match groups 
Prevents ValueErrors of datetime.strptime in most cases
 2015-09-11 23:47:35 +02:00
golderweb 6e64d8448e Prevent RED_PAGE from resetting status if parser is not fired 2015-09-11 22:29:16 +02:00
golderweb deaa1d855b Fix bug which prevents detecting of red_fam sectionheading when there is whitespace after closing heading tags (===) 2015-09-11 22:27:25 +02:00
golderweb 90da231712 Move the code for instanciating of RED_FAM_PARSER to the end of loop 2015-09-11 17:05:18 +02:00
golderweb 529ed09702 Use pywikibot.output instead of print 2015-09-11 16:42:12 +02:00
golderweb 1892c6d015 Fix wrong intendet line (RED_FAM_PARSER only was instantiated when ending-datetime workaround fired) 2015-09-11 16:41:04 +02:00
golderweb 99f050acd3 Add workaround to detect ending datetime if there is no done notice 2015-09-11 16:22:19 +02:00
golderweb 8b7dc5dbf2 Improve sectionheading regex to reduce false positives, require at least two links 2015-09-11 16:18:46 +02:00
golderweb 5e39ea0b06 Correct use of pywikibot page.title() 2015-09-11 14:12:26 +02:00
golderweb f8dacb53e1 Add param to RED_PAGE for predefining page status archived when using cat of archives 2015-09-11 14:10:33 +02:00
golderweb ebf7a8fe0b Add alternative syntax for done_notice 2015-09-11 13:56:16 +02:00