77d1de4473
Add a tablename prefix depending on Site
...
To be able to run the bot on different wikis the db tables should be
named pywikibot.Site dependend and changed automatically
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=79 FS#79]
2016-08-24 23:53:10 +02:00
cac04f344f
Merge branch 'fs#74-helpermodules-lib' into fs#70-refactoring
2016-08-24 23:09:33 +02:00
e28acf88d1
Introduce new directory structure
...
To clarify which is a bot and which are helper scripts
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=74 FS#74]
2016-08-24 22:41:41 +02:00
af48888535
Merge branch 'fs#78-redfam-section-false-positives' into fs#70-refactoring
2016-08-24 20:05:04 +02:00
ac54aea698
Use callback to detect redfam.section
...
Detecting redfam-Sections via RegExp caused some false positives due to
wrong formated things in wikisyntax. See Task
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=78 FS#78]
2016-08-24 20:02:48 +02:00
2deb02fe47
Merge branch 'fs#77-errors-on-old-archives' into fs#70-refactoring
2016-08-24 19:59:53 +02:00
1e4c8646bf
Reparse redfam-heading with mwparser
...
See related ticked for detailed failure explanation
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=77 FS#77]
2016-08-24 19:57:25 +02:00
fe2810f07c
Merge branch 'fs#76-redfam-without-dates' into fs#70-refactoring
2016-08-24 17:01:00 +02:00
ab430e0085
Use month of reddisc as beginning if missing
...
Construct a fictive but sensfull beginning if we cant detect one
Needed since beginning is mandatory
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=76 FS#76]
2016-08-24 16:56:54 +02:00
95be313859
Pass reddisc pywikibot.page object to redfam
...
To access page information like page title (eg. to get dates from it)
of the reddisc page
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=76 FS#76]
2016-08-24 16:53:45 +02:00
0bb0b2d957
Make sure var beginning is always defined
...
To prevent unbound Errors caused by using undeclared variable beginning
if the redfam-section does not contain any timestamp
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=76 FS#76]
2016-08-24 16:51:23 +02:00
db32c9e8f6
Merge branch 'fs#75-mysql-flush-error-false-reddisc' into fs#70-refactoring
2016-08-24 16:49:37 +02:00
bd2d221c48
Prevent flush from creating cursor without con
...
MysqlRed.flush() tried to create a cursor in any case. If there was no
connection (because the subclasses haven't been instantiated an oursql
Error occured.
Instead, check before if there is a connection and otherwise raise an Error
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]
2016-08-24 16:46:51 +02:00
ee8ebbc8bc
Make sure only flush db if there are redfams
...
To prevent from doing unnecessary stuff and trying to use not existing
db connection
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]
2016-08-24 15:47:43 +02:00
dcc4851513
Check reddisc page titles against regex
...
To prevent parsing Pages which have been categorized in configured cats
wrong or are given via cmd params
Parsing them results in unexpected behaviour
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=75 FS#75]
2016-08-24 15:27:42 +02:00
0ea1b0039d
Merge branch 'fs#72-rewrite-reddiscparser' into fs#70-refactoring
2016-08-24 11:23:38 +02:00
2f878ee901
Correct filename in header
...
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72]
2016-08-24 11:20:28 +02:00
17bfb32ded
Building generators of config cats in sep Function
...
Since the main()-Function was too complex the logic to build generators
out of categories provided in jogobot.conf was moved in a separate
function
[https://fs.golderweb.de/index.php?do=details&task_id=73 FS#73]
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72]
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72]
2016-08-24 11:17:00 +02:00
6cb92c1da7
Rewrite parse control using pywikibot.bot classes
...
To use the default pywikibot.classes making life easier at some point
Beeing standardconform with pywikibot in handling args
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72]
2016-08-23 21:53:44 +02:00
a8605bcee6
Mv pages-parser.py to reddiscparser.py
...
New, more meaningfull naming conventions, from redpage to reddisc (page)
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=72 FS#72]
2016-08-23 21:50:22 +02:00
5d31bdd7eb
Jogobot submodule updated
2016-08-23 21:28:13 +02:00
7f8ab1897e
Merge branch 'fs#69-deprecated-decorators-param-str' into test-v3
2016-08-23 21:26:47 +02:00
79dbde2413
Provide Replacement to @deprecated() as str
...
Since use of pywikibot-master (or Python3.5 @see ticket below)
the @deprecator requires a str as param and no callable object like
done before
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=69 FS#69]
2016-08-23 21:23:24 +02:00
36a480a042
Merge branch 'fs#68-mysql-db-port' into test-v3
2016-08-23 21:16:09 +02:00
bd9dbdfa17
Make use of declared db_host_port
...
The port to connect to MySQL-Server was previously always assumed as
the default one. So the library was incompatible to db's on nonstandard
ports
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=68 FS#68]
2016-08-23 21:12:07 +02:00
944bea488a
Merge branch 'restucture-parsers' into test-v3
2016-03-05 15:02:31 +01:00
7cac294181
Merge branch 'parser-script' into restucture-parsers
2016-03-05 15:01:09 +01:00
a24f208449
Add parse-pages.py Script
2016-03-04 20:39:41 +01:00
0af7eb11d6
Move parsing of redfams from RedPageParser to RedFamParser.parser so RedPageParse won't do anything with redfams
...
except for returning a generator of text-sections
2016-03-03 20:41:14 +01:00
7422307985
Rewrite RedPage.parse using mwparserfromhell to make it simpler
2016-03-03 17:37:46 +01:00
b81694c6d3
Rewrite heading_parser using mwparserfromhell to make it simpler
2016-03-03 17:30:39 +01:00
a2dfffc74b
Let old date-extracting methods use dates_extract and mark them as deprecated
2016-03-03 17:23:44 +01:00
163972c924
New method dates_extract which finds begining and ending at once
2016-03-03 17:20:57 +01:00
baf4ae2a07
Merge branch 'new-structure' into test-v3
2016-03-02 17:22:04 +01:00
10f64199ab
Remove relativ imports as we don't are in a package anymore
2016-03-02 17:19:11 +01:00
24f1a7f516
Remove __init__.py as we won't use it as a package
2016-03-02 17:14:30 +01:00
9113a40704
Merge branch 'warning-non-flushed-mysql-cache' into test-v3
2016-03-02 17:13:08 +01:00
f53a5b3745
Output a warning if there are update/insert querys cached when exit programm
2016-03-02 17:10:08 +01:00
673e49c55a
Merge branch 'jogobot' into test-v3
...
Use new jogobot package
2016-02-29 11:37:40 +01:00
24adafeee7
Changes for new jogobot-module
2016-02-29 11:35:48 +01:00
b26f04db8c
Use updated version of jogobot with ast.literal_eval parsed config entrys
2016-02-29 11:13:14 +01:00
f29dfd5003
Use new jogobot module
2016-02-28 18:00:46 +01:00
ef9c13324a
Improve documentation of MysqlRed.flush()
2015-09-20 18:17:59 +02:00
e186f2f22b
Use dictionary with page_id / fam_hash as key for cached_insert_data to prevent double entrys
2015-09-20 17:45:07 +02:00
7d6cd8bb30
Strip leading and trailing whitespace in Links to prevent wrong fam_hashes (when receiving redfam from db) since MySQL drops it
2015-09-19 22:44:43 +02:00
4e21b6696a
Remove unnecessary whitespace from error messages
2015-09-19 20:51:52 +02:00
6992f82f02
Start Implementing of RedFamWorker
2015-09-19 20:51:21 +02:00
dbcfe8f106
Add a generator to MysqlRedFam to retrieve redfams from db by status
2015-09-19 19:50:38 +02:00
8059bb9992
Change behavior of MysqlRedFam to be able to get instance without knowen fam_hash
2015-09-19 19:49:20 +02:00
b5ca69077c
Remove double appearence of heading parameter in repression of RedFam
2015-09-19 19:47:09 +02:00
