2c105336b0
RedFamWorker: Exclude users and user talkpages
...
Users can't be part of valid redundances
Issue #68 (#68 )
2018-05-17 12:35:38 +02:00
6e119ea98f
RedFamWorker: Improve talkpagetoggling
...
Do not toggle to main page if we have already a talkpage and vice versa
Issue #69 (#69 )
2018-05-17 12:26:37 +02:00
fa13e2a5cf
Follow moved pages
...
Keep notice together with content
https://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:Jogo.obb&oldid=176464377#Redundanzhinweis_zu_zwischenzeitlich_verschobenen_Artikeln
Issue #70 (#70 )
2018-05-17 12:18:13 +02:00
93447d8dc6
Prepare release v1.0
...
Update Copyright Notices
Version information
2017-11-05 12:25:13 +01:00
1b6faf9e53
Use own db for red-task
...
Since we have several tables and sometimes need to create a copy on
replication servers.
2017-11-05 12:17:05 +01:00
b4c193eedc
Disable echoing of SQLAlchemy Egine
...
We don't need this extensive output for production
2017-11-05 12:07:38 +01:00
20103d589d
redfam: article_generator add redfam info to page
...
Add reference to redfam object and article title from db to Page object
since Page.title() may differe (short Namespaces, anchors, special chars)
Related Task: [FS#161](https://fs.golderweb.de/index.php?do=details&task_id=161 )
2017-11-05 11:18:53 +01:00
e18aa96a84
redfam: article_generator can return talkpage
...
To make pywikibot.pagegenerators.PageWithTalkPageGenerators unneccessary
so we can manipulate talkpage object directly
Related Task: [FS#161](https://fs.golderweb.de/index.php?do=details&task_id=161 )
2017-11-05 11:15:04 +01:00
5f4640d5ff
Replace urlencoded chars with unicode equivalent
...
Otherwise we get value errors while marking since pwb replaces those
Related Task: [FS#160](https://fs.golderweb.de/index.php?do=details&task_id=160 )
2017-10-28 22:35:25 +02:00
84802cf521
Remove leading or trailing spaces in articles
...
Some articles contain spaces between title and anchor part which will
be stripped now
Related Task: [FS#159](https://fs.golderweb.de/index.php?do=details&task_id=159 )
2017-10-28 18:41:06 +02:00
02e53475f1
Prevent lowercase article titles in Parser
...
Since real lowercase article titles are not allowed, make sure to
convert all first letters of article titles to uppercase. This is
neccessary since pywikibot will return article titles like this.
Related Task: [FS#157](https://fs.golderweb.de/index.php?do=details&task_id=157 )
2017-09-09 21:35:36 +02:00
ff03ca8f13
Explicitly set charset for PyMySQL-Connection
...
Since PyMySQL-Connection otherwise uses charset 'latin-1', explicitly
set connection charset to 'utf8'
http://docs.sqlalchemy.org/en/rel_1_0/dialects/mysql.html#charset-selection
http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html?highlight=url#sqlalchemy.engine.url.URL
Related Task: [FS#156](https://fs.golderweb.de/index.php?do=details&task_id=156 )
2017-09-02 22:10:25 +02:00
d9b4fcc0bd
Strip spaces before adding articles to redfam
...
Some article links have surounding spaces in their linktext. Remove them
before adding article to RedFam to have a cannonical title
Related Task: [FS#155](https://fs.golderweb.de/index.php?do=details&task_id=155 )
2017-09-02 22:06:30 +02:00
ec8f459db5
Merge branch 'fs#138-marked-articles-shown-again' into develop
2017-08-24 12:19:24 +02:00
3b2cb95f36
Do not fetch marked redfams from db
...
Exclude marked Redfams from DB-Query to prevent marking them again
Related Task: [FS#138](https://fs.golderweb.de/index.php?do=details&task_id=138 )
2017-08-24 12:09:43 +02:00
14ec71dd09
Rewrite get_disc_link to handle special cases
...
Use methods of pywikibot site-object and mwparser to get rid of any
special elements like templates or links in headings for construction
of our disc link.
Replace by hand as it otherwise will occur as normal space and
wont work
Related Task: [FS#147](https://fs.golderweb.de/index.php?do=details&task_id=147 )
2017-08-23 14:53:22 +02:00
cd87d1c2bb
Fix already marked articles was reshown bug
...
Since we search for matching states for articles to include or exclude
in a loop, we could not control the outer loop via default break/
continue. Python docs recommend using Exceptions and try/except
structures to realise that most conveniently.
https://docs.python.org/3/faq/design.html#why-is-there-no-goto
Related Task: [FS#138](https://fs.golderweb.de/index.php?do=details&task_id=138 )
2017-08-22 21:45:58 +02:00
30de2a2e12
Replace oursql with PyMySQL
...
Since this is prefered on toolsforge and works out of the box after
installing via pip, replace oursql which caused some problems.
Especially oursql was not able to connect to db via ssh tunnel.
Related Task: [FS#144](https://fs.golderweb.de/index.php?do=details&task_id=144 )
2017-08-21 13:55:33 +02:00
0f930082b4
Also canonicalise anchor parts of articles
...
Replace spaces in anchors with underscores as spaces are not correct
there
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=114 FS#114]
2017-03-11 11:40:41 +01:00
80c94ccf4f
Replace underscores in article titles
...
Remove underscores in article titles and replace with spaces to have
canonical state for all articles
Therefore we need to split title and posible anchors in heading parser
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=114 FS#114]
2017-03-11 11:30:19 +01:00
3e69a1c77e
Remove problem indicating stati when set marked
...
Remove states which are indicating problems in previous runs if
successfully marked article and also whole RedFam
[https://fs.golderweb.de/index.php?do=details&task_id=112 FS#112]
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=110 FS#110]
2017-03-11 00:03:42 +01:00
56f326b568
Fix error all current redfams marked when quit
...
Restructure update_status to make sure, marked is only set when all
articles are marked or gone (means deleted or redirect)
[https://fs.golderweb.de/index.php?do=details&task_id=111 FS#111]
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=110 FS#110]
2017-03-10 23:45:48 +01:00
147e96d388
Add Wrapperclass for Parser to RedPage
...
Add a wrapper class to overwrite type of Items returned by
RedPage.redfams relationship
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=109 FS#109]
2017-03-09 15:30:51 +01:00
4aaacf1443
Add redfams to redpage-obj after parsing
...
To have redfams available for updates immediately after parsing. Double
redfams then will be seen as Update.
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=108 FS#108]
2017-03-09 10:25:52 +01:00
281f1c49a8
mysqlred: Set family via pywikibot
...
Get family/language part of table names from PyWikiBot Site
2017-03-09 00:12:41 +01:00
3fe47e666f
Fix polymorphism problem with relationships
...
Since we are using subclasses of the ORM mapped classes, disable
typechecks for ORM relations
2017-03-09 00:12:41 +01:00
e16925197c
Fix pep8.. compliance
...
To be concordant with the coding styles fix pep8 compliance
2017-03-09 00:12:31 +01:00
9ba7d2e517
Change redfam generator filters
...
Change and clear up the filters in redfam generator to keep track of
article status and use positive conditionals
2017-03-09 00:10:51 +01:00
844fee52ae
Make markpages using new DB/Class structure
...
Update markpages and RedFamWorker-Code to use the new sqlalchemy based
DB ORM Interface
2017-03-09 00:10:51 +01:00
43e31c108a
Working RedFamWorker query
...
Modify RedfamWorker class to work with new DB API
2017-03-09 00:10:50 +01:00
89b50e3312
Remove old status API
...
Now we use the methods of status object directly
2017-03-09 00:10:50 +01:00
bf8e47f916
Improve new status API
...
Make sure state changes are only detected as such by sqlalchemy if they
are real changes
2017-03-09 00:10:50 +01:00
467f829af2
Some cleanups
...
Remove old commented out code from manual mysql solution
2017-03-09 00:10:50 +01:00
6e973369cd
sqlalchemy working for parser
...
Needs some testing, presumably contains some bugs
2017-03-09 00:08:48 +01:00
0ebf307bb8
Add markpages as subtask
...
Markpages is a subtask of our Red-Bot
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]
# The following line will be added automatically
# Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]
2016-11-05 19:41:22 +01:00
65fb2ecb28
Generate Fam status based on article status
...
Some article states should be reflected in the RedFam status
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]
2016-11-05 19:27:56 +01:00
870ed4bf25
Update redfam.article_generator use article status
...
To be able to filter articles by status of that article
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]
2016-08-30 17:47:02 +02:00
e13320820c
Add API to manage status per article
...
To be able to track changes to articles to update redfam status
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]
2016-08-30 17:45:18 +02:00
6149dcdb8b
Apply changes to data structure
...
See related ticket
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=94 FS#94]
2016-08-30 14:28:28 +02:00
8c56125a7b
Update talkpage notice template
...
Exact date is not necessary and end could be ommited if of the same
month
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=93 FS#93]
2016-08-30 12:07:11 +02:00
c0b18f88e5
Add filter options to redfam.article_generator
...
To give the posibility to filter not existing pages or redirect pages or
vice versa.
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=87 FS#87]
2016-08-28 15:06:17 +02:00
e5989305a4
Add a generator to redfam yielding article pages
...
To work on articles of a redfam a generator which yields belonging
articles is necessary
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=87 FS#87]
2016-08-28 13:20:13 +02:00
6717fa4fba
Add method to generate notice for article discpage
...
We need a method to generate the template to add to article discpages
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=29 FS#29]
2016-08-27 19:49:50 +02:00
3723aba578
Add a method to get link to related reddisc
...
To generate notices or other stuff it is necessary to add links to the
related reddisc.
This method returns a wikilink to text the redfam's reddisc.
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=81 FS#81]
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=81 FS#81]
2016-08-27 19:48:03 +02:00
b36dc250d2
Request information about reddisc page for redfams
...
To generate links to related reddisc it is necessary to get at least the
Title of the related reddisc page. As saving the same data in db is
worse, we retreive it via join from red_pages table
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]
2016-08-27 19:44:33 +02:00
4055dc52d8
Make it possible to get a RedPage-Object by pageid
...
When working on redfams it is necessary to have information about redpage
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]
2016-08-27 19:44:33 +02:00
594130c8a6
Restore changes from 45df35431
...
Documented to prevent deleting again
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]
2016-08-27 19:44:33 +02:00
b271a0b0b1
Add generator wrapper to fetch RedFams by status and ending
...
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]
2016-08-27 19:44:33 +02:00
ad088126e7
Define method to update Status after Working with RedFam
...
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]
2016-08-27 19:44:33 +02:00
151c22a735
Add fetched mysql_data to _mysql-Object of parent class for using change-method to update db
...
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=26 FS#26]
2016-08-27 19:44:33 +02:00
