golderweb
67aaf3cbbe
Merge branch 'i#70-follow-moved-pages' into develop
2018-05-17 12:24:00 +02:00
golderweb
fa13e2a5cf
Follow moved pages
...
Keep notice together with content
https://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:Jogo.obb&oldid=176464377#Redundanzhinweis_zu_zwischenzeitlich_verschobenen_Artikeln
Issue #70 (#70 )
2018-05-17 12:18:13 +02:00
golderweb
562e689418
Merge branch 'release-1.0' back into develop
2017-11-05 12:30:18 +01:00
golderweb
93447d8dc6
Prepare release v1.0
...
Update Copyright Notices
Version information
2017-11-05 12:25:13 +01:00
golderweb
1b6faf9e53
Use own db for red-task
...
Since we have several tables and sometimes need to create a copy on
replication servers.
2017-11-05 12:17:05 +01:00
golderweb
b4c193eedc
Disable echoing of SQLAlchemy Egine
...
We don't need this extensive output for production
2017-11-05 12:07:38 +01:00
golderweb
788a3df0cd
Update jogobot-submodule to v0.1
2017-11-05 12:00:28 +01:00
golderweb
04f591b466
Merge branch 'fs#161-add-article-titles' into develop
2017-11-05 11:24:15 +01:00
golderweb
9640467f69
markpages: Use redarticle attribute of Page
...
Instead of trying to reconstruct our db article title, use the one added
to Page-object by redfam.article_generator
Related Task: [FS#161](https://fs.golderweb.de/index.php?do=details&task_id=161 )
2017-11-05 11:22:43 +01:00
golderweb
bfec2abf98
markpages: Get rid of PageWithTalkPageGenerator
...
Since redfam.article_generator can yield talkpage with additional
information about redfam and current article from db, we do not need it
anymore.
Related Task: [FS#161](https://fs.golderweb.de/index.php?do=details&task_id=161 )
2017-11-05 11:20:55 +01:00
golderweb
20103d589d
redfam: article_generator add redfam info to page
...
Add reference to redfam object and article title from db to Page object
since Page.title() may differe (short Namespaces, anchors, special chars)
Related Task: [FS#161](https://fs.golderweb.de/index.php?do=details&task_id=161 )
2017-11-05 11:18:53 +01:00
golderweb
e18aa96a84
redfam: article_generator can return talkpage
...
To make pywikibot.pagegenerators.PageWithTalkPageGenerators unneccessary
so we can manipulate talkpage object directly
Related Task: [FS#161](https://fs.golderweb.de/index.php?do=details&task_id=161 )
2017-11-05 11:15:04 +01:00
golderweb
1dd4c7f87e
Merge branch 'test-v7' back into develop
2017-11-02 18:57:59 +01:00
golderweb
33b2e47312
Describe version test-v7
2017-10-28 22:43:53 +02:00
golderweb
3bd17ce692
Merge branch 'fs#160-urlencoded-chars' into develop
2017-10-28 22:36:55 +02:00
golderweb
5f4640d5ff
Replace urlencoded chars with unicode equivalent
...
Otherwise we get value errors while marking since pwb replaces those
Related Task: [FS#160](https://fs.golderweb.de/index.php?do=details&task_id=160 )
2017-10-28 22:35:25 +02:00
golderweb
7e0456ae4f
Merge branch 'test-v6' back into develop
2017-10-28 22:34:30 +02:00
golderweb
108b7aa331
Describe version test-v6
2017-10-28 18:46:30 +02:00
golderweb
a3adf31b89
Merge branch 'fs#86-activate-status-api' into develop
2017-10-28 18:44:42 +02:00
golderweb
614f288bb9
Activate jogobot status api for onwiki disabling
...
Related Task: [FS#86](https://fs.golderweb.de/index.php?do=details&task_id=86 )
2017-10-28 18:44:05 +02:00
golderweb
c450a045bf
Merge branch 'fs#159-space-before-anchor' into develop
2017-10-28 18:43:13 +02:00
golderweb
84802cf521
Remove leading or trailing spaces in articles
...
Some articles contain spaces between title and anchor part which will
be stripped now
Related Task: [FS#159](https://fs.golderweb.de/index.php?do=details&task_id=159 )
2017-10-28 18:41:06 +02:00
golderweb
5f6c443ba8
Merge branch 'test-v5' back into develop
2017-10-28 18:17:01 +02:00
golderweb
0c135ef1bb
Describe version test-v5
2017-09-23 23:50:42 +02:00
golderweb
8b8221cfcd
Merge branch 'fs#152-respect-always-flag' into develop
2017-09-23 23:49:59 +02:00
golderweb
bdccc8417c
Set always in Pywikibot.Basebot
...
If cmdline param -always is set, set the related option in
Pywikibot.Basebot Object for automatic edits with out further requests
Related Task: [FS#152](https://fs.golderweb.de/index.php?do=details&task_id=152 )
2017-09-23 23:49:41 +02:00
golderweb
a70835c58a
Merge back branch 'test-v4' into develop
2017-09-23 23:48:25 +02:00
golderweb
ec2b84df2a
Add requirements
...
To make setup of environment for this module easier
2017-09-23 21:09:58 +02:00
golderweb
88848cb084
Prepare Version test-v4 for release
...
Add a README.md file for this project
2017-09-23 20:32:13 +02:00
golderweb
5057aed0d3
Merge branch 'fs#157-lowercase-title' into develop
2017-09-09 21:47:03 +02:00
golderweb
02e53475f1
Prevent lowercase article titles in Parser
...
Since real lowercase article titles are not allowed, make sure to
convert all first letters of article titles to uppercase. This is
neccessary since pywikibot will return article titles like this.
Related Task: [FS#157](https://fs.golderweb.de/index.php?do=details&task_id=157 )
2017-09-09 21:35:36 +02:00
golderweb
d6f9b460c9
Merge branch 'fs#156-dbapi-charset' into develop
2017-09-02 22:13:20 +02:00
golderweb
ff03ca8f13
Explicitly set charset for PyMySQL-Connection
...
Since PyMySQL-Connection otherwise uses charset 'latin-1', explicitly
set connection charset to 'utf8'
http://docs.sqlalchemy.org/en/rel_1_0/dialects/mysql.html#charset-selection
http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html?highlight=url#sqlalchemy.engine.url.URL
Related Task: [FS#156](https://fs.golderweb.de/index.php?do=details&task_id=156 )
2017-09-02 22:10:25 +02:00
golderweb
88692ca678
Merge branch 'fs#155-article-surouding-space' into develop
2017-09-02 22:08:31 +02:00
golderweb
d9b4fcc0bd
Strip spaces before adding articles to redfam
...
Some article links have surounding spaces in their linktext. Remove them
before adding article to RedFam to have a cannonical title
Related Task: [FS#155](https://fs.golderweb.de/index.php?do=details&task_id=155 )
2017-09-02 22:06:30 +02:00
golderweb
22ff78ea98
Merge branch 'fs#154-categorie-colons-missing' into develop
2017-09-02 16:02:45 +02:00
golderweb
b3cfcdc259
Improve title detection to get correct behaviour
...
Make sure that categorie links are starting with colon and non article
pages are returned with namespace.
Related Task: [FS#154](https://fs.golderweb.de/index.php?do=details&task_id=154 )
2017-09-02 15:59:34 +02:00
golderweb
b3e0ace2f4
Merge branch 'fs#153-nested-templates' into develop
2017-09-02 14:25:21 +02:00
golderweb
f8002c85da
Do not search for templates recursivly
...
Since nested templates did not get an index in global wikicode object
searching for index of an nested template results in ValueError
Related Task: [FS#153](https://fs.golderweb.de/index.php?do=details&task_id=153 )
2017-09-02 14:23:25 +02:00
golderweb
49bc05d29b
Merge branch 'fs#151-normalize-article-titles-anchor' into develop
2017-09-02 13:36:17 +02:00
golderweb
8a26b6d92a
Normalize article titles with anchors
...
In our db article titles with anchors are stored with underscores in
anchor string. Therefore we need to replace spaces in anchor string
given by pywikibot.Page.title().
Related Task: [FS#151](https://fs.golderweb.de/index.php?do=details&task_id=151 )
2017-08-25 18:11:41 +02:00
golderweb
49a8230d76
Merge branch 'fs#141-place-notice-after-comment' into develop
2017-08-25 17:11:28 +02:00
golderweb
31c10073a2
Prevent index errors searching for comments
...
Make sure not to exceed existing indexes of wikicode object while trying
to search for comments
Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141 )
2017-08-25 17:09:38 +02:00
golderweb
642a29b022
Improve regex for blank lines
...
Do not match consecutive linebreaks as one
Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141 )
2017-08-24 18:47:18 +02:00
golderweb
2f90751dc2
Merge branch 'fs#146-famhash-generator' into develop
2017-08-24 12:27:54 +02:00
golderweb
024be69fe1
Use famhash as generator
...
If famhash is defined, fetch explicitly that redfam from db and work
only on this
Related Task: [FS#146](https://fs.golderweb.de/index.php?do=details&task_id=146 )
2017-08-24 12:27:13 +02:00
golderweb
b6d7268a7f
select by famhash: Add methods to get param in bot
...
We need a method as callback to get bot specific params passed through
to our bot class.
Introduce -famhash parameter to work on specific famhash
Related Task:[FS#146](https://fs.golderweb.de/index.php?do=details&task_id=146 )
2017-08-24 12:27:13 +02:00
golderweb
526184c1e1
Merge branch 'fs#148-articles-mixed-up' into develop
2017-08-24 12:26:53 +02:00
golderweb
3aa6c5fb1c
Disable PreloadingGenerator temporarily
...
PreloadingGenerator mixes up yielded Pages. This is very unconvenient
for semi-automatic workflow with manual checks as the articles of the
RedFams were not following each other.
Related Task: [FS#148](https://fs.golderweb.de/index.php?do=details&task_id=148 )
2017-08-24 12:23:17 +02:00
golderweb
ec8f459db5
Merge branch 'fs#138-marked-articles-shown-again' into develop
2017-08-24 12:19:24 +02:00
golderweb
3b2cb95f36
Do not fetch marked redfams from db
...
Exclude marked Redfams from DB-Query to prevent marking them again
Related Task: [FS#138](https://fs.golderweb.de/index.php?do=details&task_id=138 )
2017-08-24 12:09:43 +02:00
golderweb
41e5cc1a9d
Merge branch 'fs#141-place-notice-after-comment' into develop
2017-08-24 12:06:03 +02:00
golderweb
9b9d50c4d2
Improve detection of empty lines
...
Search with RegEx as empty lines could also contain spaces
Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141 )
2017-08-24 12:04:45 +02:00
golderweb
a755288700
Merge branch 'fs#147-templates-in-heading' into develop
2017-08-23 14:55:43 +02:00
golderweb
14ec71dd09
Rewrite get_disc_link to handle special cases
...
Use methods of pywikibot site-object and mwparser to get rid of any
special elements like templates or links in headings for construction
of our disc link.
Replace by hand as it otherwise will occur as normal space and
wont work
Related Task: [FS#147](https://fs.golderweb.de/index.php?do=details&task_id=147 )
2017-08-23 14:53:22 +02:00
golderweb
e283eb78ac
Merge branch 'fs#140-also-mark-redirects' into develop
2017-08-22 21:59:22 +02:00
golderweb
cc02006fd2
Do not exclude redirects from beeing marked
...
In accordance with Zulu55 redirect discussion pages should also get
a notice, therefore do not exclude redirects.
Related Task: [FS#140](https://fs.golderweb.de/index.php?do=details&task_id=140 )
2017-08-22 21:59:07 +02:00
golderweb
37b0cbef08
Merge branch 'fs#138-marked-articles-shown-again' into develop
2017-08-22 21:58:22 +02:00
golderweb
4137d72468
Look for existing notice by simple in-check
...
To detect maybe uncommented notices already present, check for them
using just a simple python x in y check over whole wikicode
Related Task: [FS#138](https://fs.golderweb.de/index.php?do=details&task_id=138 )
2017-08-22 21:56:43 +02:00
golderweb
cd87d1c2bb
Fix already marked articles was reshown bug
...
Since we search for matching states for articles to include or exclude
in a loop, we could not control the outer loop via default break/
continue. Python docs recommend using Exceptions and try/except
structures to realise that most conveniently.
https://docs.python.org/3/faq/design.html#why-is-there-no-goto
Related Task: [FS#138](https://fs.golderweb.de/index.php?do=details&task_id=138 )
2017-08-22 21:45:58 +02:00
golderweb
456b2ba3d4
Merge branch 'fs#141-place-notice-after-comment' into develop
2017-08-21 22:11:51 +02:00
golderweb
47b85a0b5e
Add missing line break if there is no template
...
To make sure our notice template resides in its own line in every case
Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141 )
2017-08-21 22:09:59 +02:00
golderweb
a6fdc974bd
Merge branch 'fs#144-PyMySQL-instead-oursql' into develop
2017-08-21 13:58:34 +02:00
golderweb
30de2a2e12
Replace oursql with PyMySQL
...
Since this is prefered on toolsforge and works out of the box after
installing via pip, replace oursql which caused some problems.
Especially oursql was not able to connect to db via ssh tunnel.
Related Task: [FS#144](https://fs.golderweb.de/index.php?do=details&task_id=144 )
2017-08-21 13:55:33 +02:00
golderweb
4a6855cf7b
Merge branch 'fs#141-place-notice-after-comment' into develop
2017-08-21 13:51:32 +02:00
golderweb
8422d08cb6
Keep comments and leading templates together
...
Prevent spliting up existing comments and templates as often those are
documenting archiv templates behaviour
Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141 )
2017-08-21 13:49:34 +02:00
golderweb
ed78501821
Merge branch 'fs#115-lsec-no-template' into test-v3
2017-08-21 13:16:28 +02:00
golderweb
34e7e0d3be
Prevent index Error if no template in leadsec
...
Check if there is a template in leadsec before accessing list item to
prevent IndexErrors
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=115 FS#115]
2017-03-11 12:22:10 +01:00
golderweb
f9f081d072
Merge branch 'fs#114-remove-underscores-in-articles' into test-v3
2017-03-11 11:43:49 +01:00
golderweb
0f930082b4
Also canonicalise anchor parts of articles
...
Replace spaces in anchors with underscores as spaces are not correct
there
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=114 FS#114]
2017-03-11 11:40:41 +01:00
golderweb
80c94ccf4f
Replace underscores in article titles
...
Remove underscores in article titles and replace with spaces to have
canonical state for all articles
Therefore we need to split title and posible anchors in heading parser
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=114 FS#114]
2017-03-11 11:30:19 +01:00
golderweb
37704c6661
Replace pywikibot.showDiff with patched version
...
Pywikibot.bot.userPut does not support setting the value of diff context
so it is always zero. Therefore we need to patch either userPut or
showDiff to get some context.
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=113 FS#113]
2017-03-11 10:39:31 +01:00
golderweb
4e4be1c6d0
Merge branch 'fs#110-markpages-status-problems' into test-v3
2017-03-11 00:06:00 +01:00
golderweb
3e69a1c77e
Remove problem indicating stati when set marked
...
Remove states which are indicating problems in previous runs if
successfully marked article and also whole RedFam
[https://fs.golderweb.de/index.php?do=details&task_id=112 FS#112]
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=110 FS#110]
2017-03-11 00:03:42 +01:00
golderweb
56f326b568
Fix error all current redfams marked when quit
...
Restructure update_status to make sure, marked is only set when all
articles are marked or gone (means deleted or redirect)
[https://fs.golderweb.de/index.php?do=details&task_id=111 FS#111]
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=110 FS#110]
2017-03-10 23:45:48 +01:00
golderweb
868894a38b
Format fixes
...
Set locale to de_DE.utf-8 for whole Task
Make sure Template is added in own source line
2017-03-10 23:28:24 +01:00
golderweb
65de6decb2
markpages: Filter redirects
...
Do not mark redirects discussion pages
2017-03-10 21:51:59 +01:00
golderweb
889be30a47
Merge branch 'fs#109-redpage-redfam-join-type' into test-v3
2017-03-09 15:33:05 +01:00
golderweb
147e96d388
Add Wrapperclass for Parser to RedPage
...
Add a wrapper class to overwrite type of Items returned by
RedPage.redfams relationship
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=109 FS#109]
2017-03-09 15:30:51 +01:00
golderweb
76666aa294
Merge branch 'fs#25-mark-done' into test-v3
2017-03-09 10:50:14 +01:00
golderweb
db39bb5ff4
Merge branch 'fs#88-mark-pages-bot' into fs#25-mark-done
2017-03-09 10:49:16 +01:00
golderweb
ec7880207b
Merge branch 'fs#95-sqlalchemy' into fs#88-mark-pages-bot
2017-03-09 10:48:10 +01:00
golderweb
4aaacf1443
Add redfams to redpage-obj after parsing
...
To have redfams available for updates immediately after parsing. Double
redfams then will be seen as Update.
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=108 FS#108]
2017-03-09 10:25:52 +01:00
golderweb
281f1c49a8
mysqlred: Set family via pywikibot
...
Get family/language part of table names from PyWikiBot Site
2017-03-09 00:12:41 +01:00
golderweb
3fe47e666f
Fix polymorphism problem with relationships
...
Since we are using subclasses of the ORM mapped classes, disable
typechecks for ORM relations
2017-03-09 00:12:41 +01:00
golderweb
e16925197c
Fix pep8.. compliance
...
To be concordant with the coding styles fix pep8 compliance
2017-03-09 00:12:31 +01:00
golderweb
9ba7d2e517
Change redfam generator filters
...
Change and clear up the filters in redfam generator to keep track of
article status and use positive conditionals
2017-03-09 00:10:51 +01:00
golderweb
844fee52ae
Make markpages using new DB/Class structure
...
Update markpages and RedFamWorker-Code to use the new sqlalchemy based
DB ORM Interface
2017-03-09 00:10:51 +01:00
golderweb
43e31c108a
Working RedFamWorker query
...
Modify RedfamWorker class to work with new DB API
2017-03-09 00:10:50 +01:00
golderweb
89b50e3312
Remove old status API
...
Now we use the methods of status object directly
2017-03-09 00:10:50 +01:00
golderweb
bf8e47f916
Improve new status API
...
Make sure state changes are only detected as such by sqlalchemy if they
are real changes
2017-03-09 00:10:50 +01:00
golderweb
467f829af2
Some cleanups
...
Remove old commented out code from manual mysql solution
2017-03-09 00:10:50 +01:00
golderweb
6e973369cd
sqlalchemy working for parser
...
Needs some testing, presumably contains some bugs
2017-03-09 00:08:48 +01:00
golderweb
0ebf307bb8
Add markpages as subtask
...
Markpages is a subtask of our Red-Bot
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]
# The following line will be added automatically
# Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=88 FS#88]
2016-11-05 19:41:22 +01:00
golderweb
4e4d5005fd
Merge branch 'fs#89-article-status' into fs#88-mark-pages-bot
2016-11-05 19:39:30 +01:00
golderweb
65fb2ecb28
Generate Fam status based on article status
...
Some article states should be reflected in the RedFam status
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]
2016-11-05 19:27:56 +01:00
golderweb
d55c81c97b
Set article status when worked on talkpage
...
To detect whole redfam status after run over all articles
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]
2016-08-30 18:05:51 +02:00
golderweb
870ed4bf25
Update redfam.article_generator use article status
...
To be able to filter articles by status of that article
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]
2016-08-30 17:47:02 +02:00
golderweb
e13320820c
Add API to manage status per article
...
To be able to track changes to articles to update redfam status
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=89 FS#89]
2016-08-30 17:45:18 +02:00
golderweb
4ae562590e
Merge branch 'fs#94-data-structure' into fs#88-mark-pages-bot
2016-08-30 17:37:31 +02:00
golderweb