Jonathan Golder
31c10073a2
Prevent index errors searching for comments
Make sure not to exceed existing indexes of wikicode object while trying
to search for comments
Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141 )
7 years ago
Jonathan Golder
642a29b022
Improve regex for blank lines
Do not match consecutive linebreaks as one
Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141 )
7 years ago
Jonathan Golder
2f90751dc2
Merge branch 'fs#146-famhash-generator' into develop
7 years ago
Jonathan Golder
024be69fe1
Use famhash as generator
If famhash is defined, fetch explicitly that redfam from db and work
only on this
Related Task: [FS#146](https://fs.golderweb.de/index.php?do=details&task_id=146 )
7 years ago
Jonathan Golder
b6d7268a7f
select by famhash: Add methods to get param in bot
We need a method as callback to get bot specific params passed through
to our bot class.
Introduce -famhash parameter to work on specific famhash
Related Task:[FS#146](https://fs.golderweb.de/index.php?do=details&task_id=146 )
7 years ago
Jonathan Golder
526184c1e1
Merge branch 'fs#148-articles-mixed-up' into develop
7 years ago
Jonathan Golder
3aa6c5fb1c
Disable PreloadingGenerator temporarily
PreloadingGenerator mixes up yielded Pages. This is very unconvenient
for semi-automatic workflow with manual checks as the articles of the
RedFams were not following each other.
Related Task: [FS#148](https://fs.golderweb.de/index.php?do=details&task_id=148 )
7 years ago
Jonathan Golder
ec8f459db5
Merge branch 'fs#138-marked-articles-shown-again' into develop
7 years ago
Jonathan Golder
3b2cb95f36
Do not fetch marked redfams from db
Exclude marked Redfams from DB-Query to prevent marking them again
Related Task: [FS#138](https://fs.golderweb.de/index.php?do=details&task_id=138 )
7 years ago
Jonathan Golder
41e5cc1a9d
Merge branch 'fs#141-place-notice-after-comment' into develop
7 years ago
Jonathan Golder
9b9d50c4d2
Improve detection of empty lines
Search with RegEx as empty lines could also contain spaces
Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141 )
7 years ago
Jonathan Golder
a755288700
Merge branch 'fs#147-templates-in-heading' into develop
7 years ago
Jonathan Golder
14ec71dd09
Rewrite get_disc_link to handle special cases
Use methods of pywikibot site-object and mwparser to get rid of any
special elements like templates or links in headings for construction
of our disc link.
Replace by hand as it otherwise will occur as normal space and
wont work
Related Task: [FS#147](https://fs.golderweb.de/index.php?do=details&task_id=147 )
7 years ago
Jonathan Golder
e283eb78ac
Merge branch 'fs#140-also-mark-redirects' into develop
7 years ago
Jonathan Golder
cc02006fd2
Do not exclude redirects from beeing marked
In accordance with Zulu55 redirect discussion pages should also get
a notice, therefore do not exclude redirects.
Related Task: [FS#140](https://fs.golderweb.de/index.php?do=details&task_id=140 )
7 years ago
Jonathan Golder
37b0cbef08
Merge branch 'fs#138-marked-articles-shown-again' into develop
7 years ago
Jonathan Golder
4137d72468
Look for existing notice by simple in-check
To detect maybe uncommented notices already present, check for them
using just a simple python x in y check over whole wikicode
Related Task: [FS#138](https://fs.golderweb.de/index.php?do=details&task_id=138 )
7 years ago
Jonathan Golder
cd87d1c2bb
Fix already marked articles was reshown bug
Since we search for matching states for articles to include or exclude
in a loop, we could not control the outer loop via default break/
continue. Python docs recommend using Exceptions and try/except
structures to realise that most conveniently.
https://docs.python.org/3/faq/design.html#why-is-there-no-goto
Related Task: [FS#138](https://fs.golderweb.de/index.php?do=details&task_id=138 )
7 years ago
Jonathan Golder
456b2ba3d4
Merge branch 'fs#141-place-notice-after-comment' into develop
7 years ago
Jonathan Golder
47b85a0b5e
Add missing line break if there is no template
To make sure our notice template resides in its own line in every case
Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141 )
7 years ago
Jonathan Golder
a6fdc974bd
Merge branch 'fs#144-PyMySQL-instead-oursql' into develop
7 years ago
Jonathan Golder
30de2a2e12
Replace oursql with PyMySQL
Since this is prefered on toolsforge and works out of the box after
installing via pip, replace oursql which caused some problems.
Especially oursql was not able to connect to db via ssh tunnel.
Related Task: [FS#144](https://fs.golderweb.de/index.php?do=details&task_id=144 )
7 years ago
Jonathan Golder
4a6855cf7b
Merge branch 'fs#141-place-notice-after-comment' into develop
7 years ago
Jonathan Golder
8422d08cb6
Keep comments and leading templates together
Prevent spliting up existing comments and templates as often those are
documenting archiv templates behaviour
Related Task: [FS#141](https://fs.golderweb.de/index.php?do=details&task_id=141 )
7 years ago
Jonathan Golder
ed78501821
Merge branch 'fs#115-lsec-no-template' into test-v3
7 years ago
Jonathan Golder
34e7e0d3be
Prevent index Error if no template in leadsec
Check if there is a template in leadsec before accessing list item to
prevent IndexErrors
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=115 FS#115]
7 years ago
Jonathan Golder
f9f081d072
Merge branch 'fs#114-remove-underscores-in-articles' into test-v3
7 years ago
Jonathan Golder
0f930082b4
Also canonicalise anchor parts of articles
Replace spaces in anchors with underscores as spaces are not correct
there
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=114 FS#114]
7 years ago
Jonathan Golder
80c94ccf4f
Replace underscores in article titles
Remove underscores in article titles and replace with spaces to have
canonical state for all articles
Therefore we need to split title and posible anchors in heading parser
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=114 FS#114]
7 years ago
Jonathan Golder
37704c6661
Replace pywikibot.showDiff with patched version
Pywikibot.bot.userPut does not support setting the value of diff context
so it is always zero. Therefore we need to patch either userPut or
showDiff to get some context.
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=113 FS#113]
7 years ago
Jonathan Golder
4e4be1c6d0
Merge branch 'fs#110-markpages-status-problems' into test-v3
7 years ago
Jonathan Golder
3e69a1c77e
Remove problem indicating stati when set marked
Remove states which are indicating problems in previous runs if
successfully marked article and also whole RedFam
[https://fs.golderweb.de/index.php?do=details&task_id=112 FS#112]
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=110 FS#110]
7 years ago
Jonathan Golder
56f326b568
Fix error all current redfams marked when quit
Restructure update_status to make sure, marked is only set when all
articles are marked or gone (means deleted or redirect)
[https://fs.golderweb.de/index.php?do=details&task_id=111 FS#111]
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=110 FS#110]
7 years ago
Jonathan Golder
868894a38b
Format fixes
Set locale to de_DE.utf-8 for whole Task
Make sure Template is added in own source line
7 years ago
Jonathan Golder
65de6decb2
markpages: Filter redirects
Do not mark redirects discussion pages
7 years ago
Jonathan Golder
889be30a47
Merge branch 'fs#109-redpage-redfam-join-type' into test-v3
7 years ago
Jonathan Golder
147e96d388
Add Wrapperclass for Parser to RedPage
Add a wrapper class to overwrite type of Items returned by
RedPage.redfams relationship
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=109 FS#109]
7 years ago
Jonathan Golder
76666aa294
Merge branch 'fs#25-mark-done' into test-v3
7 years ago
Jonathan Golder
db39bb5ff4
Merge branch 'fs#88-mark-pages-bot' into fs#25-mark-done
7 years ago
Jonathan Golder
ec7880207b
Merge branch 'fs#95-sqlalchemy' into fs#88-mark-pages-bot
7 years ago
Jonathan Golder
4aaacf1443
Add redfams to redpage-obj after parsing
To have redfams available for updates immediately after parsing. Double
redfams then will be seen as Update.
Related Task: [https://fs.golderweb.de/index.php?do=details&task_id=108 FS#108]
7 years ago
Jonathan Golder
281f1c49a8
mysqlred: Set family via pywikibot
Get family/language part of table names from PyWikiBot Site
7 years ago
Jonathan Golder
3fe47e666f
Fix polymorphism problem with relationships
Since we are using subclasses of the ORM mapped classes, disable
typechecks for ORM relations
7 years ago
Jonathan Golder
e16925197c
Fix pep8.. compliance
To be concordant with the coding styles fix pep8 compliance
7 years ago
Jonathan Golder
9ba7d2e517
Change redfam generator filters
Change and clear up the filters in redfam generator to keep track of
article status and use positive conditionals
7 years ago
Jonathan Golder
844fee52ae
Make markpages using new DB/Class structure
Update markpages and RedFamWorker-Code to use the new sqlalchemy based
DB ORM Interface
7 years ago
Jonathan Golder
43e31c108a
Working RedFamWorker query
Modify RedfamWorker class to work with new DB API
7 years ago
Jonathan Golder
89b50e3312
Remove old status API
Now we use the methods of status object directly
7 years ago
Jonathan Golder
bf8e47f916
Improve new status API
Make sure state changes are only detected as such by sqlalchemy if they
are real changes
7 years ago
Jonathan Golder
467f829af2
Some cleanups
Remove old commented out code from manual mysql solution
7 years ago
