Repository navigation
DOC: Please describe all arguments so their functionality is clearly understood. #265
Description
Example questions include:
- What is the "mining subsystem?"
- What are the details regarding the "Continuous Integration mechanism" and how is it configured?
- How is a source determined to ever be "MALICIOUS" or "SANE?"
- Is there a way to disable "splitted" output to prevent data duplication?
- What are "Analytic" files supposed to represent?
Metadata
Metadata
Assignees
Projects
- Status
Activity
[-]DOC: Please format the documentation page so all arguments are clearly understood.[/-][+]DOC: Please describe all arguments so their functionality is clearly understood.[/+]T145 commentedon Nov 6, 2021
Analyticdescription, it sounds likesplittedalready does exactly that and adds more details. It seems like removingAnalyticoutput altogether and renamingsplittedtoanalyticwould be a simple solution. Then the analytic config option would control splitted generation.These questions are just examples to encourage thought behind how the documentation explains PyFunceble's functionality. It appears that there are options that serve more than one purpose and/or deliver multiple output artifacts, in which case what goes on needs more detail than what could be read from the variable name.
Introduction of a note about the switch mechanism.
funilrys commentedon Oct 6, 2022
Hi @T145 Thanks for the feedback. I plan to improve the documentation for the next maintenance session starting with your bullet points.
As English isn't my primary language please excuse the unclarities.
It is "documented" here.
It is called "mining" for historical reason. Back in 2017 @mitchellkrogza and I talked about what we called a "mining" mechanism for funceble.
The principle was simple: When active, PyFunceble will spawn some extra processes that will be fed with all the
ACTIVEdomains caught by the main processes. The mining process consists of registering, following, storing, and testing all domains in an HTTP redirection path.What does it mean?
Let's assume the following:
Without the "mining" mechanism you won't even know the existence of
example.org,example.deandexample.co.jp. With the mining mechanism, PyFunceble will test them and therefore produce outputs with them. In a highly automated environment, this gave us the ability to keep track of moving or new redirections behind really bad actors - back then.It is "documented" here.
This is a feature that is only relevant if you run PyFunceble within the following "CI Engines":
If PyFunceble runs within one of those "engines" and the
--cioption or its configuration counterpart is active, PyFunceble will run for around 30 minutes before stopping and committing the filesystem changes to your Git Repository. If used along the auto continue and a database, the next "run" after the commit will restart PyFunceble with the same parameter. But on the 2nd execution, PyFunceble will just try to continue where it stopped. It's a tricky feature but when your CI engine is under your control, this is a nice feature 😄That's indeed something that is not "deeply" documented. My excuse for this. It all started with the #68 issue.
A source is
MALICIOUSwhen listed in Alienvault's IP Reputation dataset.Yes! You can use the
--unified-resultsor its configuration counterpart.This is something I really need to do better ... There is room for improvement on that side.
That's also something that isn't documented enough ... It's another historical "feature".
Back in 2017-2018, I was really biased when it came to the status of the subject. I indeed was following RFCs as much as possible. But as soon as I started to work with @mitchellkrogza on bigger projects we soon realized that the technical power of mankind surpasses the power of the most powerful RFCs.
Most people think that the worldwide DNS infrastructure follows the RFCs but we had our fun back then.... When cross-checking the results of Py-Funceble with some of our very own "human inputs" we noticed for example that around ~100 domains didn't have NS records but still had A records .... That's actually impossible you may think ... Well, that's the kind of fun we had back then ...
Therefore, PyFunceble tries to follow the RFCs, but when it doesn't, a record within the
analyticoutput folder is generated so that "humans" are informed of something that PyFunceble found "weird" but is somehowACTIVEorINACTIVE.Squashed commit of the following: