# Changelog ## v6.7 (2026-03-12) ### Changes * [ail_stats] get all subtypes objects nb. [terrtia] * [ui objects] refactor object dashboard + add investigations section. [terrtia] * [chat + correlation] add function to remove chat object + fix chat correlation. [terrtia] * [retro hunt] add hunted objects status: viewed, done, rejected (fp) [terrtia] * [tracker] add tracked objects status: viewed, done, rejected (fp) + improve invalid yara-rule error message. [terrtia] * [items + chats messages] add to top sticky btn. [terrtia] * [investigation] dowmload investigation objects. [terrtia] * [message iterator] add option to iterate on chat type as sources. [terrtia] * [mail + json feeder] use meta ID for item obj_id if don't exists + mail, extract email if not default zmq. [terrtia] * [tracker] add btn to create retro hunt from tracker. [terrtia] * [tracker + retro hunt] item, only extract current tracker/retro hunt. [terrtia] * [tracker + retro hunt] item, only extract current tracker/retro hunt. [terrtia] * [stats] add fct to reset feeders names. [terrtia] * [codereader] chg gif logger level. [terrtia] * [onion lookup] remove non onion strings. [terrtia] * [show correlation] add btns: Show Direct Correlations (Level 0) + Clear All Object Types + Select All Object Types. [terrtia] * [trackers] match objs, add view pdf btn if correlation filename == 1 pdf. [terrtia] * [onion lookup] hide titles fron unsafe domains. [terrtia] ### Fix * [investigations ui] remove duplicate inf field. [terrtia] * [correlation] add missing correlations. [terrtia] * [tracker] view tracker change view event click to mousedown. [terrtia] * [chat feeder] message id of pdf and image, the channel ID was added to the global message ID. fix invalid chat correlation + image/pdf not linked in the same message in the UI. [terrtia] * [crawler] add domhash extraction timeout on malformed html. [terrtia] * [crawler] add domhash extraction timeout on malformed html. [terrtia] * [chat monitoring] fix typo. [terrtia] * [investigation] fix global datatable. [terrtia] * [investigation download] item filename. [terrtia] * [acl investigation] edit created. [terrtia] * [acl investigation] edit created. [terrtia] * [investigation] download, remove item .gz extension. [terrtia] * [mail] none source. [terrtia] * [passive ssh] except connection error. [terrtia] * [tracker] show traker to retro hunt only if yara type. [terrtia] * [item] fix extractor tracker uuid. [terrtia] * [correlation engine] load direct correlations first in the correlation graph. [terrtia] * [tracker] fix none pdf btn when multiple pdfs correlate with the same file-name. [terrtia] * [flask] fix error 500 logger for not logged users. [terrtia] * [chat importer] fix none images if size image > limit. [terrtia] * [pdf] fix translation overlapping bbox. [terrtia] ## v6.6 (2025-12-05) ### Changes * [pdf] reduce translated pdf size. [terrtia] * [pdf + translation] improve pdf translation + save translated pdf + pdf translation progress. [terrtia] * [pdf card] show Author. [terrtia] * [tracker regex] improve perf. [terrtia] * [v6.6] add update. [terrtia] * [file-name] process file-name: trackers + modules. [terrtia] * [pdf] add option to translate a PDF. [terrtia] * [file metadata] save pdf file metadata + new author object + correlation. [terrtia] * [PDF] add support for PDF Files. processing + correlation + content to markdown. [terrtia] * [crawler] crawling request, user cookiejar local storage. [terrtia] * [cookiejar] add UI local storage view + fix add cookies acl. [terrtia] * [merge] merge master. [terrtia] * [requirements] add faup-rs. [terrtia] * [users settings] add admin user view. [terrtia] * [crawler importer] fix endpoint + check required capture field. [terrtia] * [HOWTO.md] add meilisearch install. [Thirion Aurélien] * [crawler capture] add bew import to manually import crawler capture. [terrtia] * [ail_core] improve perf, enfore set usage. [terrtia] * [queue] create a new queue for tracker modules. [terrtia] * [crawler] update tor browser user agent. [terrtia] * [ail objects] add function to add custom metas to objects. [terrtia] * [chat and message viewer] show message reply forwarded_from meta. [terrtia] * [request chat monitoring] improve description. [terrtia] * [chats monitoring requests] track new requests + mark as done/rejected + delete chat monitoring requests. [terrtia] * [ail users] enable/disable user accounts. [terrtia] * [images_engine] improve domain description prompt. [terrtia] * [images_engines] improve prompts + ad prompt to detect csam domains and images. [terrtia] * [tag] add fct to check if a single tag in safe. [terrtia] * [screenshot] also tag domain if tag is unsafe. [terrtia] * [screenshot] also tag domain if tag is unsafe. [terrtia] * [cookiejar] import lacus local storage. [terrtia] ### Fix * [chat importer] None if image size > 5Mb. [terrtia] * [pdf] fix downloaded pdfa name. [terrtia] * [pdf] fix temp dir. [terrtia] * [card pdf] hide author if None. [terrtia] * [Translation module] fix launcher. [terrtia] * [Images/Screenshots] Add missing return statement in get_description_models() [David Curran] The get_description_models() method was building a list of description model names but never returning it, causing the method to implicitly return None. This fix adds the missing return statement to both Images.py and Screenshots.py. * [test] fix crawler test rel path + kwargs. [terrtia] * [trackers] fix remove object + improve url navigation. [terrtia] * [pdf] fix max_size_config. [terrtia] * [tracker] add space between : and url. [terrtia] * [pdf] fix max_pdf_size config name. [terrtia] * [pdf] fix pdf limit size to 100 mb by default. [terrtia] * [correlation] fix file-name -> pdf correlation. [terrtia] * [pdf] fix empty author set. [terrtia] * [file-name] fix base url. [terrtia] * [file-name] fix tracker file-name filtering + template. [terrtia] * [v6.6 update] update pylacus. [terrtia] * [Global + Tracker term] fix warning + content of invalid text mimetype. [terrtia] * [pdf template] add missing template. [terrtia] * [api] remove crawler duplicate cookiejar importer. [terrtia] * [feeders dashbard] fix feeders stats. [terrtia] * [pdf] fix correlation show object. [terrtia] * [tracker decoded] decoded term amd regex tracker, Ensure only string content is processed. [terrtia] * [language] gcld3 use deprecated language code. [terrtia] * [import crawler capture] fix API response. [terrtia] * [queue] send objs in global module to trackers queue. [terrtia] * [crawler] fix cache domain dashboard list. [terrtia] * [crawler] fix cache domain dashboard list. [terrtia] * Fix: [crawler] fix new capture status @rafiot. [terrtia] * [message viewer] fix forward meta. [terrtia] * [message viewer] fix forward meta. [terrtia] * [chat viewer] some forwarded_from meta were missing. [terrtia] * [images_engine] LLM do not respect specified case. [terrtia] * [images_engine] fix reponse dict. [terrtia] * [images_engine] LLM do not fully respect requested anwsers. [terrtia] * [domain explorer] fix i2p domain explorer pagination. [terrtia] ### Other * Chg [pdf] improve translation layout. [terrtia] * Merge branch 'master' of github.com:ail-project/ail-framework. [terrtia] * Merge pull request #312 from cavedave/fix/images-screenshots-issues-v2. [Thirion Aurélien] fix: [Images/Screenshots] Add missing return statement in get_descrip… * Merge branch 'master' of github.com:CIRCL/AIL-framework. [terrtia] * Merge pull request #614 from cavedave/feature/api-crawler-tests. [Thirion Aurélien] Add some more tests for crawler API endpoints * Fix GitHub Actions workflow: add disk cleanup and optimize matrix. [David Curran] - Add disk cleanup step to prevent 'No space left on device' errors * Clears APT cache and lists * Removes unused tools (dotnet, ghc, android SDK) * Removes hosted tool cache (safe since workflow uses system Python) - Comment out unused Python version matrix (was creating 4 identical jobs) * Makes tests 4x faster without changing results * Add TODO with instructions on how to enable multi-version testing properly - All cleanup operations check for directory existence before removing * Add some more tests for crawler API endpoints. [David Curran] - Add tests/test_api_crawler.py with 10 test cases covering: * POST /api/v1/add/crawler/task endpoint (success, errors, auth, validation) * POST /api/v1/add/crawler/capture endpoint (success, errors, auth) * Authentication and authorization scenarios * Input validation and error handling - Update bin/LAUNCH.sh to include test_api_crawler in test runner - All tests use Flask test client with mocked dependencies for isolation * Merge branch 'master' into local_storage. [terrtia] * Merge pull request #290 from ail-project/dependabot/npm_and_yarn/var/www/static/codemirror/codemirror-yara/vite-6.4.1. [Steve Clement] build(deps-dev): bump vite from 6.3.6 to 6.4.1 in /var/www/static/codemirror/codemirror-yara * Build(deps-dev): bump vite in /var/www/static/codemirror/codemirror-yara. [dependabot[bot]] Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 6.3.6 to 6.4.1. - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/create-vite@6.4.1/packages/vite) --- updated-dependencies: - dependency-name: vite dependency-version: 6.4.1 dependency-type: direct:development ... * Merge branch 'master' into local_storage. [terrtia] * Merge branch 'master' of github.com:ail-project/ail-framework. [terrtia] * Merge pull request #288 from alexandercronin/fix/typo_in_hunter_tamplates. [Thirion Aurélien] fix typo `ans` to `and` * Fix typo `ans` to `and` [Alex Cronin] ## v6.5 (2025-09-29) ### New * [crawler] add I2P crawler + auto discovery crawler. [terrtia] * [search engine] search description of images, screenshots and domain. [terrtia] * [search engine] add description indexes of domains, images and screenshots. [terrtia] * Add support for kvrocks installed from the deb package. [Raphaël Vinot] * Skip SSL context in Flask if the cert/keys are missing. [Raphaël Vinot] ### Changes * [v6.5] add v6.5 update. [terrtia] * [correlation engine] fct to search for ids in one depth correlation, return intermediate obj. [terrtia] * [correlation engine] add fct to search for ids in one depth correlation. [terrtia] * [i2p crawler] filter I2P Website Unreachable and Website Unknown - Website Not Found in Addressbook domains. [terrtia] * [i2p crawler] filter I2P Website Unreachable and Website Unknown - Website Not Found in Addressbook domains. [terrtia] * [psl_faup] add support for additionnal tlds like b32.i2p + improve performance. [terrtia] * [settings] show numbers of active orgs + numbers of logged and active users. [terrtia] * [image engine] add fct to describe all images. [terrtia] * [image description] index iamge and domain description. [terrtia] * [search engine images description] show image. [terrtia] * Move yara install from var/www/update_thirdparty.sh to installing_deps.sh. [Raphaël Vinot] ### Fix * [crawler] i2p localhost redirection. [terrtia] * [search domain by daterange] fix UI domain type selector. [terrtia] * [Onion module] fix already crawled domains. [terrtia] * [Onion module] fix module. [terrtia] * [psl_faup] set IP address as domain. [terrtia] * [crawler dashboard] fix html tag. [terrtia] * [crawler dashboard] improve stats crawler type layout. [terrtia] * [i2p crawler] only filter errored I2P pages + fix i2P crawler stats. [terrtia] * [i2p crawler] filter I2P Invalid Destination. [terrtia] * [i2p crawler] filter I2P Website on domhash. [terrtia] * [i2p crawler] filter I2P Website on domhash. [terrtia] * [image engine] fct to describe all images. [terrtia] * [image engine] fct to describe all images. [terrtia] * [search engine images description] fix pagination + domain image. [terrtia] * [search engine images description] fix domain image. [terrtia] * [search engine images description] fix template. [terrtia] * [search engine image description] fix obj link baseurl. [terrtia] * [search engine image description] fix obj link. [terrtia] * [search engine] fix blueprint redirect. [terrtia] * [search engine] search description of images, screenshots and domain. [terrtia] * [search engine] fix indexs name. [terrtia] * [search engine] fix indexs name. [terrtia] * [test] fix API/UI test. [terrtia] * [test] fix API/UI test. [terrtia] * [doc] fix pystemon config dir name. [terrtia] ### Other * Merge pull request #286 from ail-project/dependabot/npm_and_yarn/var/www/static/codemirror/codemirror-yara/vite-6.3.6. [Thirion Aurélien] build(deps-dev): bump vite from 6.3.5 to 6.3.6 in /var/www/static/codemirror/codemirror-yara * Build(deps-dev): bump vite in /var/www/static/codemirror/codemirror-yara. [dependabot[bot]] Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 6.3.5 to 6.3.6. - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/v6.3.6/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/v6.3.6/packages/vite) --- updated-dependencies: - dependency-name: vite dependency-version: 6.3.6 dependency-type: direct:development ... * Merge branch 'master' of github.com:ail-project/ail-framework. [terrtia] * Merge pull request #280 from ail-project/kvrocks-deb. [Thirion Aurélien] new: Add support for kvrocks installed from the deb package * Merge pull request #281 from ail-project/skip_sslcontext. [Thirion Aurélien] new: Skip SSL context in Flask if the cert/keys are missing * Merge pull request #282 from ail-project/move_yara_install. [Thirion Aurélien] chg: Move yara install from var/www/update_thirdparty.sh to installing_deps.sh ## v6.4 (2025-08-22) ### Changes * [v6.4] add v6.4 update. [terrtia] * [crawler] add function to recrawl onion by month or to crawl all onion. [terrtia] * [cve] migrate UI cve info to vulnerability lookup. [terrtia] * [module extractor] improve email extraction. [terrtia] * [search mail] increase cache. [terrtia] * [mail search] improve search perf, cache search, paginate mail domain search. [terrtia] * [zmq importer] display triggered filter. [terrtia] * [zmq importer] add filter content. [terrtia] * [zmq importer] add option to filter by file start/end. [terrtia] * [absteact object] improve id iterator perf. [terrtia] * [search mail] refector mails search: Domain + user fragment, User + optional domain fragment, Exact address, Domain fragment only. [terrtia] * [language] add function to ping libretranslate. [terrtia] * [crawlers/dashboard] Crawler is now Lacus. [Alexandre Dulaunoy] ### Fix * [thehive] adapt to unexpected breaking change. [terrtia] * [crawler recrawl] fix last parent. [terrtia] * [cve] fix typo. [terrtia] * [cve] fix UI cve vulnerability lookup url. [terrtia] * [module extractor] limit email extraction. [terrtia] * [module extractor] limit email extraction. [terrtia] * [zmq importer] fix content filters. [terrtia] * [language] improve exception handling of unknown iso1 language code. [terrtia] * [onion lookup] fix parsing of invalid url. [terrtia] * [onion lookup] fix parsing of invalid url. [terrtia] * [onion lookup] fix parsing of invalid url. [terrtia] ### Other * Merge branch 'master' of github.com:ail-project/ail-framework. [terrtia] ## v6.3 (2025-07-16) ### Changes * [flask] log 500 errors user_id. [terrtia] * [language translation] do not sent unsupported languages to libretranslate. [terrtia] * [Onion module] improve perf. [terrtia] * [Onion module] improve perf. [terrtia] * [Onion module] improve perf. [terrtia] * [Onion module] improve perf. [terrtia] * [Onion module] print task uuid only id a new task is created. [terrtia] * [Onion module] reduce duplicate check. [terrtia] * [Onion module] reduce duplicate check. [terrtia] * [faup] replace and remove faup. [terrtia] * [ip] print deanomized hosts. [terrtia] * [ip] add ip object and ssh key correlation. [terrtia] * [qrcode extractor] extract color inverted qrcode. [terrtia] * [ssh-key] add ssh-key object and domains correlation + passive ssh search + passive ssh lookup. [terrtia] * [images engines] create domains descritpions, improve progress. [terrtia] * [images engines] create domains descritpions, print progress. [terrtia] * [stats] new fct, get ail stats. [terrtia] * [ZMQImporter] improve module prints. [terrtia] * [ZMQImporter] Add filter by feeder_name, item content is filtered if each line starts and ends with the specified pattern. [terrtia] * [tracker] add option and cache to prevent duplicate notifications on the same content. [terrtia] * [api] get onions up by month. [terrtia] * [tracker] add heatmap number of matches by years. [terrtia] * [images engine] add function to describe domains. [terrtia] * [image engine] describe domain. [terrtia] ### Fix * [v6.3] do not shutdown DB. [terrtia] * [crawler psl_faup] fix invalid parsing of domain without scheme. [terrtia] * [object sidebar] fix ip icon. [terrtia] * [v6.3] add v6.3 update. [terrtia] * [search domain name] fix domain extraction. [terrtia] * [crawler dashboard] none capture meta. [terrtia] * [unpack_url] fix defferent behaviour between faup and psl_faup. [terrtia] * [language] add be code. [terrtia] * [Onion module] fix invalid url. [terrtia] * [Onion module] fix exception if extracted domain is None. [terrtia] * [psl_faup] fix invalid port number exception. [terrtia] * [CE Detector] fix module message. [terrtia] * [CE Detector] debug. [terrtia] * [CE Detector] avoid retagging. [terrtia] * [CE Detector] avoid retagging. [terrtia] * [faup] remove faup from crawler. [terrtia] * [retro hunt] delete retro hunt meta. [terrtia] * [retro hunt] delete retro hunt meta. [terrtia] * [correlation] fix ip correlation. [terrtia] * [card ip] remove sparkline. [terrtia] * [correlation] add ip to default list. [terrtia] * [ssh-keys] remove sparkline. [terrtia] * [ip] fix get ssh key ip. [terrtia] * [ip] fix get ssh key ip. [terrtia] * [ip] fix get ssh key ip. [terrtia] * [retro-hunt] fix objects filters. [terrtia] * [message translation] fix list of translation languages + chat messages by language and user by messages by language translation. [terrtia] * [manual message language] languages not translated by libretranslate. [terrtia] * [manual message language] select lang from the full list of languages. [terrtia] * [ssh key] fix template title. [terrtia] * [ssh-key] ssh-key object sidebar. [terrtia] * [passive ssh] do not print 404. [terrtia] * [passivessh] add missing image. [terrtia] * [Host + Domclassifier] lower domains to avoid duplicate. [terrtia] * [ZMQImporter] filtered content string. [terrtia] * [images engine] typo. [terrtia] * [images engine] typo. [terrtia] * [image engine] fix domain description image type. [terrtia] * [updater] remove debug. [terrtia] * [updater] fix tag subversion. [terrtia] ## v6.2 (2025-05-28) ### New * [yara-editor] Added codemirror editor with Yara rule support. [Sami Mokaddem] * [images_engine] get screenshot description with ollama. [terrtia] ### Changes * [v6.2] add v6.2 update. [terrtia] * [image engine] add image description + save image description by model + fix screenshot correlation description. [terrtia] * [search mail + gtracker] improve perf. [terrtia] * [create user] send mail to new a user. [terrtia] * [create user] debug send mail to new user. [terrtia] * [create user] test send mail to new user. [terrtia] * [search emgine] improve search dashboard + add search helper. [terrtia] * [images engine] save image description in DB. [terrtia] * [search] add search chats messages. [terrtia] * [search] new crawled search with meilisearch+ add new search dashboard. [terrtia] * [module queue] show FeederModuleImporter Nb in Queue. [terrtia] * [trackers] move create new tracker btn to top. [terrtia] * [doc] add tracker doc. [terrtia] * [language] remove special characters before language detection. [terrtia] * [show message] add message card message subchannels. [terrtia] * [show message] add message card + protocol. [terrtia] * [retro-hunt] new endpoints: generate retro-hunt report for chat messages + ocrs (from messages) [terrtia] * [correlation] add card file-name, mail and gtracker. [terrtia] * [reprocess] add TrackingId module. [terrtia] * [google tracking] add googke tracking module + object. [terrtia] * [mail] debug punycode encoding. [terrtia] * [mail] add new mail object + correlation chats/domains/crawled. [terrtia] * [LanguageDetector] improve perf. remove old lexilang detector. [terrtia] * [language] fix template title. [terrtia] * [language] refactor language engine + get chat messages by language + get user messages by language + improve lexiang detection. [terrtia] * [settings module] add stats modules. [terrtia] * [chat viewer] add languages stats. [terrtia] * [chat participants] show the numbers of messages by participant in a chat. [terrtia] * [reprocess tools] reprocess object message by tags. [terrtia] * [chats] add corelation chat->cve, cryptocurrencies, ... + correlation domain->chat, message. [terrtia] * [MailExporter] debug: add EHLO. [terrtia] * [chat feeder] add matrix feeder. [terrtia] * [domain search] add btn to crawl unknown onion. [terrtia] * [settings + misp export] add new user config engine + Users can create and save MISP accounts/API keys to seamlessly export data to MISP or generate a JSON file. [terrtia] * [trackers dashboard] change Trackers last matches table. [terrtia] * [login OTP] increase valid window by 1. [terrtia] * [user settings HOTP] add button to print the next 50 HOTP tokens. [terrtia] ### Fix * [image engine] fix image description tooltip. [terrtia] * [image engine] fix image description tooltip. [terrtia] * [image engine] fix image description tooltip. [terrtia] * [image engine] fix image description tooltip. [terrtia] * [image engine] fix image description tooltip. [terrtia] * [image engine] fix image description tooltip. [terrtia] * [crawler dashboard] fix none task capture time. [terrtia] * [translation] fix translation bg,el,hi,ja,zh + user chat messages translate messages. [terrtia] * [translation] fix russian translation. [terrtia] * [misp export] show error if no misp instance is specified. [terrtia] * [mail exporter] fix circular import. [terrtia] * [mail search] fix empty mail content. [terrtia] * [search emgine] add missing sidebar. [terrtia] * [item] fix show item meta. [terrtia] * [misp export] fix empty relation error. [terrtia] * [installing_deps.sh] remove portable mode for default install. [Thirion Aurélien] * [flask] Removed debug. [Sami Mokaddem] * [installing_deps.sh] remove portable mode for default install. [Thirion Aurélien] * [install] Fixed installion, init submodules and added missing python dep. [Sami Mokaddem] * [images engine] fix none screenshot_id. [terrtia] * [search] update requirements + fix search card. [terrtia] * [block_img_ollama] fix image description css. [terrtia] * [block_img_ollama] show image description btn. [terrtia] * [settings] fix module template if no modules lanched. [terrtia] * [modules queues] remove old unused queue. [terrtia] * [doc] fix anchors. [terrtia] * [api] fix crawl url check proxy default web. [terrtia] * [module] only use config file. TODO: move to UI and rename config field name. [terrtia] * [module] send onion to crawler if lacus is connected and activate_crawler is set to False. [terrtia] * [correlation graph] urlencode object id. [terrtia] * [show message] empty chat first/last seen. [terrtia] * [show message] empty card. [terrtia] * [language] add missing gcld3 iso1 code + get chat instance lamguages stats. [terrtia] * [language] add missing gcld3 iso1 code + get chat instance lamguages stats. [terrtia] * [mail] filter mail containing % to avoid issue with url encoded mails. [terrtia] * [translation] fix libretranslate iso code before translation. [terrtia] * [mail] UnicodeError. [terrtia] * [mail] limit max mail length. [terrtia] * [language] fix manual language selector, add missing language codes. [terrtia] * [Updater] fix submodule update. [terrtia] * [pytaxonomie] fix missing uuid in new pytaxoonmie version, bump taxonomies + galaxies. [terrtia] * [crawlers] fix domain onion cache. [terrtia] * [crawler] fix update_last_crawled_domain domain id. [terrtia] * [chat viewer] remove unused endpoint. [terrtia] * [domain search] check if onion is valid before sending to spider. [terrtia] * [domain search] fix typo. [terrtia] * [mail module] reduce disk usage, remove faup. [terrtia] * [username search] check if username is valid + add missing discord subtype. [terrtia] * [investigation misp export] fix JSON export. [terrtia] * [module extractor] fix object value extraction + invalid retrohunt. [terrtia] * [create user] set default user role to user by default. [terrtia] * [item blur screenshot] check domain tags. [terrtia] ### Other * Merge branch 'master' of github.com:ail-project/ail-framework into langs. [terrtia] * Merge pull request #273 from ail-project/feature/yara-editor. [Thirion Aurélien] Added YARA text editor support * Merge pull request #272 from ail-project/fix/install. [Thirion Aurélien] fix: [install] Fixed installation, init submodules and added missing python dep * Merge branch 'master' into langs. [terrtia] * Merge pull request #267 from aaronkaplan/master. [Thirion Aurélien] fix #266 * Fix #266. [Aaron Kaplan] * Merge branch 'master' of github.com:ail-project/ail-framework. [terrtia] * Merge pull request #260 from NMD03/master. [Thirion Aurélien] Fix [Lacus] new dependencies * Fix [Lacus] new dependencies. [NMD03] ## v6.1 (2025-02-06) ### Changes * [v6.1] add v6.1 update. [terrtia] * [crawler] add option to controls whether the crawler should proceed with crawling onion domains that have not yet been classified as safe or unsafe. [terrtia] * [Filter unsafe onion] add a new unsafe onion filter option. [terrtia] * [show tracker] add btn to hide/show long rule. [terrtia] * [module mail ectractor] remove DNS check for UI extraction. [terrtia] * [doc] add roles functionality. [terrtia] * [auto-update] update submodules. [terrtia] * [file-names] search file by name + fix search username case sensitive. [terrtia] * [files names] add file name dashboard. [terrtia] * [show item] show file-name and message content. [terrtia] * [chat explorer] show files tags. [terrtia] * [mail exporter] add object url only if email is ail user. [terrtia] * [module extractor] add link to tracker. [terrtia] * [process chat text] process text files from chats + correlations. [terrtia] * [crawler] update blocklist. [terrtia] * [module extractor] signal timeout global extraction + reduce regex and yara timeout. [terrtia] * [gitchangelogrc] match sub version of tags. [Alexandre Dulaunoy] * [tracker] collapse long rule. [terrtia] * [retro hunt] show object match last seen date. [terrtia] ### Fix * [retro hunt] fix retro hunt item restart. [terrtia] * [MailExporter] correct typo. [Jean-Louis Huynen] * [show tracker] add btn to hide/show long rule. [terrtia] * [show tracker] add btn to hide/show long rule. [terrtia] * [doc] fix scheduler role. [terrtia] * [chat explorer] fix show codebar message content. [terrtia] * [create user] fix template error if an invalid password is provided. [terrtia] * [module extractor] remove SIGALRM usage to prevent Flask server termination. [terrtia] * [view org] fix sidebar. [terrtia] * [get str item content] fix some weird encoding issue. [terrtia] * [message file] remove file link if the file is not downloaded. [terrtia] * [api] fix add crawler task user_org. [terrtia] ### Other * Merge branch 'master' of github.com:ail-project/ail-framework. [terrtia] ## v6.0.1 (2025-01-23) ### Changes * [update v6.0.1] login lowercase. [terrtia] * [uodate v6.0.1] add update version. [terrtia] * [taxomomies + galaxy] update. [terrtia] * [show tracker] filter result by object type. [terrtia] * [updater] use tag subversion, ex:v6.0.1. [terrtia] * [crawler] improve queued error log. [terrtia] * [retro hunt + update] add ocr retro hunt + new update. [terrtia] * [user] force lowercase + add v6.0.1 update. [terrtia] * [chat explorer] show message, open message in chat/subchannel/thread. [terrtia] * [flask] update Flask_config.py. [Ikko Eltociear Ashimine] accross -> across * [chat explorer] show chats tags. [terrtia] * [flask] set proxy ProxyFix. [terrtia] * [test] log if flask is reachable. [terrtia] * [user-account viewer] add messages heatmap by year. [terrtia] * [chat viewer] heatmap messages, add year selector. [terrtia] * [chat viewer] add chat messages by current year heatmap. [terrtia] * [message description] add chat name and username. [terrtia] * [chat explorer] improve basic chat card, add chat particpants btn + improve btn css. [terrtia] * [cryptocurrency] add ripple address subtype + correlation. [terrtia] * [misp taxonomies galaxy] bump. [terrtia] * [blur unsafe images] blur violence and pornography-illicit-or-illegal tags + add warning message. [terrtia] * [search domain by name] sanityze domain name + show domain. [terrtia] * [README] update dashboard image. [terrtia] ### Fix * [ail updater] remove update between tags. [terrtia] * [ail updater] fix upper tags list + is_fork. [terrtia] * [show Domain] unblur default image if domain is down. [terrtia] * [module extrsctor] log invalid object meta. [terrtia] * [module extrsctor] fix invalid object meta. [terrtia] * [retro hunt] fix object to resume. [terrtia] * [retro hunt] fix item iterator. [terrtia] * [delete retro hunt] force pause state before delete. [terrtia] * [retro hunt] fix retro hunt resume. [terrtia] * [crawler] increase timeout QUEUED captures. [terrtia] * [crawler] increase timeout QUEUED captures. [terrtia] * [crawler] timeout QUEUED captures. [terrtia] * [crawler] title extraction, sigalarm raised by signal.alarm and sleep. [terrtia] * [crawler] debug signal timeout. [terrtia] * [crawler] log timeout + debug signal timeout. [terrtia] * [title beautifullsoup] sigalrm. [terrtia] * [title beautifullsoup] add signal, BeautifulSoup html.parser is stuck. [terrtia] * [crawler] crawler queued capture loop. [terrtia] * [crawler] debug crawler queued capture loop. [terrtia] * [chat messages by year] fix subchannel nb_max. [terrtia] * [chat forum] fix get years daterange. [terrtia] * [chat explorer] fix protocols name list order. [terrtia] * [crawler] fix reload_crawlers_stats queues stats. [terrtia] * [exifs module] Mp4 UnidentifiedImageError. [terrtia] * [investigation] add object with spaces in id to an investigation. [terrtia] * [module_extractor] fix onion extraction. [terrtia] * [module_extractor] don't extract onion from crawled items. [terrtia] * [crawler] avoid crawler loop if a capture end up in an invalid state. [terrtia] ### Other * Merge branch 'master' of github.com:ail-project/ail-framework. [terrtia] * Merge pull request #253 from eltociear/patch-1. [Thirion Aurélien] chg: [flask] update Flask_config.py * Merge pull request #250 from FafnerKeyZee/patch-1. [Thirion Aurélien] Update Tracker.py * Update Tracker.py. [Fafner [_KeyZee_]] If self._get_field('level') is None, trying to convert it into int will crash the system :) 2024-12-23 14:22:46,232 Flask_server ERROR:Exception on /trackers [GET] Traceback (most recent call last): File "/home/user/ail-framework/AILENV/lib/python3.12/site-packages/flask/app.py", line 1511, in wsgi_app response = self.full_dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/ail-framework/AILENV/lib/python3.12/site-packages/flask/app.py", line 919, in full_dispatch_request rv = self.handle_user_exception(e) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/ail-framework/AILENV/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request rv = self.dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/ail-framework/AILENV/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/ail-framework/AILENV/lib/python3.12/site-packages/flask_login/utils.py", line 290, in decorated_view return current_app.ensure_sync(func)(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/ail-framework/var/www/Role_Manager.py", line 81, in decorated_view return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/user/ail-framework/var/www/blueprints/hunters.py", line 74, in trackers_dashboard trackers = Tracker.get_trackers_dashboard(user_org, user_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/ail-framework/bin/lib/Tracker.py", line 785, in get_trackers_dashboard if not tracker.check_level(user_org, user_id): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/ail-framework/bin/lib/Tracker.py", line 220, in check_level level = self.get_level() ^^^^^^^^^^^^^^^^ File "/home/user/ail-framework/bin/lib/Tracker.py", line 183, in get_level level = int(self._get_field('level')) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Merge pull request #251 from FafnerKeyZee/patch-2. [Thirion Aurélien] Update abstract_chat_object.py * Update abstract_chat_object.py. [Fafner [_KeyZee_]] https://github.com/ail-project/ail-framework/commit/9a388dc9cb05eec9f612343fcdf4579b18bf1e00 broke the Forum view, we can not do stat on forum, only on subchan :) * Merge pull request #249 from vncloudsco/master. [Alexandre Dulaunoy] Update Install silent and Dockerfile update * Update Install silent and dockerfile update. [Mạnh Tường Solo] ## v6.0 (2024-12-09) ### Changes * [v6.0] add v6.0 update script. [terrtia] * [pubsublogger] remove old redis_logger. [terrtia] * [dashboard] filter submission tags. [terrtia] * [dashboard] show tracker description. [terrtia] * [queue] save module start time + pid. [terrtia] * [dashboard] add day date. [terrtia] * [flask] remove old modules + modules loader. [terrtia] * [dashboard] add crawler stats. [terrtia] * [dashboard] add objects tooltip. [terrtia] * [dashboard] nb objects per days + websocket update. [terrtia] * [dashboard] add echart feeder graph + cleanup olg graph libs + dashboard, show today nb objects. [terrtia] * [flask modules] remove old modules. [terrtia] * [fontawesome] v6.6.0 migration. [terrtia] * [barcode] add and extract barcode object. [terrtia] ### Fix * [dashboard] fix day format. [terrtia] * [dashboard] fix feeeder names. [terrtia] * [retro hunts] remove objects. [terrtia] * [retro hunt] fix add tags. [terrtia] * [object item] fix show item url. [terrtia] * [barcode] fix message card. [terrtia] * [show organisation] fix sidebar. [terrtia] ## v5.9 (2024-10-18) ### Changes * [qrcodes] daterange search: sort qrcode by content. [terrtia] * [dom-hash] add dom-hash object compute dom-hash for domains and crawled items. [terrtia] * [CEDetector] tag domains. [terrtia] * [CEDetector] add detect message. [terrtia] * [titles] setup Titles queues + CEDetector module. [terrtia] * [tools] add reprocess title + CEDetector. [terrtia] * [crawler] disable unsafe title auto tagging. [terrtia] * [module] debug. [terrtia] * [module] add CEDetector. [terrtia] * [crawler] tag domain by vanity. [terrtia] * [crawler] crawl list urls: filter duplicates. [terrtia] * [crawler] submit free text of urls to crawl. [terrtia] * [onion module] filter onion v2. [terrtia] * [show domain] show titles content + fix item tags. [terrtia] * [crawler] update tor user agent. [terrtia] * [message image] show qrcodes extracted. [terrtia] * [domain lookup] extract domain from url input. [terrtia] * [api] rename domain lookup. [terrtia] ### Fix * [module] typo. [terrtia] * [tag] tag delete: fix update first/last seen. [terrtia] * [show domain] fix correlation button if nb correl == 0. [terrtia] * [crawler] filter lookup tags. [terrtia] * [crawler] filter lookup parent + domain daterange. [terrtia] ## v5.8 (2024-10-03) ### Changes * [tags search] search qrcode by tags. [terrtia] * [api] add domain lookup. [terrtia] * [chats explorer] image lazy loading. [terrtia] * [favicon] favicons by daterange, lazy loading. [terrtia] * [qrcode] improve qrcode extractor + add v5.8 update. [terrtia] * [debug] debug empty qr content. [terrtia] * [qrcode] extract qrcode content from images and screenshots + qrcode object + correlation. [terrtia] * [hunter sidebar] allign icons. [terrtia] * [show item] replace canevas by image blur + add a button to wrap text. [terrtia] * [create edit user] add org selector search. [terrtia] * [username] add username search. [terrtia] * [tags search] show last tag(s) date if no result. [terrtia] * [show domain screenshots] replace canevas by img blur. [terrtia] ### Fix * [qrcode extractor] fix catch cv2 exception. [terrtia] * [qrcode extractor] fix empty qreader. [terrtia] * [qrcode extractor] filter code39. [terrtia] * [qrcode extractor] fix image/screenshot tag. [terrtia] * [bitcoin bech32] fix address validator. [terrtia] * [qrcode extractor] filter invalid image. [terrtia] * [qrcode extractor] filter invalid image. [terrtia] * [qrcode extractor] debug. [terrtia] * [qrcode extractor] fix invalid jpeg. [terrtia] * [qrcode install] add missing lib. [terrtia] * [mail module] fix typo. [terrtia] * [tags] fix invalid tag obj_id function. [terrtia] * [crwaler dashboard] fix current day domains up/down links. [terrtia] * [investigation] fix object table long object id. [terrtia] * [v5.7 update] add missing python packages. [terrtia] * [show domain] avoid overlaping table/image. [terrtia] * [crawler barchart] up/down colors. [terrtia] ## v5.7 (2024-09-17) ### Changes * [submodule] update. [terrtia] * [module] disable indexer queue. [terrtia] * [orgs list] show numbers of users. [terrtia] * [modules] disable LibInjection. [terrtia] * [chat request] add other chat type + placeholders. [terrtia] * [chats] show username. [terrtia] * [flask] use the same cookie name + cache git commits, tags and branch name. [terrtia] * [user roles] rename coordinator as org_admin. [terrtia] * [search] disable whoosh index and search. [terrtia] * [create user] new account qr code. [terrtia] * [create user] new account qr code. [terrtia] * [trackers] check acl of last trackers matches. [terrtia] * [ocr] ocr card, add first seen and last seen. [terrtia] * [2fa] add instance name in config. [terrtia] * [MISP Export] filter None MISP objects. [terrtia] * [chats explorer] add chat monitoring request. [terrtia] * [access logs] add user_agent. [terrtia] * [access logs] add user_agent. [terrtia] * [UI] abort 403 and 404. [terrtia] * [cookiejars] show organisation. [terrtia] * [trackers + retro_hunts] show organisation. [terrtia] * [investigations] show organisation. [terrtia] * [UI] replace remote_addr by access_route. [terrtia] * [user profile] show user organisation. [terrtia] * [UI + API] update endpoints ACL. [terrtia] * [acl] refactor acl cookiejars, trackers, retro_hunts, investigation + refactor users roles. [terrtia] * [update] create default org + update trackers/retro_hunts/investigations creator_org. [terrtia] * [items] check obj invesigations acl. [terrtia] * [module extractor] check tracker and retro hunt acl. [terrtia] * [user] refactor user creation and edit + edit user organisation. [terrtia] * [trackers + retro_hunts] add org level to retro_hunt + update acl to support org + refactor trackers org. [terrtia] * [cookiejar] add org level to cookiejar + update acl to support org. [terrtia] * [ail orgs] refactor organisations + add org level to investigations. [terrtia] * [core] add users organisation + tracker acl by organisation. [terrtia] * [logs] add access logs. [terrtia] * [ail users] cleanup Role_Manager. [terrtia] * [ail users] save last api usage. [terrtia] * [tests] improve test output, show all running screens. [terrtia] * [ail users] remove old User lib + improve API test. [terrtia] * [crawler] add button to manually delete lacus queues. [terrtia] * [misp-taxonomies] updated. [Alexandre Dulaunoy] * [misp-galaxy] updated to the latest version. [Alexandre Dulaunoy] * [chat viewer] show message forwarded from. [terrtia] * [usernames timeline] remove empty usernames. [terrtia] * [chats explorer] filter Chats that don't contain messages. [terrtia] * [user-account] chat user-accounts: show usernames list + usernames timeline. [terrtia] * [dashboard] replace interval requests by event stream. [terrtia] * [update] add v5.7 update. [terrtia] * [users] manually logout users/kill user session + add users meta:creator, creation date, last_edit, last_login, last_seen, is_logged. [terrtia] * [2FA] user + admin manage 2FA OTP. [terrtia] * [login] add 2FA TOTP and HOTP. [terrtia] * [chord] add parameters to filtersby min and max value. [terrtia] * [chats viewer] add relationships buttons. [terrtia] * [relationships] add messages mentions -> chats + user-accounts. [terrtia] * [user-account] show chord diagram, Numbers of Messages Posted by Chat. [terrtia] * [d3 heatmap] v7 migration + fix tooltip. [terrtia] * [circos] add number of IN/OUT message in tooltip. [terrtia] * [d3] get last D3 js version. [terrtia] * [relationships] messages foraward between chats/user-account + chat chord diagram. [terrtia] * [relationship] get object relationhips by relationship name + object types. [terrtia] * [crawler] add function to delete schedules. [terrtia] * [crawler stats] add previous month stats by domain type. [terrtia] * [user message] show user messages by chat. [terrtia] * [settings] refactor old settings blueprint module. [terrtia] * [settings] refactor UI settings. [terrtia] * [user session] refactor login + AIL users, use alternative IDs, kill user(s) session. [terrtia] ### Fix * [crawler] fix crawler queue stats. [terrtia] * [crawler] fix crawler queue stats. [terrtia] * [crawler] fix crawler queue stats. [terrtia] * [delete user] remove user from org. [terrtia] * [crawler] crawler capture with empty task. [terrtia] * [user] fix role edit. [terrtia] * [user] user check. [terrtia] * [object subtype dashboard] fix None first/last seen. [terrtia] * [objects] get_object, None pgp name. [terrtia] * [objects] get_object, filter None string. [terrtia] * [trackers] typo. [terrtia] * [trackers] blueprint user_role. [terrtia] * [2fa] fix name. [terrtia] * [org] fix trackers None orgs. [terrtia] * [update v5.7] create roles list. [terrtia] * [MISP Export] fix Decoded objects export. [terrtia] * [user] fix users self password change. [terrtia] * [users settings] fix user edit. [terrtia] * [tracker] firce delete objects filter on edit. [terrtia] * [showDomain] typo fixed. [Alexandre Dulaunoy] * [debug] [terrtia] * [user-account] fix chord graph tooltip. [terrtia] * [trackers] fix webhook export. [terrtia] * [dashboard] Filter out dashboard event-stream SSL.SSLEOFError from the log if a client disconnects. [terrtia] * [trackers] fix tracker view objects acl for global trackers. [terrtia] * [trackers] fix tracker view objects acl for global trackers. [terrtia] * [trackers] fix post filters selector + fix tracker webhooks creation + fix set edit. [terrtia] * [cookiejar] fix add cookie cookiejar uuid. [terrtia] * [relationships] objects link. [terrtia] * [user-account] fix last username timestamp. [terrtia] ### Other * Merge branch 'master' into otp. [terrtia] * Merge branch 'master' into otp. [terrtia] * Merge branch 'master' of github.com:ail-project/ail-framework. [terrtia] * Update README.md. [Alexandre Dulaunoy] * Merge branch 'relationships' into otp. [terrtia] * Merge branch 'master' into relationships. [terrtia] * Merge branch 'master' into otp. [terrtia] ## v5.6 (2024-05-10) ### Changes * Chg: [tron] add TRON crytocurrency detection + correlation. Thanks @pventuzelo for the contribution. [terrtia] * [tags] search ocrs and images by tags + fix ocr, filter invalid image. [terrtia] * [correlation graph] add date first/last seen separator + fix display of ocr object tags. [terrtia] ### Fix * [language detector] fix exception if the libretranslate url is not specified. [terrtia] * [ocr] catch cv2 errors. [terrtia] * [pgpdump] fix installer, new pgpdump version require to launch autoreconf. [terrtia] * [ocr] catch OSError on MP4 files. [terrtia] * [ocr] fix ocr supported languages. [terrtia] * [Language] Updated the language detector to return a empty list when no language is detected. [terrtia] * [ocr] filter ocr supported languages + fix type of object accepted by the tracker. [terrtia] ## v5.5 (2024-04-25) ### Changes * [doc] add overview. [terrtia] * [ocr] ocr daterange object, get ocrs by daterange + fixs. [terrtia] * [reprocess tool] add OcrExtractor module + filter image gif. [terrtia] * [ocr] add cache + correlation ocr-chats-messages + launch ocr extractor by default. [terrtia] * [domains] card domain, fix last check. [terrtia] * [ocr] detect and translate language + show ocr view + add languages blueprint. [terrtia] * [ocr] get languages model + group extracted content by line + process ocr objects + get all images. [terrtia] * [perf] reduce memory usage. [terrtia] * [ocr] extract text from image + add ocr object. [terrtia] * [tracker] add experimental report generator. [terrtia] * [user-account] add heatmap nb user messages. [terrtia] * [message] show trackers + modules matches. [terrtia] * [trackers] UI: remove object from tracker. [terrtia] * [user-account] show chats + subchannels. [terrtia] ### Fix * [ocr] fix None copy date. [terrtia] * [correlation] fix objects selector. [terrtia] * [correlation] fix objects selector. [terrtia] * [correlation] fix objects selector. [terrtia] * [ocr] UI correlation, fix language bloc. [terrtia] * [thehive] fix export logger. [terrtia] * [UI matches extractor] fix empty matches. [terrtia] * [UI matches extractor] handle overlapping matches. [terrtia] * [languages] fix items language min probability. [terrtia] ### Other * Merge branch 'master' into ocr. [terrtia] * Merge branch 'master' into ocr. [terrtia] ## v5.4 (2024-03-25) ### Changes * [language] add thread languages stats. [terrtia] * [language messages] add nb languages stats by chat/subchannel objects. [terrtia] * [lang] merge master. [terrtia] * [lang] improve language detection + UI: manual translation and detection. [terrtia] * [requirements] add lexilang. [terrtia] * [trackers match] show objects date. [terrtia] * [requirements] add demoji. [terrtia] * [user-account] add default card. [terrtia] * [chat-subchannel] add basic + default card. [terrtia] * [chat] add chat default + basic card. [terrtia] * [image object] add default + basic card. [terrtia] * [global module] perf. [terrtia] * [tools] reprocess objects by type. [terrtia] * [modules] improve error output. [terrtia] * [RetroHunt] retro hunt on messages. [terrtia] * [correlation UI] change Direct Correlations position. [terrtia] * [correlation] description: get message content + don't show etag, cookie_name and hhash by default. [terrtia] * [chats] add heatmap nb messages/hour for all messages. [terrtia] * [api] get chat/subchannel/thread messages. [terrtia] * [api] get object + get investigation. [terrtia] * [domains] add crawler status stats by domain type pie chart. [terrtia] * [d3 js] add barchart_stack graph function. [terrtia] * [crawler] add monthly crawled domains stats. [terrtia] * [test] add basic API test. [terrtia] * [git] update gitignore. [terrtia] * [api] refactor blueprint. [terrtia] * [favicon] add v5.4 update + update pylacus. [terrtia] * [favicon] add favicons objects + correlation. [terrtia] * [vanity] vanity tree, add links. [terrtia] * [vanity] add vanity domain explorer + fix blurred screenshot + fix languages search filter. [terrtia] * [domain] sort onion vanity name by prefix. [terrtia] * [titles] add api to get unsafe titles. [terrtia] * [titles] add api to get all titles. [terrtia] * [tracker] list of last matchs, show description. [terrtia] * [tracker] list of last matchs, show description. [terrtia] * [retro_hunt] show match + fix None subtype. [terrtia] * [titles] add endpoints to get all titles. [terrtia] * [messages] manual translation by language + add new kvrocks lang namespace. [terrtia] * [chat] add endpoints to download chat, subchannel and thread, + fix message translated by default. [terrtia] * [doc] add screenshot. [terrtia] * [exif] add debug. [terrtia] * [doc] add discord/telegram chats JSON fields. [terrtia] * [HOWTO] Libretranslate Chat translation. [terrtia] * [requirement] bump flask requirement. [terrtia] ### Fix * [chat] fix subchannel-message correlation + fix empty message language detection. [terrtia] * [chat-subchannel] card empty chat. [terrtia] * [dashboard] fix objects links. [terrtia] * [tracker] fix yara content error on empty message. [terrtia] * [FeederImporter] add all objects in queue. [terrtia] * [module] fix SQLInjectionDetection object ID. [terrtia] * [chats] fix templates, chat subtype. [terrtia] * [api] remove debug output. [terrtia] * [domains] add missing JS file. [terrtia] * [crawler] fix monthly crawled domains stats svg size. [terrtia] * [crawler] fix monthly crawled domains stats. [terrtia] * [tests] fix tests + global new file content. [terrtia] * [favicon] fix misp object export. [terrtia] * [favicon] crawler favicon. [terrtia] * [vanity] fix title. [terrtia] * [domain] add screenshot img onerror. [terrtia] * [vanity] length > 5 clusters. [terrtia] * [vanity] debug. [terrtia] * [vanity] debug. [terrtia] * [vanity] length+1 min vanity = 2. [terrtia] * [vanity] vanity builder. [terrtia] * [vanity] update id domains is up + min vanity = 4. [terrtia] * [api] get titles domain. [terrtia] * [tracker] fix typo. [terrtia] * [tracker] fix message trackers. [terrtia] * [titles] add api to get all titles. [terrtia] * [titles] add api to get all titles. [terrtia] * [retro_hunt] api delete paused retro_hunt. [terrtia] * [retro_hunt] delete paused retro_hunt. [terrtia] * [doc] fix pystemon install. [terrtia] * [doc] fix pystemon install. [terrtia] * [flask] fix escape import. [terrtia] * [chats] fix chats image importer. [terrtia] * [language] catch libretranslate exception. [terrtia] * [language] crawled items, force gcld3 detection. [terrtia] * [LAUNCH] update screen status. [terrtia] * [LAUNCH] fix killall. [terrtia] ### Other * Merge lang. [terrtia] * Merge branch 'master' of github.com:ail-project/ail-framework. [terrtia] * Merge pull request #215 from NMD03/master. [Thirion Aurélien] Add [ailbuilder] cleanup * Add [ailbuilder] cleanup. [niclas] ## v5.3 (2024-02-05) ### Changes * [LAUNCH] change restart flags. [terrtia] * [UI] If version None do not show release note link. [Steve Clement] * [LAUNCH] make sure reload works. [Steve Clement] * [LAUNCH] Addded a "restart" option, killAll + launchAuto. [Steve Clement] * [tags] add Tag class. [terrtia] * [Hosts] improve perf + regex timeout + cache DNS results. [terrtia] * [kvrocks] j -4 install + update to latest version. [terrtia] * [translation] translate chats info, users info and subchannels names. [terrtia] * [chats] show NB messages by chat. [terrtia] * [v5.3] add v5.3 update. [terrtia] * [relationships] add relationship engine + WIP relationships between forwarded messages/chats. [terrtia] * [message] UI translate message object. [terrtia] * [Language] replace pycld3 by gcld3 + clean text before language detection. [terrtia] * [hosts] filter onion. [terrtia] * [install] remove python package from thirdparty. [terrtia] * [thirdparties] remove sb-admin + debug #194. [terrtia] * [queues] timeout obj after 2 days. [terrtia] * [chats] get user message ids by chat. [terrtia] * [correlation] UI chats filters + correation user-account/username. [terrtia] * [crawlers] add endpoints to reset captures. [terrtia] * [crawlers] add endpoints to reset captures. [terrtia] * [chats] add pagination. [terrtia] * [chats] add participants corrlation shortcut. [terrtia] * [tags] searech messages tags by daterange. [terrtia] * [tags] search messages by tags. [terrtia] * [categ] messages, bypass categ module + fix correlation. [terrtia] * [crawler] push onion discovery capture_uuid to another AIL. [terrtia] * [chats] add chats participants + user-accounts basic template. [terrtia] * [chats] translate messages on demand. [terrtia] * [chats] add discord threads, Forum channel. [terrtia] * [chats] add messages threads. [terrtia] * [chats] add message file-name object + str emoticon reactions. [terrtia] * [chat users] add users icons. [terrtia] * [chats] factorise heatmap + chat icon. [terrtia] * [chats] add message and blur blocks. [terrtia] * [queues] add new image queue + add exif module. [terrtia] * [images] add sidebar shortcut. [terrtia] * [chats] add image object + show message image. [terrtia] * [chats] add heatmap nb week messages by hour. [terrtia] * [chats] message object template. [terrtia] * [submodules] bump. [terrtia] * [messages] refactor get_messages_meta + add basic message template. [terrtia] * [chats explorer] show chats/subchannels creation date. [terrtia] * [chats] add UI shortcut + networks list + show chats/subchannels info. [terrtia] * [chats] improve UI + fix importer. [terrtia] * [chats] add chats explorer v0. [terrtia] * [core] merge master + fix object subtype correlation stats. [terrtia] * [crawlers] submit cookies to the crawler task API. [Jean-Louis Huynen] * [crawlers] submit a single cookie to the crawler task API. [Jean-Louis Huynen] * [mail exporter] add obj content extract for each regex match. [Terrtia] * [mail exporter] add obj content extract for each yara rule match. [Terrtia] * [titles] add yara tracker on title + tags domains if unsafe title tags. [Terrtia] * [modules] crawl pasties domains. [Terrtia] * [module + queues] track + rename object global ID by module. [terrtia] * [importers obj_type] importers queues: add feeder source + object global ID. [terrtia] * [ail queues] merge. [terrtia] * [chat + user-account] correlations + usernames timeline. [Terrtia] * [core] add telegram importer + Chat object + message Object + add timeline engine. [Terrtia] * [module extrator] add debug. [Terrtia] * [module extrator] add debug. [Terrtia] * [telegram fedeer] use meta of the new feeder. [Terrtia] * [flask] cleanup. [Terrtia] * [flask] cleanup, remove unused import. [Terrtia] * [crawler ui] last crawled domains, show last check timestamp. [Terrtia] * [tracker ui] improve show typo squatting button + add tooltip. [Terrtia] * [api crawler] fix response + add cookiejar, proxy and frequency parameters. [Terrtia] * [HHHash] add HHHash object and correlation https://www.foo.be/2023/07/HTTP-Headers-Hashing_HHHash. [Terrtia] * [objs processed] xxhash messages. [terrtia] * [queues] track object + check if object processed. [Terrtia] ### Fix * [LAUNCH] fix ENV error message. [terrtia] * [crawler] fix capture None domain name. [terrtia] * [tags] fix invalid tags. [terrtia] * [README] api.md link typo. [fukusuket] * [tags] fix typo. [terrtia] * [docs + UI] fix repository links. [terrtia] * [ui] Do not show relNotes link if not on a tag. [Steve Clement] * [repo] Point to the correct ail-project repo. [Steve Clement] * [D4] fix module cache. [terrtia] * [misp export] fix empty event on module start. [terrtia] * [crawlers] fix capture return error code. [terrtia] * [crawlers] fix errored capture queue. [terrtia] * [crawlers] fix errored capture start time. [terrtia] * [chat view] fix created_at + filter. [terrtia] * [heatmap] fix tooltip position. [terrtia] * [crawler] log UNKNOWN timeout. [terrtia] * [Language] fix language source. [terrtia] * [hosts] fix number of hosts extracted. [terrtia] * [DomClassifier] improve perf. [terrtia] * [DomClassifier] improve perf. [terrtia] * [DomClassifier] improve perf. [terrtia] * [queue] save last timout in cache. [terrtia] * [crawler] fix api create_task. [terrtia] * [tags] debug. [terrtia] * [tags] debug. [terrtia] * [tags] debug. [terrtia] * [tags] debug. [terrtia] * [tags] debug. [terrtia] * [tags] debug. [terrtia] * [tags] debug. [terrtia] * [tags] debug. [terrtia] * [tags] debug. [terrtia] * [tags] remove invalid tags. [terrtia] * [keys module] fix tags. [terrtia] * [crawlers] fix task start. [terrtia] * [crawler] debug. [terrtia] * [crawler] debug. [terrtia] * [crawler] debug. [terrtia] * [libinjection] memory leak, disable module. [terrtia] * [zmq importer] fix object source name. [terrtia] * [chat] fix translation pagination. [terrtia] * [crawler] fix capture start time. [terrtia] * [sidebar] typo. [terrtia] * [translate] fix exception. [terrtia] * [api] fix add crawler capture return. [terrtia] * [language] libretranslate unreachable. [terrtia] * [requirement] fix libretranslate, fix #189 + fix #190. [terrtia] * [retro_hunt] fix daterange. [terrtia] * [chat d3 heatmap] add missing function. [terrtia] * [doc] simplify ail feeder. [terrtia] * [investigations] delete obj. [terrtia] * [chats] fix username meta. [terrtia] * [languages] fix language module. [terrtia] * [chats] fix chat username. [terrtia] * [tags] fix galaxies synonyms. [terrtia] * [chats] fix messages replies. [terrtia] * [dep] Pinning flask to < 3.0 due to Werkzeug 3.0 issues: https://stackoverflow.com/a/77215455. [Steve Clement] * [tracker] delete yara rule, fix filter by object type. [terrtia] * [items source] filter invalid item sources. [Terrtia] * [json importer] fix empty source name. [Terrtia] * [items source] fix empty sources list. [Terrtia] * Typo. [Terrtia] * [title] fix None title. [Terrtia] * [tracker objs filter] fix title icon. [Terrtia] * [module pasties] fix module name. [Terrtia] * [module] fix module obj type: language + mail. [terrtia] * [queues] fix ended duplicate + sync queue. [terrtia] * [chats] message css + reply ID. [terrtia] * Correlations. [Terrtia] * [module extractor] fix invalid yara offset. [Terrtia] * [module extractor] fix invalid yara offset. [Terrtia] * [tags ui] fix galaxy, get number of tags enabled + add toolip helper. [Terrtia] * [settings] fix edit user. [Terrtia] * [Tracker] edit tracker filters. [Terrtia] * [MailExporter] improve logs + SMTP over SSL: add an option to validate self signed CA This exporter use STARTTLS by defaults. If the SMTP server does not support the STARTTLS extension, the client will switch to SSL instead. [Terrtia] * [tools crawler] fix helper. [Terrtia] * [tools crawler] improve helper. [Terrtia] * [crawler] fix down domain + domain redirection history. [Terrtia] * [domain search] returns an empty string when searched with invalid characters. [fukusuket] * [doc] fix LAUNCH.sh typo. [fukusuket] ### Other * Merge branch 'dev' of github.com:CIRCL/AIL-framework into dev. [terrtia] * Merge pull request #595 from shadow2033/patch-3. [Thirion Aurélien] Update Categ CreditCards keyworlds * Update CreditCards. [shadow2033] ///English added (Mir; QIWI; WebMoney) ///Russian добавлены (Mir; QIWI; WebMoney) * Merge remote-tracking branch 'origin/master' into dev. [terrtia] * Merge pull request #205 from NMD03/patch-1. [Thirion Aurélien] Update README.md * Update README.md. [Niclas Dauster] * Merge pull request #203 from NMD03/LXD-imagebuilder. [Thirion Aurélien] Add [autobuild] build script + systemd config * Add [autobuild] build script + systemd config. [niclas] * Merge remote-tracking branch 'circl/master' [Alexandre Dulaunoy] * Merge pull request #604 from NMD03/LXD-installer. [Thirion Aurélien] Add LXD installer * Update [README] add installer options. [niclas] * Add [interactive install] interactive install. [niclas] * Fix [tor install] tor install on lacus. [niclas] * Add [install] install tor on lacus. [niclas] * Add [install] container setup ail + lacus. [niclas] * Initial commit. [niclas] * Merge pull request #198 from fukusuket/fix-readme-api-document-link-typo. [Alexandre Dulaunoy] fix: [README] api.md link typo * Merge remote-tracking branch 'refs/remotes/origin/dev' into dev. [terrtia] * Merge pull request #202 from SteveClement/dev. [Steve Clement] chg: [LAUNCH] Addded a "restart" option, killAll + launchAuto * Merge branch 'dev' into dev. [Steve Clement] * Merge pull request #201 from SteveClement/UI_fix. [Steve Clement] fix: [repo] Point to the correct ail-project repo * Merge branch 'ail-project:dev' into dev. [Steve Clement] * Merge pull request #1 from SteveClement/UI_fix. [Steve Clement] fix: [repo] Point to the correct ail-project repo * Merge branch 'master' into dev. [terrtia] * Merge pull request #196 from markus-lassfolk/master. [Thirion Aurélien] Fix IndexError in get_last_tag_from_remote Function * Fix IndexError in get_last_tag_from_remote function. [Markus Lassfolk] This commit adds a check to ensure that the output from the subprocess command in the get_last_tag_from_remote function has a sufficient number of lines before attempting to access specific indices. This change prevents the IndexError that occurred when the git command's output was shorter than expected. * Merge pull request #195 from markus-lassfolk/master. [Alexandre Dulaunoy] Typo in CRAWLED_SCREENSHOT * Typo in CRAWLED_SCREENSHOT. [Markus Lassfolk] There was a typo in the CRAWLED_SCREENSHOT * Fix [api] fix add_crawler_capture endpoint. [terrtia] * Merge branch 'master' into dev. [terrtia] * Merge branch 'master' into dev. [terrtia] * Merge pull request #183 from SteveClement/flask_fix. [Thirion Aurélien] fix: [dep] Pinning flask to < 3.0 due to Werkzeug 3.0 issues * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Update crawler_add_task.py. [Alexandre Dulaunoy] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Merge pull request #179 from fukusuket/fix-500-error-when-invalid-domain-search. [Thirion Aurélien] fix: [domain search] prevent exceptions when search domain by invalid characters * Merge pull request #177 from fukusuket/fix-readme-launch-sh-typo. [Alexandre Dulaunoy] fix: [doc] fix README `LAUNCH.sh` typo ## v5.2 (2023-07-12) ### Changes * [update] remove old updates < 5.0. [Terrtia] * [updater] refactor background updater + add v5.2 update. [Terrtia] * [crawler har] compress HAR. [Terrtia] * [correlation] correlation graph, add an option to hide an object/node by pressing H + reset correlation graph. [Terrtia] * [etag] add new etag object. [Terrtia] * [investigation] add object comment. [Terrtia] ### Fix * [updater] fix env. [Terrtia] * [background update] fix logger. [Terrtia] * [crawler] add timeout to Unknown captures. [Terrtia] * [crawler] add exception handing for ping_lacus. [fukusuket] * [crawler] added existence check for screenshot set. [fukusuket] * [decoded] fix download file. [Terrtia] * [updater] fix db checker. [Terrtia] * [correlation tags] fix tag all objects. [Terrtia] * [correlation card decoded meta] mimetype + size. [Terrtia] * [correlation card decoded meta] mimetype + size. [Terrtia] * [updater] remove old ARDB env. [Terrtia] * [hunter + misp export] fix misp event json export + retro hunt date search and description. [Terrtia] ### Other * Merge pull request #174 from fukusuket/fix-500-erro-when-invalid-lacus-url. [Thirion Aurélien] fix: [crawler] add exception handing for `ping_lacus` * Merge pull request #176 from fukusuket/fix-500-error-when-crawler-screenshot-setting-off. [Thirion Aurélien] fix: [crawler] added existence check for screenshot set ## v5.1 (2023-06-26) ### Changes * [pystemon importer] fix gzipped pastes. [Terrtia] * [correlation graph] show message if max_nodes reached + fix cookie-name sparkline. [Terrtia] * [crawler] auto tag crawled domains. [Terrtia] * [correlation] add an option to remove max number of nodes if max_node == 0. [Terrtia] * [object cookie-name] add new cookie-name object + correlation. [Terrtia] * [title search] add pagination. [Terrtia] * [titles] add title IDs and contents search. [Terrtia] * [favicon object] add favicon object. [Terrtia] * [sow item] show item investigations. [Terrtia] * [kvrocks migration] mv update/v.50. [Terrtia] * [redis] update minimal version. [Terrtia] * [doc] add AIL v5.0 + objects + Importers + sync. [Terrtia] * [correlation] filter blank screenshots. [Terrtia] * [importers] improve abstract class and logs. [Terrtia] * [domains explorer] unsafe tag default image. [Terrtia] * [README.md] update. [Terrtia] * [HOWTO] improve HOWTO. [Terrtia] * [correlation graph] update node legend. [Terrtia] * [correlation graph] select correlation depth. [Terrtia] * [correlation] correlation graph: filter title objects. [Terrtia] * [correlation] add direct correlation stats. [Terrtia] * [new title object] add new title object + correlation on page title. [Terrtia] * [Phone module] Filter Invalid Phone numbers + UI Show extracted. [Terrtia] * [importers] add Dir/File Importer. [Terrtia] ### Fix * [pystemon importer] fix base64 encoding. [Terrtia] * [crawler] same capture uuid if a domain is already crawled. [Terrtia] * [IPAdress module] empty queue if no IP ranges provided. [Terrtia] * [retro hunt] fix object tag queue + decoded content. [Terrtia] * [daterange object] fix objects by date. [Terrtia] * [title] fix title search empty result. [Terrtia] * [crawler] fix incomplete response. [Terrtia] * [user] fix get user token #163. [Terrtia] * [user] fix user delete #163. [Terrtia] * [MISP export] fix ail object first/last seen + obj logger. [Terrtia] * [MISP export] fix empty event. [Terrtia] * [d4] change enable d4. [Terrtia] * [kvrocks migration] [Terrtia] * [objects] fix investigation + ail2ail + screenshot MISP export. [Terrtia] * [domains explorer] None screeenshot. [Terrtia] * [show domains] fix down domains. [Terrtia] * [domains explorer] domain screeenshot. [Terrtia] * [domains explorer] fix empty screenshots. [Terrtia] * [correlation] fix tagging nb nodes. [Terrtia] * [README.md] fix CI badge. [Terrtia] * [README.md] fix logo. [Terrtia] * [module.cfg] fix templateModule example. [Terrtia] * [module extractor] fix tracker extractor. [Terrtia] * [tracker] fix tracker delete. [Terrtia] * [tracker] fix webhook. [Terrtia] * [crawler] fix undefined capture status. [Terrtia] * [correlation btc info] catch btc txs error. [Terrtia] * [Phone module] Filter Invalid Phone numbers. [Terrtia] * [phone] fix phone module. [Terrtia] * [domain search] fix template domain types filter. [Terrtia] * [domain search] fix template domain types filter. [Terrtia] * [MISP auto export] fix module input message. [Terrtia] * [tests] replace unmaintened nose by nose2. [Terrtia] * [tests] fix tests. [Terrtia] * [instaler] fix yara and pycld3 install. [Terrtia] * [tests] github workflow. [Terrtia] * [tests] github workflow. [Terrtia] * [flask] remove old import. [Terrtia] ### Other * Merge github.com:CIRCL/AIL-framework. [Terrtia] * Merge pull request #592 from shadow2033/patch-2. [Thirion Aurélien] Update Categ Mail * Update Mail. [shadow2033] ///English added (inbox; zoho) ///Russian добавлен (inbox; zoho) ## v5.0 (2023-05-17) ### Changes * [update] bump version < 5.0 roll back commit. [Terrtia] * [kvrocks] add migration notice. [Terrtia] * [update] older versions: filter 5.0 updates. [Terrtia] * [ail] cleanup. [Terrtia] * [MISP] refactor MISP thehive auto push. [Terrtia] * [IPAddress module] migrate IPAddress module. [Terrtia] * [logs] add new logger. [Terrtia] * [trackers UI] add admin endpoint. [Terrtia] * [trackers] refactor trackers. [Terrtia] * [crawler] add unsafe tag if domain contain unsafe screenshot. [Terrtia] * [objects + retro hunt] refactor retro hunt + objects retro hunts + get objects generator by filters (date_from, sources, mimetypes, ...) [Terrtia] * [trackers] refactor trackers: track objects + filter by object types/sources/subtypes. [Terrtia] * [correlation] blur image by default. [Terrtia] * [corellation graph] hide unsafe screenshots. [Terrtia] * [pystemon importer] create new PystemonImporter module, refactor pystemon feeder. [Terrtia] * [cleanup] [Terrtia] * [UI settings] add ail modules svg + fix admin view. [Terrtia] * [AIL queues] rewrite module queues. remove PUBSUB. [Terrtia] * [correlation] new admin enpoint delete correlation + add screenshot correlation filter. [Terrtia] * [correlation] tags all objects. [Terrtia] * [tags] remove old flask blueprint. [Terrtia] * [config] Kvrocks config + reset DB. [Terrtia] * [DB] remove ARDB + temp disable MISP import and auto export. [Terrtia] * [crawler] update default user agent. [Terrtia] * [Mixer] refactor Mixer + fix ZMQImporter config. [Terrtia] * [crawle] migrate domains settings. [Terrtia] * [stats] disable statistics. [Terrtia] * [core] merge conflict. [Terrtia] * [feeder] rename bgp_monitor. [Terrtia] * [feeder] add bgp_monitor importer. [Terrtia] * [pgpdump] check trackers on extracted metadata. [Terrtia] * [api] send url to crawler. [Terrtia] * [trackers] add warnings logs. [Terrtia] * [logs] add syslog level + facility + script startup message. [Terrtia] * [crawler] add crawler scheduler. [Terrtia] * [subtype objects] migrate UI cryptocurrency, pgp, username. [Terrtia] * [extractor] add cache + UI extractor + word/set extractor. [Terrtia] * [crawler] refactor crawler tasks + migrate cookiejars + add proxy option. [Terrtia] * [crawler] cookies migration + refactor. [Terrtia] * [exporter] add TheHive exporter. [Terrtia] * [exporter] refactor + add a new way to create and use exporters. [Terrtia] * [importer] refactor ZMQ + Feeder importer. [Terrtia] * [cleanup] remove ARDB + fix hive case. [Terrtia] * [MISP export + redis] refactor MISP, export all relationships between objects, relationships between investigated objects + bump redis-py version + theHive export. [Terrtia] * [Item] change parent-child key. [Terrtia] * [decoded migration] migrate Decoded Objects. [Terrtia] * [CVE] CVE search integration. [Terrtia] * [CVE] migrate CVE + get CVEs by daterange. [Terrtia] * [DB Migration] UI: Extract + highlight leaks and trackers match, Data Retention save object first/last date, Refactor Tools. [Terrtia] * [migration] migrate Item + Domain metas. [Terrtia] * [migration] migrate Screenshots + Trackers + Duplicates + fix zadd zincrby. [Terrtia] * [modules] migrate D4_client + PgpDump modules. [Terrtia] * [all] remove old objects + migrate cryptocurrencies module + cleanup code. [Terrtia] * [updates] remove v2.4 update + cleanup. [Terrtia] * [telegram] migration. [Terrtia] * [tags] refactor tags + cleanup. [Terrtia] * [crawler + core + cve] migrate crawler to lacus + add new CVE object and correlation + migrate core. [Terrtia] * [modules + correlation] migrate Cve,Iban,Language + fix correlation graph. [Terrtia] * [module] refactor Iban module + fix pgp migration. [Terrtia] * [statistics] ARDB migration. [Terrtia] * [Kvrocks migration] rewrite obj tags + migration. [Terrtia] * [launcher] disable SentimentAnalysis. [Terrtia] * [Kvrocks migration] add users + updates + trackers + retro_hunts + investigation migration. [Terrtia] * [custom tags] create + select custom tags. [Terrtia] * [kvrocks] launch + install. [Terrtia] ### Fix * [HOWTO] add missing screenshots. [Terrtia] * [tag auto push] fix navbar. [Terrtia] * [cves] fix cve search timeout. [Terrtia] * [Onion module] fix kvrocks sismeber. [Terrtia] * [core] fix instance ail:uuid. [Terrtia] * [telegram] fix regex. [Terrtia] * [logger] rename default logger file + fix log messages. [Terrtia] * [trackers] fix pgp yara tracker. [Terrtia] * [cves] catch cve search errors. [Terrtia] * [trackers] fix trackers edit. [Terrtia] * [UI feeder] fix logger. [Terrtia] * [domain explorer] increase default blur. [Terrtia] * [search domains] fix search domains by daterange. [Terrtia] * [domain explorer] fix search by daterange. [Terrtia] * [correlation] fix objects tagging. [Terrtia] * [api] fix token length. [Terrtia] * [modules] log level. [Terrtia] * [flask senf_file] fix decoded download. [Terrtia] * [domain UI] fix correlation card + language search. [Terrtia] * [investigation] fix create investigation. [Terrtia] * [module extractor] fix cache + fix yara match. [Terrtia] * [MISP exporter] fix MISP exporter CIRCL/AIL-framework#590. [Terrtia] * [objects] show/add subtype objects tags. [Terrtia] * [kvrocks config] fix default dir. [Terrtia] * [tracker modulew] fix duplicate module stats. [Terrtia] * [crawler] typo. [Terrtia] * [tags] fix delete subtype tags. [Terrtia] * [correlation] fix UI cve, decoded tags. [Terrtia] * [correlation] fix related bitcoins. [Terrtia] * [subtypes dashboard] fix obj base url. [Terrtia] * [credential module] fix logging message. [Terrtia] * [tags] enabled tags selector. [Terrtia] * [tags] fix UI enabled tags selector. [Terrtia] * [tags] fix UI enabled tags selector. [Terrtia] * [cve + crawler] fix crawler lacus client refresh + typo. [Terrtia] * [submit] fix UI submit. [Terrtia] * [queues] fix Mixer queue. [Terrtia] * [queues] fix Mixer queue. [Terrtia] * [submit] fix UI submit. [Terrtia] * [modules] small fixs. [Terrtia] * [modules] small fixs. [Terrtia] * [LAUNCH] remove ardb check. [Terrtia] * [tracker] fix exporters. [Terrtia] * [core] fix merge. [Terrtia] * [feeder] bgp_monitor class. [Terrtia] * [api] fix crawler api response. [Terrtia] * [api] fix crawler api. [Terrtia] * [exporter] thehive exporter, create case. [Terrtia] * [user] fix user creation. [Terrtia] * [modules] fix modules. [Terrtia] * [data_rentention_engine] fix object first/last seen. [Terrtia] * [Tracker migration] clean old Term library. [Terrtia] * [blueprints] remove old user library. [Terrtia] ### Other * Merge master. [Terrtia] * DB migration. [Terrtia] * Chg; [Duplicates module] refactor module + DB keys. [Terrtia] ## v4.2.1 (2022-09-08) ### Changes * [v4.2.1] add pubsublogger update. [Terrtia] * [logs] add syslog server option. [Terrtia] ### Fix * [tracker typo squatting] fix perf. [Terrtia] * [tracker typo squatting] fix cache. [Terrtia] * [tracker typo squatting] [Terrtia] ### Other * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Merge pull request #151 from cudeso/patch-1. [Alexandre Dulaunoy] Remove CIRCL NS / ZMQ from sample config file * Remove CIRCL NS / ZMQ from sample config file. [Koen Van Impe] ## v4.2 (2022-06-24) ### Changes * [Tracker} Tracker_Typo_Squatting. [David Cruciani] * [v4.2] add v4.2 update. [Terrtia] * [investigation] fix investigation by user + delete an obj from all investigation. [Terrtia] * [install vitualenv] remove travis env. [Terrtia] * [Retro Hunt] add logs. [Terrtia] * [Retro Hunt] add logs. [Terrtia] * [Retro Hunt] add logs. [Terrtia] * [AIL2AIL Sync] update exchange format. [Terrtia] * [AIL2AIL Sync] update exchange format. [Terrtia] * [add Hosts module] [Terrtia] * [sync module] debug. [Terrtia] * [sync client] debug. [Terrtia] * [websockets client] bind client ip. [Terrtia] * [websocket server] add host and port config. [Terrtia] * [telegram importer] add username correlation. [Terrtia] * [UI subtype objs] get obj by subtype + name. [Terrtia] * [misp export] add username. [Terrtia] ### Fix * [typosquatting] remove unused import. [Thirion Aurélien] * [tracker] clean import. [Thirion Aurélien] * [tracker term] fix typosquatting key. [Thirion Aurélien] * [Typo] tracker typo. [David Cruciani] * [tracker] UI for other than typosquat. [David Cruciani] * [typo] UI. [David Cruciani] * [Language] fix cld3 import. [Terrtia] * [launcher] kill AIL_2_AIL screen. [Terrtia] * [cld3] enable cld3. [Terrtia] * [cld3 python3.10] temp disable cld3. [Terrtia] * [launcher] remove Travis test. [Terrtia] * [Retro Hunt] item directory. [Terrtia] * [Retro Hunt] item directory. [Terrtia] * [Retro Hunt] fix item directory. [Terrtia] * [AIL exchange mime-type] [Terrtia] * [Hosts module] module + launcher. [Terrtia] * [abstract module] exception traceback #145. [Terrtia] * [ui tag selector] force custom tags. [Terrtia] * [installer] remove old tor install. [Terrtia] * [sync module] fix redis tag queue. [Terrtia] * [sync module] fix tags filter. [Terrtia] * [sync client] debug. [Terrtia] * [sync client] debug. [Terrtia] * [sync module] debug. [Terrtia] * [websockets client] fix client bind. [Terrtia] * [websockets] remove size limit. [Terrtia] * [UI subtype objs] fix form. [Terrtia] * [misp config] https. [Thirion Aurélien] ### Other * Merge pull request #147 from ail-project/typo. [Thirion Aurélien] Integration of the typo-squatting tracker * Fix; [set tracker] missing function. [Thirion Aurélien] * Merge branch 'master' into typo. [David Cruciani] * Add: [tracker] typo-squatting. [David Cruciani] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Merge pull request #146 from gallypette/master. [Thirion Aurélien] add: [modules] zerobinz * Add: [modules] zerobinz. [huynenjl@gmail.com] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] ## v4.1 (2022-03-11) ### Changes * [flask] updated. [Alexandre Dulaunoy] * [flask] requirements for higher version of flask. [Alexandre Dulaunoy] * [v4.1] add Investigation with MISP Export + v4.1 update. [Terrtia] * [Telegram module] refactor module + fix str format. [Terrtia] ### Fix * [Investigation] edit misp event + add misp instance url. [Terrtia] * [Investigation] fix MISP Export + UI sidebar. [Terrtia] * [UI inestigations] add items link. [Terrtia] * [UI inestigations] add objects link. [Terrtia] * [telegram launcher] [Terrtia] * [items] abstract class. [Terrtia] * [Investigation] UI sidebar. [Terrtia] * [v4.1] fix ardb # tracking DB. [Terrtia] * [username] user icon. [Thirion Aurélien] * [Term tracker] fix item date. [Terrtia] * [Telegram module] fix launcher. [Terrtia] * [pybgpranking] package install. [Terrtia] * [popper install] rename popper repository. [Terrtia] https://github.com/floating-ui/floating-ui/discussions/1425 * [UI] remove update note. [Terrtia] * [trackers] fix get_all_items_sources. [Terrtia] * [crawler] fix is_splash_manager_connected #133. [Terrtia] ### Other * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Merge pull request #139 from gallypette/jabber-feeder. [Thirion Aurélien] add: [username] jabber support * Add: [username] jabber support. [Jean-Louis Huynen] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Create SECURITY.md. [Alexandre Dulaunoy] * Merge branch 'dev' [Terrtia] ## v4.0 (2021-12-01) ### Changes * [sync UI] disable pull. [Terrtia] * [sync UI] dashboard, show nb of imported items + launch/kill ail servers when a queue is subscribed/unsubscribed. [Terrtia] * [ail sync UI] restarr/launch/kill sync connections + show sync mode api/pull/push. [Terrtia] * [ail sync server] add server controller + list connected clients ail_uuid->sync_modes. [Terrtia] * [ail sync ui] copy to clipboard ail_uuid, ail server key. [Terrtia] * [ail sync] edit ail_servers/sync_queues + fix logs. [Terrtia] * [api] rename endpoints. [Terrtia] * [ail sync] add sync api (ping, version) + UI/client error handler. [Terrtia] * [doc] GI Badge. [Steve Clement] * [v4.0 AIL SYNC / AIL 2 AIL] SYNC Manager + fixs + views. [Terrtia] * [crawler] add auto crawler functions. [Terrtia] ### Fix * [sync client] don't launch client if ail server not linked with a sync queue. [Terrtia] * [sync server] remove hardcoded host. [Terrtia] * [sync server] host. [Terrtia] * [sync client] fix websockets client connect for python >= 3.8. [Terrtia] * [ail sync] fix refresh_ail_instance_connection. [Terrtia] * [ail sync] fix refresh_ail_instance_connection. [Terrtia] * [ail sync] server + client: resend object in queue on ConnectionClosedError. [Terrtia] * [crawler] add comment. [Terrtia] * [UI ail sync] fix nav. [Terrtia] * [UI ail sync] add missing ail icon. [Terrtia] * [doc] Remove Travis. [Steve Clement] * [py] Minor python dependency change. [Steve Clement] * Inherit AbstractModule to prevent stuck queues. [osagit] regex compiled only at start, not in the loop no duplicate warning string comments * Error message contains http protocol twice. [osagit] Error Can't connect to AIL Splash Manager, http://https://localhost:7001/ ### Other * Merge pull request #130 from TonyJabbour/master. [Thirion Aurélien] New restAPIs * Merge branch 'dev' into master. [Thirion Aurélien] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Merge branch 'master' of github.com:ail-project/ail-framework. [Alexandre Dulaunoy] * Merge pull request #569 from SteveClement/master. [Steve Clement] * Chg_ [AIL 2 AIL] add backend. [Terrtia] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Add tracker fixed api function replaced it with internal function. [TonyJabbour] * Added get_tracker_metadata_api Removed unnecessarily parentheses. [TonyJabbour] * New API Endpoint: Fixed get_item_content_encoded_text Added get_item_sources Added get_check_item_source Added get_default_yara_rule_content. [TonyJabbour] * Removed unnecessarily parentheses. [TonyJabbour] * New API Endpoint: Return Item Content in base64 in non JSON format. [TonyJabbour] * Merge remote-tracking branch 'origin/master' [TonyJabbour] * Merge branch 'ail-project:master' into master. [Tony] * Merge pull request #129 from TonyJabbour/master. [Thirion Aurélien] Webhook implementation * Unnecessarily parenthesis removed. [TonyJabbour] * Base64 Problem. [TonyJabbour] * Base64 Problem. [TonyJabbour] * Type fixed. [TonyJabbour] * Type fixed. [TonyJabbour] * Error Fixed. [TonyJabbour] * Add new API endpoint that return only content encoded in base64. [TonyJabbour] * Webhook unnecessarily line removed Removed unnecessarily parentheses. [TonyJabbour] * Type fixed. [TonyJabbour] * Added a try/catch to handle exceptions Replaced the raise to send message to redis_logger. [TonyJabbour] * Fixed tracker_metadata. [TonyJabbour] * Typo Fixed. [TonyJabbour] * Fixed UI Datatable in showTracker Page Removed "Advanced Search" text from menu misleading button. [TonyJabbour] * Fixed Webhook integration with Trackers. [TonyJabbour] * -Fixed "get_tracker_metedata" typo -Typo Fixed. [TonyJabbour] * -Fixed "description" arg -Typo Fixed. [TonyJabbour] * -Fixed the 500 error issue when installing new instance of ail when adding new trackers -Fixed missing arguments -Typo Fixed. [TonyJabbour] * Fix webhook. [TonyJabbour] * Remove dict from Trackers. [TonyJabbour] * Add webhook post support in yara and regex trackers. [TonyJabbour] * Fix get_term_webhook. [TonyJabbour] * Add some changes for webhook. [TonyJabbour] * Add initial support for Webhook in Term Trackers. [TonyJabbour] * Fix spelling issue in Webhook. [TonyJabbour] * Add more support for Webhook URL. [TonyJabbour] * Add initial UI support for Webhook in tracker. [TonyJabbour] * Merge pull request #127 from osagit/patch-3. [Thirion Aurélien] fix: inherit AbstractModule to prevent stuck queues * Merge pull request #126 from osagit/patch-2. [Thirion Aurélien] fix: error message contains http protocol twice ## v3.7 (2021-08-27) ### Changes * [tracker + update] add update v3.7 + add map item_id:tracker_uuid (data retention) + fix tracker first_seen/last_seen. [Terrtia] * [tracker] typo fixed. [Alexandre Dulaunoy] * [Credential + tags] add misp-taxonomies submodule + fix typo. [Terrtia] * [gitchangelog.rc] updated to output Markdown. [Alexandre Dulaunoy] ### Fix * [tracker] global tracker list: fix first/last seen. [Terrtia] * [v3.7] add missing file. [Terrtia] * [trackers] items by daterange. [Terrtia] * [correlation graph] fix legend, remove icon text. [Terrtia] * [Credential] fix moduleStats. [Terrtia] * [Credemtial module] fix stats. [Terrtia] * [Yara + regex trackers] remove tests. [Terrtia] * [Decoder] log level. [Terrtia] * [abstract_modules + Global] log message on error + fix Global exception on invalid gzip. [Terrtia] * [Credential] fix old funct call. [Terrtia] * [UI Item submit] tags input: avoid browser and password managers autocomplete. [Terrtia] * [term tracker] typo. [Terrtia] ### Other * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Merge pull request #123 from TonyJabbour/master. [Alexandre Dulaunoy] CentOS 8 installation script Fixed a problem * CentOS 8 installation script Fixed a problem Added centos installation guide in README.md. [Tony Jabbour] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Merge pull request #122 from TonyJabbour/master. [Alexandre Dulaunoy] Added CentOS 8 installation script * Added CentOS 8 installation script. [Tony Jabbour] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Merge pull request #119 from Relega/patch-1. [Alexandre Dulaunoy] Update HOWTO.md * Update HOWTO.md. [Relega] * replaced pystemon url (from circl repository to the original repository) * refined pystemon instructions ## v3.6 (2021-07-14) ### Changes * Chg: [README] add Olivier Sagit @osagit to contributors list. [Terrtia] * [requirements] minimal version + remove old packages. [Terrtia] * Add Retro Hunt. [Terrtia] * [UI trackers] add/edit/remove tracker source/target #43 #102. [Terrtia] * [Trackers regex + yara] filter by item source. [Terrtia] * [Tracker term] track terms by sources. [Terrtia] * [test] add new workflow. [Thirion Aurélien] * [requirements] minimal version + remove old packages. [Terrtia] * [modules + tests] fix modules + test modules on samples. [Terrtia] * [update v3.6 + installer] fix faup install + add v3.6 update. [Terrtia] * [Launcher] update modules directory. [Terrtia] * [SQLInjectionDetection LibInjection modules] add module class. [Terrtia] * [Urls (Web) module] fix regex + rename. [Terrtia] * [Tracker_Regex module] create module class + mv module. [Terrtia] * [Tracker_Yara module] create module class. [Terrtia] * [modules] create new modules repository + small fixs. [Terrtia] * [Keys] add test. [Terrtia] * [modules] add tests: CreditCard, Global, DomClassifier. [Terrtia] * [Categ] tests + docs. [Terrtia] * [ApiKey] refactor module + tests. [Terrtia] * [launcher + modules] add module tests (Onion module) [Terrtia] * [AIL items + Onion] create AIL item objects + Onion module refactor. [Terrtia] * [tools extract_cryptocurrency] filter by correlation objects + errors messages. [Terrtia] * [tools extract_cryptocurrency] new input: list of addresses to extract. [Terrtia] * [tools] add 2 new tools: extract cryptocurrency correlation by type + trigger manual update. [Terrtia] * [modules] set log level to critcal on unexpected errors. [Terrtia] ### Fix * [Term tracker] fix fct args. [Terrtia] * [crawler] fix new crawled item id. [Terrtia] * [Retro Humt UI] clarify tags input. [Terrtia] * URLs www word to match. [osagit] There is 4 'w' chars instead of 3 * [crawlers] get_all_splash return type. [Terrtia] * [Splash_Manager errors] catch invalid response. [Terrtia] * [Trackers UI] fix sources logo + tracker metadata. [Terrtia] * Change module name Web to Urls. [lpwm9803] * [Splash_Manager errors] catch invalid response. [Terrtia] * [github workflows] fix test. [Terrtia] * [github workflows] clone depth. [Thirion Aurélien] * [installer] pyfaup install. [Terrtia] * [installer] pyfaup install. [Terrtia] * [UI crawler dashboard] block_languages_search: fix domains_types #110. [Terrtia] * [UI crawler dashboard] block_domains_name_search: fix domains_types #110. [Terrtia] * [modules] print + save traceback in logs. [Terrtia] * [paste_submit] restrict source characters. [Terrtia] * [submit_paste] restrict source name. [Terrtia] * [tools extract_cryptocurrency] argparse flags. [Terrtia] * [tags] invalid tags. [Terrtia] * Replace redis init with generic ConfigLoader. [osagit] StrictRedis() replaced by ConfigLoader.get_redis_conn() * Debug() string takes 1 positional argument. [osagit] * FILE_ALLOWED_EXTENSIONS without quotes. [osagit] * Stuck queues and submit paste. [Olivier SAGIT] * Name pystemon feeder in feeder monitor dashboard. [Olivier SAGIT] * [tools extract cryptocurrency] correlation type. [Terrtia] * [Updater] don't check if modified config files (redis and ardb config) [Terrtia] * [Web module] resolver, change log level. [Terrtia] * [WebStats] typo. [Terrtia] * [Indexer] debug messages. [Terrtia] * [WebStats] typo. [Terrtia] * [terms tracker] refresh Tracked terms. [Terrtia] * [redis cache] remove old paste_name db. [Terrtia] * [crawler] typo: splash restart. [Terrtia] ### Other * Merge branch 'master' of https://github.com/ail-project/ail-framework into dev. [Terrtia] * Merge pull request #115 from My-WAF/master. [Thirion Aurélien] Remove Block Copy Git Directory * Update .dockerignore. [VNC Company] * Merge branch 'dev' of https://github.com/ail-project/ail-framework into dev. [Terrtia] * Merge pull request #116 from osagit/patch-1. [Thirion Aurélien] fix: URLs www word to match * Merge. [Terrtia] * Merge branch 'dev' of https://github.com/ail-project/ail-framework into dev. [Terrtia] * Merge pull request #114 from osagit/dev. [Thirion Aurélien] fix: Change module name Web to Urls * Merge branch 'master' into dev. [Terrtia] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] * Merge pull request #559 from lesleyxyz/patch-1. [Thirion Aurélien] Submit paste contents to TheHive * Submit file contents to TheHive. [Lesley De Keyser] * Merge pull request #103 from osagit/scriptsenhance. [Thirion Aurélien] fix: stuck queues and submit paste * Merge branch 'dev' into scriptsenhance. [Thirion Aurélien] * Merge branch 'master' of https://github.com/ail-project/ail-framework. [Terrtia] * Perf: use defined compiled regex. [osagit] re.compile(regex) definition was not used use compile_regex.findAll() directely instead of re.findall(regex) * Log message split error + perf. [osagit] fix: log message split errors perf: string affected at start doc: comments * Merge pull request #101 from osagit/namedfeeder. [Alexandre Dulaunoy] fix: name pystemon feeder name in feeder monitor dashboard * Merge pull request #97 from osagit/abstract-module. [Thirion Aurélien] feat: module factorization * Feat: module factorization. [lpwm9803] ## v3.5 (2021-03-31) ### Changes * [passivedns] D4server port. [Terrtia] * [v3.5] install pyAIL. [Terrtia] * [passivedns] D4server port. [Terrtia] * [passiveDns] add dns records. [Terrtia] * [passiveDns D4 Client] add passiveDns D4 Client. [Terrtia] * [crawler] add test + relaunch crawlers + major fixs. [Terrtia] * [update] add v3.5 update. [Terrtia] * [doc] Splash Manager Configuration. [Terrtia] * [UI] add update note. [Terrtia] * [IPAddress] Add Debug + Check if option is empty. [Terrtia] * [splash manager] update enpoints + use Splash name to restart docker. [Terrtia] * [merge master] [Terrtia] * [domains search] search domains by name. [Terrtia] * [screenshot + har directory] add option to change screenshots directory. [Terrtia] * [crawler_manager] UI edit config + fix crawler queues. [Terrtia] * [config] save config on DB + default values. [Terrtia] * [crawlers manager] show setings. [Terrtia] * [crawler] show all crawlers type on dashboard. [Terrtia] * [crawler] manage crawlers. [Terrtia] * [crawler] crawler queue + restart docker on error. [Terrtia] * [core module] disable phone module by default. [Terrtia] * Merge master -> crawler_manager. [Terrtia] * [crawler manager] get all splash dockers, proxies and launch all crawlers. [Terrtia] ### Fix * [crawler] crawler test: remove print. [Terrtia] * [crawler UI] remove unused crawlers configuration. [Terrtia] * [crawler] user agent + splash restart. [Terrtia] * [crawler UI] crawler by domain type, remove old dashboard. [Terrtia] * [MISP Export UI] object type selector. [Terrtia] * [UI] fix navbar: invalid url. [Terrtia] * [crawler] typo. [Terrtia] * [crawler] typo. [Terrtia] * [crawler] typo. [Terrtia] * [Crawler] faup. [Terrtia] * [Launcher] launch core module: Crawler_manager. [Terrtia] * [cralers] remove debug. [Terrtia] * [Domain search] fix regex. [Terrtia] * [API ACL] avoid user_no_api users to access the API. [Terrtia] * [Module Queue] module without subscriber. [Terrtia] * [term Tracker] TimeoutException. [Terrtia] * [API ACL] read_only user role. [Terrtia] * [OVERVIEW] [Terrtia] * [gitignore] [Terrtia] * [showDomain] empty father field. [Terrtia] * [showDomain] empty father field. [Terrtia] * [redis cache] remove old Redis_Data_Merging db. [Terrtia] * [redis cache] remove old paste_name db. [Terrtia] * [background updater] Don't launch updates on fresh install. [Terrtia] * [v3.4 update] fix update progress. [Terrtia] * [background updater] remove completed updates. [Terrtia] * [install] force virtual environmemt activation. [Terrtia] ### Other * Merge branch 'master' into crawler_manager. [Terrtia] * Merge pull request #89 from ail-project/crawler_manager. [Alexandre Dulaunoy] Crawler manager * Merge branch 'master' into crawler_manager. [Terrtia] * Merge branch 'master' into crawler_manager. [Terrtia] * Merge pull request #536 from simonsigre/patch-1. [Alexandre Dulaunoy] Typo in placeholder "Optionnal" --> "Optional" * Typo in placeholder "Optionnal" --> "Optional" [Simon Sigré] Updated placeholder value to correct typo; "Optionnal" --> "Optional" * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Alexandre Dulaunoy] * Merge pull request #534 from simonsigre/patch-1. [Alexandre Dulaunoy] Added 'wget' as a dependency * Added 'wget' as a dependency. [Simon Sigré] 'wget' is a dependency as without this additional components won't download as part of initial install, one such example is; ``` wget -O /ail-framework/ardb/src/../deps/jemalloc-5.1.0.tar.bz2 https://github.com/jemalloc/jemalloc/releases/download/5.1.0/jemalloc-5.1.0.tar.bz2 && \ ``` * Merge branch 'master' into crawler_manager. [Terrtia] * Merge branch 'master' into crawler_manager. [Terrtia] * Merge branch 'master' into crawler_manager. [Terrtia] ## v3.4 (2020-12-20) ### Changes * [Languages]detect + search domains languages. [Terrtia] * [Tracker] edit tracker. [Terrtia] * [web] we process items nowadays not only pastes ;-) [Alexandre Dulaunoy] ### Fix * [install] pycld3 dependency. [Jean-Louis Huynen] Without this Flask fails starting * [update v3.4] updater. [Terrtia] * [Languages] import + update message. [Terrtia] * [Languages] update. [Terrtia] * [UI trackers] None trackers values. [Terrtia] * [Tracker] edit tracker ACL. [Terrtia] * [Yara Tracker] catch yara timeout. [Terrtia] * [Terms Trackcers] fix item link. [Terrtia] ### Other * Merge pull request #70 from ail-project/gallypette-patch-1. [Jean-Louis Huynen] fix: [install] pycld3 dependency * Merge branch 'master' of https://github.com/ail-project/ail-framework. [Terrtia] ## v3.3 (2020-10-13) ### Changes * [UI show Item] refactoring + bootstrap 4 migration. [Terrtia] * [UI show domain] add last origin table: domain + item. [Terrtia] * [travis] -> bionic. [Alexandre Dulaunoy] * [yara trackers UI] add yara trackers, show default yara rule. [Terrtia] * [bitcoin tags + correlation] add bech32 adresses. [Terrtia] ### Fix * [UI correlation graph] fix item links. [Terrtia] * [UI correlation graph] fix item links. [Terrtia] * [users management] hide API keys by default + fix delete/edit users. [Terrtia] * [crawler] fix ResponseNeverReceived retry time. [Terrtia] * [crawler] fix ResponseNeverReceived hanlder, check if splash restarted. [Terrtia] * [Yara tracker] fix mail notification. [Terrtia] * [Yara tracker] fix mail notification. [Terrtia] * [RegexTracker] fix missing hostname in mail notification. [Terrtia] * [Flask server] change default host. [Terrtia] * [ardb] switch to ail-project ardb fork #38. [Jean-Louis Huynen] * [Launcher] fix virtualenv loader. [Terrtia] ### Other * Merge branch 'master' of https://github.com/ail-project/ail-framework. [Terrtia] * Merge branch 'master' of https://github.com/ail-project/ail-framework. [Terrtia] ## v3.2 (2020-08-27) ### Changes * [update] add v3.2. [Terrtia] ## v3.1.1 (2020-08-27) ### Changes * [yara trackers] add debug. [Terrtia] * [tracker yara] show rule content. [Terrtia] * [correlation] basic correlation: get_correlation_all_object function. [Terrtia] * [submodule + YARA] add submodule auto update + update v3.1.1. [Terrtia] * [trackers] add yara trackers. [Terrtia] * [Trackers] email notifications: add tracker description in email subject. [Terrtia] * [username correlation + login redirection] add twitter username correlation + redirect to the requested page on login. [Terrtia] * [Item delete] delete father/child link + remove from domain tree + delete all child from the same domain. [Terrtia] * [feeder pystemon] add debug. [Terrtia] * [whosh index] add data retention fct. [Terrtia] * [core] disable Release module. [Terrtia] * [tor crawler] nyt added. [Alexandre Dulaunoy] * [API + item library] add a way to check if a user can access the API + item import (use item_basic) [Terrtia] * [black-list onion] keybase added. [Alexandre Dulaunoy] * [core crawler] add screen script create screen + windows + kill them + get list. [Terrtia] * [crawler proxy] interact with the splash manager API (get list of proxies + splash containers + launch them) [Terrtia] * [launcher] add option to reset UI admin password. [Terrtia] * [UI dashboard] sort module by name. [Terrtia] * [LAUNCH] launch telegram module by default. [Terrtia] ### Fix * [install] canevasjs. [Terrtia] * [UI tags] fix add tags template. [Terrtia] * [install] travis git unshallow. [Terrtia] * [trackers yara] remove sleep time. [Terrtia] * [trackers] typo. [Terrtia] * [updater] tag subversion. [Terrtia] * [update v3.1.1] init submodule. [Terrtia] * [travis] [Terrtia] * [crawler] fix auto crawler creation. [Terrtia] * [travis] fetch git last tag. [Terrtia] * [update thirparty] fix canevasjs install. [Terrtia] * [install crawler] remove old python requirement. [Terrtia] * [TheHive feeder] create alert, get item full path. [Terrtia] * [TheHive feeder] create alert. [Terrtia] * Typo. [Terrtia] * Typo. [Terrtia] * Typo. [Terrtia] * [username correlation] fix domain correlation. [Terrtia] * [RegexTracker] fix search regex helper. [Terrtia] * [regex_helper] [Terrtia] * [regex tracker] fix timeout. [Terrtia] * [reset_AIL] add helper + fix soft reset. [Terrtia] * [Exporter] ad missing file. [Terrtia] * [update_thirparty] fix popper install. [Terrtia] * [UI popper version] use popper v1 (https://github.com/twbs/bootstrap/issues/29842) [Terrtia] * [MISP auto Export] fix import path. [Terrtia] * [MISP auto export] fix MISP_TheHive_feeder. [Terrtia] * [travis] virtualenv. [Terrtia] * [install] unshallow clone. [Terrtia] * [requirement] remove old crawler requirements (installed by default) [Terrtia] * [crawler] option to disable screenshots and har. [Terrtia] * [crawlers lib] missing config file. [Terrtia] * [create au to crawler] save crawler_delta time. [Terrtia] * [updater] return boolean. [Terrtia] * [Credential] fix timeout error. [Terrtia] * [Credential] fix timeout error. [Terrtia] * [Credential] force lowercase of mail address. [Terrtia] * [Credential] fix timeout message. [Terrtia] * [Mail module] regex timeout. [Terrtia] * [update v1.5] [Terrtia] * [Mail module] remove debug. [Terrtia] * [Mail module] fix dns caching + use redis queue. [Terrtia] * [UI Decoded items] fix download decoded file. [Terrtia] ### Other * Merge pull request #519 from myasn1k/docker_fixes. [Thirion Aurélien] Docker fixes * Update update_thirdparty.sh. [Pietro Mazzini] * Remove useless copy. [Pietro Mazzini] * Uncomment update and explain in docker readme. [Pietro Mazzini] * Add tz as argument. [Pietro Mazzini] * Update docker readme (cp) [Pietro Mazzini] * Update docker readme. [Pietro Mazzini] * Fix canvas folder name. [Pietro Mazzini] * Add to docker start crawlers launch. [Pietro Mazzini] * Update Dockerfile: use ubuntu 18, remove redundant update_thirdparty, remove old crawler requirements pip install. [Pietro Mazzini] * Comment git repo tag in installing_deps, git repo not recognised. [Pietro Mazzini] * LAUNCH update function fail because the folder isn't a git repo, comment it. [Pietro Mazzini] * Merge pull request #22 from C00kie-/master. [Alexandre Dulaunoy] allow to configure binding interface * Update Flask_server.py. [C00kie-] * Update core.cfg.sample. [C00kie-] * Typo fixed as ail repo is now lower-case. [Alexandre Dulaunoy] Typo fixed as ail repo is now lower-case * Merge pull request #16 from ChilliSec/patch-1. [Thirion Aurélien] Update HOWTO.md * Update HOWTO.md. [ChilliSec] * Merge branch 'master' of https://github.com/ail-project/ail-framework. [Terrtia] * Merge pull request #497 from ronaldtf/fix-case-thehive. [Thirion Aurélien] Fix create case with TheHive * Fix create case with TheHive. [Ronald Teijeira Fernandez] ## v3.1 (2020-05-12) ### Changes * [update] add update v3.1 + install crawler python requirements by default. [Terrtia] * [UI correlation] add username correlation card. [Terrtia] * [UI correlation] add username correlation graph. [Terrtia] * [UI correlation] add username correlation: daterange pages. [Terrtia] * [telegram module] add new tag: telegram invite code. [Terrtia] * [telegram backend] add username correlation + save invite hash. [Terrtia] * [importer] add map twitter id - item id, add parents link between twitter and url extracted. [Terrtia] * [travis install] shallow clone. fetch tags + commit id. [Terrtia] * [travis install] shallow clone. fetch tags. [Terrtia] * [install] debug travis. [Terrtia] * [README] update gitter link. [Terrtia] * [import er url_extract] add item twitter parent. [Terrtia] * [importer url_extract] fix item id. [Terrtia] * [importer] add url_extract importer. [Terrtia] * [API json import] take list as imput. [Terrtia] * [core import] add AIL JSON import format, API endpoint added (AIL feeders) [Terrtia] * [update doc] update doc install + logo + fix updater. [Terrtia] * [Updater] change default branch origin. [Terrtia] * [Updater] change default branch origin. [Terrtia] * [update Readme logo + links] [Terrtia] * [add new logo] [Terrtia] * [AIL logo UI] [Terrtia] * [doc] add cookiejar screenshot. [Terrtia] * [Crawler] default docker memory usage. [Terrtia] * [MISP export] export domain as domain-crawled object. [Terrtia] * [domain explorer UI] add shortcut button: misp export + correlation. [Terrtia] * [crawler] edit cookie and cookiejar + add cookie to cookiejar + fix screenshot duplicate. [Terrtia] * [Crawler core + UI] crawler lua: handle retry + fix cookie loader and selector. [Terrtia] * [cookiejar UI] add cookiejar + show all. [Terrtia] * [crawler cookies] use cookiejar. [Terrtia] * [crawler - cookies] add/show/select cookies. [Terrtia] * [crawler] add cookies list by user/global, save cookies from file + dict(name, value), TODO: API + handle errors. [Terrtia] * [crawler] bypass login: use cookie provided by user and accept cookie from server + refractor. [Terrtia] * [Splash Crawler] use cookies to bypass login. [Terrtia] * [UI] bump jquery to 3.4.1. [Terrtia] * [import_dir] fix is_gzip test, use magic number. [Terrtia] * [MISP Importer] files: handle missing sha1/sha256 attributes + fix Items and Screenshots dir. [Terrtia] * [Updater] relauch updater on change (git pull) [Terrtia] ### Fix * [Mail module] add debug output. [Terrtia] * [Mail module] replace signal by multiprocessing (https://docs.python.org/3.4/library/signal.html#execution-of-python-signal-handlers) [Terrtia] * [Mail] debug signal. [Terrtia] * [Mail] debug. [Terrtia] * [Mail module] remove test time.wait. [Terrtia] * [Mail module] debug signal timeout. [Terrtia] * [Mail module] debug signal timeout. [Terrtia] * [telegram module] typo. [Terrtia] * [telegram module] fix tagging. [Terrtia] * [telegram module] check username length. [Terrtia] * [telegram module] remove debug. [Terrtia] * [Mails] regex timeout. [Terrtia] * [Mails] change module output. [Terrtia] * [Mails] remove print + test. [Terrtia] * [Mails] refactor Mail module. [Terrtia] * [urlextract importer] fix parent map + replace set JSON queue by list. [Terrtia] * [import urlextract parent] fix typo. [Terrtia] * [travis install] shallow clone. fetch missing commit id. [Terrtia] * [Mails regex timeout] reduce default timeout. [Terrtia] * [Mails dns resolver] update timeout exception. [Terrtia] * [Mails] typo. [Terrtia] * [Mails] import. [Terrtia] * [Mails] import. [Terrtia] * [Mails] add regex timeout. [Terrtia] * [Credential] add regex timeout. [Terrtia] * [Credential] add regex timeout. [Terrtia] * [urlexport importer] fix item name + redis config. [Terrtia] * [urlexport importer] item_id, force str type. [Terrtia] * [urlexport importer] add missing import. [Terrtia] * [urlextract importer] class name. [Terrtia] * [importer] fix typo. [Terrtia] * [Global] extend ungzip error catching, catch invalid compressed file. [Terrtia] * [json import API] remove list input. [Terrtia] * [install create default user] default passwd file: add missing new line. [Terrtia] * [installer] get last git version. [Terrtia] * [Readme travis badge] [Terrtia] * [Readme travis badge] [Terrtia] * [Readme logo size] [Terrtia] * [show item UI] return 404. [Terrtia] * [Cookiejar UI] fix typo. [Terrtia] * [Flask session cookie name] add uuid to cookie name. [Terrtia] * [crawler] typo. [Terrtia] * [Crawler splash ResponseNeverReceived] add retry. [Terrtia] * [crawler] error catcher. [Terrtia] * [MISP export UI] fix input: name + value overwrite. [Terrtia] * [update thirdparty] update taxonomies. [Terrtia] * [crawler] cleanup. [Terrtia] * [PgpDump parser] remove header comment (rfc4880) + remove empty lines. [Terrtia] * [Pgp Dump] remove tool version. [Terrtia] * [ZMQ Feeder] performance: replace zmq recv NOBLOCK by Poller. [Terrtia] * [pgpdump] fix subtype save. [Terrtia] * [Updater] force updater update. [Terrtia] * [Updater] fix current_tag parser. [Terrtia] * [import_dir] remove special characters. [Terrtia] * [import_dir] remove dir whitespaces #475. [Terrtia] ### Other * Merge pull request #492 from sunil3590/master. [Thirion Aurélien] crawler_time -> crawler_delta * Crawler_time -> crawler_delta. [Sunil D S] * Chg [telegram + correlation] new module: telegram (username + login code + join_chat) + add simple_correlation backend. [Terrtia] * Chg [telegram + correlation] new module: telegram (username + login code + join_chat) + add simple_correlation backend. [Terrtia] * Merge pull request #487 from CIRCL/crawler_v2. [Thirion Aurélien] fix: [crawler] error catcher * Merge pull request #486 from CIRCL/crawler_v2. [Thirion Aurélien] Crawler v2 - Add cookiejar - use cookie to bypass login form * Merge branch 'master' into crawler_v2. [Terrtia] ## v3.0 (2020-02-21) ### Changes * [MISP export] remove pythonify. [Terrtia] * [MISP export] save event uuid - global_id. [Terrtia] * [presentation] add MISP export screenshot. [Terrtia] * [core] Drop python 3.5 support. [Terrtia] * [update v3.0] add update v3.0. [Terrtia] * [UI MISP export] add object export picker by user. [Terrtia] * [SentimentAnalysis] clean. [Terrtia] * [UI MISP export] export to MISP instance. [Terrtia] * [UI MISP import]show error +imported uuid/global_id. [Terrtia] * [UI MISP export] add error output. [Terrtia] * [UI MISP export] add MISP export. [Terrtia] * [UI MISP import] add MISP import. [Terrtia] * [MISP import export] item with relationships (domain not handled) [Terrtia] * [MISP import export] decoded with relationships. [Terrtia] * [MISP import-export] screenshots with relationships. [Terrtia] * [MISP import-export] pgp and cryptocurrency with relationships. [Terrtia] * [MISP import] import files obj: decoded + screenshot. [Terrtia] * [MISP import] import files obj: decoded + screenshot. [Terrtia] * [add MISP import] correlation TODO: create correlation. [Terrtia] * [add MISP import](no correlations) import item + pgp(incomplete) + screenshot(incomplete) [Terrtia] * [MISP export] add pgp object + add relationship: item + pgp + cryptocurrency. [Terrtia] * [export MISP] export Items + Screenshots objects with relationships. [Terrtia] ### Fix * [MISP export] get full list. [Terrtia] * [MISP export] misp response parsing. [Terrtia] * [MISP export import] [Terrtia] * [misp export] force event uuid. [Terrtia] * [MISP export] fix pgp-meta template. [Terrtia] * [Update] force str tags. [Terrtia] * [Update] force str tags. [Terrtia] * [update thirdparty] #473. [Terrtia] * [misp keys import] fix #471. [Terrtia] * [Update] support subversion + fix #472. [Terrtia] * [Pytaxonomies install] [Terrtia] * [SentimentAnalysis] download nltk punkt. [Terrtia] * [IPAdress] remove leading zeros #465. [Terrtia] * [module Webstats + BankAccount-Decoder] fix faup return type + remove old Paste library from BankAccount-Decoder #465. [Terrtia] * [Global catch incomplete file error] fix #464. [Terrtia] * [search items tag] fix daterange. [Terrtia] * [core Global] catch and log incomplete files. [Terrtia] * [domain explorer web] fix pagination by daterange. [Terrtia] * [domain explorer web] fix pagination by daterange. [Terrtia] ### Other * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] * Merge pull request #467 from petmi627/master. [Thirion Aurélien] Added characters limitation in trackersManagement.html * Added characters limition in trackersManagement.html. [Mike Peters] * Merge master => import_export_v1. [Terrtia] * Merge pull request #463 from petmi627/master. [Thirion Aurélien] Added support for username as notification credentials * Added support for username as notification credentials. [Mike Peters] ## v2.9 (2020-02-03) ### Changes * [domain explorer] domains explorer v2, filter domains by daterange. [Terrtia] ### Fix * [IPAddress] catch empty config error. [Terrtia] * [Global: already saved filename] save updated + filter duplicated items. [Terrtia] * [Global: filename provided by all feeders] avoid path tranversal. [Terrtia] * [Domain explorer UI] fix daterange pagination links. [Terrtia] * [Tag core] check if item_date type is an integer. [Terrtia] ## v2.8 (2020-01-23) ### Changes * [domain core + UI] add domain explorer v1. [Terrtia] * [test api] increase import timeout. [Terrtia] * [UI + core] tag decoded items, fix: #455. [Terrtia] * [UI item tags search] refractor: search item tag by object, use new functions. [Terrtia] * [UI tags] add + delete image (screenshot) tags. [Terrtia] * [UI tags] add + delete image (screenshot) tags. [Terrtia] ### Fix * [MISP export] force pymisp version. [Terrtia] * [tag core] typo. [Terrtia] * [tag core] item date type. [Terrtia] * [Tag core] add tag, update tag last seen. [Terrtia] * [Flask server + cookie session] chg default cookie name (also use port number) + add Flask port number to config. [Terrtia] * [expand btc adress] filter empty addr fields. [Terrtia] * [UI navbar] chg icon decoded tags. [Terrtia] * [UI show item min] fix empty modal. [Terrtia] * Change name popper.js-1.14.3 to popper-core-1.14.3. [mangelft] * [UI] screenshot url. [Terrtia] ### Other * Merge pull request #453 from mangelft/master. [Thirion Aurélien] fix: change name popper.js-1.14.3 to popper-core-1.14.3 * Merge pull request #450 from CIRCL/tags_v2. [Thirion Aurélien] Tags v2 ## v2.7 (2020-01-13) ### Changes * [UI domain] add input: show domain by name. [Terrtia] * [tags blueprint] clean code. [Terrtia] * [update v2.7] sort domain full_onion_up and full_regular_up. [Terrtia] * [UI tags] search domains by tags. [Terrtia] * [core + UI] search domain by tags. [Terrtia] * [Update v2.7] add update v2.7 scripts. [Terrtia] * [tags UI] edit object tags (delete tags) [Terrtia] * [Tag core] objects tagging, Part 2/2 TODO: UI tags domain + screenshot + object rename paste=>item. [Terrtia] * [Tag core] objects tagging, Part 1/2 TODO UI (tags) + rename paste=>item. [Terrtia] * [Tag core] objects tagging, Part 1/2. [Terrtia] * [README] remove top terms. [Terrtia] * [correlation graph UI] add json error handler + add loading status. [Terrtia] * [UI correlation graph + UI domain] correlation screenshot: show img in toolip + show hash in ShowDomain TODO: pixelate images. [Terrtia] ### Fix * [UI tags] fix domain links. [Terrtia] * [UI term] remove deprecated trending charts, fix #446 #447. [Terrtia] * [Crawler] typo. [Terrtia] * [UI decoded item] sort mimetype. [Terrtia] * [Crawler] fix screenshot-domain typo. [Terrtia] * [Crawler] fix screenshot-domain typo. [Terrtia] * [Crawler] fix screenshot-domain map. [Terrtia] * [UI showDomain] fix screenshot accordeon. [Terrtia] ### Other * Merge pull request #449 from CIRCL/tags_v2. [Alexandre Dulaunoy] Tags v2 - Tagging system refractoring * Merge branch 'master' into tags_v2. [Terrtia] * Update README.md. [Thirion Aurélien] ## v2.6 (2019-12-17) ### New * [slides] source code added. [Alexandre Dulaunoy] ### Changes * [screenshot correlation + v2.6] add screenshot-domain correlation + v2.6 update. [Terrtia] * [API] get domain min metadata (first up, last up) + get crawled domain by daterange and status. [Terrtia] * [Domain + Date] get domain up range + get date days and months by daterange. [Terrtia] * [Domain] get all/by month domains up. [Terrtia] * [API] get domain metadata (minimal) [Terrtia] * [UI correlation graph] tooltip: show domain tags. [Terrtia] * [UI correlation graph] popover: add loading status + chg css. [Terrtia] * [correlation UI] add basic popover. [Terrtia] * [slide] update slide. [Terrtia] * [pgpdump] add debug. [Terrtia] * Linked TOR installation instruction in the README. [Sami Mokaddem] * [Onion] add discovery queue. [Terrtia] * [Showpaste] check if tags are safe (img) + fix domain link. [Terrtia] * [crawler dashboard UI] add UP/Down domains url. [Terrtia] ### Fix * [update v2.6] fix faup encoding. [Terrtia] * [update v2.6] performance. [Terrtia] * [UI correlation graph] typo. [Terrtia] * [UI showDomain] fix down domain history. [Terrtia] * [Domain] domain was up. [Terrtia] * [Domain] is_domain_up. [Terrtia] * Typo. [Terrtia] * [Update] force manual update, fix #443. [Terrtia] * [UI] fix show paste modal. [Terrtia] * [screenshot canevas + domain link] fix item domain link + screenshot canevas: chg colors and icons for unsafe tags. [Terrtia] * [PgpDump] catch bs4 error. [Terrtia] * Placed Tor installation instruction in the installation section. [Sami Mokaddem] * [Update v2.4] fix empty set. [Terrtia] * [Update v2.4] fix empty set. [Terrtia] * [Item lib] fix import. [Terrtia] * [Paste submit] fix tags unpack. [Terrtia] * [Show Domain UI] fix screenshot link, fix #431. [Terrtia] * [Update] filter invalid tags. [Terrtia] ## v2.5 (2019-11-25) ### Changes * [MispModules] use ConfigLoader. [Terrtia] * [MispModules] mv misp modules to lib. [Terrtia] * [merge] merge master in misp_modules. [Terrtia] * [misp_modules] parse modules result. [Terrtia] * [misp_modules] add enrichment request. [Terrtia] * [misp_modules] create misp modules + generate config. [Terrtia] * [Correlation UI] btc address: get related transaction input/output. [Terrtia] * [Domain UI] add total_nb_correlation. [Terrtia] * [Domain UI] add unsafe_tags_check (taxo: dark-web + copine-scale) + hide unsafe domain img. [Terrtia] * [Corrrelation UI] add 404 return. [Terrtia] * [Correlation UI] add metadata card and item link: paste + domain. [Terrtia] * [Correlation UI + default taxonomies] add dark-web to default taxonomies + fix domain correlation url. [Terrtia] * [UI] show correlation (pgp. decoded, cryptocurrency) migration. [Terrtia] * [Correlation UI] add correlation blueprint + UI graph correlation. [Terrtia] * [doc] fix a typo. [Jean-Baptiste Rouault] * [Correlation Graph] add correlation by pgp, cryptocurrency, decoded and item + request correlation by type(pgp, ...) + objects(domain, paste) [Terrtia] * [Correlation Graph] add limit nb max nodes. [Terrtia] * [Correlation domain] add correlation graph by domain (union + intersection) [Terrtia] ### Fix * [update v2.5] [Terrtia] * [RoleManager] remove debug print. [Terrtia] * [correlation] merge conflict. [Terrtia] * [show paste min UI] typo. [Terrtia] * [show paste min UI] fix correlation url. [Terrtia] * [show paste min UI] fix correlation url. [Terrtia] * [Correlation UI] fix decoded file type + exist_correlation_id. [Terrtia] * [showDomain UI] fix template. [Terrtia] * [showDomain UI] fix template. [Terrtia] * [showDomain UI] fix template. [Terrtia] * [correlation UI] clean script. [Terrtia] * [showpaste url] fix decoded url. [Terrtia] * [Domain tag] avoid crawled tags. [Terrtia] * [Cryptocurrency] fix import conflict. [Terrtia] ### Other * Merge pull request #422 from CIRCL/misp_modules. [Alexandre Dulaunoy] Misp modules * Merge branch 'master' into misp_modules. [Terrtia] * Chg; [User role] add roles: user + user_no_api + read_only. [Terrtia] * Merge branch 'master' into misp_modules. [Terrtia] * Merge pull request #421 from jbrouault/patch-1. [Thirion Aurélien] chg: [doc] fix a typo ## v2.4 (2019-11-08) ### Changes * [update + show decoded items] add background update. [Terrtia] * [Domain] Show last origin. [Terrtia] * [Update v2.4] cp config. [Terrtia] * [Update] add v2.4 update. [Terrtia] * [core] mv bin/packages/config.cfg configs/core.cfg + use ConfigLoader. [Terrtia] * [UI Domain] UI: tag domain. [Terrtia] * [core + UI] refractor Domain + Show Domain, better handle of ports, select screenshot on the UI, ... [Terrtia] * [Domain + UI Crawler] refractor show domain. [Terrtia] * [Correlation] get correlation (crypto + pgp) by domain. [Terrtia] * [Domain] move Domain to lib/ [Terrtia] * [lib ConfigLoader] add configLoader class. [Terrtia] * [Domain] add domain object: tag + correlation (decoded items + tags + pgp + cryptocurrency) [Terrtia] ### Fix * [Decoded] add Decoded file. [Terrtia] * [showDomain] fix url pgp + cryptocurrency. [Terrtia] * [tests] add lib path. [Terrtia] ### Other * Merge pull request #419 from CIRCL/domain_object. [Thirion Aurélien] fix: [Decoded] add Decoded file * Merge pull request #418 from CIRCL/domain_object. [Thirion Aurélien] Domain object * Merge branch 'master' into domain_object. [Terrtia] ## v2.3 (2019-10-29) ### Changes * [Cryptocurrency + Tools] launch by default + remove old Bitcoin module. [Terrtia] * [Keys module] detect public key. [Terrtia] * [Tools detection] add tool detection module. [Terrtia] * [Cryptocurrency, RegexTracker] uptdate cryptocurrency list + fix: RegexTracker typo. [Terrtia] * [Cryptocurrency] add private_key entry + fix dash regex. [Terrtia] * [Cryptocurrency] add new Cryptocurrency module. [Terrtia] * [Tracker] add optionnal description field. [Terrtia] ### Fix * [Tool] fix searchsploit regex. [Terrtia] * [Tools] typo. [Terrtia] * [Tools] typo. [Terrtia] * [Tools] fix loop. [Terrtia] * [url_prefix] add root blueprint, fix:#403. [Terrtia] * [TermTracker] fix performance: disable token stats. [Terrtia] * [SQL module] fix typo. [Terrtia] ### Other * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] * Merge pull request #417 from andurin/master. [Alexandre Dulaunoy] Fix pybgpranking dependency in requirements * Fix pybgpranking dependency in requirements. [Hendrik] Relates #334 * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] * Merge pull request #404 from WimpyMan/master. [Thirion Aurélien] * LAUNCH.py: Added execution of script IPAddress.py. [Bastien Schils] * IPAddress.py: use ipaddress module. [Bastien Schils] Improved readibility, maintainability and use of standard module * Modules.cfg: Minor: Added \n to seperate sections. [WimpyMan] * Config.cfg.sample: Improved example for IP module. [WimpyMan] By default, the list of networks to monitor is now empty. The previous value is now given as example. * Added: IP matching module. [Bastien Schils] * Merge pull request #411 from krial057/patch-1. [Alexandre Dulaunoy] Fixed some typos * Fixed some typos. [krial057] Fixed some typos in the readme * Merge pull request #408 from stamparm/master. [Thirion Aurélien] Adding more tools * Adding more tools. [Miroslav Stampar] * Merge pull request #407 from stamparm/patch-1. [Thirion Aurélien] Covering special cases (on pastebin) * Covering special cases (on pastebin) [Miroslav Stampar] There is no need for checking `()` in case of (e.g.) Litecoin and Dash as those are also additionally checked with Bitcoin address verifier * Merge pull request #406 from stamparm/master. [Thirion Aurélien] Adding tool regexes * Adding tool regexes. [Miroslav Stampar] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] * Merge pull request #405 from stamparm/master. [Thirion Aurélien] Enforcing Base58 check on Litecoin and Dash addresses * Enforcing Base58 check on Litecoin and Dash addresses. [Miroslav Stampar] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] * Merge pull request #401 from stamparm/master. [Thirion Aurélien] Improvement of crypto-address regexes (lesser FPs) * Improvement of crypto-address regexes (lesser FPs) [Miroslav Stampar] * Merge pull request #398 from stamparm/master. [Thirion Aurélien] Implementation for different cryptocurrencies * Implementation for different cryptocurrencies. [Miroslav Stampar] * Merge pull request #396 from stamparm/master. [Thirion Aurélien] Improvement of SQLi detection * Improvement of SQLi detection. [Miroslav Stampar] ## v2.2 (2019-09-13) ### Changes * [API] add Tracker documentation. [Terrtia] * [Tracker] add more info. [Terrtia] * [update] add v2.2 update + fix default update + fix Empty Item.get_tags() + add new LAUNCHER options. [Terrtia] * [tracker] add missing btn. [Terrtia] * [trackers] filter trackers list by type + minor fix. [Terrtia] * [UI sparklines] sparklines: fix + factory. [Terrtia] * [merge] master. [Terrtia] * [UI term tracker] refractor term management: trackers list + show trackers + add new trackers. [Terrtia] * [api] add endpoint: get tracked item_id by uuid and daterange. [Terrtia] * [api] add endpoint: delete tracker term (regex/set/word) [Terrtia] * [Term Tracker] refractor term tracker word/set/regex modules + remove old modules. [Terrtia] * [Term tracker] add term tracker module (word + set) + API: add new term to track (word + set + regex) [Terrtia] * [term] refractor + add new tracked word/set. [Terrtia] * [README] add link to API documentation. [Thirion Aurélien] * [tests API] use argv api key. [Terrtia] * [api] add new endpoints: get bitcoin/pgp name/pgp keys/pgp mail metadata + items list. [Terrtia] ### Fix * [sparkline] datatable drawing. [Terrtia] * [d3 graph] fix script path. [Terrtia] * Add missing file. [Terrtia] * [d3 js plugin] [Terrtia] * [template] add trackers. [Terrtia] * [Term Tracker module] chg module flow. [Terrtia] * [BankAccount] fix #385. [Terrtia] * [API doc] get bitcoin metadata + list of items. [Terrtia] * [UI caching] avoid domain archive caching. [Terrtia] * Typo. [Terrtia] * [UI caching] fix: #373 avoid screenshot caching. [Terrtia] ### Other * Merge pull request #391 from CIRCL/term_trackerv1. [Thirion Aurélien] fix: add missing file * Merge pull request #390 from CIRCL/term_trackerv1. [Thirion Aurélien] fix: [d3 js plugin] * Merge pull request #389 from CIRCL/term_trackerv1. [Alexandre Dulaunoy] Term trackerv1 ## v2.1 (2019-08-14) ### Changes * [api] 405 response: add url to api endpoint documentation. [Terrtia] * [api doc] test anchor link. [Terrtia] * [api doc] test anchor link. [Terrtia] * [api] add bruteforce protection. [Terrtia] * [api] use POST with parameters + add API unittest. [Terrtia] * [api] use POST with parameters + add API unittest. [Terrtia] * [api] add api versioning. [Terrtia] * [api] add new endpoints: get tag metadata + get all tags. [Terrtia] * [api] add advanced get item via POST + use same query for each get item. [Terrtia] * [api] add 404 errors code + add future endpoints in doc. [Terrtia] * [API v1] add API documentation + update/delete items tags + Flask_tags refractor. [Terrtia] * [API] add GET: item metadata + item content + item tags. [Terrtia] * [API] import item (text) [Terrtia] * [API + import] add API format + item_import refractor. [Terrtia] * [rest API] access: check token user role. [Terrtia] * [reset_AIL] add echo message. [Terrtia] * [Crawler] add default crawler config + update default user_agent. [Terrtia] * [requirements] fix pycountry force 18.12.8 as recommended in #366. [Alexandre Dulaunoy] * [README] add python minimal supported version. [Thirion Aurélien] * [doc] add huge screencast. [Alexandre Dulaunoy] * [doc] screenshot added. [Alexandre Dulaunoy] * [doc] cleanup. [Alexandre Dulaunoy] * [doc] move the Docker junk. [Alexandre Dulaunoy] * [doc] old Python 3 upgrade added. [Alexandre Dulaunoy] ### Fix * [api doc] typo. [Thirion Aurélien] * [api doc] typo. [Thirion Aurélien] * [doc] typo. [Terrtia] * [api] chg api role. [Terrtia] * [test api] typo. [Terrtia] * [api doc] typo. [Terrtia] * [api] fix errors handler 404 405, return json. [Terrtia] * [api] fix documentation errors output. [Terrtia] * [Tags + api] fix dict keys name + fix documentation errors output. [Terrtia] * Typo. [Terrtia] * [reset_AIL] launch ardb + remove dir. [Terrtia] * [reset_AIL] launch ardb. [Terrtia] * [reset_AIL] fix #368. [Terrtia] * [reset_AIL] #368 fix. [Terrtia] * [reset_ail] fix directory name. [Terrtia] * [ansible docker] move files, fix 365. [Terrtia] * [Crawler] incorrect config. [Terrtia] ### Other * Merge pull request #370 from CIRCL/api_v1. [Alexandre Dulaunoy] Api v1 * So many spelling mistakes have been corrected in this commit. It's totally worth writing so much explanation... [Sascha Rommelfangen] * Consistency and wording. [Sascha Rommelfangen] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] ## v2.0 (2019-07-05) ### Changes * [helper] dump crawler history by daterange. [Terrtia] * [UI submit items] bootstrap 4 migration. [Terrtia] * [Flask login] add brute force protection + log login errors. [Terrtia] * [helper] generate self signed certificates. [Terrtia] * [Flask server] https support + create self signed certificate. [Terrtia] * [user_management 2.0] add update scripts + fix create_default_user. [Terrtia] * [UI user_management] user_role acl: hide admin panel. [Terrtia] * [UI user_management] incorrect passwords: display errors. [Terrtia] * [user_management endpoint] check user roles + add 503 template. [Terrtia] * [UI dashboard + search] bootstrap 4 migration: dashbaord + fix search input. [Terrtia] * [UI crawler, show_domain] domain history: remove target blank. [Terrtia] * [UI crawler, show_domain] add domain history list + navigation. [Terrtia] * [user_managemant] clean code + check password and email length. [Terrtia] * [user_management UI] add admin section: edit + create users. [Terrtia] * [user_management UI] edit my_profile + renew api tokens. [Terrtia] * [user_management] create default admin user (temp passwd save in AIL_HOME) + change password UI + logout UI + create random password. [Terrtia] * [restapi] add rest api authentification + create default user. [Terrtia] * [user_management] add user role_management. [Terrtia] * [user_management] create + check user password. [Terrtia] * [UI] add basic user management. [Terrtia] ### Fix * [role_manager] add password characters. [Terrtia] * [secrets import] use urandom. [Terrtia] * [items submit UI] fix tags dropdown. [Terrtia] * [helper dump_crawler] fix files not found. [Terrtia] * [helper dump_crawler] fix empty dict. [Terrtia] * [MISP export] fix event creation. [Terrtia] * [UI items_submit] add active tag + fix template name. [Terrtia] * [UI login/change_password] add missing ail-logo. [Terrtia] * [Update] clean output. [Terrtia] * [Update] add default update script. [Terrtia] * [install_dep] create update current_version. [Terrtia] * [UI settings] fix toggle_sidebar. [Terrtia] * [install_dep] create default user. [Terrtia] * [user_management] fix tokens duplicate + check user_acl_integrity + add login errors messages. [Terrtia] * [server endpoint] unknow users: avoid endpoint enumeration. [Terrtia] * [TheHive feeder] create_alert: push all items tags TODO check items status + add more item metadata. [Terrtia] * [domain history] fix domain status. [Terrtia] ### Other * Update README.md. [Thirion Aurélien] * Merge pull request #359 from CIRCL/user_management. [Alexandre Dulaunoy] User/role management * Merge branch 'master' into user_management. [Terrtia] * Merge branch 'master' into user_management. [Terrtia] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] * Update README.md. [Thirion Aurélien] * Merge branch 'master' into user_management. [Terrtia] * Merge branch 'user_management' of https://github.com/CIRCL/AIL-framework into user_management. [Terrtia] * Update README.md. [Thirion Aurélien] Update install instructions * Merge branch 'master' into user_management. [Terrtia] * Merge branch 'master' into user_management. [Terrtia] ## v1.8 (2019-06-12) ### Changes * [UI crawled domains] Download all domain content (HTML + HAR + screenshot) [Terrtia] * [backend crawler] domains: download 1 archive by crawled (most recent) [Terrtia] * [paste_submit UI] add debug. [Terrtia] * [template] the "item" project. [Alexandre Dulaunoy] ### Fix * [PgpDump] catch UnicodeDecodeError error. [Terrtia] * [backend crawler] rename downloaded archive. [Terrtia] * [paste_submit UI] filter empty file field. [Terrtia] * [PgpDump] process large pgp blocks. [Terrtia] * [paste_submit UI] filter empty file field. [Terrtia] * [UI crawler endpoints] display crawler status + fix #353. [Terrtia] * [update v1.7] add bs4 requirement. [Terrtia] ## v1.7 (2019-05-24) ### Changes * [PgpDump] PMF fix - longer keys are truncated. [Alexandre Dulaunoy] * [correlation] clean files. [Terrtia] * [update v1.7] update thirdparty. [Terrtia] * [correlation] add cryptocurrency + refractor correlation. [Terrtia] * [Bitcoin] map cryptocurrency: bitcoin (DB pivot) [Terrtia] * [update v1.7] add update scripts. [Terrtia] * [pgpdump] reprocess tagged items + fix pgpdump. [Terrtia] * [Update] force update order. [Terrtia] * [PgpDump] fix graph + add new tags: pgp-signature pgp-public-key-block + avoid keys injection in pgp user_id. [Terrtia] * [decoded UI] add PgpDump UI + fix hashdecoded js. [Terrtia] * [decoded items] bootstrap 4 migration. [Terrtia] * [PgpDump] add PgpDump backend TODO: UI. [Terrtia] * [crawler] manual/auto crawler: always save screenshots. [Terrtia] * [crawler] manual/auto crawler: always save screenshots. [Terrtia] ### Fix * [correlation] fix graph node endpoint. [Terrtia] * [correlation] fix endpoint. [Terrtia] ### Other * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] * Update README.md. [Thirion Aurélien] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] * Merge pull request #349 from kovacsbalu/fix-paste-encoding. [Thirion Aurélien] Fix #314 * Use default encoding error from redis. [kovacsbalu] * Fix #314 Replace char on redis encoding error. Try to use local file on other error. [kovacsbalu] * Merge pull request #350 from kovacsbalu/fix-crawler-rotation. [Thirion Aurélien] fix: [crawler] rotation * Hopp, single quote :) [kovacsbalu] * Fix crawler rotation. [kovacsbalu] Before this, crawler processed prioritized onions and after all starts prioritized regular. ## v1.6 (2019-05-13) ### Changes * [travis] Travis has his own venv where it installs "stuff". Now we detect and us it in the launcher. [Steve Clement] * [travis] Require Python 3.6 to make build faster. [Steve Clement] * [doc] Some stats on build status/gitter etc. [Steve Clement] * [hashDecoded] cleanup for the VT message + PEP. [Alexandre Dulaunoy] ### Fix * [faup] fix new return types (bytes to str) [Terrtia] * [Crawler] force domains/subdomains lower case (rfc4343) [Terrtia] * [showpaste] fix: #346, avoid None screenshots. [Terrtia] * [python requirements] rename file. [Terrtia] * [crawler] typo: domains down. [Terrtia] * [travis] LAUNCH.sh needs -l to launch... [Steve Clement] * [travis] Next round of travis fixes. LAUNCH.sh is the only launch script needed. chg: [installer] Be way more quiet, watching unzips is only fun during development. chg: [installer] Make the arch one +x. [Steve Clement] * [installer] Avoid doing funky **sudo pip install** moves, it breaks python on package managed python installs, if ever, use the **--user** flag. [Steve Clement] * [travis] Try and require xenial (16.04) and see if it works better. [Steve Clement] * [travis ] There are issues on the 14.04 build system of Travis. This fixes it temporarily. [Steve Clement] * [Onion] typo. [Terrtia] ### Other * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] * Chg; [doc crawler] use the torproject torrc. [Terrtia] ## v1.5 (2019-04-26) ### New * [slides] Added training slides december 2018. [Sami Mokaddem] ### Changes * [crawler UI] bootstrap 4 migration: show domains by daterange + bug fix hidden services tags. [Terrtia] * [overview] update db doc. [Terrtia] * [update v1.5] add background update: screenshots_crawled. [Terrtia] * [crawled screenshot] use sha256 as filepath. [Terrtia] * [config update] refractor, auto update, create config if file doesnt exist. [Terrtia] * [UI crawler] add splash dashboard. [Terrtia] * [Onion, crawler config] auto crawler: add config by url, fix onions tagging + filter subdomains. [Terrtia] * [UI crawler] status/remove auto crawler. [Terrtia] * [UI DOC] add dynamic update message + add bootstrap 4 migration message + add update doc (DB) [Terrtia] * [v1.5 background update] create update stats. [Terrtia] * [settings UI] add background update progress. [Terrtia] * [git directory] add git helper. [Terrtia] * [UI] add server Management, check if new update/version are available + background update status + link to new realease/commits. [Terrtia] * [UI] update nav bar endpoint. [Terrtia] * [Update v1.5] update tracked terms path. [Terrtia] * [OVERVIEW] add doc termfred DB. [Terrtia] * [Update + Launcher] use background update + add new options in launcher. [Terrtia] * [update] add update script Redis ARDB + add background update. [Terrtia] * [DB] update items tags metadata. [Terrtia] * [DB] remove absolute path. [Terrtia] * [merge] merge update into advanced_crawler. [Terrtia] * [merge] merge master into update branche. [Terrtia] * [Overview] add doc. [Terrtia] * Update Overview. [Terrtia] * [DB Migration] add hash dynamic update. [Terrtia] * [update DB] add migration script. [Terrtia] * [Update] add debug. [Terrtia] * [Launch] remove old launchers. [Terrtia] * [Launch] remove old launchers. [Terrtia] * [update 1.5] add update bash. [Terrtia] * [Update] add 1.5 update, remove absolute paths from DB. [Terrtia] * [Update] add additonal update. [Terrtia] * [Update] generate current version. [Terrtia] * [gitignore] add update version. [Terrtia] * [LAUNCH] update: launch + check BDD. [Terrtia] * [LAUNCH] add AIL update by default. [Terrtia] * [Update] add update script. [Terrtia] * [CVE] add stat script. [Terrtia] * [doc] badge release updated. [Alexandre Dulaunoy] * [Crawler] handle port: crawling + history. [Terrtia] * [UI + item submit] remove browse paste from UI + fix pasteSubmit tagging. [Terrtia] * [Tags alertHandler] remove alertHandler + add quick tag search sidebar + use tag last_seen to display tagged items. [Terrtia] * [tags] remove/add tags by daterange + improve item loading time. [Terrtia] * [Tags] add backend, daterange selector. [Terrtia] * [UI preview items] b4 migration + new template + better display + decrease item loading time + add basic animation/transition. [Terrtia] * [UI Tags search] bootstrap 4 migration. [Terrtia] * [UI Tags] start b4 migration + add date range selector. [Terrtia] * [Flask UI] refractor crawler frontend + show crawled domain. [Terrtia] * [Crawler UI] add basic tooltips. [Terrtia] * [Crawler UI Flask] refractor functions, templates. [Terrtia] * [css] add basic toggle sidebar. [Terrtia] * [Crawler UI] Crawler major refractor (end) + basic UI for manual crawler. [Terrtia] * [Crawler] major refractor. [Terrtia] * [Crawler] refractor. [Terrtia] * [Crawler] refractor. [Terrtia] * [thirdparty_update] get b4 datatable. [Terrtia] * [Crawler UI Tags] add tag by day + add crawler status + UI onion blacklist. [Terrtia] * [Overview] add db map tag. [Terrtia] * [Tags] tags by day. [Terrtia] * [Crawler UI] fix css + bootstrap4 migration. [Terrtia] * [UI] font icon v5 + navbar + sidebar. [Terrtia] * [UI] basic navbar + sidebar + refractor. [Terrtia] * [Crawler] add bootsrap4 src + refractor crawler. [Terrtia] * [Crawler] add onion to blacklist. [Thirion Aurélien] * [doc] how to cite AIL in academic publications. [Alexandre Dulaunoy] * [Crawler] add priority queue, fix #263. [Terrtia] * [hiddenServices] crawlers status, dynamic update. [Terrtia] * [hiddenServices] auto update last crawled domains. [Terrtia] * [OVERVIEW] update db map. [Terrtia] * [hashDecoded] add top5 types piechart (v0.1) [Terrtia] * [hashDecoded] add option to display decoded files, fix #271. [Terrtia] * [UI Crawled Domains] show crawled domains by date range (options: domains tags, up domains, down domains) todo: Shows domain history. [Terrtia] * [bash] Small PATH precedence issue. [Steve Clement] * [bash] Replace '' with "" [Steve Clement] * [doc] Added note that redis-server is bundled during install. [Steve Clement] * [python] Added necessary checks in LAUNCH.sh to be able to launch the script without doing bin/activate chg: [doc] Added correct pystemon path fix: [zmq] Small ZMQ test fix. fix: [doc] Moved the print statement in feeder so it is less verbose but still useful. [Steve Clement] * [Mixer] config default feeder name #291. [Terrtia] * [terms_management] delete terms tags. [Terrtia] * [terms_management] delete email. [Terrtia] * [tag export] UI, add custom tags. [Terrtia] * [auto push tag] delete unwanted tags. [Terrtia] * [Terms] tag tracked terms. [Terrtia] * [NotificationHelper] add error traceback. [Terrtia] * [doc] cannot type in the train... [Alexandre Dulaunoy] * [doc] Tor hidden services updated. [Alexandre Dulaunoy] * [doc] Tor crawler screenshot added. [Alexandre Dulaunoy] ### Fix * [submit paste] remove item full path, use rel path. [Terrtia] * [paste submit] update broken endpoint + get item date. [Terrtia] * [update v1.5] create update config file. [Terrtia] * [update v1.5] fix update thirdparty. [Terrtia] * [Update] typo. [Terrtia] * [v1.5 background update screenshot] remove duplicate files. [Terrtia] * [update v1.5] remove unbuffer. [Terrtia] * [Crawler] unpack_url. [Terrtia] * [LAUNCH] fix typo. [Terrtia] * [update] fix is_fork check. [Terrtia] * [UI dashboard] typo. [Terrtia] * [UI show domain] fix domain/port unpacking. [Terrtia] * [Update tags + UI] use srandmember + display update warning. [Terrtia] * [tracked terms] show mime type. [Terrtia] * [dynamic update v1.5] make sure updates are excuted in the correct order + fix nb_seen_hash dynamic update. [Terrtia] * [Update DB] remove indexErrors. [Terrtia] * [background update] avoid multiple update on the same key. [Terrtia] * [Global] remove absolute path from item path. [Terrtia] * [UI showpaste] min preview. [Terrtia] * [AIL] use only relative paths pastes duplicates are fixed on the fly. [Terrtia] * [Scripts] Remove absolute path. [Terrtia] * [update_thirdparty] detect and use virtualenv. [Terrtia] * [update v1.5] [Terrtia] * [DB fix] performance. [Terrtia] * [update 1.4] typo. [Terrtia] * [LAUNCH] add update in helper. [Terrtia] * [Update] cleaning. [Terrtia] * [CVE] fix stat regex. [Terrtia] * [Onion] filter too many subdomain. [Terrtia] * [Crawler] typo. [Terrtia] * [Crawler] typo. [Terrtia] * [Crawler] typo. [Terrtia] * [Global Crawler] max filename size. [Terrtia] * [Crawler] fix index + redis history key. [Terrtia] * [UI terms] update tags endpoint. [Terrtia] * [Tags] cast nb_max_page to int. [Terrtia] * [UI Tags] paste preview. [Terrtia] * [UI crawler manual] fix min input. [Terrtia] * [UI] fix flot js install. [Terrtia] * Reset alarm timeout. [Xavier Mertens] * [Crawler] change max page crawled. [Terrtia] * [Crawler] first_seen. [Terrtia] * [hiddenServices] catch empty data. [Terrtia] * [Onion] typo. [Terrtia] * [Crawler] limit max crawled pages. [Terrtia] * [Onion] loop. [Terrtia] * [Crawler] typo. [Terrtia] * [Crawler] fix crawler cache info. [Terrtia] * [Crawler] fix onion blacklist + add crawler info. [Terrtia] * [Onion] add timeout signal. [Terrtia] * [hiddenServices] typo. [Terrtia] * [Crawler] Restart Splash on failure, limit unbound in memory cache (maxrss) [Terrtia] * Cleaning ... [Terrtia] * [hiddenServices UI] fix none type. [Terrtia] * [Splash server] add debug output. [Terrtia] * [Crawler] retry when splash is not available. [Terrtia] * [requirements] remove duplicate entry, #296. [Thirion Aurélien] * [requirement] force redis version. [Thirion Aurélien] * [install script] use virtualenv package. [Terrtia] * [bash] fixed log_subscriber path. [Steve Clement] * [bash] evoking bash -c and splitting vars in this fashion wants to be changed in future revisions. [Steve Clement] * [bash] As we instantiate from a subdirectory, we need to take this into account. [Steve Clement] * [lib_refine] add dns configuration. [Terrtia] * [MISP_The_Hive_feeder] handle not saved pastes. [Terrtia] * [term_management] delete tags. [Terrtia] * [term_management] delete emails. [Terrtia] * [tag export] limit custom tag lenght. [Terrtia] * [MISP_the_Hive_feeder] force absolute path. [Terrtia] * [update_thirdparty] upgrade github source. [Terrtia] * [MISP_The_Hive_feeder] typo. [Terrtia] * Typo. [Terrtia] * [NotificationHelper] add config. [Terrtia] * [Search] tags display. [Terrtia] * [Search] Improve search performance note: search by paste name is temporary disabled. [Terrtia] * [MISP_The_Hive_feeder] set default values. [Terrtia] * [Duplicate] show duplicate with relative path. [Terrtia] * [README] typo docker install. [Thirion Aurélien] * [Paste] create absolute path. [Terrtia] ### Other * Merge pull request #342 from CIRCL/advanced_crawler. [Alexandre Dulaunoy] Advanced_crawler + tag by daterange + bootstrap 4 migration + bugs fix * Merge branch 'master' into update. [Terrtia] * Merge master. [Terrtia] * Merge branch 'master' into advanced_crawler. [Terrtia] * Merge pull request #336 from 5thColumn/docker. [Thirion Aurélien] Add indexdir volume to docker-compose * Add indexdir volume to docker-compose. [Jason Hedden] Make the indexdir data persistent and shared with Flask for searching pastes. * Merge pull request #335 from 5thColumn/docker. [Alexandre Dulaunoy] Add docker-compose support * Add docker-compose support. [Jason Hedden] This commit separates the AIL Framework services into individual Docker containers that can be managed with docker-compose. It's intended to ease the installation, development and troubleshooting procedures for all-in-one deployments. No changes to architecture or base code are required in this commit. Changes to existing files: .gitignore * Ignore pystemon archive directory Dockerfile: * Add AIL shell environment variables * Install pystemon and crawler pip requirements into AIL virtual environment Docker-compose notes: * All containers are participating in a service network `network_mode: service:flask`. This allows the containers to share the same IP namespace to accommodate hard-coded localhost entries. * By default persistent data is saved to the following directories in the local AIL framework git directory: PASTES, HASHS, CRAWLED_SCREENSHOTS, pystemon/archives This was tested with docker-ce on Ubuntu 16.04 and MacOS. A typical deployment would look like: ``` git clone https://github.com/CIRCL/AIL-framework.git cd AIL-framework cp bin/packages/config.cfg.docker-compose-sample bin/packages/config.cfg (optionally enable activate_crawler in config.cfg) docker-compose build docker-compose up -d ``` * Merge pull request #334 from 5thColumn/pybgpranking. [Alexandre Dulaunoy] Add pybgpranking to install deps * Add pybgpranking to install deps. [Jason Hedden] The python module DomainClassifier requires pybgpranking which is not available in the community PyPI repository. This commit installs pybgpranking from git using the same version required by DomainClassifier https://github.com/adulau/DomainClassifier Fixes the following installation error when using installing_deps.sh: ``` Could not find a version that satisfies the requirement pybgpranking (from DomainClassifier->-r pip3_packages_requirement.txt (line 50)) (from versions: ) No matching distribution found for pybgpranking (from DomainClassifier->-r pip3_packages_requirement.txt (line 50)) ``` * Merge pull request #333 from 5thColumn/fix_modulesinformationv2. [Thirion Aurélien] Fix current_selected amount and action vars * Fix current_selected amount and action vars. [Jason Hedden] This commit fixes issue #268 ``` File "./bin/ModulesInformationV2.py", line 319, in _setValue action = current_selected_action + " " + str(current_selected_amount) + "x" ``` `current_selected_action` was defined twice. The second occurance should be `current_selected_amount`. This also fixes the generic exception from failing when `current_selected_value` is not defined. * Merge pull request #338 from CIRCL/pre_update. [Thirion Aurélien] Merge Pre update * Merge branch 'master' into pre_update. [Terrtia] * Chg [Update] update AIL clone and fork. [Terrtia] * Merge branch 'master' into advanced_crawler. [Terrtia] * Merge pull request #319 from xme/master. [Thirion Aurélien] fix: [Onion] Timeout error in Onion.py * Cleanup. [Xavier Mertens] * Merge remote-tracking branch 'upstream/master' [Xavier Mertens] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] * Merge pull request #318 from raggadhub/master. [Thirion Aurélien] [Redis] by default just listen on loopback interface [security hardening] * 6381 - by default just listen on loopback interface [security hardening] [raggadhub] user will change explicitly the bind option if necessary * 6380 - by default just listen on loopback interface [security hardening] [raggadhub] user will change explicitly the bind option if necessary * 6379 - by default just listen on loopback interface [security hardening] [raggadhub] user will change explicitly the bind option if necessary * Blacklist onion debian manpages. [Thirion Aurélien] * Merge pull request #311 from raggadhub/patch-2. [Thirion Aurélien] fix path_to_send path * Fix path_to_send path. [raggadhub] Hi, Missing slash between pastes_directory and paste in path_to_send = pastes_directory+paste path error example: /opt/AIL-framework/PASTESarchive/pastebin.com_pro/2019/01/07/cAgqsY2U.gz * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] * Merge pull request #297 from swedishmike/Dockerfile. [Alexandre Dulaunoy] Change to the Dockerfile * Changed the Dockerfile so that it starts with installing any and all outstanding updates for Ubuntu. [Mike Eriksson] * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Alexandre Dulaunoy] * Training added. [Alexandre Dulaunoy] * Merge pull request #287 from CIRCL/term_management. [Sami Mokaddem] Term management upgrade * Merge pull request #282 from xme/master. [Alexandre Dulaunoy] Added Tor binding config * Update LAUNCH.sh. [Xavier Mertens] * Fixed line counter (count all lines) [Xavier Mertens] * Added timeout for Regex module. [Xavier Mertens] * Added timeout to avoid blocking regexes. [Xavier Mertens] * Added module Regex.py. [Xavier Mertens] * Added Tor binding config. [Xavier Mertens] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] * Merge pull request #266 from kovacsbalu/fix-regexp-timeout. [Sami Mokaddem] Stop regexp processing after timeout (60sec) * Fix config section name. [kovacsbalu] * RegexpForTerm timeout from config Critical log removed. [kovacsbalu] * Stop regexp processing after timeput (60sec) Minor pep8 fixes. [kovacsbalu] * Common onion addresses. [Alexandre Dulaunoy] * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Alexandre Dulaunoy] * Merge pull request #262 from kovacsbalu/feat-smtp-test. [Alexandre Dulaunoy] Test notification * Test notification Minor pep8 fixes. [kovacsbalu] ## v1.4 (2018-10-02) ### Changes * [HOWTO] add a warning regarding the Tor configuration and SOCKS5 proxy. [Alexandre Dulaunoy] * [HOWTO] clarify that AIL framework is required for the crawler part. [Alexandre Dulaunoy] * [config] Tor crawler disabled by default in sample configuration (to match HOWTO when enabling it) [Alexandre Dulaunoy] * [Crawler] add domains blacklist. [Terrtia] * [Crawler] domains stats + logs + clean. [Terrtia] * [Crawler] css + limit splash RAM. [Terrtia] * [Crawler] add docs. [Terrtia] * [crawler] add infos. [Terrtia] * [Flask] add prefix in config to flask routes. [Terrtia] * [Flask] add prefix in config to flask routes. [Terrtia] * [statistics] clean scripts. [Terrtia] * [statistics] add domain statistics. [Terrtia] * [statistics] add script to generate pie chart png. [Terrtia] * [sglinjection Phone] add tld statistic, fix phone regex. [Terrtia] * [Mail Credential] add tld statistic. [Terrtia] * [Mail] add country statistic. [Terrtia] * Slides from workshop in Luxembourg added. [Alexandre Dulaunoy] * [Crawler] add launcher and install. [Terrtia] * [HiddenServices] get domain related to other domains. [Terrtia] * [UI] pixelate crawled screenshot by default. [Terrtia] * [Crawler] catch server response. [Terrtia] * [Crawler UI] display domain information. [Terrtia] * [Crawled Domain] show crawled pastes by domain. [Terrtia] * [Domain crawled] add random screenshot. [Terrtia] * [I2P] add default config. [Terrtia] * [Crawler] crawler accept all kind of domains. [Terrtia] * [Crawler] change BDD, save i2p links. [Terrtia] * [Showpaste] add screenshot + improve onion db. [Terrtia] * [Onion] change onion regex, fix crawler. [Terrtia] * [Onion] add onion splash crawler. [Terrtia] ### Fix * Typo. [Terrtia] * [pystemon-feeder] paste save path. [Terrtia] * [Crawler] remove hardcoded url. [Terrtia] * [Flask] remove hardcoded url. [Terrtia] * [Flask] remove hardcoded url. [Terrtia] * [Flask] remove hardcoded url. [Terrtia] * [Crawler] detect splash connection to proxy error. [Terrtia] * [pystemon] pystemon encoding. [Terrtia] * [MISP feeder] typo filename-config name. [Terrtia] * [ModulesInformationV2] fix #244. [Terrtia] * [hashDecoded] range_type_json. [Terrtia] * [config] add virus total API key sample, fix #253. [Terrtia] * [import_dir] fix #251, fix paste import if the date directory tree don't exist, he will be generated. [Terrtia] * [LAUNCH.sh] make pystemon optionnal. [Terrtia] AIL and Pystemon are independent, pystemon shouldn't be launch by default. * [LAUNCH.sh] remove old decoders. [Terrtia] * [Install] add python and package requirements. [Terrtia] * [Install] add python requirements. [Terrtia] * [statistics] remove unused graph. [Terrtia] * [showpaste] path transversal. [Terrtia] * [Crawler] clean. [Terrtia] * [Crawler] clean. [Terrtia] * [crawled screenshot] fix img error. [Terrtia] * [Crawler] save domain to crawl on splash error. [Terrtia] * Merge. [Terrtia] * [doc] IBAN feature added. [Alexandre Dulaunoy] ### Other * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Alexandre Dulaunoy] * Merge pull request #260 from CIRCL/onion_crawler. [Alexandre Dulaunoy] Onion crawler * Merge branch 'master' into onion_crawler. [Terrtia] * Merge branch 'onion_crawler' of https://github.com/CIRCL/AIL-framework into onion_crawler. [Terrtia] * Update HOWTO.md. [Sami Mokaddem] * Update HOWTO.md. [Sami Mokaddem] * Merge branch 'master' into onion_crawler. [Terrtia] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] * Merge pull request #243 from kovacsbalu/docker-with-pystemon. [Alexandre Dulaunoy] Docker build with pystemon * Fix docekrfile CMD. [kovacsbalu] * Install pystemon and start pystemon-feeder in docker. [kovacsbalu] * Merge pull request #250 from CIRCL/statistic. [Alexandre Dulaunoy] Statistics * Merge branch 'master' into statistic. [Terrtia] * Add iban country statistic. [Terrtia] * Merge branch 'master' into statistic. [Terrtia] * Merge pull request #248 from CIRCL/revert-242-moreComments. [Thirion Aurélien] Revert "Added more documentation/descriptions/..." * Revert "Added more documentation/descriptions/..." [Thirion Aurélien] * Merge pull request #242 from mokaddem/moreComments. [Thirion Aurélien] Added more documentation/descriptions/... * Added more documentation/descriptions/... [Sami Mokaddem] * Pixelate paste screenshot. [Terrtia] * Merge branch 'master' into onion_crawler. [Terrtia] * Merge branch 'master' into onion_crawler. [Terrtia] ## v1.3 (2018-08-24) ### New * Add base64 display on paste + base64 statistics. [Terrtia] * Added pass the SALT workshop slides. [Sami Mokaddem] ### Changes * [doc] screenshots for 1.3 release. [Alexandre Dulaunoy] * [doc] new screenshots for hashed file functionality in 1.3. [Alexandre Dulaunoy] * [Hash] add Help on graph node. [Terrtia] * [Hash] add reprocess regex order. [Terrtia] * [UI-Hash] add decoder pie. [Terrtia] * [UI-Hash] add decoded type on graph. [Terrtia] * Refractor base64 encoded to hash. [Terrtia] * Use json to submit hash. [Terrtia] * Change base64 icon. [Terrtia] * [Mixer] add submitted paste. [Terrtia] * [UI] add AIL icon, fix li links. [Terrtia] * [Keys] typo. [Terrtia] * [Keys] detect ssh2 private key. [Terrtia] * [BankAccount] regex. [Terrtia] * [UI-dashboard] add logs number configuration. [Terrtia] * [UI-dashboard] convert time to local. [Terrtia] * Display last logs. [Terrtia] * [doc] Marked unmaintained efforts as such. [Steve Clement] * [docker] allow arg as a parameter. [Steve Clement] * [UI] Typo fixed as requested by our favourite German. [Alexandre Dulaunoy] * Screenshots updated to the latest version. [Alexandre Dulaunoy] ### Fix * [doc] badges are cool. [Alexandre Dulaunoy] * [doc] new 1.3 features. [Alexandre Dulaunoy] * [HashDecoded] typo. [Terrtia] * [HashDecoded] change template title. [Terrtia] * [UI-Hash] pie chart json. [Terrtia] * [UI-Hash] pie decoder. [Terrtia] * Remove experimental modules. [Terrtia] * Decoder bdd. [Terrtia] * Binary decoder base2 string to bytes. [Terrtia] * Paste submit css. [Terrtia] * Release tiemeout alarm, chg:icon graph node. [Terrtia] * [pystemon-feeder] fix #234. [Terrtia] * Add miss config. [Thirion Aurélien] * [UI-Showpaste] show raw paste, force text/plain mimetype. [Terrtia] * Typo. [Terrtia] * [Queues] module with no subscriber. [Terrtia] * [hive auto export] fix #221. [Terrtia] * [LAUNCH] script checking. [Terrtia] * [LAUNCH] use envar, add options (launch all, flask, killall) [Terrtia] * Remove hardcoded mail url. [Terrtia] * Add LibInjection to LAUNCH.sh. [Terrtia] * Date format https://github.com/CIRCL/AIL-framework/issues/219. [Terrtia] * Term_management UI bug ,https://github.com/CIRCL/AIL-framework/issues/214. [Terrtia] * Hive_feeder hiveApi declaration. [Thirion Aurélien] * File submission, content type. [Thirion Aurélien] * Catch file error in paste submission. [Terrtia] * Update submitted tags link. [Terrtia] * LevelDB not more used. [Alexandre Dulaunoy] ### Other * Merge pull request #237 from CIRCL/base64. [Alexandre Dulaunoy] Hashed Files * Merge branch 'master' into base64. [Terrtia] * [fix: [UI-hash] fix graph node zoom. [Terrtia] * Merge branch 'master' into base64. [Terrtia] * Add: Decoder module. [Terrtia] * Add: Decoder Module, decode binary, hex and base64. [Terrtia] * Add: Binary decoder. [Terrtia] * Add: hash download. [Terrtia] * Add: hash line_graph (nb/day) [Terrtia] * Add: base 64 node graph. [Terrtia] * Add: show hash info. [Terrtia] * Add: stacked barchart, resize chart. [Terrtia] * Add date_range stacked bar. [Terrtia] * Add: base64 date and type selector. [Terrtia] * Change import d3.js to d3.min.js. [Terrtia] * Add d3 and daterange js. [Terrtia] * Fix submit paste. [Terrtia] * Merge branch 'master' into base64. [Terrtia] * Update OVERVIEW. [Terrtia] * Merge pull request #235 from P4rs3R/patch-2. [Thirion Aurélien] Very little typo * Very little typo. [A. Cristallo] [Grammar Nazi mode] Sorry guys, I just HAD to :D [/Grammar Nazi mode] * Merge pull request #232 from CIRCL/iban. [Thirion Aurélien] Add Iban detection * Clean. [Terrtia] * Add: [BankAccount] iban detection. [Terrtia] * Merge pull request #231 from CIRCL/logs. [Thirion Aurélien] chg: [UI-dashboard] display last logs, fix #41 * Merge pull request #230 from CIRCL/launch. [Thirion Aurélien] fix: [LAUNCH] use envar, add options (launch all, flask, killall) * Update README.md. [Steve Clement] * Merge pull request #229 from SteveClement/master. [Steve Clement] chg: [doc] Marked unmaintained efforts as such. * Merge pull request #226 from schutzwerk/sw-direct-io-note. [Steve Clement] chg: [doc] README.md: Add note regarding Direct I/O * README.md: Add note regarding Direct I/O. [Philipp Schmied] * Add: new mail body and starttls. [Terrtia] * Merge pull request #215 from buixor/master. [Thirion Aurélien] [WIP] LibInjection Module add: LibInjection Module * Add libinjection bindings to pip requirements. [bui] * Add LibInjection Module. [bui] * Initial LibInjection module. [bui] * Merge pull request #222 from raw-data/master. [Thirion Aurélien] [fix] switch from git:// to https:// while cloning tlsh.git * [fix] switch from git:// to https:// while cloning tlsh.git. [raw-data] * Fix submit paste. [Terrtia] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] * Change: feeder name separator. [Terrtia] * Merge pull request #211 from GoSecure/docker-fix-encoding. [Alexandre Dulaunoy] Fixes encoding issue with MISP Galaxies JSON files * Fixes encoding issue with MISP Galaxies JSON files. [Olivier Bilodeau] * Merge pull request #212 from GoSecure/docker-fix-ardb-not-redis. [Alexandre Dulaunoy] Fixes the docker start script to use ardb instead of redis * Fixes the docker start script to use ardb instead of redis. [Olivier Bilodeau] The switch happened but the docker_start.sh script was left behind. * Merge pull request #210 from GoSecure/docker-fix-pip-issue. [Alexandre Dulaunoy] Using pip3 instead of pip fixes docker issues * Using pip3 instead of pip fixes docker issues. [Olivier Bilodeau] ## v1.2 (2018-06-20) ### Changes * README improvement. [Alexandre Dulaunoy] * GDPR document reference added. [Alexandre Dulaunoy] ### Fix * Tp and fp, errors messages on paste submission. [Terrtia] ### Other * Add module manager screenshot. [Thirion Aurélien] * Add api submit doc + add module information screenshot. [Terrtia] * Merge pull request #208 from CIRCL/tags. [Alexandre Dulaunoy] Tags * Merge branch 'master' into tags. [Thirion Aurélien] * Merge pull request #207 from garanews/patch-1. [Thirion Aurélien] removed -y from pip install command * Removed -y from pip install command. [garanews] removed -y from pip install command * Update README.md. [Thirion Aurélien] * Add screenshot. [Terrtia] * Fix hive and misp error. [Terrtia] * Add tag fp and tp. [Terrtia] * Fix path-event id map. [Terrtia] * Fix db init. [Terrtia] * Fix. [Terrtia] * Fix hive connection + add keys sample. [Terrtia] * Fix hive connection + submitted dir creation. [Terrtia] * Update ail-leak object. [Terrtia] * Add img + fix upload api. [Terrtia] * Change default submit tag, event-path map, publish events. [Terrtia] * Fix duplicate export attribute + tag export whitelist. [Terrtia] * Misp event creation, auto and manual + the hive alert auto creation, manual case creation. [Terrtia] * Modify navbar + decompress file password. [Terrtia] * Add feature, user can submit paste on the web interface. [Terrtia] * Verify file upload extention. [Terrtia] * Submit users input paste. [Terrtia] ## v1.1 (2018-06-06) ### Changes * New author Thirion Aurélien added as copyright holder. [Alexandre Dulaunoy] ### Other * Fix tags enabled by default. [Thirion Aurélien] * Merge pull request #206 from CIRCL/tags. [Sami Mokaddem] Addition of tagging system * Clean Flask_tags + gitignore. [Terrtia] * Add tag screenshot. [Terrtia] * Clean. [Terrtia] * Merge branch 'tags' of https://github.com/CIRCL/AIL-framework into tags. [Terrtia] * Update README.md. [Thirion Aurélien] * Clean. [Terrtia] * Add screenshots. [Terrtia] * Fix install. [Terrtia] * Fix install. [Terrtia] * Fix install. [Terrtia] * Fix install. [Terrtia] * Fix install. [Terrtia] * Taxonomies and galaxies update via LAUNCH. [Terrtia] * Add taxonomies and galaxies update and install. [Terrtia] * Fix duplicate. [Terrtia] * Galaxy tag info + fix. [Terrtia] * Galaxies synonyms. [Terrtia] * Galaxy description and tags. [Terrtia] * Taxonomie + add tags + tags display. [Terrtia] * Add and delete tags features. [Terrtia] * Display tag on important paste and search. [Terrtia] * Add tag js and css. [Terrtia] * Tags. [Terrtia] * Merge pull request #204 from CIRCL/python3. [Alexandre Dulaunoy] Python3 * Merge branch 'python3' of https://github.com/CIRCL/AIL-framework into python3. [Terrtia] * Fix Duplicate, save list of duplicates on disk + prevent empty hash creation. [Terrtia] * Merge pull request #202 from meauses/patch-1. [Alexandre Dulaunoy] tor install fails in Docker build * Tor install fails in Docker build. [Niels] added -y parameter to tor install ## v1.0 (2018-05-11) ### Fix * Features updated. [Alexandre Dulaunoy] * Correct package names and more privilege for installing tlsh. [Sami Mokaddem] * Added missing python3 dependencies for alertHandler (running under p3 for pymisp) [Sami Mokaddem] * Renamed Credentials into Credential sectiont title config change: By default, zmq listen to both local and CRF. [Sami Mokaddem] * Asciimatics introduced the '_required_height' property in the Label class. Just updating here to mirror the change. [Sami Mokaddem] * Create a directory for the last year (named as such) in LEVEL_DB_DATA/. This is done to prevent Duplicates.py to fail connecting to the database, as directories are created when starting install_dependencies.sh. [Sami Mokaddem] * Added a backoff time if an IOError happens. [Sami Mokaddem] * We are in 2018. [Alexandre Dulaunoy] * Canvasjs changed its folder hierarchy. [Sami Mokaddem] * As JSON decode needs a string, the empty list is a string. [Alexandre Dulaunoy] ### Other * Merge pull request #201 from CIRCL/python3. [Alexandre Dulaunoy] Python 3 migration + many new features + fixes * Fix duplicate + get raw paste. [Terrtia] * Fix duplicate + fix issue #200. [Terrtia] * Update README.md. [Thirion Aurélien] * Python 3 upgrade mv old redis db. [Terrtia] * Python 3 upgrade instructions. [Terrtia] * Clean. [Terrtia] * Change leveldb to ardb. [Terrtia] * Clean. [Terrtia] * Decode with redis connection. [Terrtia] * Fix bash python 2 to 3. [Terrtia] * Add bash python 2 to 3. [Terrtia] * Use feeder name. [Terrtia] * Add python 2 to 3 bash + fix Duplicate + clean. [Terrtia] * Improve Duplicate + tlsh + add time out handler + debug + clean. [Terrtia] * Create json metadata for base64 module. [Terrtia] * Fix travis tests 2. [Terrtia] * Fix travis tests. [Terrtia] * Add apiKeys module. [Terrtia] * Create new dashbord alert. [Terrtia] * Fix track term and avoid duplicate mail address. [Terrtia] * Merge branch 'python3' of https://github.com/CIRCL/AIL-framework into python3. [Terrtia] * Delete config.cfg.backup. [Thirion Aurélien] * Fix Onion module. [Terrtia] * Add bitcoin module. [Terrtia] * Add Base64 module. [Terrtia] * Fix python 3 compqtibility issues. [Terrtia] * Frontend python 3.5 upgrade. [Terrtia] * Add python 3 requirements. [Terrtia] * Clean. [Terrtia] * Fix ModulesInnformationV2. [Terrtia] * Fix python3 types issue. [Terrtia] * Fix python3 types issue. [Terrtia] * Python 3 backend upgrade. [Terrtia] * Fix incoming encoding file issue. [Terrtia] * Python3.5 Mixer and Global module. [Terrtia] * Fix travis. [Terrtia] * Fix travis. [Terrtia] * Fix travis. [Terrtia] * Fix travis build. [Thirion Aurélien] * Create first test. [Terrtia] * Merge pull request #196 from Terrtia/master. [Sami Mokaddem] add optional tor install for onion module * Add optional tor install for onion module. [Terrtia] * Merge pull request #194 from Terrtia/master. [Sami Mokaddem] fix doc graph generator and add a data flow graph * Improve doc graphs display. [Terrtia] * Fix doc graph generator and add a data flow graph. [Terrtia] * Merge pull request #193 from Terrtia/master. [Sami Mokaddem] Keys module improvement * Keys module improvement. [Terrtia] * Merge pull request #192 from Terrtia/master. [Sami Mokaddem] add python 3 requirement langid * Add python 3 requirement langid. [Terrtia] * Merge pull request #191 from drnguyen/master. [Sami Mokaddem] support non-TLS SMTP servers for email notifications * Refactor NotificationHelper code. [CD Nguyen] * Support non TSL SMTP servers. [CD Nguyen] * Merge pull request #184 from schutzwerk/ansible. [Sami Mokaddem] Install AIL using Ansible * Ansible folder: Add Ansible role, README.md: Add reference to ansible readme. [Philipp Schmied] * Merge pull request #189 from P4rs3R/patch-1. [Alexandre Dulaunoy] launching Flask_server.py * Launching Flask_server.py. [A. Cristallo] someone just forgot `./` ;) * Merge pull request #183 from schutzwerk/email-notifications. [Sami Mokaddem] New feature: E-Mail notifications for tracked terms * Fixed typo. [Philipp Schmied] * README: Reset to upstream file for PR. [Philipp Schmied] * README: Reset to upstream file for PR. [Philipp Schmied] * Fixed e-mail notificatins for sets and regexes. [Philipp Schmied] * Flask_terms.py: Fixed mapping of notifcation e-mails to terms with uppercase letters. [Philipp Schmied] * Config.cfg.sample: Merged overwritten changes from upstream. [Philipp Schmied] * Removed previously copied files. [Philipp Schmied] * Implemented email notifications (bin: config.cfg additions, email sending via analyzer scripts; var: Changes to add notifications via terms management); terms_management: Fixed click handlers not being added to all tracked terms. [Philipp Schmied] * README.md: Add note. [Philipp Schmied] * Merge pull request #181 from mokaddem/fix-dependencies. [Sami Mokaddem] Added missing python3 dependencies * Merge pull request #164 from mokaddem/showDiff. [Alexandre Dulaunoy] Show differences between duplicates * Merge branch 'iss160' into showDiff Preemptively preventing merge-conflict with branch iss160 (moving to configuration) [Sami Mokaddem] * Typo: added word + deleted trailing spaces. [Sami Mokaddem] * Update: Possibility to change the authorized max line length in the config for the diff of duplicates. [Sami Mokaddem] * Feature: diff in the web interface for duplicates pastes. [Sami Mokaddem] * Merge pull request #163 from mokaddem/iss160. [Alexandre Dulaunoy] Moving more to config * Typo: added word + deleted trailing spaces. [Sami Mokaddem] * Update: Moved filtering operation (thresholds, number of matching in the categ file) in the configuration file. It permits to better control the flow of pastes. Also set default mixer duplicate filtering to 3 (Do not filter) [Sami Mokaddem] * Merge pull request #179 from mokaddem/fix-177. [Sami Mokaddem] fix: Added _required_height in CLabel class * Merge pull request #178 from mokaddem/fix-176. [Sami Mokaddem] fix: Create a directory for the last year in ``LEVEL_DB_DATA/`` * Merge pull request #170 from mokaddem/iss_169. [Alexandre Dulaunoy] Iss 169 * Fixing no such file. [Sami Mokaddem] * Duplicate diff features added. [Alexandre Dulaunoy] * Merge pull request #166 from mokaddem/fix_165. [mokaddem] fix: canvasjs changed its folder hierarchy * Merge pull request #162 from mokaddem/sentimentBugfix. [mokaddem] * Fixed from front-end date representation bug. [Sami Mokaddem] * Merge pull request #159 from eurodude/patch-1. [Alexandre Dulaunoy] Allow for multiple entries like 1 2 3 4 5 * Allow for multiple entries like 1 2 3 4 5. [Fabien Mathey] Should replace the need to type 12 etc. and allows for 1 2 3 4 5 * Add: Python script to test if the ZMQ feed works as expected. [Alexandre Dulaunoy] * Merge pull request #153 from mokaddem/ail_training_november. [mokaddem] update slides * Updated: slides. [Sami Mokaddem] * Update slides. [Sami Mokaddem] * Merge pull request #152 from mokaddem/ail_training_november. [mokaddem] Added slides of AIL Training November Edition * Added slides. [Sami Mokaddem] * Features updated to add MISP export + random minor fixes. [Alexandre Dulaunoy] * Merge pull request #151 from mokaddem/misp_obj. [Alexandre Dulaunoy] Alert to MISP * Update: removed useless comments. [Sami Mokaddem] * Added pip3 in dependencies. [Sami Mokaddem] * Feature: try to send data to MISP if keys are provided, else do nothing. [Sami Mokaddem] * Update/feature: Max number of duplicate push to MISP + duplicate are pushed as attachment. [Sami Mokaddem] * Added a seemingly working version of ailleak to misp. [Sami Mokaddem] * Better support of python3 in Helper.py. [Sami Mokaddem] * Support of add_object to a MISP instance. [Sami Mokaddem] * Starting support of python3. [Sami Mokaddem] * Updated alertHandler and ailleakObject. [Sami Mokaddem] * Merge branch 'launcher_update' into misp_obj. [Sami Mokaddem] * Added draft support of MISP ail-leak object. [Sami Mokaddem] * Renamed BrowseWarningPaste into alertHandler. [Sami Mokaddem] * Added seconds args in import_dir. [Sami Mokaddem] * Merge pull request #150 from mokaddem/launcher_update. [mokaddem] Launcher update * Added '_AIL' to the end of screen session name to better check an kills sessions. Also, gracefully close redis server + Check 1 time if redis-servers are running. [Sami Mokaddem] * Merge remote-tracking branch 'upstream/master' [=] * Merge branch 'import_dir' [=] * Improvement of import_dir.py. Added feed name parameter, send the correct path (without potential ../) and gzipEncode it if needed. [=] * Merge branch 'bugfix' [=] * Merge remote-tracking branch 'upstream/master' [=] * Merge branch 'bugfix' [=] * Updated AIL presentation from SHA2017. [=] * Merge pull request #146 from PaulSec/master. [mokaddem] refactor (without deleting) * Changed the README.md file. [Paul A] * Quick refactor for the JavaScript. [Paul A] * Merge pull request #145 from PaulSec/master. [mokaddem] Removed duplicate function update_values() * Modified the README.md. [Paul A] * Fixed some code for the JavaScript (coding-style) [Paul A] * Removed duplicate function. [Paul A] * Merge pull request #143 from ngsimon/master. [mokaddem] Updated Docker image + Fixed Web UI dependencies installation * Updated Docker documentation typo. [Simon Tran] * Updated Dockerfile - Migrated Dockerfile to Ubuntu 16.04 LTS - Fixed dependency to "sudo" - Fixed AIL Web dependencies installation - Added Docker documentation. [Simon Tran] * Migrated Dockerfile to Ubuntu 16.04 LTS Fixed AIL Web dependencies installation Added Docker documentation. [Simon Tran] * Merge pull request #139 from mokaddem/indexer_register_fix. [Alexandre Dulaunoy] Fixed bug: No CR between indexes in all_index.txt file * Typo2.. [=] * Typo. [=] * Fixed bug: No CR between indexes in all_index.txt file. [=] * Merge pull request #138 from mokaddem/import_dir_update. [Alexandre Dulaunoy] Improvement of import_dir * Improvement of import_dir.py. Added feed name parameter, send the correct path (without potential ../) and gzipEncode it if needed. [=] * Merge pull request #137 from mokaddem/rst_script. [Alexandre Dulaunoy] Added AIL reset script * Added AIL reset script. [=] * Merge pull request #136 from mokaddem/bugfix. [Alexandre Dulaunoy] LevelDB auto-directory * Removed useless import. [=] * Removed 301x Dir in install_deps. [=] * Updated individual launch scripts. [=] * Flask: Added support of LevelDB database depending of the year. [=] * Automatically start redis_levelDB with the correct date. [=] * Dynamically choose correct port for LevelDB in script. [=] * Added LevelDB Curve for 2017 (3017, used for backward compatibility) [=] * Semi-automated directory creation with date as name in LAUNCH and installing_deps. [=] * Merge pull request #135 from mokaddem/bugfixes. [Alexandre Dulaunoy] Bugfixes and slides * Restored adns dependency. [=] * Updated KillAll in LAUNCH. [=] * BrowseImportantPaste: Fixed a bug where onClickListener was not set when the table was only 1 page long. [=] * Deleted data-flow graph and header.html from the index + Added fixed typo where 10min was in fact 3min. [=] * Updated slides. [=] * Updated workshop slides. [=] * Added sha2017-workshop-slides. [=] * Merge pull request #133 from mokaddem/quickNewFeature. [mokaddem] Quick new features * Added operation mode 3 for Mixer (do not filter) and added possibility to recurse through dir for import_dir. [Mokaddem] * Merge pull request #132 from mokaddem/quickFix. [mokaddem] Quick fix * Quick fix: commented not working dependencies and added support of python3 in generate_flow_graph. [=] * Merge pull request #131 from mokaddem/credentialsTracker. [Alexandre Dulaunoy] Credentials seeker * Added comments, cleanup and added credentials search progress in Flask. [Mokaddem] * Added comments. [Mokaddem] * Finished basic link web interface + solved some bugs. [Mokaddem] * Added similarity checks + started link webInterface. [Mokaddem] * Draft module + web link. [Mokaddem] * Dirty draft webpage credential_tracking. [Mokaddem] * Merge pull request #129 from mokaddem/master. [mokaddem] Avoid loosing the first message in import_dir.py * Restaured old example webpage. [Mokaddem] * Prevent the loss of the 1 message in import_dir.py. [Mokaddem] * Merge pull request #128 from mokaddem/doc. [mokaddem] Doc * Merged with previous changes for PR. [Mokaddem] * Merge pull request #127 from mokaddem/auto-web-interface. [Alexandre Dulaunoy] Auto web interface * Merge branch 'master' into auto-web-interface. [Mokaddem] * Merge pull request #126 from ninoseki/Add-a-missing-screenshot. [Alexandre Dulaunoy] Add a missing screenshot * Add a missing screenshot. [ninoseki] `./doc/screenshots/sentiment.png` is missing and I fixed it. * Merge pull request #120 from PaulSec/master. [Alexandre Dulaunoy] Added Python Phonenumbers module to parse numbers and retrieves stats… * Added Python Phonenumbers module to parse numbers and retrieves stats from countrycode. [Paul A] * Merge pull request #115 from mokaddem/regexInTerms. [Alexandre Dulaunoy] New features for terms frequency * Merge pull request #116 from mokaddem/auto-conf. [Alexandre Dulaunoy] Auto-configuration * Merge pull request #114 from mokaddem/newIndexer. [Raphaël Vinot] New indexer behavior * Merge pull request #113 from mokaddem/master. [Alexandre Dulaunoy] Improvement of terms - Behavior and webpage * Merge remote-tracking branch 'origin/auto-conf' into auto-web-interface. [Mokaddem] * Added possibility to keep a backup of the old config + check if config is valid before starting scripts. [Mokaddem] * Added all_modules.txt rule in gitignore. [Mokaddem] * Merge branch 'newIndexer' into auto-conf. [Mokaddem] * Merge remote-tracking branch 'origin/newIndexer' into auto-web-interface. [Mokaddem] * Modified check_size timeout + utilisation of shutil instead of os.system() [Mokaddem] * Added link towards HOWTO in README.md. [Mokaddem] * Added ore info on new webpage creation. [Mokaddem] * Removed screenshot for module information (will be added later) [Mokaddem] * Added new information to Howto. [Mokaddem] * Improved description of modules inside the scripts. [Mokaddem] * Improved overview. [Mokaddem] * Modified README order. [Mokaddem] * Split README with HOWTO and OVERVIEW. [Mokaddem] * Improved phone regex to handle end with 4 digits and presence of parenthesis. [Mokaddem] * Added support of onion in browseWarningPaste. [Mokaddem] * Bug fixes, jinja2 iter0 and typo. [Mokaddem] * Deleted print + Added curve fill for Mixer graph in Index. [Mokaddem] * New script create_new_web_module permits to rapidly create a new pre-configured webpage from a template. [Mokaddem] * Modified skeleton template name into rawSkeleton + Added possibility to hide modules in the header. [Mokaddem] * Auto organisation of the navbar -> Web page are auto-inserted. Created skeleton webpage. [Mokaddem] * Draft of new organisation of the web interface. Usage of Flask's blueprint. [Mokaddem] * Merge branch 'regexInTerms' into production. [Mokaddem] * Updated TermsFrequency related modules + Fixed bugs (encoding, behaviors, ...) [Mokaddem] * New feature in Terms: regex and set of words support (draft) [Mokaddem] * Merge branch 'terms-improvements' [Mokaddem] * Merge pull request #111 from mokaddem/update-archlinux-installer-script. [Alexandre Dulaunoy] Updated archlinux installer * Updated docker_start dataset year + Mixer. [Mokaddem] * Updated archlinux instaler. [Mokaddem] * Merge pull request #108 from mokaddem/newModuleInformation. [Alexandre Dulaunoy] New version of ModuleInformation * Merge pull request #110 from mokaddem/bugfix_categ. [mokaddem] Fixed bug in Categ and Mail * Merge branch 'newIndexer' into auto-conf. [Mokaddem] * Remplaced usage of global variable by a function param. [Mokaddem] * Removed wierd line. [Mokaddem] * Typo. [Mokaddem] * Keep original section order while updating. [Mokaddem] * Added script that checks configuration and may auto-fix it + added missing field in config.cfg.sample. [Mokaddem] * Fixed all_index newline bug + added print of what's happening. [Mokaddem] * Changed os.system by os.mkdir + fixed bug during rotation. [Mokaddem] * Indexer max size now in config. [Mokaddem] * Added backward support + renamed index with timestamp. [Mokaddem] * Added display of number of elements inside the index + changed variables names. [Mokaddem] * Added possibility to choose the index in search + Updated search page with jinja2 iter0. [Mokaddem] * Indexer now create an index_dir when it became too large (search in index not updated yet) [Mokaddem] * Merge branch 'terms-improvements' into production. [Mokaddem] * Added support of per_paste for term-management and plot-term. [Mokaddem] * Fixed a bug in manage_term (usage of loop.index0) + fixed bug in top_set where graph was not displaying the correct per_paste value. [Mokaddem] * Merge branch 'production' into terms-improvements. [Mokaddem] * Merge branch 'production' into terms-improvements. [Mokaddem] * Added possibility to select in top_terms the data per_paste or not (need module testing) [Mokaddem] * Merge branch 'newModuleInformation' into production. [Mokaddem] * Added a catch for psutils if process get stopped by user. [Mokaddem] * Merge branch 'newModuleInformation' into production. [Mokaddem] * Fixed bug - If we move the cursor from a line that has dissapear. [Mokaddem] * Merge branch 'newModuleInformation' into production. [Mokaddem] * ModuleInformation create the all_modules files if missing. [Mokaddem] * Added possibility to show or not in dashboard text that is logged. [Mokaddem] * Merge branch 'bugfix_categ' into production. [Mokaddem] * Fixed bug closing parenthesis in categ + changed behavior of sending to browseWarningPaste in Mail. [Mokaddem] * Merge pull request #107 from mokaddem/fix-linkPaste-debug. [Alexandre Dulaunoy] Fixed Link to download a paste is undefined when in DEBUG * Merge pull request #106 from mokaddem/master. [mokaddem] Updated readme - feeding AIL with pystemon * Merge branch 'newModuleInformation' into production. [Mokaddem] * Fixed a bug (str+float) [Mokaddem] * Merge branch 'fix-linkPaste-debug' into production. [Mokaddem] * Added path for debug in SQLInjection. [Mokaddem] * Merge branch 'newModuleInformation' into production. [Mokaddem] * Fixed str and int concat + Added sorting by queue name in queue not running. [Mokaddem] * Deleted all_modules.txt added by error. [Mokaddem] * Merge branch 'newModuleInformation' into production. [Mokaddem] * Added all_modules file. [Mokaddem] * Added support of ENTER for navigation. [Mokaddem] * Moved logging into a function. [Mokaddem] * Renamed variables + added comments. [Mokaddem] * Removed useless comments + commented. [Mokaddem] * Added comments. [Mokaddem] * Added comments and moved portion of code for slightly better visibility. [Mokaddem] * Added a way to quit application with 'q' more or less properly. [Mokaddem] * Added end of paste marker + handling screen resize. [Mokaddem] * Merge branch 'newModuleInformation' into production. [Mokaddem] * Fixed screen bug by adding an input before starting asciimatics + improved path localisation in the logs from modules. [Mokaddem] * Merge branch 'newModuleInformation' into production. [Mokaddem] * Added support of display of the last paste processed by a module. [Mokaddem] * Merge branch 'fix-linkPaste-debug' into production. [Mokaddem] * Added path of the paste in the log of Categ.py. [Mokaddem] * Updated feeding AIL with pystemon. [Mokaddem] * Merge remote-tracking branch 'upstream/master' [Mokaddem] * Merge remote-tracking branch 'upstream/master' [Mokaddem] * Updated feeding AIL with pystemon. [Mokaddem] * Merge remote-tracking branch 'upstream/master' into production. [Mokaddem] * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Alexandre Dulaunoy] * (partially fix #90 too) using a simple Alarm (SIGNAL) when exec-timeout. [Alexandre Dulaunoy] Introducing a timer (in this case 60 seconds) to ensure that the execution time of the analysis takes less than 60 seconds. This is a simple and standard POSIX signal handler. If the timeout is reached, the module will process the next one. This approach fixes the specific issues we have currently with some inputs where the sentiment analysis takes too much time. This fix should be improved and be more generic: - Introducing statistics of content which timeouts. - Keeping a list/queue to further process those files using a different analysis approach. Maybe a set of "dirty" processes to handle the edge cases and to not impact the overall processing and analysis. - Make the timer configurable per module (at least for this one). * Merge remote-tracking branch 'upstream/master' into production. [Mokaddem] * Merge pull request #102 from ptitdoc/bugfix1. [mokaddem] Bug fixes in Workers and improvement of pystemon-feeder * Pystemon-feeder: add the option bind in configuration file instead of address. [Olivier MEDOC] This is essentially because multiple feeds can be used in the address configuration variable. * SentimentAnalysis: use lexicon file path from the ail configuration file. [Olivier MEDOC] * Add pystemon path in configuration sample. [ptitdoc] * Import pystemon-feeder configuration from the central configuration file. [Olivier MEDOC] * Add all_modules.txt file so that the WebGUI can do proper cleanup. [Olivier MEDOC] * Fix invalid variable propagation. [Olivier MEDOC] * Fix to use the new pycountry API. [Olivier MEDOC] * (partially) Fix #91 using a simple Alarm (SIGNAL) when exec-timeout. [Alexandre Dulaunoy] Introducing a timer (in this case 5 seconds) to ensure that the execution time of the tokenizer takes less than 5 seconds. This is a simple and standard POSIX signal handler. This approach fixes the specific issues we have currently with some inputs where the tokenization takes too much time. This fix should be improved and be more generic: - Introducing statistics of content which timeouts. - Keeping a list/queue to further process those files using a different tokenizer approach. Maybe a set of "dirty" processes to handle the edge cases and to not impact the overall processing and analysis. - Make the timer configurable per module (at least for this one). * Merge pull request #99 from mokaddem/more-feed. [Alexandre Dulaunoy] Usage of sha1 to filter paste content in Mixer.py * Merge pull request #98 from mokaddem/more-feed. [Alexandre Dulaunoy] Config 2017 * Merge pull request #96 from mokaddem/more-feed. [Alexandre Dulaunoy] Handling of multiple feeders * Merge pull request #95 from mokaddem/module_handling. [Alexandre Dulaunoy] Module handling * Merge pull request #94 from mokaddem/website-refacto. [Alexandre Dulaunoy] Website refacto, support and fixes * Merge pull request #88 from mokaddem/Issue-Feeding-AIL. [Alexandre Dulaunoy] Pystemon tuto * Fixed typo 5. [Mokaddem] * Fixed typo 4. [Mokaddem] * Fixed typo 3. [Mokaddem] * Fixed typos 2. [Mokaddem] * Fixed typos. [Mokaddem] * Added pystemon instruction section. [Mokaddem] * Added requirement, better behaviors and confirmation box. [Mokaddem] * Improved module + Added support of click and keyboard navigation for killing and starting modules. [Mokaddem] * Added action button and support of logs. [Mokaddem] * Added color in front of queue related to its state. [Mokaddem] * Overwritted listbox update function to add formating. [Mokaddem] * Added new ModuleInformation. [Mokaddem] * Usage of sha1 to filter paste content in Mixer.py. [Mokaddem] * Merge branch 'more-feed' of github.com:mokaddem/AIL-framework into more-feed. [Mokaddem] * Added a line separator between the two graphs in dashboard. [Mokaddem] * Added 2017. [Mokaddem] * Added sleep in helper for multiple listening interfaces (reduce useless work), Added a simple way to pre-process feed before sending it to the global feed. [Mokaddem] * BUG FIX in Flask_trendingmodule, Fixed member_set when get_top_relevant_data is None. [Mokaddem] * Commented Mixer.py for further multiple feeds support. [Mokaddem] * Improved interface for multiple feeds and refresh_script. [Mokaddem] * Fixed bug multiple refresh instances and harmonized interface. [Mokaddem] * Reduced refresh rate of processed_pastes, synchro graphs and adjusted max on each graphs. [Mokaddem] * Updated web interface to handle new mixer module. and fixed one dependency bug. [Mokaddem] * Added basic mixer with confirugable behavior. It handles muliple feeders and performs some basic stats on them. [Mokaddem] * Merge branch 'module_handling' into production. [Mokaddem] * Prepared moduleInfo to handle module click with curses (will come back later). [Mokaddem] * Added legend + added more info if restarting disabled. [Mokaddem] * Added new config. [Mokaddem] * Removed typo in the number of remaining pastes to display in browseImportantPaste. [Mokaddem] * Added sub-flask in a directory. [Mokaddem] * Added missing depency datetime in dashboard. [Mokaddem] * Added missing json dependency in dashboard. [Mokaddem] * Splitted Flask_server into module related to website sections. [Mokaddem] * Merge branch 'website-improvement' into production. [Mokaddem] * Webstats should correctly updates top_progression_zset (Not fully tested because not enough data. Will be tested latter) [Mokaddem] * Terms top_sets correctly supports blacklisted terms. [Mokaddem] * Added support of re-plotting the plotted terms. [Mokaddem] * Started plotting same data with different dates. [Mokaddem] * Merge branch 'module_handling' into production. [Mokaddem] * Do not try to restart not running queue if autokill is not set. [Mokaddem] * ModuleInformation sort by name as well as elapsed time. [Mokaddem] * Added row coloring in ModuleInformation. [Mokaddem] * ModuleInfo correctly handle CurveManageTopSets, Changed queue coloring in dashboard, ModuleInformation now have a history of executed command and better handle module killing. [Mokaddem] * Better handle stuck modules. Differentiate between not running and no info and tries to restart stuck ones. [Mokaddem] * Merge branch 'website-improvement' into production. [Mokaddem] * Added date in Duplicate for better distinguish duplicate. Also, added a timeout for the dataTable in duplicate inside the modal. [Mokaddem] * Added dynamic data loading in dataTable in search.html. [Mokaddem] * Prevent browser to request reconnection for the eventStream + Added dynamic data loading in the dataTable of BrowseImportantPastes. [Mokaddem] * Removed server overhead for paste preview data. [Mokaddem] * Fixed bug in show concerned pastes previex en (terms managements) [Mokaddem] * Fixed bun in terms management, where action_button was not bind in other dataTable pages. [Mokaddem] * Pastes dataTable now display only the latest clicked paste even if there were discarded queries. [Mokaddem] * Fixed modal background stacking bug + button leading to saved paste bug fixed in search. [Mokaddem] * Tryed to fix html display problem in tooltip + fixed problem button showPaste in modal not pointing to the paste. [Mokaddem] * Partially solved the browser freezing problem + tootlip is also binded at a new page generation of the DataTable. [Mokaddem] * Added event source for getImportantPaste. [Mokaddem] * Dynamic table in search now load all the data + fixed bugs where tooltip where not shown on other than the first page displayed and tooltip interpret html (not supposed to...) [Mokaddem] * Added support of html formatting in search result dynamic loading table. [Mokaddem] * Added draft of dynamic update table with search result. [Mokaddem] * Fixed bug in modal show paste (show more button not showing) [Mokaddem] * Added dataTable for duplicate in show saved paste. [Mokaddem] * Bootstrap support in show saved paste. [Mokaddem] * Added support of browsing concerned paste in dashboard for all modules + Show in the page what was the concerned module. [Mokaddem] * Added support of local paste view in dashboard for Mails event only. [Mokaddem] * Modified provider name into no data when there is no data. [Mokaddem] * Added msgs when queues or feed is not running/coming. [Mokaddem] * Merge branch 'installation-fix' into production. [Mokaddem] * Fixed missing dependency in sentimentAnalysis. [Mokaddem] * Use latest version off tlsh. [Raphaël Vinot] #82 fixed upstream * Merge pull request #89 from mokaddem/installation-fix. [Raphaël Vinot] Installation fix * Fixed path issues in doc + added dependencies related to ssdeep. [Mokaddem] * Merge branch 'production' of github.com:mokaddem/AIL-framework into production. [Mokaddem] * Stuck not stucked. [Alexandre Dulaunoy] * Merge pull request #84 from jumojer/fix/web-interface. [Raphaël Vinot] Fixed dependency fetching for the web interface * Fixed dependency fetching for the web interface. [Jerome St-Hilaire] * Fix tlsh python module. [Raphaël Vinot] * Fix tlsh installer. [Raphaël Vinot] * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Raphaël Vinot] * Merge pull request #81 from madhuakula/master. [Raphaël Vinot] Added -y option for packages installation * Added -y option for packages installation. [Madhu Akula] This will break docker installation, because it's a non-interactive. * Update download link for adns. [Raphaël Vinot] * Merge pull request #77 from rivenz/docker. [Alexandre Dulaunoy] Added Dockerfile to automate the build of a Docker image based on Ubuntu 14.04 LTS. * Added Dockerfile to automate the build of a Docker image based on Ubuntu 14.04 LTS. [Simon Tran] * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Alexandre Dulaunoy] * Typo fixed and screencast added. [Alexandre Dulaunoy] * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Alexandre Dulaunoy] * Update Flask_server.py. [Raphaël Vinot] * Update update_thirdparty.sh. [Raphaël Vinot] * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Alexandre Dulaunoy] * Merge pull request #75 from mokaddem/multiple-modif. [Raphaël Vinot] Multiple modif v4 * Updated screenshot. [Mokaddem] * Fixed introduced bug relative to chart in index webpage. [Mokaddem] * Added support of PID in ModuleInformation and index.html. [Mokaddem] * Removed auto-reload in dashboard if window in undefined. [Mokaddem] * Updated dependencies. [Mokaddem] * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Alexandre Dulaunoy] * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Alexandre Dulaunoy] * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Alexandre Dulaunoy] * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Alexandre Dulaunoy] Conflicts: bin/launch_scripts.sh * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Alexandre Dulaunoy] Conflicts: bin/packages/modules.cfg * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Alexandre Dulaunoy] Conflicts: bin/packages/modules.cfg * Prod config. [Alexandre Dulaunoy] * Fix update 3rd party website updater. [Raphaël Vinot] * Fix travis. [Raphaël Vinot] * Update install script. [Raphaël Vinot] Fix #76 * Add dependency. [Raphaël Vinot] * Fix install script. [Raphaël Vinot] * Update travis. [Raphaël Vinot] * Merge pull request #74 from mokaddem/multiple-modif. [Raphaël Vinot] Multiple modif v3 * Added comments. [Mokaddem] * Added comment in moduleInformation. [Mokaddem] * Deleted useless parsing function. [Mokaddem] * Added not running module info + swapped order of module launch. [Mokaddem] * Added dataTable support in browse paste modal. [Mokaddem] * Changed lots of code into zrank. [Mokaddem] * Updated individual launch script. [Mokaddem] * Suppressed metismenu import. [Mokaddem] * Added module number info in index webpage. [Mokaddem] * Added number of same module running in ModuleInformation. [Mokaddem] * Auto generate graph div + possibility to query all sentiments data. [Mokaddem] * Individualized sentiments-graph loading. [Mokaddem] * Merge pull request #71 from mokaddem/multiple-modif. [Raphaël Vinot] Multiple modif v2 * Removed autokill feature if not specified. [Mokaddem] * Removed autokill feature if not specified. [Mokaddem] * Added directory creation. [Mokaddem] * Fixed bug in WebStats not deleting correct key + fixed typo in CurveSetManager preventing connecting to redis + modified display in moduleInfo. [Mokaddem] * Added color in web-index reprenting modules states. [Mokaddem] * Removed useless import. [Mokaddem] * Added pip dependency. [Mokaddem] * Added moduleInfo + Renamed modules. [Mokaddem] * ModuleInfo support autokill and auto restart after logging. [Mokaddem] * ModuleInfo now support logs. [Mokaddem] * Fixed typo in creditcard + fixed bug in ModuleInformation (relative to date). [Mokaddem] * Added new module 'ModuleInformation' which gives real time information on running module. [Mokaddem] * Added script module-dataflow-graph + fixed bug in module.cfg preventing the credential module to receive data. [Mokaddem] * Added directory creation. [Mokaddem] * Merge pull request #70 from mokaddem/multiple-modif. [Alexandre Dulaunoy] Update of Readme * Added install script nltk.vader. [Mokaddem] * New screenshot + Added titles in readme. [Mokaddem] * Added title spacing. [Mokaddem] * Draft modification of Readme. [Mokaddem] * Fixed typo. [Mokaddem] * Modified dashboard screenshot. [Mokaddem] * Added screenshots. [Mokaddem] * Merge pull request #69 from mokaddem/multiple-modif. [Raphaël Vinot] Added config redis 6382 * Added config redis 6382. [Mokaddem] * Merge pull request #68 from mokaddem/multiple-modif. [Raphaël Vinot] Multiple modif * Added icon in header. [Mokaddem] * Added new module in individual launch scripts. [Mokaddem] * Added show/hide curve with legend in terms-top-set. [Mokaddem] * Added position of each top term in the other sets + highligth ones not present. [Mokaddem] * Auto hide button add_to_chart when no data plotted. [Mokaddem] * Added tooltip in graph for terms-plot-tool. [Mokaddem] * Added multiple terms plot in terms-plot-tool. [Mokaddem] * Harmonized tooltip and popover display. [Mokaddem] * Added autohide curves in terms-top-set. [Mokaddem] * Moved legend place in term-plot-top. [Mokaddem] * Added config.sample. [Mokaddem] * Fixed bug in curve preventing to track concerned_pastes. Now all tracked words are put in lower case. [Mokaddem] * Fixed potential XSS problem with pastes display in 'search' and 'important_paste_by_module'. [Mokaddem] * Added save of paste_baseName in set for searches. This is done in ModuleStats to not surcharge the Global module. [Mokaddem] * Replaced negative hour by 24h based. [Mokaddem] * In sentiment analysis. if today top provider set is empty, try to take yesterday data and then takes from all_providers. [Mokaddem] * Suppressed useless comms. [Mokaddem] * Suppressed prints. [Mokaddem] * Removed commented code. [Mokaddem] * Fixed stupid line remove in index.html... [Mokaddem] * Fixed bug in flask with zrange (was taking the bottom of the set instead of the top leading to get only the min instead of max) [Mokaddem] * Fixed bug with redis where the inserted key was too long, causing a crash + fixed bug taking min instead of max in terms-top-set. [Mokaddem] * Fixed bug in manage-terms-top-set + added auto sorting in important-paste-by-module. [Mokaddem] * Fixed bug in terms-management-interface (only display one paste) + removed get_language of all pastes due to the huge overhead. [Mokaddem] * Added terms trending module and web interface: - Curve module has been rewritten - Added new module curve_manage_top_set - Added all web-pages. [Mokaddem] * Fixed bug in sentiment trending, if there was no data for the avg -> there was one NaN leading to no display in the gauge. [Mokaddem] * In sentiment trending, if there are no providers in the today\'s set, get data from random providers. [Mokaddem] * Fixed typo. [Mokaddem] * Deleted alert() [Mokaddem] * Enter simulate button click on input field for terms management and plot tool. [Mokaddem] * Added terms-plot tool. [Mokaddem] * Finnished terms-plot-top-set. Also added possibility to add terms to track or blacklist. [Mokaddem] * Added top set webpage. [Mokaddem] * Correctly handle and display add or remove a term in term-frequency + started sketch of terms-plot tool. [Mokaddem] * Added query of the data for the new tracked word. [Mokaddem] * Added draft of terms_management + Added copyright in readme. [Mokaddem] * - Modified redis connection (from levelDB to redis). - Added term frequency in curve. - Modified ModuleStats and WebStats to use redis special command (incrby instead of get-set and zset) and Flask to perform the correct queries. - Added panel color in sentiment trending. [Mokaddem] * Probably fixed bug in creditcard (not sending to duplicate) [Mokaddem] * Added support of pasteName in search + added header in search.html + supressed mixed indent and trailing in Flask.py. [Mokaddem] * Updated isolated launch scripts + Added switch displaying queue in index. [Mokaddem] * Modified + into append. [Mokaddem] * Auto unchecked info checkbox in index. [Mokaddem] * Modified icon sentiment in web-header. [Mokaddem] * Added dependencies. [Mokaddem] * Added script flexgauge. [Mokaddem] * Added tooltips + slight gauge modification. [Mokaddem] * Added drop of really long line in sentiment-analysis module + Added description of sentiment module. Also, fixed bug in webpage sentiement-trending concerning avg and date range. [Mokaddem] * Added draft of filter in sentiment analysis (Discard syntaxical languages) + Added nice tooltip for sparkline. Trending displays avg in function of the number of elements processed and not for the complete week + fixed bug in gauge and canvasjs (was performing avg with only 1 graph instead of all 8). [Mokaddem] * Added autoreload in sentiment-trending. [Mokaddem] * Added avg text in sentiment-trending + fixed date range bug. [Mokaddem] * Added all providers for plot_tool in moduleStats + fixed date range bug. [Mokaddem] * Added stacked choice or not for plot-tool (sentiments) [Mokaddem] * Tool-plot for sentiment working + Splitted script and webpage. [Mokaddem] * Created draft of search for plot tool. [Mokaddem] * Linked plot button with server. [Mokaddem] * Added provider checkbox. [Mokaddem] * Added date on slider. [Mokaddem] * Added link to data in analysis trending + fixed tooltip bug in sparkline. [Mokaddem] * Added support of week and day average. [Mokaddem] * Added sentiment analyser module (draft) [Mokaddem] * Modified webpage display and added js. [Mokaddem] * Splited js from html + Added some ID + small display modification. [Mokaddem] * Added page Plot-tool. [Mokaddem] * Separated options from the plot funcion for sparkline. [Mokaddem] * Finished first panel in sentiment-today. [Mokaddem] * Added empty webpages for sentiments modules. [Mokaddem] * Merge branch 'webpage-header-splitted' into sentiment-analysis. [Mokaddem] * Added module sentimentAnalyser. [Mokaddem] * Merge pull request #66 from mokaddem/webpage-header-splitted. [Raphaël Vinot] Separated webpage header code from all webpage * Separated webpage header code from all webpage. [Mokaddem] * Merge pull request #65 from mokaddem/production. [Raphaël Vinot] New modules and upgraded front-end * Fixed small bug in SQLInjection module. [Mokaddem] * Added bar chart hover binder in trending charts. [Mokaddem] * Added the part 'other' in Trending chartscd. [Mokaddem] * Added new file in gitignore. [Mokaddem] * Added trendingchart.js. [Mokaddem] * Slight refacto + sql comments are now placed in injection suspision. [Mokaddem] * Removed console log. [Mokaddem] * In index webpage, reset value graph if no paste received. [Mokaddem] * Fixed one of the small display bug in DUplicate_ssdeep_v2 (was displaying [tlsh, tlsh] instead of simply tlsh) [Mokaddem] * Added whole config in config.sample. [Mokaddem] * Fixed bug where Listener was not placed if the row was not displayed (BrowseWarningPaste) [Mokaddem] * Restaured loading gif when new tab is shown (BrowseWarningPaste) [Mokaddem] * Bug fix related with redis: Fixed typo key in redis for module creditcard and sqlinjection Modified Curve redisLvlDb server Modified Url.py so that it forwards name of protocol from saved protocolsfile Added Cache control in Flask Modified key-tab name into keys-tab. [Mokaddem] * Fixeed bug in Duplicate (The comparison value was not saved correctly in redis) + Modified to progression detection algo. [Mokaddem] * Fixed bug in CVE (always sending to browse_warning even if there was no match) [Mokaddem] * Fixed bug in flask not correctly displaying the number of paste/providers. [Mokaddem] * Fixed bug in credential (not sending correctly to ModuleStat) + Fixed date bug in ModuleStats. [Mokaddem] * Fixed bug CVE, CVE was not sending its data to BrowseWarningPaste. [Mokaddem] * Merge branch 'master' into production. [Mokaddem] * Merge pull request #63 from mokaddem/uptodate-duplicate-module. [Raphaël Vinot] Uptodate duplicate module * Fix path to tldsfile. [Raphaël Vinot] * Added warning_paste module and created related webpages. Fixed a Faup bug in credential (multiple instanciation) and added correc populate_set_out in concerned modules (creditcard, credential, ...). Linked browse_warning_paste module and Flask function with redis (created new sets). [Mokaddem] * Started browse important paste feature. Still need to link with redis. + Fixed a bug in Flask (String to list parser) [Mokaddem] * Added few comments. [Mokaddem] * Draft: added new duplicate hash comparison - tlsh. [Mokaddem] * Tryed to fix the glob_tabvar bug. [Mokaddem] * Merge branch 'sql-injection' into production. [Mokaddem] * Better display printing. [Mokaddem] * Added SQLInjectionDetection module. [Mokaddem] * Fixed a bug in the hover of provider graph (happened when there was an 'other' in other graphs) [Mokaddem] * Fixed a division by 0 in Flask + Harmonized webpage item positioning. [Mokaddem] * Fixed a bug in the hover of providers graph. When we click on a part in providers graphs, we hover an item, then we click on a random part not belonging to providers graphs, and we re-hover one part of the providers graphs, there was an undefined variable (The random click overrided it). [Mokaddem] * Fixed bug in case of null elements arrising because of a new month. [Mokaddem] * Removed prints. [Mokaddem] * Fixed size bug, which in some cases display the total size instead of the average + few cleanup + commented out log scale in moduleStats.js and added large number division in case of div by 0. [Mokaddem] * Big refactoring and added lot of comments + fixed bug attaching multiple binder on barChart. [Mokaddem] * Fixed small race condition bug in ModuleStat.py and Added highlight of bar belonging to the same serie in moduleStats.js. [Mokaddem] * Added tooltip in moduleStat above each bar chart + Added highlight of the selected bar. (Still not working for category 'Other') [Mokaddem] * Added tooltip for moduleStats bar chart. [Mokaddem] * Fixed multiple binding bug in moduleStats + Added multiple bar display for 'other pie part' + Added ModuleStats tab in all other pages. [Mokaddem] * Clean up and refactoring for back-end moduleStats. [Mokaddem] * Clean up and refactoring for front-end moduleStats. [Mokaddem] * Added number of paste by provider in module stats. [Mokaddem] * In moduleStats, if there is data which have small percentage, display it in an 'other' part + Added stacked chart for them (still draft) [Mokaddem] * Added the click on a part in module stats. [Mokaddem] * In moduleStat, display old data if there is no data today and notify the user. [Mokaddem] * Removed useless comments + prevent trying to add an already present element in the set. [Mokaddem] * Fixed variable bug in moduleStats and added draft of credential stats. [Mokaddem] * Added Draft of ModuleStats + Paste size average. [Mokaddem] * Merge branch 'uptodate-duplicate-module' into production. [Mokaddem] * Clean not compatible ssdeep hash encountered. [Mokaddem] * Added ssdeep in pip_requirments. [Mokaddem] * Fixed bug introduced in merge-conflict. [Mokaddem] * Added dependency flot.time. [Mokaddem] * Merge branch 'statistics-v2' into uptodate-statistics-v2. [Mokaddem] * Re-ordered tab in trending charts. [Mokaddem] * Added top_progression chart for tld, domain and scheme + Small modification in config file. [Mokaddem] * Merge branch 'duplicate-module' into uptodate-duplicate-module. [Mokaddem] * Added related functions and display of duplicated paste in search.py. [Mokaddem] * Duplicate module takes its messages from other modules and no more from Global. [Mokaddem] * Added default config. [Mokaddem] * Added two new version of duplicate module. One with hashes are saved in json on disk The other with only leveldb. [Mokaddem] * Added new module for Duplicate paste. Seems working but has some small bug (re-check same paste twice) [Mokaddem] * Added SimHash library. [Mokaddem] * Bug global_tabvar seems solved: Caused by a race condition, Just switched two functions execution. [Mokaddem] * Restored deleted part of a failed merge-conflict. [Mokaddem] * Merge remote-tracking branch 'upstream/master' [Mokaddem] * Fix default config. [Raphaël Vinot] * Fix path in config. [Raphaël Vinot] * Add WebStats in launcher. [Raphaël Vinot] * Fix pep8. [Raphaël Vinot] * Update travis. [Raphaël Vinot] * Update web thirdparty modules. [Raphaël Vinot] * Fix travis. [Raphaël Vinot] * Update travis & install script. [Raphaël Vinot] * Cleanup and remove useless deps. [Raphaël Vinot] * Added dependency for flot chart. [mokaddem] * Merge branch 'webpage-splitting' of https://github.com/mokaddem/AIL-framework into mokaddem-webpage-splitting. [Raphaël Vinot] * Added default configuration. [Mokaddem] * Merge pull request #59 from mokaddem/processed-pastes-graph. [Raphaël Vinot] Processed pastes graph * Fixed a flexibility issue. [Mokaddem] * Added default configuration. [Mokaddem] * Fixed a bug in processed_pastes graph. [Mokaddem] * In index: Added number of processed pastes chart. [Mokaddem] * Commented out get_language because it adds too much overhead. [Mokaddem] * Merge pull request #57 from mokaddem/master. [Raphaël Vinot] New web pages and statistics modules * Added dependency for flot chart. [mokaddem] * Fixed a bug in processed_pastes graph. [Mokaddem] * In index: Added number of processed pastes chart. [Mokaddem] * Commented out get_language because it adds too much overhead. [Mokaddem] * Display the correct active tab in navbar. [Mokaddem] * Searching now works in trending graphs. [Mokaddem] * Deleted redundant code and separeted all trendings chart from the main page. [Mokaddem] * Merge branch 'mokaddem-testing' [Mokaddem] * Modified variable name str. [Mokaddem] * Optimized create_plot and removed test commemts. [Mokaddem] * Merge branch 'statistics-on-url' [Mokaddem] Merge for pull request * Restored working search>html + Added max curves display to all graphs. [Mokaddem] * Added a dynamic limit on the number of elements to display. Not fully tested. (recovered old doc, searching not working) [Mokaddem] * Graphs now displayed the highest value. Plotting and filtering is done in its own script. [Mokaddem] * Renamed tldstrending into trending and added graphics into their respective pannels. Separated graph generation from html source code for wordtrending and protocoltrending. [Mokaddem] * Added loading GIF when searching and fixed small display bug in modal. [Mokaddem] * Added comments and renamed variables and separeted chunks of codes into function. [Mokaddem] * Fixed small sorting problem (date) [Mokaddem] * Added dataTable resources in update-third-party.sh. [Mokaddem] * Added query search display. [Mokaddem] * Added dynamic table sorting in search page. (Still need to add dependencies) [Mokaddem] * Search preview modal more dynamic and responsive. [Mokaddem] * Search preview modal do not duplicate anymore but still needs improvements. [Mokaddem] * Added load more content in search modal preview. Works fine but may duplicate content (need to fix). [Mokaddem] * Added pannel in search modal + small fix related to config. [Mokaddem] * Searching now working with preview, details and redirection to the paste display page. [Mokaddem] * Added DomainTrending seems working. Started search features with related html pages, not finish yet. [Mokaddem] * Created new graph pannel for further url stats and dissociate graph script from html. [Mokaddem] * Merge pull request #56 from mokaddem/mokaddem-testing. [Alexandre Dulaunoy] New statistics modules and usage of Faup * Added few comments. [Mokaddem] * Added template tld. Modified URL using Faup and refactored WebStats. [Mokaddem] * Added new modules and started WebTrending web interface. [Mokaddem] * Getting Started: Initial configuration working on the laptop. [Mokaddem] * Merge pull request #52 from MaximeStor/master. [Alexandre Dulaunoy] Added module SourceCode * Improve SourceCode, keywords and add description in /doc. [MaximeStor] * Add first version of Source Code. [MaximeStor] * Because 0MQ fails. [Raphaël Vinot] * Fix logging, fix URL regex. [Raphaël Vinot] * Add Credential in the scripts to launch. [Raphaël Vinot] * Split filepath and count in credential module. [Raphaël Vinot] * Add new category (Credential) [Raphaël Vinot] * Refactoring on Credential, Phone and Release. [Raphaël Vinot] * Few corrections/additions/changes. [Sascha Rommelfangen] * How to contribute updated. [Alexandre Dulaunoy] * Basic feature list added. [Alexandre Dulaunoy] * Logo added. [Alexandre Dulaunoy] * CIRCL AIL logo added. [Alexandre Dulaunoy] * Installation of AIL simplified and documentation updated. [Alexandre Dulaunoy] * Broken links fixed. [Alexandre Dulaunoy] * Document the simplified version of creating modules. [Alexandre Dulaunoy] Describe the new template location and how to create your new AIL module. The old and outdated description has been removed. * Merge branch 'pgp' of https://github.com/Rafiot/AIL-framework. [Alexandre Dulaunoy] Conflicts: bin/packages/modules.cfg * Add module to find PGP encrypted blobs. [Raphaël Vinot] * Merge pull request #48 from MaximeStor/module. [Alexandre Dulaunoy] Add modules Credential and Release * Merge branch 'master' of https://github.com/CIRCL/AIL-framework into module. [MaximeStor] * Merge pull request #46 from blackbern/blackbern. [Alexandre Dulaunoy] Blackbern * Autre correction de installing_deps.sh. [Blackbern] * Correction de installing_deps.sh. [Blackbern] * Ajout d'un script d'installation pour archlinux via pacman et yaourt. [Blackbern] * Merge pull request #49 from Alainfou/master. [Alexandre Dulaunoy] Phone module added (regex needs optimization) * Phone regex updated. [Alain] Might still need to be fixed / optimized, in case of maths or random numbers starting with a 0. Do not capture dates, hours, coordinates anymore. Captured formats are: e.g. +331234567890 ; 09 12 34 56 78 ; +4177/123.45.69 ; +352(0)6-23-23-23... * Update module.cfg (adding Keys and Phone section) [Alain] * Starting Phone number recognition. [Alain] * Add modules Credential and Release. [MaximeStor] * Listen locally for 0MQ. [Raphaël Vinot] * Launch redis and leveldb from local directory. [Raphaël Vinot] * Update install script. [Raphaël Vinot] * Add script to import from local directory, use local python from env. [Raphaël Vinot] * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Raphaël Vinot] * Add Travis logo. [Raphaël Vinot] * Merge pull request #45 from Rafiot/travis. [Raphaël Vinot] Travis, print logs * Travis, print logs. [Raphaël Vinot] * Add template file for writing a new module. [Raphaël Vinot] * Merge pull request #44 from Rafiot/travis. [Raphaël Vinot] Add initial Travis file * Add initial Travis file. [Raphaël Vinot] * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Alexandre Dulaunoy] * Merge pull request #38 from cudeso/master. [Alexandre Dulaunoy] libsnappy-dev dependecy * Libsnappy-dev dependecy. [Koen Van Impe] * This fix #36 update distribute version before install matplotlib. [Debra Jules] * -v option added to list the path. [Alexandre Dulaunoy] * Web: basic search functionality added. [Alexandre Dulaunoy] * Font awesome link fixed. [Alexandre Dulaunoy] * Logo added and layout changed. [Alexandre Dulaunoy] * Onion fetching loop deactivated by default. [Alexandre Dulaunoy] * Re-add config option. [Raphaël Vinot] * Revert changes on the config file due to merging messup. [Raphaël Vinot] * Update start-bootstrap urls. [Alexandre Dulaunoy] * Remove double requirement. [Alexandre Dulaunoy] * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Raphaël Vinot] * Add contributors. [Raphaël Vinot] * Update README.md. [Alexandre Dulaunoy] Framework for Analysis of Information Leaks * Update README.md. [Alexandre Dulaunoy] * Update README.md. [Alexandre Dulaunoy] * Merge pull request #30 from adulau/master. [Alexandre Dulaunoy] A simple feeder script feeding data from pystemon to AIL. * A simple feeder script feeding data from pystemon to AIL. [Alexandre Dulaunoy] The configuration matches the default Redis parameters used in the pystemon configuration. https://github.com/cvandeplas/pystemon/blob/master/pystemon.yaml#L16 * Spelling/language corrections. [Sascha Rommelfangen] * Merge pull request #29 from adulau/master. [Alexandre Dulaunoy] Old tag cloud removed * Ultag removed. [Alexandre Dulaunoy] * Removed ultag. [Alexandre Dulaunoy] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Alexandre Dulaunoy] * Merge pull request #27 from adulau/master. [Alexandre Dulaunoy] Add proper publisher for classified domains/hostnames * Add proper publisher for classified domains/hostnames. [Alexandre Dulaunoy] * Merge pull request #26 from adulau/master. [Alexandre Dulaunoy] DomainClassifier module added * DomainClassifier sample configuration added. [Alexandre Dulaunoy] * First version of the DomainClassifier. [Alexandre Dulaunoy] * DomainClassifier requirements added. [Alexandre Dulaunoy] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Alexandre Dulaunoy] * Merge pull request #23 from adulau/master. [Alexandre Dulaunoy] ZMQ Publisher removed * Global queue for DomainClassifier. [Alexandre Dulaunoy] * ZMQ Publisher removed. [Alexandre Dulaunoy] ZMQ Publisher removed to allow concurrent use of the scripts. In short term, we would replace all publishing part within AIL into pub-sub Redis to avoid ZMQ limitation. * Add hotfixes. [Raphaël Vinot] * Publish the fetched onions on a ZMQ feed. [Raphaël Vinot] * Ultag removed. [Alexandre Dulaunoy] * Removed ultag. [Alexandre Dulaunoy] * Add config file for DomainClassifier, proper reporting. [Raphaël Vinot] * Fix onions, cc and domain classifier modules. [Raphaël Vinot] * Categ now listen to the Global queue. [Raphaël Vinot] * More cleanup. [Raphaël Vinot] * Add Domain Classifier module. [Raphaël Vinot] Cleanup in the config files. * Fix the exceptions. [Raphaël Vinot] * Stop killing the disk when creating the word curve. [Raphaël Vinot] * Update starting script. [Raphaël Vinot] * Fix the onion module, log the valid onions. [Raphaël Vinot] * The onion module now fetches the URLs it finds. [Raphaël Vinot] * Big refactoring, make the queues more flexible. [Raphaël Vinot] * Cleanup. [Raphaël Vinot] * Remove useless subscriber * Fix typo in the config file * Update Helper accordingly * Make sure the webserver fails properly if there is no config file. [Raphaël Vinot] Create the queue list in a more pythonesque fashion. * Merge pull request #22 from adulau/master. [Alexandre Dulaunoy] * Cleanup: removed unused parameter. [Alexandre Dulaunoy] * Readme updated. [Starow] * Add missing dependencies. [Starow] * Automatically copy the config file if there is no. [Raphaël Vinot] * Merge pull request #21 from adulau/master. [Jules] cymruwhois uses dotted decimal format * Cymruwhois uses dotted decimal format. [Alexandre Dulaunoy] * Merge pull request #20 from CIRCL/testing. [Jules] Big refactoring, make the workflow much easyer to follow. * Update deps and install script to work out of the box. [Raphaël Vinot] * Disable the sockets and the pid files from the redis config files. [Raphaël Vinot] * Small fixes to make the refactoring production ready. [Raphaël Vinot] * the port for the logging is 6380 * use os.environ properly * fix typos * Update gitignore to have a clean dev directory. [Raphaël Vinot] * Use env variables everywhere. [Raphaël Vinot] * Cleanup, add FIXME. [Raphaël Vinot] * Move Redis_Data_Merging to Paste. [Raphaël Vinot] * Update config. [Raphaël Vinot] Make all paths in the config file relative to the home directory. * Fix subscriber names, update default config. [Raphaël Vinot] * Merge remote-tracking branch 'origin/master' into testing. [Raphaël Vinot] Conflicts: bin/ZMQ_Sub_Urls.py * Merge branch 'rewrite_zmq' into testing. [Raphaël Vinot] Conflicts: bin/indexer_lookup.py * Completely remove ZMQ_PubSub.py. [Raphaël Vinot] * Do not create a ZMQ sub if it is not required. [Raphaël Vinot] * Improve the cleanup. Still some to do. [Raphaël Vinot] * Cleanup the queues. [Raphaël Vinot] * Merge pull request #19 from adulau/master. [Alexandre Dulaunoy] Redis explanation added * Markdown fixed. [Alexandre Dulaunoy] * Overview of Redis database added. [Alexandre Dulaunoy] * Merge pull request #18 from adulau/master. [Alexandre Dulaunoy] Fix one assignment before use and remove duplicate call to libmagic * MIME type guessing - removed one duplicate call to libmagic. [Alexandre Dulaunoy] * Remove undeclared variable. [Alexandre Dulaunoy] * Assignment before use (if Enumerate fails) [Alexandre Dulaunoy] * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Raphaël Vinot] * Merge pull request #15 from adulau/master. [Alexandre Dulaunoy] Configuration path fixed * Configuration path fixed. [Alexandre Dulaunoy] * Fix install script. [Raphaël Vinot] * Big cleanup, pep8. [Raphaël Vinot] * Merge pull request #14 from adulau/master. [Jules] Requirements updated for CC/ASN lookup in URL module * Requirements updated for CC/ASN lookup in URL module. [Alexandre Dulaunoy] * Merge pull request #13 from adulau/master. [Jules] Log where URLs are hosted - cc_critical option added * Log where URLs are hosted - cc_critical option added. [Alexandre Dulaunoy] It logs where the hostname of the URL is hosted (ASN and geographic location). A simple option cc_critical added to set the country code to log as critical. * Cleanup (remove unused imports, more pep8 compatible) [Raphaël Vinot] * Maxi cleanup old code :'( [Starow] * Adding a js library for the tag cloud (to install with update_thirdparty.sh. [Starow] * Javascript modif (Tagclouds) ... [Starow] * Printing set of domain for debugging. [Starow] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Starow] * Cleanup of main Paste module. [Raphaël Vinot] * Remove script installed by pubsublogger. [Raphaël Vinot] * Remove 3rd party code (pubsublogger), add it in the deps. [Raphaël Vinot] * Logs display modifications. [Starow] * Minor logs modifications. [Starow] * Splitting the graph in two one for words, one for paste monitoring. [Starow] * Padding log hours + adding some super nice bootstrap icon :) [Starow] * Minor javascript update. [Starow] * Harmonising logs messages + Changing some dygraph options. [Starow] * Adding an option to have better view of the curves. [Starow] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Starow] * Merge pull request #12 from adulau/master. [Jules] Indexer script updated to dump documents and find similar documents * -s option added to find similar documents. [Alexandre Dulaunoy] By default, the index is not storing the vector of the document (Whoosh document schema). It won't work if you don't change the schema of the index for the content. It depends of your storage strategy. * -f option added: dump full document for each match. [Alexandre Dulaunoy] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Alexandre Dulaunoy] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Alexandre Dulaunoy] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Alexandre Dulaunoy] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Alexandre Dulaunoy] * Improving javascript and Fixing minor curves drawing bug Everything now is fully "dynamic" [Starow] * Major javascript update: Queues curves are now being set dynamically. Now the javacript part is a little bit less disgusting than before. [Starow] * Adding logging proc time inside the table. [Starow] * Updating the screenshot. [Starow] * Checkbox default checked at true. [Starow] * Merge pull request #11 from adulau/master. [Jules] Indexer tool - statistics added * Indexer: Some index statistics added. [Alexandre Dulaunoy] usage: indexer_lookup.py [-h] [-q Q] [-n] [-t] [-l] Fulltext search for AIL optional arguments: -h, --help show this help message and exit -q Q query to lookup (one or more) -n return number of indexed documents -t dump top 500 terms -l dump all terms encountered in indexed documents * -l added -> dumping all terms indexed. [Alexandre Dulaunoy] * Return the number of indexed documents. [Alexandre Dulaunoy] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Starow] * Javascript modification dashboard log table. [Starow] * Index.html log table modification. [Starow] * Merge pull request #10 from adulau/master. [Jules] Whoosh requirements added for Indexer modules * Configuration that should not be there... [Alexandre Dulaunoy] * Whoosh requirements added. [Alexandre Dulaunoy] * Whoosh added for the indexer modules. [Alexandre Dulaunoy] * Indexer queue and script added to "BBS-like" LAUNCH script. [Alexandre Dulaunoy] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Alexandre Dulaunoy] * Merge pull request #9 from adulau/master. [Jules] Indexer module: script to query the index * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Alexandre Dulaunoy] * Making bigger the graph. [Starow] * Removing "Dashboard" Title to win some place for displaying infos.. [Starow] * Hardcoded path from ZMQ_Curve are now referring correctly in config.cfg.sample fix #6. [Starow] * Indexer module: script to query the index. [Alexandre Dulaunoy] Test script to query the index generated from the Indexer module. python indexer_lookup.py -q Visa -q Mastercard * Merge pull request #8 from adulau/master. [Jules] Indexer module added - initial version with Whoosh full-text indexer * Indexer module added - initial version with Whoosh full-text indexer. [Alexandre Dulaunoy] The indexer module indexes all the pastes using Whoosh. The module can be extended to support additional full-text indexers in the future. * Queue name tab moved to left side of the dashboard. [Starow] * Updating dashboard navigation menu list. [Starow] * Updating installing script + adding another required package to the list. [Starow] * Update of installing_deps.sh and pip_requirement.txt. [Starow] * Importing dns.exeption fix #4 fix #7. [Starow] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Starow] * Catching the exception dns.exception.Timeout fix #4. [Starow] * Fixing a bug about caching paste inside Redis :) [Starow] * Adding a letter (s) [Starow] * Adding a . [Starow] * Fixing last commit ... [Starow] * Fixing last commit. [Starow] * Improving cache code. [Starow] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Starow] * Merge branch 'master' of github.com:CIRCL/AIL-framework. [Alexandre Dulaunoy] * Clarification for the env variable. [Alexandre Dulaunoy] * Redis cache added fix #5. [Starow] The paste will be add in Redis during 5min and also saved on disk. Now if a module want to get the paste for further processing, it will first try to get it in the cache instead of getting it directly on the disk and wasting I/O. * Removing the hardcoded IP adress. [Starow] * README fixed for bare bone install. [Alexandre Dulaunoy] * Catching the exception dns.exception.Timeout fix #7. [Starow] * Editing redis .conf file I change the dir as it was originally to avoid editing these 3 files during the installation. [Starow] * Update pubsublogger with the last version. [Starow] * Changing ZMQ Curve Module comment. [Starow] * Initial import of AIL framework - Analysis Information Leak framework. [Starow] AIL is a modular framework to analyse potential information leak from unstructured data source like pastes from Past ebin or similar services. AIL framework is flexible and can be extended to support other functionalities to mine sen sitive information