AIL 6.2: Smarter Analysis, Search and Enhanced User Experience

We’re excited to release AIL Framework v6.2, a major update with new features and improved performance. This version makes analysis easier and the overall experience faster and more user-friendly.

Among the highlights are a fully revamped search engine powered by MeiliSearch, improved language detection for short text, local AI-driven image descriptions, and a yara-hunting editor tool.

What’s New in AIL 6.2

• Integrated YARA Editor: AIL now includes a built-in CodeMirror editor with full YARA rule support. Analysts can write, edit, and manage YARA rules directly within AIL, complete with syntax highlighting for an improved editing experience. (Thanks to Sami Mokaddem for this contribution!)
• AI-Powered Screenshot Descriptions with Ollama: The images engine now leverages Ollama to generate descriptions for screenshots and images. This AI-powered capability helps you quickly understand the content of images without needing to view them directly, adding another layer of insight. Descriptions are also saved in the database.
• Expanded Search Horizons:
  • Chat Message Search: You can now search the content of chat messages using MeiliSearch, making it easier to find specific conversations or keywords.
  • Tor Content Search: The new search engine also supports full-text search of crawled Tor pages, helping you quickly locate relevant information across hidden services.
• Enhanced Data Ingestion & Processing:
  • Matrix Feeder: AIL now supports ingesting data from Matrix export via a new chat feeder.
  • Google Tracking Module: A new module and object have been added for tracking Google analytics ID.
• Streamlined User Management & Configuration:
  • Welcome Emails for New Users: New users can now automatically receive a welcome email upon account creation.
  • New User Configuration Engine: Users can now create and save MISP accounts and API keys directly within AIL, enabling seamless data export to MISP instances or generation of MISP JSON files.
• Richer Correlation Capabilities: Discover new connections with added correlations for chat-to-CVE, chat-to-cryptocurrencies, and domain-to-chat/message.
• New Mail Object: A new mail object has been introduced, improving search and correlations with chats, domains, and crawled data.
• Module Statistics: The settings module now includes statistics for your AIL modules.
• Language Statistics in Chat Viewer: Gain insights into language distribution within chats directly in the chat viewer.

Key Enhancements in This Release

Alongside the new features, AIL 6.2 brings a wave of improvements to existing functionalities:

• Performance Boosts:
  • Significant performance improvements for mail and Gtracker searches.
  • Language detection is now more performant.
• Improved Search and Dashboards:
  • The general search dashboard has been revamped for better usability and now includes a helpful search assistant.
• Advanced Language Processing:
  • Language detection accuracy has been increased by removing special characters before analysis and improving the old Lexilang detector. new language detector for short text
  • The language engine has been refactored, allowing retrieval of chat messages and user messages filtered by language.
• Deeper Data Analysis & Correlation:
  • The reprocess functionality now includes a TrackingId module.
  • Enhanced correlation cards for file names, mail, and Gtracker entries.
• User Experience & System Management Refinements:
  • The module queue now displays the number of FeederModuleImporters.
  • The “Create New Tracker” button has been conveniently moved to the top of the trackers page.
  • HOTP users can now easily print their next 50 tokens via a new button in user settings.
  • The number of messages per participant is now shown in the chat participants view.
• Mail & Chat Improvements:
  • Punycode encoding issues in mail have been addressed.
  • Message cards now display subchannels and protocol information for better contextual understanding.
• Image Engine Refinements:
  • Beyond the new Ollama integration, image descriptions are stored along with the model used to generate them.
• Domain Analysis:
  • A new button has been added in domain search to directly crawl unknown onion sites.

Important Fixes

As with every release, AIL 6.2 includes a multitude of bug fixes to improve stability and reliability. Some notable areas include:

• Improved translations for several languages (BG, EL, HI, JA, ZH, RU) and user chat message translation.
• The installation process has been made smoother, with fixes for dependencies and submodule initialization.
• MISP export errors and empty relation issues have been resolved.
• Fixes for mail search and mail content display.
• Several fixes related to language detection, manual language selection, and LibreTranslate ISO codes.
• Crawler dashboard and domain onion cache fixes.
• And many more under-the-hood tweaks for a better overall experience!

Documentation

• We’ve added documentation for the tracker functionality to help you get the most out of it.

A Big Thank You!

This release wouldn’t have been possible without our dedicated community and contributors. Special thanks to Sami Mokaddem, Thirion Aurélien, Aaron Kaplan, NMD03 for their extensive work and valuable contributions to this version.

We encourage you to update to AIL 6.2 to take advantage of these new features and improvements.

🔗 Download & Documentation: AIL Project GitHub

💡 Feedback & Contributions: As always, we welcome community feedback and contributions to make AIL even better!

Funding

MISP-LEA, a collaborative endeavor between Shadowserver and CIRCL, is a 24-month initiative funded by the European Union. The project’s central aim is to establish operational and enduring MISP and AIL instances dedicated specifically to law enforcement agencies. This setup will facilitate a smoother exchange of evidence between law enforcement agencies and improve the onset of collaborative investigations. For this purpose, the system will ingest data from Shadowserver’s ransomware and C2 infrastructure tracking.

Law enforcement agencies willing to discover and leverage the MISP-LEA platform can apply on the misp-lea.org website.

Follow us

• Mastodon @ail_project@infosec.exchange
• LinkedIn https://www.linkedin.com/company/ail-project