initial commit

README.md

@@ -0,0 +1,58 @@
+# search-and-analyze
+
+**Author:** Victor Giers
+
+# search-and-analyze
+
+**Author:** Victor Giers
+
+## Overview
+
+This project contains a Python script that automates the process of generating search queries, performing web searches, extracting and analyzing content from web pages, and providing relevant information based on user input. The script uses advanced language models for query generation and analysis, ensures efficient web scraping with Playwright, and supports various configurations for different AI models.
+
+## Features
+
+1. **Query Generation:** Generates 5 optimized search queries based on user input using a specified LLM model.
+2. **Web Search:** Searches the web using SearXNG to find relevant URLs.
+3. **Content Extraction:** Extracts content from web pages, rendering JavaScript if necessary.
+4. **Analysis:** Analyzes extracted text for relevance and provides summaries or specific information based on user queries.
+5. **Fallback Mechanism:** Handles failures in data extraction or analysis by moving to the next URL.
+6. **Parallel Processing:** Loads and analyzes up to 5 URLs concurrently.
+7. **Language Support:** Automatically detects and supports multiple languages using `langid` and `langcodes`.
+8. **NLTK Integration:** Supports NLTK for text summarization as an alternative to external LLM models.
+
+## Requirements
+
+- Python 3.x
+- Required Python packages: `requests`, `asyncio`, `argparse`, `urllib`, `json`, `re`, `termios`, `atexit`, `signal`, `time`, `langid`, `langcodes`, `collections`, `newspaper3k`, `nltk`, `playwright`, `langchain_core`, `langchain_ollama`
+- SearXNG running locally at `http://127.0.0.1:8888`
+
+## Installation
+
+```bash
+pip install requests asyncio argparse urllib json re termios atexit signal time langid langcodes collections newspaper3k nltk playwright langchain_core langchain_ollama
+playwright install chromium
+```
+
+## Usage
+
+Run the script with a user prompt and optional arguments for analysis and query models.
+
+```bash
+python search-and-analyze.py "your search query" --analysis-model mistral-small3.1:24b --query-model mistral:latest
+```
+
+- `--analysis-model`: Specifies the AI model to use for content analysis (e.g., `mistral-small3.1:24b`). Use `NLTK` for a local summary using Newspaper3k.
+- `--query-model`: Specifies the LLM model to use for generating search queries (default is `mistral:latest`).
+
+## Example
+
+```bash
+python search-and-analyze.py "origin of Valentine’s Day" --analysis-model NLTK
+```
+
+This command will generate search queries related to the origin of Valentine's Day, perform web searches, extract and summarize content from relevant pages, and output the results.
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.